Can this assembly get faster if we do not need the Spark SQL or some other component in spark ? such as we only need the core of spark.
On Wed, Nov 26, 2014 at 3:37 PM, lihu <lihu...@gmail.com> wrote: > Matei, sorry for my last typo error. And the tip can improve about 30s in > my computer. > > On Wed, Nov 26, 2014 at 3:34 PM, lihu <lihu...@gmail.com> wrote: > >> Mater, thank you very much! >> After take your advice, the time for assembly from about 20min down to >> 6min in my computer. that's a very big improvement. >> >> On Wed, Nov 26, 2014 at 12:32 PM, Matei Zaharia <matei.zaha...@gmail.com> >> wrote: >> >>> BTW as another tip, it helps to keep the SBT console open as you make >>> source changes (by just running sbt/sbt with no args). It's a lot faster >>> the second time it builds something. >>> >>> Matei >>> >>> On Nov 25, 2014, at 8:31 PM, Matei Zaharia <matei.zaha...@gmail.com> >>> wrote: >>> >>> You can do sbt/sbt assembly/assembly to assemble only the main package. >>> >>> Matei >>> >>> On Nov 25, 2014, at 7:50 PM, lihu <lihu...@gmail.com> wrote: >>> >>> Hi, >>> The spark assembly is time costly. If I only need >>> the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need >>> the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to >>> avoid assemble the example jar. I know *export SPARK_PREPEND_CLASSES=* >>> *true* method can reduce the assembly, but I do not >>> develop locally. Any advice? >>> >>> -- >>> *Best Wishes!* >>> >>> >>> >>> >>> >> >> >> -- >> *Best Wishes!* >> >> *Li Hu(李浒) | Graduate Student* >> >> *Institute for Interdisciplinary Information Sciences(IIIS >> <http://iiis.tsinghua.edu.cn/>)* >> *Tsinghua University, China* >> >> *Email: lihu...@gmail.com <lihu...@gmail.com>* >> *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/ >> <http://iiis.tsinghua.edu.cn/zh/lihu/>* >> >> >> > > > -- > *Best Wishes!* > > *Li Hu(李浒) | Graduate Student* > > *Institute for Interdisciplinary Information Sciences(IIIS > <http://iiis.tsinghua.edu.cn/>)* > *Tsinghua University, China* > > *Email: lihu...@gmail.com <lihu...@gmail.com>* > *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/ > <http://iiis.tsinghua.edu.cn/zh/lihu/>* > > > -- *Best Wishes!* *Li Hu(李浒) | Graduate Student* *Institute for Interdisciplinary Information Sciences(IIIS <http://iiis.tsinghua.edu.cn/>)* *Tsinghua University, China* *Email: lihu...@gmail.com <lihu...@gmail.com>* *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/ <http://iiis.tsinghua.edu.cn/zh/lihu/>*
