Bump! On Wed, Aug 10, 2016 at 2:59 PM, Efe Selcuk <efema...@gmail.com> wrote:
> Thanks for the replies, folks. > > My specific use case is maybe unusual. I'm working in the context of the > build environment in my company. Spark was being used in such a way that > the fat assembly jar that the old 'sbt assembly' command outputs was used > when building a spark applicaiton. I'm trying to figure out if I can just > use the many library jars instead, but in the meantime I'm hoping to get a > fat assembly in the old way to get us unblocked in updating our application > to use 2.0. It's a proprietary build system, not maven or sbt, so it's not > straightforward and the dependencies are modeled differently. > > To be a bit more clear: the fat assembly was not used for any reason other > than to get the spark application to build. This is in the context of > running in Amazon EMR, so we don't send that spark assembly over for runs. > > Efe > > On Wed, Aug 10, 2016 at 2:15 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Hi Efe, >> >> Are you talking about creating an uber/fat jar file for your specific >> application? Then you can distribute it to another node just to use the jar >> file without assembling it. >> >> I can still do it in Spark 2 as before if I understand your special use >> case. >> >> [warn] Strategy 'discard' was applied to 349 files >> [warn] Strategy 'first' was applied to 450 files >> >> *[info] Assembly up to date: >> /data6/hduser/scala/CEP_streaming/target/scala-2.10/scala-assembly-1.0.jar*[success] >> Total time: 117 s, completed Aug 10, 2016 9:31:24 PM >> Submiting the job >> Ivy Default Cache set to: /home/hduser/.ivy2/cache >> The jars for the packages stored in: /home/hduser/.ivy2/jars >> >> >> >> HTH >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 10 August 2016 at 20:35, Holden Karau <hol...@pigscanfly.ca> wrote: >> >>> What are you looking to use the assembly jar for - maybe we can think of >>> a workaround :) >>> >>> >>> On Wednesday, August 10, 2016, Efe Selcuk <efema...@gmail.com> wrote: >>> >>>> Sorry, I should have specified that I'm specifically looking for that >>>> fat assembly behavior. Is it no longer possible? >>>> >>>> On Wed, Aug 10, 2016 at 10:46 AM, Nick Pentreath < >>>> nick.pentre...@gmail.com> wrote: >>>> >>>>> You're correct - Spark packaging has been shifted to not use the >>>>> assembly jar. >>>>> >>>>> To build now use "build/sbt package" >>>>> >>>>> >>>>> >>>>> On Wed, 10 Aug 2016 at 19:40, Efe Selcuk <efema...@gmail.com> wrote: >>>>> >>>>>> Hi Spark folks, >>>>>> >>>>>> With Spark 1.6 the 'assembly' target for sbt would build a fat jar >>>>>> with all of the main Spark dependencies for building an application. >>>>>> Against Spark 2, that target is no longer building a spark assembly, just >>>>>> ones for e.g. Flume and Kafka. >>>>>> >>>>>> I'm not well versed with maven and sbt, so I don't know how to go >>>>>> about figuring this out. >>>>>> >>>>>> Is this intended? Or am I missing something? >>>>>> >>>>>> Thanks. >>>>>> >>>>> >>>> >>> >>> -- >>> Cell : 425-233-8271 >>> Twitter: https://twitter.com/holdenkarau >>> >>> >> >
