Re: yarn, fat-jars and lib_managed

2014-01-10 Thread Alex Cozzi
gt; > -Original Message- > From: Alex Cozzi [mailto:alexco...@gmail.com] > Sent: Friday, January 10, 2014 5:32 AM > To: dev@spark.incubator.apache.org > Subject: yarn, fat-jars and lib_managed > > I am just starting out playing with spark on our hadoop 2.2 cluster and I >

RE: yarn, fat-jars and lib_managed

2014-01-09 Thread Liu, Raymond
Liu -Original Message- From: Alex Cozzi [mailto:alexco...@gmail.com] Sent: Friday, January 10, 2014 5:32 AM To: dev@spark.incubator.apache.org Subject: yarn, fat-jars and lib_managed I am just starting out playing with spark on our hadoop 2.2 cluster and I have a question. The current

yarn, fat-jars and lib_managed

2014-01-09 Thread Alex Cozzi
I am just starting out playing with spark on our hadoop 2.2 cluster and I have a question. The current way to submit jobs to the cluster is to create fat-jars with sbt assembly. This approach works but I think is less than optimal in many large hadoop installation: the way we interact with the