Let me try it... Will this be fixed if I generate a assembly file with mllib-1.1.0 SNAPSHOT jar and other dependencies with the rest of the application code ?
On Sat, Aug 2, 2014 at 10:46 AM, Xiangrui Meng <men...@gmail.com> wrote: > You can try enabling "spark.files.userClassPathFirst". But I'm not > sure whether it could solve your problem. -Xiangrui > > On Sat, Aug 2, 2014 at 10:13 AM, Debasish Das <debasish.da...@gmail.com> > wrote: > > Hi, > > > > I have deployed spark stable 1.0.1 on the cluster but I have new code > that > > I added in mllib-1.1.0-SNAPSHOT. > > > > I am trying to access the new code using spark-submit as follows: > > > > spark-job --class com.verizon.bda.mllib.recommendation.ALSDriver > > --executor-memory 16g --total-executor-cores 16 --jars > > spark-mllib_2.10-1.1.0-SNAPSHOT.jar,scopt_2.10-3.2.0.jar > > sag-core-0.0.1-SNAPSHOT.jar --rank 25 --numIterations 10 --lambda 1.0 > > --qpProblem 2 inputPath outputPath > > > > I can see the jars are getting added to httpServer as expected: > > > > 14/08/02 12:50:04 INFO SparkContext: Added JAR > > file:/vzhome/v606014/spark-glm/spark-mllib_2.10-1.1.0-SNAPSHOT.jar at > > http://10.145.84.20:37798/jars/spark-mllib_2.10-1.1.0-SNAPSHOT.jar with > > timestamp 1406998204236 > > > > 14/08/02 12:50:04 INFO SparkContext: Added JAR > > file:/vzhome/v606014/spark-glm/scopt_2.10-3.2.0.jar at > > http://10.145.84.20:37798/jars/scopt_2.10-3.2.0.jar with timestamp > > 1406998204237 > > > > 14/08/02 12:50:04 INFO SparkContext: Added JAR > > file:/vzhome/v606014/spark-glm/sag-core-0.0.1-SNAPSHOT.jar at > > http://10.145.84.20:37798/jars/sag-core-0.0.1-SNAPSHOT.jar with > timestamp > > 1406998204238 > > > > But the job still can't access code form mllib-1.1.0 SNAPSHOT.jar...I > think > > it's picking up the mllib from cluster which is at 1.0.1... > > > > Please help. I will ask for a PR tomorrow but internally we want to > > generate results from the new code. > > > > Thanks. > > > > Deb >
