Hi all! We'be been using the sbt-pack sbt plugin (https://github.com/xerial/sbt-pack) for building our standalone Spark application for a while now. Until version 1.0.0, that worked nicely.
For those who don't know the sbt-pack plugin, it basically copies all the dependencies JARs from your local ivy/maven cache to a your target folder (in target/pack/lib), and creates launch scripts (in target/pack/bin) for your application (notably setting all these jars on the classpath). Now, since Spark 1.0.0 was released, we are encountering a weird error where running our project with "sbt run" is fine but running our app with the launch scripts generated by sbt-pack fails. After a (quite painful) investigation, it turns out some JARs are NOT copied from the local ivy2 cache to the lib folder. I noticed that all the missing jars contain "shaded" in their file name (but all not all jars with such name are missing). One of the missing JARs is explicitly from the Spark definition (SparkBuild.scala, line 350): ``mesos-0.18.1-shaded-protobuf.jar``. This file is clearly present in my local ivy cache, but is not copied by sbt-pack. Is there an evident reason for that? I don't know much about the shading mechanism, maybe I'm missing something here? Any help would be appreciated! Cheers Pierre -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-sbt-pack-with-Spark-1-0-0-tp6649.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
