When I try to run spark in local mode on my Mac, it get stuck trying to fetch
my application jar.
My command line is:
SPARK_CLASSPATH=~/depot/Engineering/kenji/dadlog-filter/target/scala-2.10/dadlog-filter-assembly-1.0.jar
~/opt/spark/bin/spark-class {myMainClass} local s3n://{myInputFile}
The version of commons-io included in the Spark assembly is an old one, which
doesn't have the version of closeQuietly that takes a Closeable:
$ javap -cp
/root/spark/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.1-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar
org.apache.commons.io.IOUtils
When I run sbt assembly, I use the provided configuration in the
build.sbt library dependency, to avoid conflicts in the fat jar:
libraryDependencies += org.apache.spark %% spark-core %
0.8.1-incubating % provided
But if I want to do sbt run, I have to remove the provided, otherwise it
doesn't
