Finally I figured out the problem and fixed it. There was some inconsistency in my .ivy2 and .m2 repository. Spark resolves the dependencies using meta data in ivy2/cache by not verifies its real location. That was why Spark resolved jackson-core-asl in local-m2-cache. But when Spark tried to load the JAR file, it couldn't find it and caused the error.
To solve the problem, I deleted the corresponding directory for jackson-core-asl under .ivy2. On Fri, Feb 5, 2016 at 5:00 PM, Jeff - Data Bean Australia < [email protected]> wrote: > Hi, > > Trying to use an external package with the following command line and it > doesn't work: > > ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10 > > Please find the package info here: > http://spark-packages.org/package/harsha2010/magellan > > The error related to one of the dependencies was not found successfully: > > [NOT FOUND ] > org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms) > > And the weird part is that Spark or Ivy didn't actually try to look it up > online, but simply reported error when it didn't find the package in my > local m2 repository. Please see the complete error message below and help > me out. > > MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages > harsha2010:magellan:1.0.3-s_2.10 > Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache > The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars > :: loading settings :: url = > jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml > harsha2010#magellan added as a dependency > :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 > confs: [default] > found harsha2010#magellan;1.0.3-s_2.10 in spark-packages > found commons-io#commons-io;2.4 in local-m2-cache > found com.esri.geometry#esri-geometry-api;1.2.1 in central > found org.json#json;20090211 in local-m2-cache > found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache > :: resolution report :: resolve 282ms :: artifacts dl 9ms > :: modules in use: > com.esri.geometry#esri-geometry-api;1.2.1 from central in [default] > commons-io#commons-io;2.4 from local-m2-cache in [default] > harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default] > org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in > [default] > org.json#json;20090211 from local-m2-cache in [default] > --------------------------------------------------------------------- > | | modules || artifacts | > | conf | number| search|dwnlded|evicted|| number|dwnlded| > --------------------------------------------------------------------- > | default | 5 | 0 | 0 | 0 || 5 | 0 | > --------------------------------------------------------------------- > > :: problems summary :: > :::: WARNINGS > [NOT FOUND ] > org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms) > > ==== local-m2-cache: tried > > > > file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar > > :::::::::::::::::::::::::::::::::::::::::::::: > > :: FAILED DOWNLOADS :: > > :: ^ see resolution messages for details ^ :: > > :::::::::::::::::::::::::::::::::::::::::::::: > > :: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar > > :::::::::::::::::::::::::::::::::::::::::::::: > > > > :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS > Exception in thread "main" java.lang.RuntimeException: [download failed: > org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar] > at > org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009) > at > org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > -- > Data Bean - A Big Data Solution Provider in Australia. > -- Data Bean - A Big Data Solution Provider in Australia.
