Bo yang- I am using Spark 1.2.0 and undoubtedly there are older Guava classes which are being picked up and serialized with the closures when they are sent from the driver to the executors because the class serial version ids don't match from the driver to the executors. Have you tried doing this? Guava works fine for me when this is not the case- but as soon as a Guava class which was changed from versions <15.0 is serialized, it fails. See [1] fore info- we did fairly extensive testing last night. I've isolated the issue to Hadoop's really old version of Guava being picked up. Again, this is only noticeable when classes are used from Guava 15.0 that were changed from previous versions and those classes are being serialized on the driver and shipped to the executors.
[1] https://github.com/calrissian/mango/issues/158 On Wed, Feb 4, 2015 at 1:31 AM, bo yang <bobyan...@gmail.com> wrote: > Corey, > > Which version of Spark do you use? I am using Spark 1.2.0, and guava > 15.0. It seems fine. > > Best, > Bo > > > On Tue, Feb 3, 2015 at 8:56 PM, M. Dale <medal...@yahoo.com.invalid> > wrote: > >> Try spark.yarn.user.classpath.first (see >> https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN). >> Also thread at >> http://apache-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark-submit-td18030.html >> . >> >> HTH, >> Markus >> >> On 02/03/2015 11:20 PM, Corey Nolet wrote: >> >> I'm having a really bad dependency conflict right now with Guava versions >> between my Spark application in Yarn and (I believe) Hadoop's version. >> >> The problem is, my driver has the version of Guava which my application >> is expecting (15.0) while it appears the Spark executors that are working >> on my RDDs have a much older version (assuming it's the old version on the >> Hadoop classpath). >> >> Is there a property like "mapreduce.job.user.classpath.first' that I >> can set to make sure my own classpath is extablished first on the executors? >> >> >> >
