[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553258#comment-14553258 ] ASF GitHub Bot commented on SPARK-6069: --- Github user pferrel closed the pull request at: https://github.com/apache/mahout/pull/125 Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525267#comment-14525267 ] Pat Ferrel commented on SPARK-6069: --- Same with Mahout where we were using Guava. Using Scala collections only will solve this for us. The way to work around this is to use the spark.executor.extraClassPath, which should point to the correct jar in the native filesystem on every worker! So you have to move a dependency jar to every worker. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[
https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524372#comment-14524372
]
Russell Alexander Spitzer commented on SPARK-6069:
--
Running with --conf spark.files.userClassPathFirst=true yields a different error
{code}
scala cc.sql(SELECT * FROM test.fun as a JOIN test.fun as b ON (a.k =
b.v)).collect
15/05/01 17:24:34 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
10.0.2.15): java.lang.NoClassDefFoundError: org/apache/spark/Partition
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at
org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at
org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42)
at
org.apache.spark.executor.ChildExecutorURLClassLoader.findClass(ExecutorURLClassLoader.scala:50)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:30)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$findClass$1.apply(ExecutorClassLoader.scala:57)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$findClass$1.apply(ExecutorClassLoader.scala:57)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:57)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Partition
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[
https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524363#comment-14524363
]
Russell Alexander Spitzer commented on SPARK-6069:
--
We've seen the same issue while developing the Spark Cassandra Connector.
Unless the connector lib is loaded via spark.executor.extraClassPath,
kryoSerializaition for joins always returns a classNotFound even though all
operations which don't require a shuffle are fine.
{code}
com.esotericsoftware.kryo.KryoException: Unable to find class:
org.apache.spark.sql.cassandra.CassandraSQLRow
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)
at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at
org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:144)
at
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.joins.HashedRelation$.apply(HashedRelation.scala:80)
at
org.apache.spark.sql.execution.joins.ShuffledHashJoin$$anonfun$execute$1.apply(ShuffledHashJoin.scala:46)
at
org.apache.spark.sql.execution.joins.ShuffledHashJoin$$anonfun$execute$1.apply(ShuffledHashJoin.scala:45)
at
org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
Adding the jar to executorExtraClasspath rather than --jars solves the issue.
Deserialization Error ClassNotFoundException with Kryo, Guava 14
Key: SPARK-6069
URL: https://issues.apache.org/jira/browse/SPARK-6069
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.2.1
Environment: Standalone one worker cluster on localhost, or any
cluster
Reporter: Pat Ferrel
Priority: Critical
A class is contained in the jars passed in when creating a context. It is
registered with kryo. The class (Guava HashBiMap) is created correctly from
an RDD and broadcast but the deserialization fails with ClassNotFound.
The work around is to hard code the path to the jar and make it available on
all workers. Hard code because we are creating a library so there is no easy
way to pass in to the app something like:
spark.executor.extraClassPath /path/to/some.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521542#comment-14521542 ] ASF GitHub Bot commented on SPARK-6069: --- Github user pferrel commented on the pull request: https://github.com/apache/mahout/pull/125#issuecomment-97811136 Still would like an eye on BiMap and BiDictionary. The remaining question here is if it solves https://issues.apache.org/jira/browse/SPARK-6069 on a cluster. I'll concentrate on finding a way to test this. A user tried this on a cluster and got an infinite recursion so wouldn't want to push yet. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521664#comment-14521664 ] Pat Ferrel commented on SPARK-6069: --- didn't mean for those comments to be cross-posted. Removing the use of Javaserializer to work around this problem in Spark 1.2, which is in wide use. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519940#comment-14519940 ] ASF GitHub Bot commented on SPARK-6069: --- GitHub user pferrel opened a pull request: https://github.com/apache/mahout/pull/125 Spark 1.2 Created a Scala based BiMap and BiDictionary and removed usage of Guava HashBiMap (actually no Guava is used now), I hope this gets past the Javaserializer bug in https://issues.apache.org/jira/browse/SPARK-6069 Also found an error that did an unneeded collect, which will drastically reduce memory requirements in reading in data for cooccurrence. Passing around ```Option[BiDictionary] = None``` instead of an empty dictionary, which seems a better idiom when using an immutable BiDictionary. Questions: * Passes local build tests but haven't tried on a cluster to verify it works in spite of SPARK-6069 * Need to try it on epinions data before pushing * Where should this go, it's in 0.11.0-SNAPSHOT now * Still need to try putting Scala libs in mahout shell module only. This is to check to see if they are needed in non-shell Scala modules I'm having trouble getting HDFS+Spark running, even standalone, so if anyone wants to try this I can provide instructions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pferrel/mahout spark-1.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/125.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #125 commit 71165a54b9bc99f23c2cce6e8c63720b7fc64001 Author: Andrew Palumbo apalu...@apache.org Date: 2015-02-24T22:54:15Z added scala dependencies in math-scala to fix spark-shell commit 75054c195ed71fe51f6e3de85e598f6ce6d51c3a Author: pferrel p...@occamsmachete.com Date: 2015-04-29T14:36:58Z initial cut at making cooccurrence work on Spark 1.2--runs locally but cleanup needed, also major memory use optimizations commit 990347b16258b3bf410c130314a2225654100181 Author: pferrel p...@occamsmachete.com Date: 2015-04-29T15:49:24Z refactored BiDictionary and BiMap to carve off dicictionary specific behavior in the companion object commit 1d897176e4d2e50552d476741e5c314c072f Author: pferrel p...@occamsmachete.com Date: 2015-04-29T16:19:08Z runs build tests now removing the .create factory method, which will require creation only when keys are already available commit f3611855fbf2b06bc3faa7083484d426e84e1b73 Author: pferrel p...@occamsmachete.com Date: 2015-04-29T18:37:38Z using Option[BiDictionary] = None now instread of passing around an empty BiDictionary Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343372#comment-14343372 ] Pat Ferrel commented on SPARK-6069: --- ok, I'll try 1.3 as soon as I get a chance. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343437#comment-14343437 ] Marcelo Vanzin commented on SPARK-6069: --- I think this is a dupe of SPARK-5470, but haven't really tested. SPARK-4660 is a similar (same?) issue for the Java serializer. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343597#comment-14343597 ] Pat Ferrel commented on SPARK-6069: --- great since it's fixed in 1.3 I'll definitely try that next and resolve this if it flies. Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[ https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341819#comment-14341819 ] Sean Owen commented on SPARK-6069: -- No I set it kind of preemptively. I don't know that I serialize any Guava classes though, come to think of it. I am using YARN + Hadoop 2.5. I don't think it should be necessary in general. Guava is a strange special case, so though it worth trying. If you have the energy, you might try 1.3.0-SNAPSHOT since I see a few things fixed that may be relevant: https://issues.apache.org/jira/browse/SPARK-4877 https://issues.apache.org/jira/browse/SPARK-4660 Deserialization Error ClassNotFoundException with Kryo, Guava 14 Key: SPARK-6069 URL: https://issues.apache.org/jira/browse/SPARK-6069 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.2.1 Environment: Standalone one worker cluster on localhost, or any cluster Reporter: Pat Ferrel Priority: Critical A class is contained in the jars passed in when creating a context. It is registered with kryo. The class (Guava HashBiMap) is created correctly from an RDD and broadcast but the deserialization fails with ClassNotFound. The work around is to hard code the path to the jar and make it available on all workers. Hard code because we are creating a library so there is no easy way to pass in to the app something like: spark.executor.extraClassPath /path/to/some.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6069) Deserialization Error ClassNotFoundException with Kryo, Guava 14
[
https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341809#comment-14341809
]
Pat Ferrel commented on SPARK-6069:
---
Embarrassed to say still on Hadoop 1.2.1 and so no yarn. The packaging is not
in the app jar but a separate pruned down dependencies-only jar. I can see why
yarn would throw a unique kink into the situation. So I guess you ran into
this and had to use the {{user.classpath.first}} work around or are you saying
it doesn't occur in oryx?
Still none of this should be necessary, right? Why else would jars be specified
in to context creation? We do have a work around if someone has to work with
1.2.1 but because of that it doesn't seem like a good version to recommend.
Maybe I'll try 1.2 and install H2 and yarn--which seems like what the distros
support.
Deserialization Error ClassNotFoundException with Kryo, Guava 14
Key: SPARK-6069
URL: https://issues.apache.org/jira/browse/SPARK-6069
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.2.1
Environment: Standalone one worker cluster on localhost, or any
cluster
Reporter: Pat Ferrel
Priority: Critical
A class is contained in the jars passed in when creating a context. It is
registered with kryo. The class (Guava HashBiMap) is created correctly from
an RDD and broadcast but the deserialization fails with ClassNotFound.
The work around is to hard code the path to the jar and make it available on
all workers. Hard code because we are creating a library so there is no easy
way to pass in to the app something like:
spark.executor.extraClassPath /path/to/some.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
