Have you registered your class with kryo ? See core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala and core/src/test/scala/org/apache/spark/SparkConfSuite.scala
On Fri, Mar 13, 2015 at 10:52 AM, ilaxes <ila...@hotmail.com> wrote: > Hi, > > I'm working on a RDD of a tuple of objects which represent trees (Node > containing a hashmap of nodes). I'm trying to aggregate these trees over > the > RDD. > > Let's take an example, 2 graphs : > C - D - B - A - D - B - E > F - E - B - A - D - B - F > > I'm spliting each graphs according to the vertex A resulting in : > (B(1, D(1, C(1,))) , D(1, B(1, E(1,))) > (B(1, E(1, F(1,))) , D(1, B(1, F(1,))) > > And I want to aggregate both graph getting : > (B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,))) > > Some graph are potentially large (+4000 vertex) but I'm not supposed to > have > any cyclic references. > When I run my program I get this error : > > java.lang.StackOverflowError > at > > com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127) > at > > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) > at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) > at > > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) > at > > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) > at > > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) > at > > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at > > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) > at > > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) > > I've tried to increase the size of the stake and use the standard java > serializer but no effect. > > Any hint of the reason of this error and ways to change my code to solve it > ? > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
