I haven't register my class in kryo but I dont think it would have such an impact on the stack size.
I'm thinking of using graphx and I'm wondering how it serializes the graph object as it can use kryo as serializer. 2015-03-14 6:22 GMT+01:00 Ted Yu <yuzhih...@gmail.com>: > Have you registered your class with kryo ? > > See core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala > and core/src/test/scala/org/apache/spark/SparkConfSuite.scala > > On Fri, Mar 13, 2015 at 10:52 AM, ilaxes <ila...@hotmail.com> wrote: > >> Hi, >> >> I'm working on a RDD of a tuple of objects which represent trees (Node >> containing a hashmap of nodes). I'm trying to aggregate these trees over >> the >> RDD. >> >> Let's take an example, 2 graphs : >> C - D - B - A - D - B - E >> F - E - B - A - D - B - F >> >> I'm spliting each graphs according to the vertex A resulting in : >> (B(1, D(1, C(1,))) , D(1, B(1, E(1,))) >> (B(1, E(1, F(1,))) , D(1, B(1, F(1,))) >> >> And I want to aggregate both graph getting : >> (B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,))) >> >> Some graph are potentially large (+4000 vertex) but I'm not supposed to >> have >> any cyclic references. >> When I run my program I get this error : >> >> java.lang.StackOverflowError >> at >> >> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127) >> at >> >> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) >> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) >> at >> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) >> at >> >> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) >> at >> >> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) >> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) >> at >> >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) >> at >> >> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) >> at >> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) >> at >> >> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) >> at >> >> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) >> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) >> >> I've tried to increase the size of the stake and use the standard java >> serializer but no effect. >> >> Any hint of the reason of this error and ways to change my code to solve >> it >> ? >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
