Hi,
I'm working on a RDD of a tuple of objects which represent trees (Node
containing a hashmap of nodes). I'm trying to aggregate these trees over the
RDD.
Let's take an example, 2 graphs :
C - D - B - A - D - B - E
F - E - B - A - D - B - F
I'm spliting each graphs according to the vertex A resulting in :
(B(1, D(1, C(1,))) , D(1, B(1, E(1,)))
(B(1, E(1, F(1,))) , D(1, B(1, F(1,)))
And I want to aggregate both graph getting :
(B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,)))
Some graph are potentially large (+4000 vertex) but I'm not supposed to have
any cyclic references.
When I run my program I get this error :
java.lang.StackOverflowError
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
I've tried to increase the size of the stake and use the standard java
serializer but no effect.
Any hint of the reason of this error and ways to change my code to solve it
?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
