Hi,
I have a complex object with a MutableList and a Map containing
thousands of objects. Here is a snippet of the exception. The actual
exception is much longer. There, you can see that the exception is
caused by too many recursive calls. For each recursive call the stack
size increases because the parameters of each resursion level are saved
on the stack. I am not sure but I suggest that the Map is the problem.
Exception in thread "main" java.lang.StackOverflowError
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
Best Regards,
Hilmi
Am 21.01.2016 um 11:14 schrieb Till Rohrmann:
Hi Hilmi,
I've never seen such a problem. Could you please provide a bit more
background information. Which types are you serializing? Which serializer
are you using? What do you mean with recursion? Maybe you can provide a
simple example program which reproduces the problem.
The thing is that if you have a composite data type, then each field will
be serialized with it's own serializer. Thus, if you data type is strongly
nested, then you will end up with a long chain of nested serializer calls.
In order to cause a stack overflow exception, you would have to have a
really deeply nested data structure. Maybe you can flatten it a bit.
Cheers,
Till
On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <hilmi.yildi...@dfki.de>
wrote:
Hi,
I am using flink to train a ML model. There, I need to broadcast huge
broadcast variables in each iteration. During the serialization I get a
Stackoverflow exception. If I increase the Stack size then it works well.
The Stackoverflow could be avoided If you do not use recursion in
serialization. You could use iterations instead, I know it is more complex
but this would avoid a stack overflow.
Best Regards,
Hilmi
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: hilmi.yildi...@dfki.de <mailto:hilmi.yildi...@dfki.de>
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: hilmi.yildi...@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
