[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081733#comment-17081733
]
Julien Nioche commented on STORM-3582:
--
have commented in STORM-3620
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081584#comment-17081584
]
Ethan Li commented on STORM-3582:
-
STORM-3620 doesn't necessary change outputCollector to be thread-safe. As of
now, you can synchronize the outputCollector in your code. Or eliminate
multi-threading inside spout/bolt instance.
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081189#comment-17081189
]
Julien Nioche commented on STORM-3582:
--
Based on Simon's findings, this should be fixed with the synchronization issue.
I'll try again once STORM-3620 has been done
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080174#comment-17080174
]
Ethan Li commented on STORM-3582:
-
[~thecoop1984] Thanks. You are right. OutputCollector is not thread-safe. I
filed STORM-3620 for that issue.
I guess Kyro issue in this JIRA might still exist. [~jnioche] Could you help me
reproduce it? Thanks
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079063#comment-17079063
]
Simon Cooper commented on STORM-3582:
-
We have found the issue that was causing our exceptions - in our bolts we have
multiple threads that were accessing OutputCollectors concurrently. In storm 1,
it seems that collectors are thread-safe, but in storm 2 they are not. This
meant the serialization buffer was being written to by multiple threads at the
same time, causing data corruption which was only spotted when it was
deserialized on the target worker, resulting in seemingly random exceptions as
the system tried to interpret the random data.
Adding a mutex to all calls to output collectors has fixed this problem for us.
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078900#comment-17078900
]
Ethan Li commented on STORM-3582:
-
I am interested to see what's going on. We have been using storm 2.x without
any serious problem. Can you let me know how to reproduce this by providing a
simple example so I can look into it? Thanks
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074535#comment-17074535
]
Simon Cooper commented on STORM-3582:
-
Looking at the kryo code, this actually points towards it being a threading
issue - could the same {{KryoValuesSerializer}} be accessed by more than one
thread at once?
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074497#comment-17074497
]
Simon Cooper commented on STORM-3582:
-
Another exception which may indicate it's a problem in kryo (this is when
serializing an ack tuple):
{code}java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination
index 32 out of bounds for byte[31]
at java.lang.System.arraycopy(Native Method) ~[?:?]
at com.esotericsoftware.kryo.io.Output.toBytes(Output.java:130)
~[kryo-shaded-3.0.3.jar:?]
at
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:41)
~[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.daemon.worker.WorkerState.checkSerialize(WorkerState.java:580)
~[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.executor.ExecutorTransfer.tryTransferLocal(ExecutorTransfer.java:99)
~[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:64)
~[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.executor.LocalExecutor$1.tryTransfer(LocalExecutor.java:36)
~[storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.daemon.Task.sendUnanchored(Task.java:211)
~[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.executor.bolt.BoltOutputCollectorImpl.ack(BoltOutputCollectorImpl.java:128)
~[storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.task.OutputCollector.ack(OutputCollector.java:181)
~[storm-client-2.1.0.jar:2.1.0]{code}
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (STORM-3582) Kryo errors when using custom serialization
[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072636#comment-17072636
]
Simon Cooper commented on STORM-3582:
-
We've also come across this issue. We first encountered it by
mismatch/decompression exceptions thrown by our custom Snappy
`ListDelegateSerializer` implementation. This implies it may be an issue with a
buffer being flushed on the sending worker.
> Kryo errors when using custom serialization
> ---
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.1.0
>Reporter: Julien Nioche
>Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
>
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
