Hi, After upgrading to Storm 2.0 and running in production couple of days I saw following error:
server errors in handling the request com.esotericsoftware.kryo.KryoException: Unable to find class: j, Serialization trace: value (org.apache.storm.metric.api.IMetricsConsumer$DataPoint) at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:45) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.lang.ClassNotFoundException: j, at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_212] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212] at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_212] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ~[kryo-3.0.3.jar:?] ... 40 more and also messages similar to one below, but with different class IDs server errors in handling the request com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 108 Serialization trace: value (org.apache.storm.metric.api.IMetricsConsumer$DataPoint) at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:45) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] And also: server errors in handling the request com.esotericsoftware.kryo.KryoException: Buffer underflow. at com.esotericsoftware.kryo.io.Input.require(Input.java:182) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.io.Input.readLong(Input.java:686) ~[kryo-3.0.3.jar:?] at org.apache.storm.tuple.MessageId.deserialize(MessageId.java:56) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:44) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] After which storm process is dying. I'm not 100% that this is related to upgrade to 2.0 version of storm or if anything related to kryo was touched, but I haven't seen anything similar with previous version of storm (1.1.x) Have you seen anything similar? Best regards, Michal
