In my case I had a custom KryoSerializer and write method was pushing length of array and *wrong* number of bytes. When serializer read method input.readBytes(length) tried to read expected number of bytes it throws such exception.
2015-03-20 2:01 GMT+04:00 Luke Rohde <[email protected]>: > Yeah, apologies. The section of my topology in question is like this: > > Spout -> Bolt1 -> Bolt2 > > Bolt1 declares an outputstream with fields (A, B) of respective types > (String, [Custom type]) > This works fine. > > But if Bolt1 tacks on a third field (in this case a String), I get this > buffer underflow exception in Bolt3 when storm deserializes. > I also just discovered that this does not happen if I put the additional > string field in the middle and not at the end. > > Does that help? My sense is that this is probably a kryo bug. > > Appreciate you taking the time. > > > > On Thu, Mar 19, 2015 at 5:07 PM P. Taylor Goetz <[email protected]> wrote: > >> Hi Luke, >> >> Can you elaborate on the steps necessary to reproduce the problem? >> There’s not much to go on here. >> >> -Taylor >> >> On Mar 19, 2015, at 4:40 PM, Luke Rohde <[email protected]> wrote: >> >> How did you resolve it? This started being a problem after I added a >> third field to an output tuple, just a String. >> >> On Thu, Mar 19, 2015 at 4:20 PM Vladimir Protsenko <[email protected]> >> wrote: >> >>> Hi, I've got once the same error when in deserialization was reading the >>> wrong number of bytes that were sent in serialization code. >>> >>> 2015-03-19 19:01 GMT+04:00 Luke Rohde <[email protected]>: >>> >>>> Hi, I'm seeing this exception from kryo deep inside storm and I haven't >>>> been able to figure it out. Forums/docs on kryo seem to mention that this >>>> can happen if you use a kryo instance in multiple threads concurrently, but >>>> I'm not doing that (and this appears to be coming from storm). Has anyone >>>> seen/dealt with this before? The stack trace: >>>> >>>> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: >>>> Buffer underflow. >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.daemon.executor$fn__3441$fn__3453$fn__3500.invoke(executor.clj:748) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] >>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] >>>> Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow. >>>> at com.esotericsoftware.kryo.io.Input.require(Input.java:156) >>>> ~[kryo-2.21.jar:na] >>>> at com.esotericsoftware.kryo.io.Input.readInt(Input.java:337) >>>> ~[kryo-2.21.jar:na] >>>> at >>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109) >>>> ~[kryo-2.21.jar:na] >>>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) >>>> ~[kryo-2.21.jar:na] >>>> at >>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) >>>> ~[kryo-2.21.jar:na] >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) >>>> ~[kryo-2.21.jar:na] >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) >>>> ~[kryo-2.21.jar:na] >>>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) >>>> ~[kryo-2.21.jar:na] >>>> at >>>> backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.daemon.executor$mk_task_receiver$fn__3364.invoke(executor.clj:398) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) >>>> ~[storm-core-0.9.3.jar:0.9.3] >>>> ... 6 common frames omitted >>>> >>>> >>> >>
