Yeah, apologies. The section of my topology in question is like this: Spout -> Bolt1 -> Bolt2
Bolt1 declares an outputstream with fields (A, B) of respective types (String, [Custom type]) This works fine. But if Bolt1 tacks on a third field (in this case a String), I get this buffer underflow exception in Bolt3 when storm deserializes. I also just discovered that this does not happen if I put the additional string field in the middle and not at the end. Does that help? My sense is that this is probably a kryo bug. Appreciate you taking the time. On Thu, Mar 19, 2015 at 5:07 PM P. Taylor Goetz <[email protected]> wrote: > Hi Luke, > > Can you elaborate on the steps necessary to reproduce the problem? There’s > not much to go on here. > > -Taylor > > On Mar 19, 2015, at 4:40 PM, Luke Rohde <[email protected]> wrote: > > How did you resolve it? This started being a problem after I added a third > field to an output tuple, just a String. > > On Thu, Mar 19, 2015 at 4:20 PM Vladimir Protsenko <[email protected]> > wrote: > >> Hi, I've got once the same error when in deserialization was reading the >> wrong number of bytes that were sent in serialization code. >> >> 2015-03-19 19:01 GMT+04:00 Luke Rohde <[email protected]>: >> >>> Hi, I'm seeing this exception from kryo deep inside storm and I haven't >>> been able to figure it out. Forums/docs on kryo seem to mention that this >>> can happen if you use a kryo instance in multiple threads concurrently, but >>> I'm not doing that (and this appears to be coming from storm). Has anyone >>> seen/dealt with this before? The stack trace: >>> >>> java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: >>> Buffer underflow. >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.executor$fn__3441$fn__3453$fn__3500.invoke(executor.clj:748) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] >>> Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow. >>> at com.esotericsoftware.kryo.io.Input.require(Input.java:156) >>> ~[kryo-2.21.jar:na] >>> at com.esotericsoftware.kryo.io.Input.readInt(Input.java:337) >>> ~[kryo-2.21.jar:na] >>> at >>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109) >>> ~[kryo-2.21.jar:na] >>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) >>> ~[kryo-2.21.jar:na] >>> at >>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) >>> ~[kryo-2.21.jar:na] >>> at >>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) >>> ~[kryo-2.21.jar:na] >>> at >>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) >>> ~[kryo-2.21.jar:na] >>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) >>> ~[kryo-2.21.jar:na] >>> at >>> backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.executor$mk_task_receiver$fn__3364.invoke(executor.clj:398) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> ... 6 common frames omitted >>> >>> >> >
