Hi Jie, A few questions:
1. How many topologies do you have running on this cluster? 2. Once the topology recovers, do the errors still happen or do they disappear for the most part? 3. Have you noticed any worker failures on the cluster around the time these errors happen? -Nathan On Tue, Aug 26, 2014 at 10:04 PM, Jie Li <[email protected]> wrote: > Thanks Dmitry. We are using Trident with transactional spout/state, and > this happens in the first 10 minutes when the topolgoy starts, and recover > itself after then. Since the topology is transactional, same messages are > replayed, so this doesn't seem a data issue, and does seem some storm bug? > > Jie Li > Software engineer at Pinterest | www.linkedin.com/in/jay23jack/ > > > On Wed, Aug 13, 2014 at 2:13 PM, Dmitry Shulga <[email protected]> > wrote: > >> If you have access to the bolt, check what fields the bolt is expecting. >> You may have some coding issues when creating a Tuple for it. >> >> >> >> >> Here is example of the method that defines what fields the bolt is >> outputting taken from >> https://storm.incubator.apache.org/documentation/Tutorial.html. The Execute >> method emits a Tuple with two fields, definitions of which can be found in >> declareOutputFields. >> >> >> >> @Override >> public void execute(Tuple input) { >> int val = input.getInteger(0); >> _collector.emit(input, new Values(val*2, val*3)); >> _collector.ack(input); >> } >> >> @Override >> public void declareOutputFields(OutputFieldsDeclarer declarer) { >> declarer.declare(new Fields("double", "triple")); >> } >> >> >> >> -D >> >> >> On Wed, Aug 13, 2014 at 9:47 AM, Jie Li <[email protected]> wrote: >> >>> Hi all, >>> >>> We've been testing Storm for a while and have been happy so far, but >>> this error seems a bit unknown and concerning. One of our topologies >>> somehow came across this error during testing with nothing really changed, >>> so I have no clue how to debug it. And it seems an transient error, as the >>> topology somehow recovered itself after running for a while. >>> >>> The storm version is 0.9.1 running with most of default settings. One >>> thing I noticed suspicious was the warning message of "Received invalid >>> messages for unknown tasks. Dropping... ". Not sure if it was related and >>> if anyone has idea? Here is the full stack trace of the error (that doesn't >>> seem useful): >>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple >>> created with wrong number of fields. Expected 2 fields but got 1 fields >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539) >>> ~[na:na] >>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433) >>> ~[na:na] >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] >>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38] >>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong >>> number of fields. Expected 2 fields but got 1 fields >>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395) >>> ~[na:na] >>> at >>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58) >>> ~[na:na] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> ... 6 common frames omitted >>> 2014-08-13 01:14:58 b.s.d.executor [ERROR] >>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple >>> created with wrong number of fields. Expected 2 fields but got 1 fields >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539) >>> ~[na:na] >>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433) >>> ~[na:na] >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] >>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38] >>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong >>> number of fields. Expected 2 fields but got 1 fields >>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> at >>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395) >>> ~[na:na] >>> at >>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58) >>> ~[na:na] >>> at >>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104) >>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] >>> ... 6 common frames omitted >>> 2014-08-13 01:14:58 b.s.util [INFO] Halting process: ("Worker died") >>> Thanks, >>> Jie Li >>> Software engineer at Pinterest | www.linkedin.com/in/jay23jack/ >>> >> >> > -- Twitter: @nathanmarz http://nathanmarz.com
