Dell - Internal Use - Confidential Regarding the NotSerializableException: I would like to see what grouping you have defined between the two bolts that you have experienced the NotSerializableException with.
I am guessing it is localOrShuffleGrouping, can you change it to shuffleGrouping and retest with one worker. From: Asif Ihsan [mailto:[email protected]] Sent: Thursday, May 14, 2015 5:23 AM To: [email protected] Subject: Re: Multiple Workers in local mode gives error You are right about it. What does multiple workers means. Does it mean that when I run topology with single worker than i will see one single process handling all the bolts and when running 6 workers for 5 bolts and 1 spout than i will see 6 separate processes running each bolt and spout? M i right.If not then what does it mean? On Thu, May 14, 2015 at 2:46 PM, 임정택 <[email protected]<mailto:[email protected]>> wrote: Hi. Storm serializes tuples when tuple should be sent to other (remote) worker. In other words, Storm doesn't serialize tuples when destination is local task. That's why you didn't meet error when testing with 1 worker. MapEventBean seems to be not serializable, so you need to convert to other data type. Hope this helps. Regards. Jungtaek Lim (HeartSaVioR) 2015-05-14 18:38 GMT+09:00 Asif Ihsan <[email protected]<mailto:[email protected]>>: I am using Esper in one of the bolts of the Storm topology. Bolt emit MapEventBean array (EventBean[]). With the single worker topology run smoothly. With multiple workers it gives following error in emit function call. May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke SEVERE: Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) at backtype.storm.util$fast_list_map.invoke(util.clj:771) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) ... 6 more Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeArray(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) ... 21 more May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke SEVERE: java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) at backtype.storm.util$fast_list_map.invoke(util.clj:771) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) ... 6 more Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeArray(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) ... 21 more On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <[email protected]<mailto:[email protected]>> wrote: Asif: Would love to help you, however, we will need way more details. Why don't you start with telling us what the error is? How are you detecting the error? You will want to look over both the supervisor and worker logs. Thank you for your time! +++++++++++++++++++++ Jeff Maass<mailto:[email protected]> linkedin.com/in/jeffmaass<http://linkedin.com/in/jeffmaass> stackoverflow.com/users/373418/maassql<http://stackoverflow.com/users/373418/maassql> +++++++++++++++++++++ On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <[email protected]<mailto:[email protected]>> wrote: Hi, I recently started using Storm. I am having topology. It works well with numofworkers=1. However when I change it to 6 or more than 1. Its gives error when emitting tuples. What is the issue and how to resolve it with multiple workers. Thank you. -- Regards Muhammad Asif Ihsan -- Regards Muhammad Asif Ihsan -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior -- Regards Muhammad Asif Ihsan
