Dell - Internal Use - Confidential Assuming your parallelism hints default to 1, you have combined parallelism of 6 and with 6 workers each would get 1 executor each.
So each of the bolt and task will run on a different worker(if you have >6 available slots on your cluster) Checkout the awesome docs @ https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html From: Asif Ihsan [mailto:[email protected]] Sent: Thursday, May 14, 2015 9:47 AM To: [email protected] Subject: Re: Multiple Workers in local mode gives error If I create 6 workers for 5 bolts and 1 spout. Then how will these spout and bolts be distributed among workers. On Thu, May 14, 2015 at 5:34 PM, 임정택 <[email protected]<mailto:[email protected]>> wrote: Yes, right. Worker is a JVM instance which can contains one or more executors, each executor is spout, bolt, acker, etc. 2015년 5월 14일 목요일, Asif Ihsan<[email protected]<mailto:[email protected]>>님이 작성한 메시지: You are right about it. What does multiple workers means. Does it mean that when I run topology with single worker than i will see one single process handling all the bolts and when running 6 workers for 5 bolts and 1 spout than i will see 6 separate processes running each bolt and spout? M i right.If not then what does it mean? On Thu, May 14, 2015 at 2:46 PM, 임정택 <[email protected]<mailto:[email protected]>> wrote: Hi. Storm serializes tuples when tuple should be sent to other (remote) worker. In other words, Storm doesn't serialize tuples when destination is local task. That's why you didn't meet error when testing with 1 worker. MapEventBean seems to be not serializable, so you need to convert to other data type. Hope this helps. Regards. Jungtaek Lim (HeartSaVioR) 2015-05-14 18:38 GMT+09:00 Asif Ihsan <[email protected]<mailto:[email protected]>>: I am using Esper in one of the bolts of the Storm topology. Bolt emit MapEventBean array (EventBean[]). With the single worker topology run smoothly. With multiple workers it gives following error in emit function call. May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke SEVERE: Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) at backtype.storm.util$fast_list_map.invoke(util.clj:771) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) ... 6 more Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeArray(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) ... 21 more May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke SEVERE: java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) at backtype.storm.util$fast_list_map.invoke(util.clj:771) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) ... 6 more Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeArray(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) ... 21 more On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <[email protected]<mailto:[email protected]>> wrote: Asif: Would love to help you, however, we will need way more details. Why don't you start with telling us what the error is? How are you detecting the error? You will want to look over both the supervisor and worker logs. Thank you for your time! +++++++++++++++++++++ Jeff Maass linkedin.com/in/jeffmaass<http://linkedin.com/in/jeffmaass> stackoverflow.com/users/373418/maassql<http://stackoverflow.com/users/373418/maassql> +++++++++++++++++++++ On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <[email protected]<mailto:[email protected]>> wrote: Hi, I recently started using Storm. I am having topology. It works well with numofworkers=1. However when I change it to 6 or more than 1. Its gives error when emitting tuples. What is the issue and how to resolve it with multiple workers. Thank you. -- Regards Muhammad Asif Ihsan -- Regards Muhammad Asif Ihsan -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior -- Regards Muhammad Asif Ihsan -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior -- Regards Muhammad Asif Ihsan
