If I create 6 workers for 5 bolts and 1 spout. Then how will these spout and bolts be distributed among workers.
On Thu, May 14, 2015 at 5:34 PM, 임정택 <[email protected]> wrote: > Yes, right. > Worker is a JVM instance which can contains one or more executors, each > executor is spout, bolt, acker, etc. > > 2015년 5월 14일 목요일, Asif Ihsan<[email protected]>님이 작성한 메시지: > > You are right about it. What does multiple workers means. Does it mean >> that when I run topology with single worker than i will see one single >> process handling all the bolts and when running 6 workers for 5 bolts and 1 >> spout than i will see 6 separate processes running each bolt and spout? M i >> right.If not then what does it mean? >> >> >> On Thu, May 14, 2015 at 2:46 PM, 임정택 <[email protected]> wrote: >> >>> Hi. >>> >>> Storm serializes tuples when tuple should be sent to other (remote) >>> worker. >>> In other words, Storm doesn't serialize tuples when destination is local >>> task. That's why you didn't meet error when testing with 1 worker. >>> >>> MapEventBean seems to be not serializable, so you need to convert to >>> other data type. >>> >>> Hope this helps. >>> >>> Regards. >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> >>> 2015-05-14 18:38 GMT+09:00 Asif Ihsan <[email protected]>: >>> >>>> I am using Esper in one of the bolts of the Storm topology. Bolt emit >>>> MapEventBean array (EventBean[]). With the single worker topology run >>>> smoothly. With multiple workers it gives following error in emit function >>>> call. >>>> >>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke >>>> SEVERE: Async loop died! >>>> java.lang.RuntimeException: java.lang.RuntimeException: >>>> java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) >>>> at >>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) >>>> at >>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) >>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) >>>> at clojure.lang.AFn.run(AFn.java:24) >>>> at java.lang.Thread.run(Unknown Source) >>>> Caused by: java.lang.RuntimeException: >>>> java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at >>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) >>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) >>>> at >>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) >>>> at >>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) >>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) >>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) >>>> at >>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) >>>> at >>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) >>>> at >>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) >>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771) >>>> at >>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) >>>> at >>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) >>>> at >>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) >>>> ... 6 more >>>> Caused by: java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source) >>>> at java.io.ObjectOutputStream.writeArray(Unknown Source) >>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source) >>>> at java.io.ObjectOutputStream.writeObject(Unknown Source) >>>> at >>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) >>>> ... 21 more >>>> >>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke >>>> SEVERE: >>>> java.lang.RuntimeException: java.lang.RuntimeException: >>>> java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) >>>> at >>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) >>>> at >>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73) >>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) >>>> at clojure.lang.AFn.run(AFn.java:24) >>>> at java.lang.Thread.run(Unknown Source) >>>> Caused by: java.lang.RuntimeException: >>>> java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at >>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24) >>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) >>>> at >>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87) >>>> at >>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17) >>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) >>>> at >>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) >>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) >>>> at >>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) >>>> at >>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27) >>>> at >>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99) >>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771) >>>> at >>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99) >>>> at >>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238) >>>> at >>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43) >>>> at >>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) >>>> ... 6 more >>>> Caused by: java.io.NotSerializableException: >>>> com.espertech.esper.event.map.MapEventBean >>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source) >>>> at java.io.ObjectOutputStream.writeArray(Unknown Source) >>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source) >>>> at java.io.ObjectOutputStream.writeObject(Unknown Source) >>>> at >>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) >>>> ... 21 more >>>> >>>> >>>> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <[email protected]> >>>> wrote: >>>> >>>>> Asif: >>>>> >>>>> Would love to help you, however, we will need way more details. Why >>>>> don't you start with telling us what the error is? How are you detecting >>>>> the error? >>>>> >>>>> You will want to look over both the supervisor and worker logs. >>>>> >>>>> Thank you for your time! >>>>> >>>>> +++++++++++++++++++++ >>>>> Jeff Maass >>>>> linkedin.com/in/jeffmaass >>>>> stackoverflow.com/users/373418/maassql >>>>> +++++++++++++++++++++ >>>>> >>>>> >>>>> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I recently started using Storm. I am having topology. It works well >>>>>> with numofworkers=1. However when I change it to 6 or more than 1. Its >>>>>> gives error when emitting tuples. What is the issue and how to resolve it >>>>>> with multiple workers. Thank you. >>>>>> >>>>>> -- >>>>>> Regards >>>>>> Muhammad Asif Ihsan >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards >>>> Muhammad Asif Ihsan >>>> >>> >>> >>> >>> -- >>> Name : 임 정택 >>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >>> Twitter : http://twitter.com/heartsavior >>> LinkedIn : http://www.linkedin.com/in/heartsavior >>> >> >> >> >> -- >> Regards >> Muhammad Asif Ihsan >> > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > > -- Regards Muhammad Asif Ihsan
