Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/149
Overall ran well. Increasing the indexWriter executors to 2 didn't create
the ConcurrentModificationException. Increasing the kafkaSpout executors to 2
gave me the following exception, but the topology recovered.
I'm +1 on the patch, but I've left my test rig up in case you wish to do
some additional debugging.
Trace:
2016-06-10 22:23:36.438 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.OptionalDataException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$fn__5495$fn__5508$fn__5559.invoke(executor.clj:808)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.lang.RuntimeException: java.io.OptionalDataException
at
backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)
~[kryo-2.21.jar:?]
at
backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
Caused by: java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
~[?:1.8.0_40]
at java.util.HashMap.readObject(HashMap.java:1394) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_40]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_40]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_40]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40]
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
~[?:1.8.0_40]
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
~[?:1.8.0_40]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
~[?:1.8.0_40]
at
backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)
~[kryo-2.21.jar:?]
at
backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
2016-06-10 22:23:36.441 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.OptionalDataException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$fn__5495$fn__5508$fn__5559.invoke(executor.clj:808)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.lang.RuntimeException: java.io.OptionalDataException
at
backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)
~[kryo-2.21.jar:?]
at
backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
Caused by: java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
~[?:1.8.0_40]
at java.util.HashMap.readObject(HashMap.java:1394) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_40]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_40]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_40]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40]
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
~[?:1.8.0_40]
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
~[?:1.8.0_40]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
~[?:1.8.0_40]
at
backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
~[kryo-2.21.jar:?]
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)
~[kryo-2.21.jar:?]
at
backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
2016-06-10 22:23:36.467 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
at
backtype.storm.daemon.worker$fn__6068$fn__6069.invoke(worker.clj:636)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at
backtype.storm.daemon.executor$mk_executor_data$fn__5328$fn__5329.invoke(executor.clj:256)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:485)
[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
