Yes, allocate more memory to your Storm workers.
On Wed, Jun 14, 2017 at 1:37 PM, Laurens Vets <[email protected]> wrote:
> Hello list,
>
> One of the Storm workers dies with the following error message:
>
> 2017-06-14 11:17:32.503 o.a.s.util [ERROR] Async loop died!
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272)
> ~[stormjar.jar:?]
> at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:626)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:548)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:354)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:286)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:224)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
> ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2017-06-14 11:17:40.058 o.a.s.d.executor [ERROR]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272)
> ~[stormjar.jar:?]
> at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:626)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:548)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:354)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:286)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:224)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
> ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2017-06-14 11:18:08.898 o.a.s.util [ERROR] Halting process: ("Worker died")
>
> As far as I can see, it's automatically restarted, but then dies again.
>
> Might this be due to only having 64MB of memory available:
>
> 2017-06-14 11:06:35.034 o.a.s.d.supervisor [INFO] Launching worker with
> command: '/usr/jdk64/jdk1.8.0_77/bin/java' '-cp'
> '/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.
> 0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/
> lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-
> 3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/
> usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.
> 5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.
> 3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-
> 37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/
> objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-
> 1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.
> jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/
> hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.
> 0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/
> usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.
> 0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.
> 0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-
> supervisor/conf:/data1/hadoop/storm/supervisor
> /stormdist/indexing-5-1497457977/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf'
> '-Xmx64m' '-Dlogfile.name=worker.log' '-Dstorm.home=/usr/hdp/2.5.3.0-37/storm'
> '-Dworkers.artifacts=/var/log/storm/workers-artifacts'
> '-Dstorm.id=indexing-5-1497457977'
> '-Dworker.id=98d9fa65-b025-4924-9d4a-8f77324f9cf2'
> '-Dworker.port=6705' '-Dstorm.log.dir=/var/log/storm'
> '-Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml'
> '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector'
> 'org.apache.storm.LogWriter' '/usr/jdk64/jdk1.8.0_77/bin/java' '-server'
> '-Xmx1024m' '-javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxe
> tric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wiref
> ormat31x=true,mode=multicast,config=/usr/hdp/current/storm-
> client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6705_JVM'
> '-Djava.library.path=/data1/hadoop/storm/supervisor/stormdis
> t/indexing-5-1497457977/resources/Linux-amd64:/data1/
> hadoop/storm/supervisor/stormdist/indexing-5-
> 1497457977/resources:/usr/local/lib:/opt/local/lib:/usr/lib:
> /usr/hdp/current/storm-client/lib' '-Dlogfile.name=worker.log'
> '-Dstorm.home=/usr/hdp/2.5.3.0-37/storm'
> '-Dworkers.artifacts=/var/log/storm/workers-artifacts'
> '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/var/log/storm'
> '-Djava.io.tmpdir=/data1/hadoop/storm/workers/98d9fa65-b025-4924-9d4a-8f77324f9cf2/tmp'
> '-Dlogging.sensitivity=S3' '-Dlog4j.configurationFile=/us
> r/hdp/2.5.3.0-37/storm/log4j2/worker.xml' '-DLog4jContextSelector=org.ap
> ache.logging.log4j.core.selector.BasicContextSelector'
> '-Dstorm.id=indexing-5-1497457977'
> '-Dworker.id=98d9fa65-b025-4924-9d4a-8f77324f9cf2'
> '-Dworker.port=6705' '-cp' '/usr/hdp/2.5.3.0-37/storm/lib
> /asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.
> jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/
> hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/
> storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/
> log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-
> slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-
> impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:
> /usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.
> 5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-
> 37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/
> lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/
> slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-
> core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/
> storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/
> storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/
> ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-
> supervisor/conf:/data1/hadoop/storm/supervisor
> /stormdist/indexing-5-1497457977/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf'
> 'org.apache.storm.daemon.worker' 'indexing-5-1497457977'
> '9c90251f-54cc-42d7-8c4e-6f3bb2528752' '6705'
> '98d9fa65-b025-4924-9d4a-8f77324f9cf2'
>
> Any ideas on what might be happening?
>
