Hello all,
I am currently trying to setup a storm local cluster using the master
branch (2.0.0 snapshot) from github.
I am able to start the zookeeper, nimbus, supervisor and ui services
without any errors.
I wanted to execute WordCountTopology to verify the setup. The topology
gets submitted without any errors, but the worker process exit abruptly. The
following are the snippets of output from submit command.
<snip>
centos@centos-base ~/s/bin> ./storm jar
../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
org.apache.storm.starter.WordCountTopology ww6
Running: java -client -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/home/centos/storm -Dstorm.log.dir=/home/centos/storm/logs
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64
-Dstorm.conf.file= -cp
/home/centos/storm/*:/home/centos/storm/lib/*:/home/centos/storm/extlib/*:../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar:/home/centos/storm/conf:/home/centos/storm/bin
-Dstorm.jar=../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
-Dstorm.dependency.jars= -Dstorm.dependency.artifacts={}
org.apache.storm.starter.WordCountTopology ww6
21:30:11.814 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
payload for MD5-digest: -4869737168860327630:-9077312875146305878
21:30:11.897 [main] WARN o.a.s.v.ConfigValidation -
storm.messaging.netty.max_retries is a deprecated config please see class
org.apache.storm.Config.STORM_MESSAGING_NETTY_MAX_RETRIES for more information.
21:30:11.964 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus :
centos-base.novalocal:6627
21:30:11.965 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
21:30:12.003 [main] INFO o.a.s.StormSubmitter - Uploading dependencies -
jars...
21:30:12.003 [main] INFO o.a.s.StormSubmitter - Uploading dependencies -
artifacts...
21:30:12.003 [main] INFO o.a.s.StormSubmitter - Dependency Blob keys - jars :
[] / artifacts : []
21:30:12.014 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar to assigned
location:
/tmp/storm/nimbus/inbox/stormjar-dc6e6f5c-7d5c-4360-8aa3-3a7653296788.jar
21:30:12.410 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology
jar to assigned location:
/tmp/storm/nimbus/inbox/stormjar-dc6e6f5c-7d5c-4360-8aa3-3a7653296788.jar
21:30:12.411 [main] INFO o.a.s.StormSubmitter - Submitting topology ww6 in
distributed mode with conf
{"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-4869737168860327630:-9077312875146305878","topology.workers":3,"topology.debug":true}
21:30:13.705 [main] INFO o.a.s.StormSubmitter - Finished submitting topology:
ww6
</snip>
This is the snippet from the supervisor.log
<snip>
2018-03-29 21:30:19.261 o.a.s.d.s.BasicContainer SLOT_6701 [INFO] Launching
worker with command: 'java' '-cp'
'/home/centos/storm/lib-worker/*:/home/centos/storm/extlib/*:/home/centos/storm/conf:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/stormjar.jar'
'-Xmx64m' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log'
'-Dstorm.home=/home/centos/storm'
'-Dworkers.artifacts=/home/centos/storm/logs/workers-artifacts'
'-Dstorm.id=ww6-4-1522377012'
'-Dworker.id=a8755d1f-400c-4683-9d19-4bba2076e603' '-Dworker.port=6701'
'-Dstorm.log.dir=/home/centos/storm/logs'
'-Dlog4j.configurationFile=/home/centos/storm/log4j2/worker.xml'
'-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector'
'-Dstorm.local.dir=/tmp/storm' '-Dworker.memory_limit_mb=1152'
'org.apache.storm.LogWriter' 'java' '-server' '-Dlogging.sensitivity=S3'
'-Dlogfile.name=worker.log' '-Dstorm.home=/home/centos/storm'
'-Dworkers.artifacts=/home/centos/storm/logs/workers-artifacts'
'-Dstorm.id=ww6-4-1522377012'
'-Dworker.id=a8755d1f-400c-4683-9d19-4bba2076e603' '-Dworker.port=6701'
'-Dstorm.log.dir=/home/centos/storm/logs'
'-Dlog4j.configurationFile=/home/centos/storm/log4j2/worker.xml'
'-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector'
'-Dstorm.local.dir=/tmp/storm' '-Dworker.memory_limit_mb=1152' '-Xmx1152m'
'-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log' '-XX:+PrintGCDateStamps'
'-XX:+PrintGCTimeStamps' '-XX:+UseGCLogFileRotation'
'-XX:NumberOfGCLogFiles=10' '-XX:GCLogFileSize=1M'
'-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=artifacts/heapdump'
'-Djava.library.path=/tmp/storm/supervisor/stormdist/ww6-4-1522377012/resources/Linux-amd64:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64'
'-Dstorm.conf.file=' '-Dstorm.options='
'-Djava.io.tmpdir=/tmp/storm/workers/a8755d1f-400c-4683-9d19-4bba2076e603/tmp'
'-cp'
'/home/centos/storm/lib-worker/*:/home/centos/storm/extlib/*:/home/centos/storm/conf:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/stormjar.jar'
'org.apache.storm.daemon.worker.Worker' 'ww6-4-1522377012'
'21e05477-6017-4062-bf0f-342b9f2cceca-127.0.0.1' '6701'
'a8755d1f-400c-4683-9d19-4bba2076e603'.
2018-03-29 21:30:19.262 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE
WAITING_FOR_BLOB_LOCALIZATION msInState: 12 -> WAITING_FOR_WORKER_START
msInState: 0 topo:ww6-4-1522377012 worker:159f603a-f552-43fd-b363-d2e744cc2131
2018-03-29 21:30:19.262 o.a.s.d.s.Slot SLOT_6700 [INFO] SLOT 6700: Changing
current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012,
executors:[ExecutorInfo(task_start:22, task_end:22),
ExecutorInfo(task_start:16, task_end:16), ExecutorInfo(task_start:28,
task_end:28), ExecutorInfo(task_start:10, task_end:10),
ExecutorInfo(task_start:4, task_end:4), ExecutorInfo(task_start:19,
task_end:19), ExecutorInfo(task_start:13, task_end:13),
ExecutorInfo(task_start:7, task_end:7), ExecutorInfo(task_start:1, task_end:1),
ExecutorInfo(task_start:25, task_end:25)],
resources:WorkerResources(mem_on_heap:1280.0, mem_off_heap:0.0, cpu:100.0,
shared_mem_on_heap:0.0, shared_mem_off_heap:0.0,
resources:{offheap.memory.mb=0.0, onheap.memory.mb=1280.0,
cpu.pcore.percent=100.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:19.281 o.a.s.d.s.Slot SLOT_6701 [INFO] STATE
WAITING_FOR_BLOB_LOCALIZATION msInState: 22 -> WAITING_FOR_WORKER_START
msInState: 0 topo:ww6-4-1522377012 worker:a8755d1f-400c-4683-9d19-4bba2076e603
2018-03-29 21:30:19.281 o.a.s.d.s.Slot SLOT_6701 [INFO] SLOT 6701: Changing
current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012,
executors:[ExecutorInfo(task_start:14, task_end:14),
ExecutorInfo(task_start:20, task_end:20), ExecutorInfo(task_start:26,
task_end:26), ExecutorInfo(task_start:8, task_end:8),
ExecutorInfo(task_start:2, task_end:2), ExecutorInfo(task_start:17,
task_end:17), ExecutorInfo(task_start:11, task_end:11),
ExecutorInfo(task_start:5, task_end:5), ExecutorInfo(task_start:23,
task_end:23)], resources:WorkerResources(mem_on_heap:1152.0, mem_off_heap:0.0,
cpu:90.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0,
resources:{offheap.memory.mb=0.0, onheap.memory.mb=1152.0,
cpu.pcore.percent=90.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:19.310 o.a.s.d.s.Slot SLOT_6702 [INFO] STATE
WAITING_FOR_BLOB_LOCALIZATION msInState: 58 -> WAITING_FOR_WORKER_START
msInState: 0 topo:ww6-4-1522377012 worker:a410536b-56ef-4eb3-83d5-4810e92c2c6c
2018-03-29 21:30:19.310 o.a.s.d.s.Slot SLOT_6702 [INFO] SLOT 6702: Changing
current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012,
executors:[ExecutorInfo(task_start:18, task_end:18),
ExecutorInfo(task_start:24, task_end:24), ExecutorInfo(task_start:12,
task_end:12), ExecutorInfo(task_start:6, task_end:6),
ExecutorInfo(task_start:15, task_end:15), ExecutorInfo(task_start:3,
task_end:3), ExecutorInfo(task_start:9, task_end:9),
ExecutorInfo(task_start:21, task_end:21), ExecutorInfo(task_start:27,
task_end:27)], resources:WorkerResources(mem_on_heap:1152.0, mem_off_heap:0.0,
cpu:90.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0,
resources:{offheap.memory.mb=0.0, onheap.memory.mb=1152.0,
cpu.pcore.percent=90.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:22.548 o.a.s.d.s.BasicContainer Thread-211 [INFO] Worker
Process a410536b-56ef-4eb3-83d5-4810e92c2c6c exited with code: 1
2018-03-29 21:30:22.551 o.a.s.d.s.BasicContainer Thread-209 [INFO] Worker
Process 159f603a-f552-43fd-b363-d2e744cc2131 exited with code: 1
2018-03-29 21:30:22.725 o.a.s.d.s.BasicContainer Thread-210 [INFO] Worker
Process a8755d1f-400c-4683-9d19-4bba2076e603 exited with code: 1
2018-03-29 21:30:35.795 o.a.s.d.s.t.SupervisorHealthCheck timer [INFO] Running
supervisor healthchecks...
2018-03-29 21:30:35.795 o.a.s.h.HealthChecker timer [INFO] The supervisor
healthchecks succeeded.
</snip>
I am currently using a CentOS Linux release 7.4.1708 (Core). I installed storm
version 1.2 and it works fine. I tried the same configuration (for storm v2.0)
on an ubuntu machine, and storm-starter works fine there.
I am not sure what I'm missing. Can you let me know if redhat needs any
additional configuration steps, or any steps to help debug my setup further?
Please let me know if you need any additional data/files from me.
thanks for your time,
regards,
ravi
