All,
After further investigation, I found that the issue does not occur if I run
against Apache Zookeeper rather than Confluent Zookeeper.
I am not sure why it won't work with Confluent Zookeeper. I searched on the
Internet (both on the Confluent web site and elsewhere) for differences between
Confluent Zookeeper and Apache Zookeeper (which might explain the difference in
Storm behavior) - but have so far come up empty. So if anyone knows of reasons
for the issue with Confluent's Zookeeper and/or how to resolve it, I would much
appreciate it if you can share.
Here is my Docker-compose YAML file entry for Apache Zookeeper (I only had 1
instance):
zookeeper:
image: zookeeper
container_name: apache-zookeeper
restart: always
hostname: zookeeper
ports:
- 2181:2181
environment:
# ZOO_MY_ID: 1
# ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181
server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
NOTE: I still had to override worker.childopts (to remove GC logging options)
in the commands for starting nimbus and supervisor still required. Overriding
them via storm.yaml or via env variables in the docker-compose file did not
work even after the change in Zookeeper.
Thanks,
Shashi
From: Prabhakar, Shashi <[email protected]>
Sent: Monday, May 3, 2021 1:45 PM
To: [email protected]
Subject: RE: [Apache Storm 2.2.0] No spout activity from ExclamationTopology
topology on remote Apache Storm 2.2.0 cluster (running in Docker) due to worker
JVM creation failure (Unrecognized VM creation option 'PrintGCDateStamps')
All,
I am still looking for a solution for the issue below.
I have tried the following since I sent the original email below:
1. Loaded a different topology (word-count) - results were the same. So the
issue is not specific to the Exclamation Topology.
2. Searched all files in both nimbus and supervisor for instances of the
unrecognized VM option - I did not find any.
3. I cloned the Storm repo from Github and searched the code base - only
occurrences were in default.yaml and integration-test/config/storm.yaml.
* I believe presence in default.yaml is expected - and is supposed to be
overridden by settings in storm.yaml and the command line (I am using the
latter).
I am now looking through the code to see if I can tell what might be happening.
If anyone knows of another location where the code may be getting the GC
logging settings or any other possible reasons and/or workarounds/fixes for the
issue, I would greatly appreciate it if you could share.
Thanks,
Shashi
From: Prabhakar, Shashi
<[email protected]<mailto:[email protected]>>
Sent: Friday, April 30, 2021 12:39 PM
To: [email protected]<mailto:[email protected]>
Subject: [Apache Storm 2.2.0] No spout activity from ExclamationTopology
topology on remote Apache Storm 2.2.0 cluster (running in Docker) due to worker
JVM creation failure (Unrecognized VM creation option 'PrintGCDateStamps')
*External Message* - Use caution before opening links or attachments
Hello all,
I am trying to submit and run the ExclamationTopology topology (from the Apache
Storm 2.2 Storm starter jar file) in remote mode on an Apache Storm 2.2.0
cluster running in Docker. The topology was successfully submitted (it shows
up in Storm UI) - but I am not seeing any activity from the spout. This issue
does NOT occur (Spout activity is observed) if I submit the topology in local
mode.
Investigation findings and resolutions attempted:
* The Worker logs (on the supervisor node) show the following:
2021-04-30 14:34:10.781 STDERR Thread-0 [INFO] Unrecognized VM option
'PrintGCDateStamps'
2021-04-30 14:34:10.781 STDERR Thread-1 [INFO] [0.001s][warning][gc] -Xloggc is
deprecated. Will use -Xlog:gc:artifacts/gc.log instead.
2021-04-30 14:34:10.800 STDERR Thread-0 [INFO] Error: Could not create the Java
Virtual Machine.
2021-04-30 14:34:10.802 STDERR Thread-0 [INFO] Error: A fatal exception has
occurred. Program will exit.
* I found 2 reports of this issue on the Internet where the root causes
matched what I was seeing on my environment:
* Outdated default JVM GC Logging options in worker.childopts:
https://stackoverflow.com/questions/56902645/topology-just-idles-on-storm-in-remote-mode<https://urldefense.com/v3/__https:/stackoverflow.com/questions/56902645/topology-just-idles-on-storm-in-remote-mode__;!!In4Qlw!5ekC-DrDKASw4Z6oGRpw6qP5oqnB1ScA4dUySyomHN531RneaqYYWc_bTLBe9LdJww$>
* Mismatch in JDK versions used to build Storm and the topology jars:
https://stackoverflow.com/questions/58405388/storm-topology-not-executing-in-cluster<https://urldefense.com/v3/__https:/stackoverflow.com/questions/58405388/storm-topology-not-executing-in-cluster__;!!In4Qlw!5ekC-DrDKASw4Z6oGRpw6qP5oqnB1ScA4dUySyomHN531RneaqYYWc_bTLDvgEXDbQ$>
* I deleted PrintGCDateStamps and all other GC print settings) from the
worker.childopts configuration option as recommended in the first post and
verified that this change took effect as follows:
* Nimbus and Topology configuration settings in Storm UI show that the
worker_childopts setting no longer has these settings.
* The nimbus and supervisor logs no longer show the GC logging options
for worker.childopts in the commands when each is started up.
* NOTE: I am sending the modified value of worker.childopts in the
startup commands for nimbus and supervisor (see Docker compose file below) -
had to do this as the recommended way of overriding the default value in
storm.yaml did not work for me. I will report this in a separate email.
* I also verified (using the javap utility) that the version of JDK used to
build ExclamationTopology.class is identical to that used to build the class
files in Apache Storm 2.2.0 server, client and core (I compared with a randomly
picked class file from each of the latter).
* However, the issue is continuing to occur: The worker logs continue to
report the unrecognized VM option.
* I have tried restarting zookeeper and the 3 Storm containers as well as
completely deleting and recreating them - but neither had any effect.
Does anyone know of/have any thoughts on other possible root cause(s) and/or
what else I can try to resolve it?
Please see below for additional information on my environment. Please let me
know if you need any other information/have any questions and/or if I should
log a JIRA ticket.
Any help would be greatly appreciated.
Thanks,
Shashi
Additional Information:
* Apache Storm 2.2.0 Docker image downloaded from Docker Hub:
https://hub.docker.com/_/storm<https://urldefense.com/v3/__https:/hub.docker.com/_/storm__;!!In4Qlw!5ekC-DrDKASw4Z6oGRpw6qP5oqnB1ScA4dUySyomHN531RneaqYYWc_bTLCnIDOqgA$>
* Storm cluster configuration: 1 Nimbus instance, 1 Supervisor instance, 1
Nimbus UI instance
* ExclamationTopology topology being loaded is from the Storm Starter 2.2.0
jar downloaded from Maven Repository:
https://mvnrepository.com/artifact/org.apache.storm/storm-starter/2.2.0<https://urldefense.com/v3/__https:/mvnrepository.com/artifact/org.apache.storm/storm-starter/2.2.0__;!!In4Qlw!5ekC-DrDKASw4Z6oGRpw6qP5oqnB1ScA4dUySyomHN531RneaqYYWc_bTLAnTjblvQ$>
* Commands to submit the topology (executed in the CLI window of the nimbus
container in Docker Desktop - current directory /apache-storm-2.2.0):
* Remote (issue occurs): storm jar examples/storm-starter-2.2.0.jar
org.apache.storm.starter.ExclamationTopology
* Local (issue does not occur): storm local
examples/storm-starter-2.2.0.jar org.apache.storm.starter.ExclamationTopology
* Running everything on Windows 10 laptop.
* Docker Compose file (showing only the services related to Apache Storm -
the file does have entries for other services needed for our application):
zookeeper:
image: confluentinc/cp-zookeeper:6.0.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
nimbus:
image: storm
container_name: nimbus
command: storm nimbus -c worker.childopts="-Xmx%HEAP-MEM%m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
depends_on:
- zookeeper
links:
- zookeeper
restart: always
ports:
- 6627:6627
volumes:
- ./storm/nimbus/conf:/apache-storm-2.2.0/conf
- ./storm/nimbus/examples:/apache-storm-2.2.0/examples
- ./storm/nimbus/logs:/logs
user: root
nimbus-ui:
image: storm
container_name: nimbus-ui
command: storm ui
depends_on:
- nimbus
links:
- nimbus
restart: always
ports:
- 8080:8080
volumes:
- ./storm/nimbus-ui/conf:/apache-storm-2.2.0/conf
- ./storm/nimbus-ui/logs:/logs
- ./storm/nimbus-ui/examples:/apache-storm-2.2.0/examples
supervisor:
image: storm
container_name: supervisor
command: storm supervisor -c worker.childopts="-Xmx%HEAP-MEM%m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
depends_on:
- nimbus
- zookeeper
links:
- nimbus
- zookeeper
restart: always
volumes:
- ./storm/supervisor/conf:/apache-storm-2.2.0/conf
- ./storm/supervisor/logs:/logs
* Storm.yaml (Nimbus) - commented lines omitted
storm.zookeeper.servers:
- "localhost"
nimbus.seeds: "localhost"
### ui.* configs are for the master
ui.host: localhost
ui.port: 8080
ui.header.buffer.bytes: 4096
### worker.* configs are for task workers
#worker.childopts: "-Xmx%HEAP-MEM%m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=artifacts/heapdump"
* Storm.yaml (Supervisor) - commented lines omitted.
storm.zookeeper.servers:
- "localhost"
nimbus.seeds: "localhost"
### worker.* configs are for task workers
#worker.childopts: "-Xmx%HEAP-MEM%m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=artifacts/heapdump"
