Ethan Li created STORM-3667:
-------------------------------
Summary: Exception on submitting topology due to incorrect worker
max heap size should be more straightforward
Key: STORM-3667
URL: https://issues.apache.org/jira/browse/STORM-3667
Project: Apache Storm
Issue Type: Improvement
Reporter: Ethan Li
Submitting a topology can fail when the largest component heap size is larger
than
{code:java}
topology.worker.max.heap.size.mb
{code}
For example, when you submit a topology with the following command
{code:java}
bin/storm jar /home/y/lib64/jars/storm-starter.jar
org.apache.storm.starter.WordCountTopology -c
topology.worker.max.heap.size.mb=100 -c
topology.component.resources.onheap.memory.mb=120 wc
{code}
While the nimbus log shows
{code:java}
2020-07-06 21:51:12.026 o.a.s.d.n.Nimbus pool-29-thread-45 [WARN] Topology
submission exception. (topology name='wc')
java.lang.IllegalArgumentException: Topology will not be able to be
successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=100.0 < 120.0
(Largest memory requirement of a component in the topology). Perhaps set
TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
at
org.apache.storm.daemon.nimbus.Nimbus.validateTopologyWorkerMaxHeapSizeConfigs(Nimbus.java:1650)
~[storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.daemon.nimbus.Nimbus.submitTopologyWithOpts(Nimbus.java:3145)
[storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.daemon.nimbus.Nimbus.submitTopology(Nimbus.java:3107)
[storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
{code}
the exception returned to users is
{code:java}
21:51:12.012 [main] INFO o.a.s.StormSubmitter - Submitting topology wc in
distributed mode with conf
{"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8967725949793348049:-5020411463316947973","topology.workers":3,"topology.component.resources.onheap.memory.mb":120,"topology.worker.max.heap.size.mb":100,"topology.debug":true}
java.lang.RuntimeException: org.apache.storm.thrift.TApplicationException:
Internal error processing submitTopology
at
org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:291)
at
org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:210)
at
org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:173)
at
org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119)
at
org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58)
at
org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68)
at
org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36)
Caused by: org.apache.storm.thrift.TApplicationException: Internal error
processing submitTopology
at
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:319)
at
org.apache.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:303)
at
org.apache.storm.StormSubmitter.submitTopologyInDistributeMode(StormSubmitter.java:344)
at
org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:279)
... 6 more
{code}
which is not very helpful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
