Ethan Li created STORM-2820:
-------------------------------
Summary: validateTopologyWorkerMaxHeapSizeConfigs function never
picks up the value set by nimbus
Key: STORM-2820
URL: https://issues.apache.org/jira/browse/STORM-2820
Project: Apache Storm
Issue Type: Bug
Reporter: Ethan Li
Assignee: Ethan Li
https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2548
{code:java}
@VisibleForTesting
static void validateTopologyWorkerMaxHeapSizeConfigs(
Map<String, Object> stormConf, StormTopology topology) {
double largestMemReq = getMaxExecutorMemoryUsageForTopo(topology,
stormConf);
double topologyWorkerMaxHeapSize =
ObjectReader.getDouble(stormConf.get(Config.TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB),
768.0);
if (topologyWorkerMaxHeapSize < largestMemReq) {
throw new IllegalArgumentException(
"Topology will not be able to be successfully scheduled: Config
"
+ "TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB="
+ topologyWorkerMaxHeapSize
+ " < " + largestMemReq + " (Largest memory requirement of
a component in the topology)."
+ " Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a
larger amount");
}
}
{code}
The topologyWorkerMaxHeapSize in the above code is either the value from
topology configuration (set by the topology) or 768.0. It never picks up the
value set by nimbus .
A test:
I set
{code:java}
topology.worker.max.heap.size.mb: 2000.0
{code}
in storm.yaml.
And have the WordCountTopology to have the WordCount bolt with memory load of
1024MB.
{code:java}
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new
Fields("word")).setMemoryLoad(1024);
{code}
I got an error when submitting this topology. The nimbus log shows
{code:java}
2017-11-15 19:46:43.085 o.a.s.d.n.Nimbus pool-14-thread-2 [WARN] Topology
submission exception. (topology name='wc')
java.lang.IllegalArgumentException: Topology will not be able to be
successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 1024.0
(Largest memory requirement of a component in the topology). Perhaps set
TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
