Hey, I noticed that the topology jvm arguments and ram map are passed in as arguments to the heron executor... I'm curious if there is a reason why they are not read directly from the topology definition file that is also provided as an argument and should be available at the time the executor runs.
The reason I ask is I want to add a few more settings around setting the native memory side of the JVM https://github.com/apache/incubator-heron/blob/master/heron/executor/src/python/heron_executor.py#L550-L552. Currently the values for native memory overhead are hard coded and in practice I have found that native memory for a topology can both be quick variable depending on the implementation and also be significantly higher in the general case depending on thread usage and garbage collector used. I have found that the best way to model this overhead is as a percentage of the RAM that is passed in. My idea would be to take the amount of RAM specified in the RAM map and reduce it by some percentage to account for the native memory overhead of the JVM. The consequence of not doing this, at least in auroras case, is the containers get killed for using too much memory after a period of execution because the native memory overhead of the jvm is not accounted. My feeling is this should be a jvm specific setting with some sensible default so I was thinking about making it similar to the ram map so that it can be tuned per jvm instance instead of a single setting applying to all running instances. My thought was that pass it in as another argument to the heron_executor feels wrong which lead me down the path of determining whether this information can be read directly from the topology definition and it looks like it can. My main question is around the decision to pass in the ram map and jvm opts explicitly to the heron_executor and whether there are reasons for doing this that I haven't considered or if it was a design decision that was carried forward. This is also interesting from the perspective of topologies written in other languages (python for example) since these arguments would not apply in those cases but still exist. Best, Andrew Jorgensen @ajorgensen
