tillrohrmann commented on issue #9760: [FLINK-13982][runtime] Implement memory calculation logics URL: https://github.com/apache/flink/pull/9760#issuecomment-537474186 Yes, all explicitly configured values should be respected. I agree, the default values of * Framework Heap Memory * Task Off-Heap Memory * Min/Max Shuffle Memory * JVM Metaspace * Min/Max JVM Overhead should be respected. In the end, we are trying to solve a linear programming problem for which there are known methods how to solve it. The problem should have the following form ``` 0 = jvm_overhead + jvm_metaspace + framework + task_heap + task_off_heap + shuffle + managed_memory - process_memory min_overhead <= jvm_overhead max_overhead >= jvm_overhead min_shuffle <= shuffle max_shuffle >= shuffle 0 = fraction_overhead * (jvm_overhead + jvm_metaspace + framework + task_heap + task_off_heap + shuffle + managed_memory) - jvm_overhead ``` If `shuffle` has not been set, then we have additionally `0 = fraction_shuffle * (framework + task_heap + task_off_heap + shuffle + managed_memory) - shuffle` There might be some shortcuts given the particulars of this problem. But I'm not sure whether it is exactly as you've described. I would suggest to rely on known approaches. Concerning the backwards compatibility plan, I agree that considering the legacy `fraction`, turns the linear problem into a non-linear problem. Hence, I would be ok with dropping support for it. I guess we should add a big release note and log a warning when using the legacy `fraction` option.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
