Hi Yangze, Does this tool just parse the configuration in flink-conf.yaml ? Maybe it could be done in JobListener [1] (we should enhance it via adding hook before job submission), so that it could all the cases (e.g. parameters coming from command line)
[1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 Yangze Guo <karma...@gmail.com> 于2020年3月30日周一 上午9:40写道： > Hi, Yun, > > I'm sorry that it currently could not handle it. But I think it is a > really good idea and that feature would be added to the next version. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:21 AM Yun Tang <myas...@live.com> wrote: > > > > Very interesting and convenient tool, just a quick question: could this > tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > > > > Best > > Yun Tang > > ________________________________ > > From: Yangze Guo <karma...@gmail.com> > > Sent: Friday, March 27, 2020 18:00 > > To: user <user@flink.apache.org>; user...@flink.apache.org < > user...@flink.apache.org> > > Subject: [Third-party Tool] Flink memory calculator > > > > Hi, there. > > > > In release-1.10, the memory setup of task managers has changed a lot. > > I would like to provide here a third-party tool to simulate and get > > the calculation result of Flink's memory configuration. > > > > Although there is already a detailed setup guide[1] and migration > > guide[2] officially, the calculator could further allow users to: > > - Verify if there is any conflict in their configuration. The > > calculator is more lightweight than starting a Flink cluster, > > especially when running Flink on Yarn/Kubernetes. User could make sure > > their configuration is correct locally before deploying it to external > > resource managers. > > - Get all of the memory configurations before deploying. User may set > > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > > But they also want to know the total memory consumption of Flink. With > > this tool, users could get all of the memory configurations they are > > interested in. If anything is unexpected, they would not need to > > re-deploy a Flink cluster. > > > > The repo link of this tool is > > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > > BashJavaUtils.jar of Flink and ensures the calculation result is > > exactly the same as your Flink dist. For more details, please take a > > look at the README. > > > > Any feedback or suggestion is welcomed! > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > > > Best, > > Yangze Guo > -- Best Regards Jeff Zhang
