Hi, there, As Flink 1.11.0 released, we provide a new calculator[1] for this version. Feel free to try it and any feedback or suggestion is welcomed!
[1] https://github.com/KarmaGYZ/flink-memory-calculator/blob/master/calculator-1.11.sh Best, Yangze Guo On Wed, Apr 1, 2020 at 9:45 PM Yangze Guo <karma...@gmail.com> wrote: > > @Marta > Thanks for the tip! I'll do that. > > Best, > Yangze Guo > > On Wed, Apr 1, 2020 at 8:05 PM Marta Paes Moreira <ma...@ververica.com> wrote: > > > > Hey, Yangze. > > > > I'd like to suggest that you submit this tool to Flink Community Pages [1]. > > That way it can get more exposure and it'll be easier for users to find it. > > > > Thanks for your contribution! > > > > [1] https://flink-packages.org/ > > > > On Tue, Mar 31, 2020 at 9:09 AM Yangze Guo <karma...@gmail.com> wrote: > >> > >> Hi, there. > >> > >> In the latest version, the calculator supports dynamic options. You > >> could append all your dynamic options to the end of "bin/calculator.sh > >> [-h]". > >> Since "-tm" will be deprecated eventually, please replace it with > >> "-Dtaskmanager.memory.process.size=". > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Mar 30, 2020 at 12:57 PM Xintong Song <tonysong...@gmail.com> > >> wrote: > >> > > >> > Hi Jeff, > >> > > >> > I think the purpose of this tool it to allow users play with the memory > >> > configurations without needing to actually deploy the Flink cluster or > >> > even have a job. For sanity checks, we currently have them in the > >> > start-up scripts (for standalone clusters) and resource managers (on > >> > K8s/Yarn/Mesos). > >> > > >> > I think it makes sense do the checks earlier, i.e. on the client side. > >> > But I'm not sure if JobListener is the right place. IIUC, JobListener is > >> > invoked before submitting a specific job, while the mentioned checks > >> > validate Flink's cluster level configurations. It might be okay for a > >> > job cluster, but does not cover the scenarios of session clusters. > >> > > >> > Thank you~ > >> > > >> > Xintong Song > >> > > >> > > >> > > >> > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo <karma...@gmail.com> wrote: > >> >> > >> >> Thanks for your feedbacks, @Xintong and @Jeff. > >> >> > >> >> @Jeff > >> >> I think it would always be good to leverage exist logic in Flink, such > >> >> as JobListener. However, this calculator does not only target to check > >> >> the conflict, it also targets to provide the calculating result to > >> >> user before the job is actually deployed in case there is any > >> >> unexpected configuration. It's a good point that we need to parse the > >> >> dynamic configs. I prefer to parse the dynamic configs and cli > >> >> commands in bash instead of adding hook in JobListener. > >> >> > >> >> Best, > >> >> Yangze Guo > >> >> > >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang <zjf...@gmail.com> wrote: > >> >> > > >> >> > Hi Yangze, > >> >> > > >> >> > Does this tool just parse the configuration in flink-conf.yaml ? > >> >> > Maybe it could be done in JobListener [1] (we should enhance it via > >> >> > adding hook before job submission), so that it could all the cases > >> >> > (e.g. parameters coming from command line) > >> >> > > >> >> > [1] > >> >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > >> >> > > >> >> > > >> >> > Yangze Guo <karma...@gmail.com> 于2020年3月30日周一 上午9:40写道： > >> >> >> > >> >> >> Hi, Yun, > >> >> >> > >> >> >> I'm sorry that it currently could not handle it. But I think it is a > >> >> >> really good idea and that feature would be added to the next version. > >> >> >> > >> >> >> Best, > >> >> >> Yangze Guo > >> >> >> > >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang <myas...@live.com> wrote: > >> >> >> > > >> >> >> > Very interesting and convenient tool, just a quick question: could > >> >> >> > this tool also handle deployment cluster commands like "-tm" mixed > >> >> >> > with configuration in `flink-conf.yaml` ? > >> >> >> > > >> >> >> > Best > >> >> >> > Yun Tang > >> >> >> > ________________________________ > >> >> >> > From: Yangze Guo <karma...@gmail.com> > >> >> >> > Sent: Friday, March 27, 2020 18:00 > >> >> >> > To: user <user@flink.apache.org>; user...@flink.apache.org > >> >> >> > <user...@flink.apache.org> > >> >> >> > Subject: [Third-party Tool] Flink memory calculator > >> >> >> > > >> >> >> > Hi, there. > >> >> >> > > >> >> >> > In release-1.10, the memory setup of task managers has changed a > >> >> >> > lot. > >> >> >> > I would like to provide here a third-party tool to simulate and get > >> >> >> > the calculation result of Flink's memory configuration. > >> >> >> > > >> >> >> > Although there is already a detailed setup guide[1] and migration > >> >> >> > guide[2] officially, the calculator could further allow users to: > >> >> >> > - Verify if there is any conflict in their configuration. The > >> >> >> > calculator is more lightweight than starting a Flink cluster, > >> >> >> > especially when running Flink on Yarn/Kubernetes. User could make > >> >> >> > sure > >> >> >> > their configuration is correct locally before deploying it to > >> >> >> > external > >> >> >> > resource managers. > >> >> >> > - Get all of the memory configurations before deploying. User may > >> >> >> > set > >> >> >> > taskmanager.memory.task.heap.size and > >> >> >> > taskmanager.memory.managed.size. > >> >> >> > But they also want to know the total memory consumption of Flink. > >> >> >> > With > >> >> >> > this tool, users could get all of the memory configurations they > >> >> >> > are > >> >> >> > interested in. If anything is unexpected, they would not need to > >> >> >> > re-deploy a Flink cluster. > >> >> >> > > >> >> >> > The repo link of this tool is > >> >> >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > >> >> >> > BashJavaUtils.jar of Flink and ensures the calculation result is > >> >> >> > exactly the same as your Flink dist. For more details, please take > >> >> >> > a > >> >> >> > look at the README. > >> >> >> > > >> >> >> > Any feedback or suggestion is welcomed! > >> >> >> > > >> >> >> > [1] > >> >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > >> >> >> > [2] > >> >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > >> >> >> > > >> >> >> > Best, > >> >> >> > Yangze Guo > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Best Regards > >> >> > > >> >> > Jeff Zhang
