I think we need an interpretation of "-tm" regardless of what is in the default configuration, because we can always have a modified configuration and then a user passes the "-tm" flag.
I kind of like the first proposal: Interpret "-tm" as "override memory size config and set the Yarn TM container size." It would mean exactly ignoring "taskmanager.memory.flink.size" and using the "-tm" value as " "taskmanager.memory.process.size". That does not sound too bad to me. Best, Stephan On Mon, Jan 13, 2020 at 5:35 PM Andrey Zagrebin <azagre...@apache.org> wrote: > Hi all, > > While working on changing process memory to Flink memory in default > configuration, Xintong encountered a problem. > When -tm option is used to rewrite container memory, basically process > memory, it can collide with the default Flink memory. > For legacy users it should not be a problem as we adjusted the legacy heap > size option to be interpreted differently for standalone and container > modes. > > One solution could be to say in -tm docs that we rewrite both options under > the hood: process and Flink memory, basically unset Flink memory from yaml > config. > The downside is that this adds more magic. > > Alternatively, we can keep process memory in default config and, as > mentioned before, increase it to maintain the user experience by matching > the previous default setting for heap (now Flink in standalone) size. > The Flink memory can be mentioned in process memory comment as a simpler > alternative which does not require accounting for JVM overhead. > The downside is again more confusion while trying out Flink and tuning > memory at the same time. > On the other hand, if memory already needs to be tuned it should > quite quickly lead to the necessity of understanding the memory model in > Flink. > > Best, > Andrey > > On Thu, Jan 9, 2020 at 12:27 PM Stephan Ewen <se...@apache.org> wrote: > > > Great! Thanks, guys, for the continued effort on this topic! > > > > On Thu, Jan 9, 2020 at 5:27 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > > > Thanks all for the discussion. I believe we have get consensus on all > the > > > open questions discussed in this thread. > > > > > > Since Andrey already create a jira ticket for renaming shuffle memory > > > config keys with "taskmanager.memory.network.*", I'll create ticket for > > the > > > other topic that puts flink.size in flink-conf.yaml. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Wed, Jan 8, 2020 at 9:39 PM Andrey Zagrebin <azagre...@apache.org> > > > wrote: > > > > > > > It also looks to me that we should only swap network and memory in > the > > > > option names: 'taskmanager.memory.network.*'. > > > > There is no strong consensus towards using new 'shuffle' naming so we > > can > > > > just rename it to 'taskmanager.memory.network.*' as 'shuffle' naming > > has > > > > never been released. > > > > When we have other shuffle services and start advertising more this > > > concept > > > > in Flink, we could revisit again the whole naming for this concept. > > > > https://jira.apache.org/jira/browse/FLINK-15517 > > > > > > > > > >
