Just to clarify Xintong, you suggest that Task off-heap memory represents direct and native memory. Since we don't know how the user will allocate the memory we will add this value to -XX:MaxDirectMemorySize so that the process won't fail if the user allocates only direct memory and no native memory. Is that correct?
Cheers, Till On Wed, Sep 4, 2019 at 10:18 AM Xintong Song <tonysong...@gmail.com> wrote: > @Stephan > Not sure what do you mean by "just having this value". Are you suggesting > that having "taskmanager.memory.network" refers to "shuffle" and "other > network memory", or only "shuffle"? > > I guess what you mean is only "shuffle"? Because currently > "taskmanager.network.memory" refers to shuffle buffers only, which is "one > less config value to break". > > Thank you~ > > Xintong Song > > > > On Wed, Sep 4, 2019 at 3:42 PM Stephan Ewen <se...@apache.org> wrote: > > > If we later split the network memory into "shuffle" and "other network > > memory", I think it would make sense to split the option then. > > > > In that case "taskmanager.memory.network" would probably refer to the > total > > network memory, which would most likely be what most users actually > > configure. > > My feeling is that for now just having this value is actually easier, and > > it is one less config value to break (which is also good). > > > > On Wed, Sep 4, 2019 at 9:05 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > > > Thanks for the voting and comments. > > > > > > @Stephan > > > - The '-XX:MaxDirectMemorySize' value should not include JVM Overhead. > > > Thanks for correction. > > > - 'taskmanager.memory.framework.heap' it heap memory reserved for task > > > executor framework, which can not be allocated to task slots. I think > > users > > > should be able to configure both how many total java heap memory a task > > > executor should have and how many of the total java heap memory can be > > > allocated to task slots. > > > - Regarding network / shuffle memory, I'm with @Andrey. ATM, this > memory > > > pool (derived from "taskmanager.network.memory.[min/max/fraction]") is > > only > > > used inside NettyShuffleEnvironment as network buffers. There might be > > > other network memory usage outside the shuffle component (as @Zhijiang > > also > > > suggested), but that is not accounted by this memory pool. This is > > exactly > > > why I would suggest to change the name from 'network' to 'shuffle'. > > > - I agree that we need very good documentation to explain the memory > > pools > > > and config options, as well as WebUI to present the memory pool sizes. > I > > > would suggest to address these as follow-ups of all the three resource > > > management FLIPs, for better integration. > > > > > > @Andrey > > > - Agree with the 'shuffle' naming. See above. > > > > > > @Till > > > - My understanding is that Task Off-heap memory accounts for both > direct > > > and native memory used by the user code. I'm not sure whether we need > > > another configure option to split it. Given that we only decided (in > the > > > discussion thread) to try it out the way we set > '-XX:MaxDirectMemorySize' > > > in current design and may switch to other alternatives if it doesn't > work > > > out well, I would suggest the same for this one. I suggest that we > first > > > try it without the splitting config option, and we can easily add it if > > it > > > doesn't work well. > > > - Agree that it's really important to have good documentation for this. > > See > > > above. > > > > > > @Zhijiang > > > - Thanks for the input. My understanding is that 'shuffle memory' is a > > > portion of the task executor memory reserved for the shuffle component. > > The > > > way shuffle component use these memory (local buffer pool, netty > internal > > > memory, etc.) can be different depending on the shuffle implementation. > > The > > > task executor (outside of the shuffle implementation) should only know > > the > > > overall memory usage of the shuffle component but no need to understand > > > more details inside the shuffle implementation. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Tue, Sep 3, 2019 at 10:41 PM zhijiang <wangzhijiang...@aliyun.com > > > .invalid> > > > wrote: > > > > > > > Thanks for proposing this FLIP and also +1 on my side. > > > > > > > > @Andrey Zagrebin For the point of "network memory is actually used > more > > > > than shuffling", I guess that the component of queryable state is > also > > > > using network/netty stack atm, which is outside of shuffling. > > > > In addition, if we only consider the shuffle memory provided by > shuffle > > > > service interface, we should not only consider the memory used by > local > > > > buffer pool, but also consider the netty internal memory > > > > usages as the overhead, especially we have not the zero-copy > > improvement > > > > on dowstream read side. This issue might be out of the vote scope, > just > > > > think of we have this issue in [1]. :) > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-12110 > > > > > > > > Best, > > > > Zhijiang > > > > ------------------------------------------------------------------ > > > > From:Till Rohrmann <trohrm...@apache.org> > > > > Send Time:2019年9月3日(星期二) 15:07 > > > > To:dev <dev@flink.apache.org> > > > > Subject:Re: [VOTE] FLIP-49: Unified Memory Configuration for > > > TaskExecutors > > > > > > > > Thanks for creating this FLIP and starting the vote Xintong. > > > > > > > > +1 for the proposal from my side. > > > > > > > > I agree with Stephan that we might wanna revisit some of the > > > configuration > > > > names. > > > > > > > > If I understood it correctly, then Task Off-heap memory represents > the > > > > direct memory used by the user code, right? How would users configure > > > > native memory requirements for the user code? If it is part of Task > Off > > > > heap memory, then we need to split it to set -XX:MaxDirectMemorySize > > > > correctly or to introduce another configuration option. > > > > > > > > Given all these configuration options, I can see that users will get > > > > confused quite easily. Therefore, I would like to emphasise that we > > need > > > a > > > > very good documentation about how to properly configure Flink > processes > > > and > > > > which knobs to turn in which cases. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Sep 3, 2019 at 2:34 PM Andrey Zagrebin <and...@ververica.com > > > > > > wrote: > > > > > > > > > Thanks for starting the vote Xintong > > > > > > > > > > Also +1 for the proposed FLIP-49. > > > > > > > > > > @Stephan regarding namings: network vs shuffle. > > > > > My understanding so far was that the network memory is what we > > > basically > > > > > give to Shuffle implementations and default netty implementation > uses > > > it > > > > in > > > > > particular mostly for networking. > > > > > Are the network pools used for something else outside of the > > shuffling > > > > > scope? > > > > > > > > > > best, > > > > > Andrey > > > > > > > > > > On Tue, Sep 3, 2019 at 1:01 PM Stephan Ewen <se...@apache.org> > > wrote: > > > > > > > > > > > +1 to the proposal in general > > > > > > > > > > > > A few things seems to be a bit put of sync with the latest > > > discussions > > > > > > though. > > > > > > > > > > > > The section about JVM Parameters states that the > > > > > > -XX:MaxDirectMemorySize value is set to Task Off-heap Memory, > > Shuffle > > > > > > Memory and JVM Overhead. > > > > > > The way I understand the last discussion conclusion is that it is > > > only > > > > > the > > > > > > sum of shuffle memory and user-defined direct memory. > > > > > > > > > > > > I am someone neutral but unsure about is the separation between > > > > > > "taskmanager.memory.framework.heap" and > > > "taskmanager.memory.task.heap". > > > > > > Could that be simply combined under > "taskmanager.memory.javaheap"? > > > > > > > > > > > > It might be good to also expose these values somehow in the web > UI > > so > > > > > that > > > > > > users see immediately what amount of memory TMs assume to use for > > > what. > > > > > > > > > > > > I assume config key names and default values might be adjusted > over > > > > time > > > > > as > > > > > > we get feedback. > > > > > > - I would keep the network memory under the name > > > > > > "taskmanager.memory.network". Because network memory is actually > > used > > > > for > > > > > > more than shuffling. Also, the old config key seems good, so why > > > change > > > > > it? > > > > > > > > > > > > One thing to be aware of is that often, the Java Heap is > understood > > > as > > > > > > "managed memory" as a whole, because it is managed by the GC not > > > > > explicitly > > > > > > by the user. > > > > > > So we need to make sure that we don't confuse users by speaking > of > > > > > managed > > > > > > heap and unmanaged heap. All heap is managed in Java. Some memory > > is > > > > > > explicitly managed by Flink. > > > > > > > > > > > > Best, > > > > > > Stephan > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 3:08 PM Xintong Song < > tonysong...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I'm here to re-start the voting process for FLIP-49 [1], with > > > respect > > > > > to > > > > > > > consensus reached in this thread [2] regarding some new > comments > > > and > > > > > > > concerns. > > > > > > > > > > > > > > This voting will be open for at least 72 hours. I'll try to > close > > > it > > > > > Sep. > > > > > > > 5, 14:00 UTC, unless there is an objection or not enough votes. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html > > > > > > > > > > > > > > On Tue, Aug 27, 2019 at 9:29 PM Xintong Song < > > > tonysong...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Alright, then let's keep the discussion in the DISCUSS > mailing > > > > > thread, > > > > > > > and > > > > > > > > see whether we need to restart the vote. > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann < > > > > trohrm...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> I had a couple of comments concerning the implementation > plan. > > > > I've > > > > > > > posted > > > > > > > >> them to the original discussion thread. Depending on the > > outcome > > > > of > > > > > > this > > > > > > > >> discussion we might need to restart the vote. > > > > > > > >> > > > > > > > >> Cheers, > > > > > > > >> Till > > > > > > > >> > > > > > > > >> On Tue, Aug 27, 2019 at 11:14 AM Xintong Song < > > > > > tonysong...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Hi all, > > > > > > > >> > > > > > > > > >> > I would like to start the voting process for FLIP-49 [1], > > > which > > > > is > > > > > > > >> > discussed and reached consensus in this thread [2]. > > > > > > > >> > > > > > > > > >> > This voting will be open for at least 72 hours. I'll try > to > > > > close > > > > > it > > > > > > > >> Aug. > > > > > > > >> > 30 10:00 UTC, unless there is an objection or not enough > > > votes. > > > > > > > >> > > > > > > > > >> > Thank you~ > > > > > > > >> > > > > > > > > >> > Xintong Song > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > [1] > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > >> > [2] > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
