Re: Problems with the state.backend.fs.memory-threshold parameter
Hi，Zakelly Thank you for your answer. Best， rui Zakelly Lan 于2023年10月13日周五 19:12写道： > Hi rui, > > The 'state.backend.fs.memory-threshold' configures the threshold below > which state is stored as part of the metadata, rather than in separate > files. So as a result the JM will use its memory to merge small > checkpoint files and write them into one file. Currently the > FLIP-306[1][2] is proposed to merge small checkpoint files without > consuming JM memory. This feature is currently being worked on and is > targeted for the next minor release (1.19). > > > Best, > Zakelly > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints > [2] https://issues.apache.org/jira/browse/FLINK-32070 > > On Fri, Oct 13, 2023 at 6:28 PM rui chen wrote: > > > > We found that for some tasks, the JM memory continued to increase. I set > > the parameter of state.backend.fs.memory-threshold to 0, and the JM > memory > > would no longer increase, but many small files might be written in this > > way. Does the community have any optimization plan for this area? >
Re: Problems with the state.backend.fs.memory-threshold parameter
Hi rui, The 'state.backend.fs.memory-threshold' configures the threshold below which state is stored as part of the metadata, rather than in separate files. So as a result the JM will use its memory to merge small checkpoint files and write them into one file. Currently the FLIP-306[1][2] is proposed to merge small checkpoint files without consuming JM memory. This feature is currently being worked on and is targeted for the next minor release (1.19). Best, Zakelly [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints [2] https://issues.apache.org/jira/browse/FLINK-32070 On Fri, Oct 13, 2023 at 6:28 PM rui chen wrote: > > We found that for some tasks, the JM memory continued to increase. I set > the parameter of state.backend.fs.memory-threshold to 0, and the JM memory > would no longer increase, but many small files might be written in this > way. Does the community have any optimization plan for this area?
Problems with the state.backend.fs.memory-threshold parameter
We found that for some tasks, the JM memory continued to increase. I set the parameter of state.backend.fs.memory-threshold to 0, and the JM memory would no longer increase, but many small files might be written in this way. Does the community have any optimization plan for this area?
