Thanks Shammon for the proposal. For me it is more appropriate to leave the memory management to the computing engine.
But I have a small question about this proposal. If the engine's memory is not configured properly, is it possible to cause a deadlock when requesting for memory from the engine's managed memory? Is it necessary to add some memory checking or timeout mechanism here? Thanks, Ming Li Shammon FY <[email protected]> 于2023年4月19日周三 09:57写道： > Hi devs: > > I would like to start a discussion of PIP-1: Improve Shared Writer Buffer > Pool For Sink [1]. Currently Paimon sink task creates a heap memory pool > which is shared by writers. When there are multiple tasks in an Executor, > it may cause FullGC, performance issues and even OOM. > > This PIP aims to improve the buffer pool for writers in Paimon tasks. > Paimon tasks can create memory pools based on Executor Memory which will be > managed by Executor, such as Managed Memory in Flink TaskManager. It will > improve the stability and performance of sinks by managing writer buffers > for multiple tasks through Executor. > > Looking forward to your feedback, thanks. > > > [1] > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-1%3A+Improve+Shared+Writer+Buffer+Pool+For+Sink > > > Best, > Shammon FY >
