Thanks Shammon for the proposal. The PIP looks very good! And I think it is useful.
Can we just convert Flink MemorySegment to Paimon MemorySegment? (They are the same). Maybe we can just use some reflection method to get offHeapBuffer and heapMemory from Flink MemorySegment. Best, Jingsong On Wed, Apr 19, 2023 at 9:57 AM Shammon FY <[email protected]> wrote: > > Hi devs: > > I would like to start a discussion of PIP-1: Improve Shared Writer Buffer > Pool For Sink [1]. Currently Paimon sink task creates a heap memory pool > which is shared by writers. When there are multiple tasks in an Executor, > it may cause FullGC, performance issues and even OOM. > > This PIP aims to improve the buffer pool for writers in Paimon tasks. > Paimon tasks can create memory pools based on Executor Memory which will be > managed by Executor, such as Managed Memory in Flink TaskManager. It will > improve the stability and performance of sinks by managing writer buffers > for multiple tasks through Executor. > > Looking forward to your feedback, thanks. > > > [1] > https://cwiki.apache.org/confluence/display/PAIMON/PIP-1%3A+Improve+Shared+Writer+Buffer+Pool+For+Sink > > > Best, > Shammon FY
