neuyilan opened a new issue, #3819: URL: https://github.com/apache/paimon/issues/3819
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 0.8 ### Compute Engine None ### Minimal reproduce step According to the current code, a table's write will have an AbstractFileStoreWrite object, which will then create a containerWriter for each bucket of each partition based on the partition. All these containerWriters share a single WriteBufferPool[1]. The code is shown below. So this may lead to the following issues: 1. The creation of `WriteBufferPool` is not thread safe, as the notifyNewWriter is called when writing the first data in each partition's bucket. So if multiple bucket writing threads are called simultaneously, it is not thread safe. 2. The `nextSegment()` function in the `MemoryPoolFactory`[2] retrieves the available memory from the available memory. But when multiple threads are called simultaneously, it is not thread safe either. 3. When the memory is used up and returned to the memory pool, the `returnAll(List<MemorySegment> memory) `function in `AbstractMemorySegmentPool` [3] is called. The `flushWriteBuffer()` function in `MergeTreeWriter` will trigger the `returnAll` function, where may cause multiple possibilities for `flushWriteBuffer`, such as compact or normal writing. So it is not thread safe either. I'm not sure if this is a problem or if it was intentionally designed for it. If it's a problem, I'm willing to fix it. Thanks. [1] https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/operation/MemoryFileStoreWrite.java#L113 [2] https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/memory/MemoryPoolFactory.java#L140 [3] https://github.com/apache/paimon/blob/master/paimon-common/src/main/java/org/apache/paimon/memory/AbstractMemorySegmentPool.java#L59 ### What doesn't meet your expectations? Fix it ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
