cshuo commented on code in PR #13892:
URL: https://github.com/apache/hudi/pull/13892#discussion_r2681072437
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithBufferSort.java:
##########
@@ -79,38 +92,52 @@ public void open(Configuration parameters) throws Exception
{
GeneratedNormalizedKeyComputer keyComputer =
codeGenerator.generateNormalizedKeyComputer("SortComputer");
GeneratedRecordComparator recordComparator =
codeGenerator.generateRecordComparator("SortComparator");
MemorySegmentPool memorySegmentPool =
MemorySegmentPoolFactory.createMemorySegmentPool(config);
- this.buffer = BufferUtils.createBuffer(rowType,
+
+ this.activeBuffer = BufferUtils.createBuffer(rowType,
memorySegmentPool,
keyComputer.newInstance(Thread.currentThread().getContextClassLoader()),
recordComparator.newInstance(Thread.currentThread().getContextClassLoader()));
- log.info("{} is initialized successfully.", getClass().getSimpleName());
+ this.backgroundBuffer = BufferUtils.createBuffer(rowType,
+ MemorySegmentPoolFactory.createMemorySegmentPool(config),
Review Comment:
In this pr, these two buffers are sharing the same memory pool, and config
`WRITE_BUFFER_SIZE` is used to limit the memory usage of the one buffer. Maybe
we can set `WRITE_BUFFER_SIZE` properly to avoid memory overuse.
Or if we wanna evaluate the memory usage more accurately, we can use
`BufferSizeDetector` instead of estimating by record counts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
