yugan95 opened a new pull request, #8159:
URL: https://github.com/apache/paimon/pull/8159
### Purpose
Linked issue: #7620
`RowHelper.reuseWriter` grows its internal `MemorySegment` for large records
(e.g. 100MB+), but `BinaryRowWriter.reset()` only resets the cursor without
releasing the oversized segment. Additionally,
`InternalRowSerializer.serialize()` can exit via `EOFException` — a normal
signal when the sort buffer is full (`SimpleCollectingOutputView.nextSegment()`
throws it, caught by `BinaryInMemorySortBuffer.write()`) — skipping any cleanup
of the bloated buffer.
With many buckets (e.g. 256), each bucket's writer independently retains an
inflated buffer: 256 × 100MB+ = tens of GB, causing OOM.
#### Changes
- **`RowHelper`**: add `resetIfTooLarge()` with hysteresis — release
internal buffer only when the segment exceeds 4MB **and** the last written
record (`reuseRow.getSizeInBytes()`) is smaller than 4MB
- Sustained large records (5–10MB): buffer retained, no thrashing
- Occasional large record → back to small records: buffer released, OOM
protection
- Normal small records (< 4MB): buffer never exceeds threshold, the check
is a no-op
- **`InternalRowSerializer`**: call `resetIfTooLarge()` in `finally` blocks
of `serialize()` and `serializeToPages()` to handle the `EOFException` exit path
#### Why 4MB and why not configurable
The threshold is derived from the production scenario: 256 buckets × 4MB =
1GB baseline, leaving reasonable headroom in a typical 4–8GB TaskManager heap.
Making it configurable would require plumbing config through the entire
serializer chain — `RowHelper` and `InternalRowSerializer` live in
paimon-common which is config-free by design (all 27 serializer classes are
parameterized solely by schema types). The full chain from `CoreOptions` →
`KeyValueFileStoreWrite` → `MergeTreeWriter` → `SortBufferWriteBuffer` →
`BinaryInMemorySortBuffer` → `InternalRowSerializer` → `RowHelper` would need
changes across modules. A fixed value with hysteresis is sufficient for the
initial fix.
### Tests
`RowHelperTest` — 4 test cases covering:
- Hysteresis: buffer retained when last record is large
- Buffer released when workload transitions to small records
- Safe to call before any `copyInto`
- Reuse recreated after release, small buffer survives `resetIfTooLarge()`
### API and Format
N/A
### Documentation
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
