slfan1989 opened a new pull request, #4276:
URL: https://github.com/apache/amoro/pull/4276
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related
issue
-->
Close #4166.
`SortedPosDeleteWriter` currently only flushes buffered position deletes
based on a record count threshold, which can cause OutOfMemoryError (OOM) in
memory-constrained environments when processing large-scale delete operations.
There was a TODO comment in the code indicating the need for a heap
memory-based flush policy:
```
// TODO Flush buffer based on the policy that checking whether whole heap
memory size exceed the
// threshold.
```
This PR implements the heap-based flush mechanism to prevent OOM issues by
monitoring JVM heap usage and triggering flush when memory pressure is
detected, while maintaining backward compatibility.
## Brief change log
### Core Implementation:
- SortedPosDeleteWriter:
- Added `HeapUsageProvider` interface to monitor JVM heap memory
(max/used)
- Implemented `shouldFlushByHeap()` method with safety guards (minimum
records, valid ratio check)
- Modified `delete()` logic to flush when `records >= recordsNumThreshold
|| shouldFlushByHeap()`
- Added multiple constructor overloads to support new parameters while
maintaining backward compatibility
### Configuration:
- **TableProperties**: Added three new table properties:
- `pos-delete.flush.heap.ratio` (default: 0.8) - Heap usage ratio
threshold
- `pos-delete.flush.records` (default: Long.MAX_VALUE) - Record count
threshold
- `pos-delete.flush.heap.min-records` (default: 1000) - Minimum records
before heap flush
## How was this patch tested?
- Add some test cases that check the changes thoroughly including negative
and positive cases if possible
- Added `TestSortedPosDeleteWriterHeapFlush` with mock `HeapUsageProvider`
to test heap-based flush logic
- Test case 1: Verify flush is triggered when heap usage exceeds threshold
- Test case 2: Verify heap flush is disabled when ratio is set to 0
## Documentation
- Does this pull request introduce a new feature? **yes**
- If yes, how is the feature documented? **JavaDocs**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
