jacklong319 opened a new pull request, #10133: URL: https://github.com/apache/paimon/pull/10133
### Purpose Linked issue: close #10132 Today each Flink write subtask uses one shared async compaction thread for all `(partition, bucket)` writers. Under peak load, compaction falls behind; in Deletion Vectors (MOW) mode, delayed L0 compaction hurts read freshness (see #10132). This PR adds table option **`compaction.task-threads`** to run parallel async compaction **across buckets within the same write subtask**, while **compaction for the same bucket remains serialized**. **Semantics:** - `1` (default): `SINGLE` — unchanged behavior. - `N > 1`: `FIXED_POOL` — N threads shared by buckets in the subtask. - `-1`: `PER_BUCKET` — one thread per active `(partition, bucket)` writer (higher memory use). **Changes:** - Add `CompactionTaskExecutorMode` and `CoreOptions.COMPACTION_TASK_THREADS`. - Route executors in `AbstractFileStoreWrite.compactExecutor(partition, bucket)`; release per-bucket executors on writer cleanup where applicable. - Default remains `1`; no storage format change. ### Tests - [ ] `mvn -pl paimon-api,paimon-core -am -Pfast-build test` - [ ] Unit tests for executor mode selection (if included in this PR) - [ ] Existing write/compaction tests pass - [ ] Manual validation as described in #10132 (optional; production summary stays in the issue) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
