JingsongLi commented on PR #10133: URL: https://github.com/apache/paimon/pull/10133#issuecomment-5805809039
The linked production issue provides a strong end-to-end case for cross-bucket compaction parallelism (reported DV read lag improved from ~40 min at 1 thread to ~8 min at 3). I traced the executor routing and the per-bucket compact managers’ `taskFuture` guard; the default remains a single shared executor, and the managers appear to keep each bucket’s submitted task serialized. Merge blockers / production follow-up: 1. A normal JDK 8 `mvn -o -pl paimon-api,paimon-core -DskipTests compile` fails Spotless in `AbstractFileStoreWrite.java` at the `perBucketCompactExecutors.remove(...)` line. This must be formatted; the PR currently has 14 failing checks. 2. There are no new tests for `FIXED_POOL` or `PER_BUCKET`, cross-bucket concurrency, same-bucket serialization, executor release/recreation, or external executor ownership. The existing targeted compaction/write classes pass locally (29 tests run, 1 skipped), but they exercise mostly the default path. Please add focused behavioral tests before production merge. 3. Please document the thread/memory trade-off and monitoring behavior described in #10132; `compactionThreadBusy` can exceed 100 with parallel workers. Also reject unsupported values such as `0` and `-2` rather than silently treating them as `SINGLE`. The feature has clear value, but the current head is not ready to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
