hbgstc123 opened a new pull request, #9660: URL: https://github.com/apache/paimon/pull/9660
## Motivation Compaction of append-only tables currently rewrites every data file: all rows are decoded, passed through the writer pipeline, and re-encoded, even when the input files are perfectly mergeable as-is. For Parquet (the default file format) this is unnecessarily expensive, because row groups are self-contained compressed units — a set of Parquet files that share the same schema and codec can be merged by concatenating their row groups directly and only rewriting the file footer, skipping row decode/re-encode entirely. On high-ingestion append-only tables, compaction consumes a significant share of streaming job resources, so the savings are substantial (measurements below). ## Design An opt-in fast path in append-only compaction, controlled by `append.compaction.row-group-copy.enabled` (default `false`). When enabled, each compaction batch is checked for eligibility; if every input file qualifies, the batch is merged via row-group copy, otherwise it transparently falls back to the traditional rewrite path. The fast path therefore never changes behavior for ineligible tables and is safe to enable unconditionally. A batch is eligible only when **all** of the following hold for every input file: - table format is Parquet, and the file carries the table's current schema (same schema id and identical Parquet message type); - uniform compression codec across all row groups and columns, matching the table's configured codec; - no deletion vectors, no row tracking / data evolution, no file index or bloom filter configured, no extra files, no embedded index, no partial-column writes (`writeCols`); - file source is `APPEND` or `COMPACT`; - file is not encrypted and was not written with Parquet writer v2 (`BYTE_STREAM_SPLIT` encoding). Value stats of output files reuse the input files' stats when a file is copied in full, and are recomputed from row-group metadata for partially copied files (when target-file-size splitting cuts a file). Output row count is verified against input row count as a safety net. Parquet-specific compatibility checks live in paimon-format (`ParquetRowGroupCopyChecker`), keeping paimon-core free of Parquet internals. ## Options - `append.compaction.row-group-copy.enabled` (default `false`): enable the RowGroup copy fast path. - `append.compaction.row-group-copy.preserve-page-index` (default `false`): preserve ColumnIndex/OffsetIndex so page-level predicate pruning keeps working on compacted files, at the cost of reading and rewriting page indexes during compaction. - `append.compaction.row-group-copy.footer-read.parallelism` (default `1`): bounded concurrent footer reads while preparing a compaction batch (hard cap 8). ## Performance evidence Local micro benchmark (`RowGroupCopyCompactionBenchmark`, included in this PR, one command to reproduce): ``` mvn -pl paimon-benchmark/paimon-micro-benchmarks -Dtest=RowGroupCopyCompactionBenchmark test ``` - narrow numeric table: copy vs rewrite **6.4–6.9×** faster; - wide string table, zstd, 8MB row groups: up to **24–32×**; - preserving the page index reduces the gain, and for wide tables with small (256KB) row groups it regresses below 1× (per-column-chunk index reads dominate) — this is the main reason `preserve-page-index` defaults to `false`. Production measurements (Flink and Spark compaction jobs, three independent runs each, kernel task time excluding scheduling/commit overhead): stable **~59–68% reduction** of compaction time, all batches on the fast path with zero fallbacks. Read-side regression tests on full scans, aggregations and medium-selectivity filters showed no regression when the page index is dropped. ## Tests - `ParquetFastPathCompactRewriterTest`: hit, every miss/fallback condition, DV/row-tracking/bloom/writer-v2 rejection, partial file copy stats merging, output row-count verification. - `SimpleStatsMergerTest`: stats merging across files and row groups. - The two benchmarks double as correctness checks (content equality verification mode). ``` mvn -pl paimon-core -Dtest='ParquetFastPathCompactRewriterTest,SimpleStatsMergerTest' test # 18 tests pass mvn -pl paimon-api,paimon-format,paimon-core,paimon-benchmark/paimon-micro-benchmarks spotless:check checkstyle:check # pass ``` The benchmark classes are not executed in CI (class names do not match surefire patterns; trigger explicitly with `-Dtest`). ## Documentation - `docs/generated/core_configuration.html` regenerated for the three new options. - New "Parquet RowGroup copy fast path" section in the append-table documentation. ## Notes for reviewers 1. `preserve-page-index` defaults to `false`: full-scan/aggregation workloads (the common case for compacted historical data) don't benefit from page indexes, and preservation measurably cuts the compaction gain (up to a regression for wide tables with small row groups, per the benchmark). Happy to flip the default if the community prefers the conservative choice. 2. The fast path is opt-in for the first iteration; since fallback makes it safe, defaulting `enabled` to `true` can be considered as a follow-up once it has seen more production mileage. 3. The footer-read parallelism cap is intentionally hard-coded at 8 to bound metadata-read pressure on the storage backend; it can be made configurable if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
