hbgstc123 opened a new issue, #9664: URL: https://github.com/apache/paimon/issues/9664
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Compaction of append-only tables currently rewrites every data file: all rows are decoded, passed through the writer pipeline, and re-encoded, even when the input files are perfectly mergeable as-is. For Parquet (the default file format) this is unnecessarily expensive, because row groups are self-contained compressed units — a set of Parquet files that share the same schema and codec can be merged by concatenating their row groups directly and only rewriting the file footer, skipping row decode/re-encode entirely. In our production environment (Flink and Spark compaction jobs on append-only tables), a row-group copy fast path reduced compaction kernel task time by ~59–68% with zero fallbacks. Local micro benchmarks show 6.4–6.9× for narrow numeric tables and up to 24–32× for wide string tables (zstd, 8 MB row groups). ### Solution An opt-in fast path in append-only compaction, controlled by `append.compaction.row-group-copy.enabled` (default `false`). When enabled, each compaction batch is checked for eligibility; if every input file qualifies, the batch is merged via row-group copy, otherwise it transparently falls back to the traditional rewrite path. The fast path therefore never changes behavior for ineligible tables and is safe to enable unconditionally. A batch is eligible only when **all** of the following hold for every input file: - table format is Parquet, and the file carries the table's current schema (same schema id and identical Parquet message type); - uniform compression codec across all row groups and columns, matching the table's configured codec; - no deletion vectors, no row tracking / data evolution, no file index or bloom filter configured, no extra files, no embedded index, no partial-column writes (`writeCols`); - file source is `APPEND` or `COMPACT`; - file is not encrypted and was not written with Parquet writer v2 (`BYTE_STREAM_SPLIT` encoding). Value stats of output files reuse the input files' stats when a file is copied in full, and are recomputed from row-group metadata for partially copied files (when target-file-size splitting cuts a file). Output row count is verified against input row count as a safety net. Parquet-specific compatibility checks live in paimon-format (`ParquetRowGroupCopyChecker`), keeping paimon-core free of Parquet internals. Options: - `append.compaction.row-group-copy.enabled` (default `false`): enable the fast path. - `append.compaction.row-group-copy.preserve-page-index` (default `false`): preserve ColumnIndex/OffsetIndex so page-level predicate pruning keeps working on compacted files, at the cost of reading and rewriting page indexes during compaction. - `append.compaction.row-group-copy.footer-read.parallelism` (default `1`): bounded concurrent footer reads while preparing a compaction batch (hard cap 8). ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
