JingsongLi commented on PR #9660: URL: https://github.com/apache/paimon/pull/9660#issuecomment-6137682566
[P2] The small-row-group guard is still missing on `469b9d4a880fca16859372f1acddcd185cfcbc46`. The [proposed configurable 8MiB threshold](https://github.com/apache/paimon/pull/9660#issuecomment-6096028672) addresses a real production boundary. Current table/file/footer eligibility checks do not inspect size, so enabling the option still copies ordinary small files from checkpoint commits. `targetFileSize` only rolls output files and never combines their small row groups. I compared the actual `BaseAppendFileStoreWrite.compactRewrite` paths using the same 64 files, 32 rows per file, default Parquet block size and compression (2,048 rows). Both outputs matched every input row. After two warmup rounds, five paired rounds gave: | Result | Row-group copy | Ordinary rewrite | | --- | ---: | ---: | | Output files / row groups | 1 / 64 | 1 / 1 | | Output bytes | 54,089 | 8,901 | | Median compaction | 32.16 ms | 13.70 ms | | Median warmed full read | 2.687 ms | 0.477 ms | This is a local filesystem microbenchmark, not an HDFS/S3 result. The retained row-group count and output-size amplification are deterministic in this fixture; the timing shows why schema/codec eligibility alone is insufficient for this workload. Please implement the configurable measured size threshold, including the cheap manifest-size rejection before footer reads, and cover both the small-batch fallback and a qualifying copy hit before enabling it in production. The large-row-group optimization has end-to-end value and this PR should stay open. Current JDK 8 package verification without fast-build passed 43 focused core/format tests, including actual copy/read/predicate checks, statistics and metric lifecycle. The prior unsigned-binary, counts-stats and metric-lifecycle findings are fixed. Exact-head CI is green; `git diff --check` passed. I did not independently rerun the claimed production throughput benchmarks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
