skyyyy955 commented on issue #10458: URL: https://github.com/apache/paimon/issues/10458#issuecomment-7056678684
I ran `CompactCandidateRangeCollector` at `7a30ef580e` with the 12 × 200 MiB metadata layout from the report (512 MiB target, 4 MiB open cost). It selected no ranges with `compaction.min.file-num=5`. Changing only that setting to `2` produced four ranges of three files each. The [collector source](https://github.com/apache/paimon/blob/ee544b74f172fcb14274812161e6b411a66a2046/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/CompactCandidateRangeCollector.java#L541-L550) is identical on current master (`ee544b74f1`). I also wrote 12 batches to a local Data Evolution table and read back all 12,000 row IDs. Its `$files` metadata showed 12 Parquet files of 210,014 bytes each with contiguous row IDs. Feeding those values to the collector with a 512 KiB target and a 4 KiB open cost again gave zero ranges at a minimum of five files and four ranges at a minimum of two. Both checks stop at candidate selection, before any Spark or Flink compaction job rewrites files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
