Zouxxyy opened a new pull request, #11277: URL: https://github.com/apache/paimon/pull/11277
### Purpose Addresses the full manifest compaction case in #5683. Suppose the deleted files belong to partitions `(day=1, event=A)` and `(day=19, event=B)`. A base manifest covering days 9 through 11 can pass the combined partition bounds even though neither deleted partition falls within it. Opening such manifests can dominate the cost of an overwrite commit. After the existing bounds check, test each distinct DELETE partition tuple against the manifest's per-field statistics. Reuse the base manifest when every tuple can be ruled out by at least one field. Stop at the first possible match and handle unknown bounds and null counts conservatively. Both the block and legacy full-compaction mergers opt into the check. The existing `mustChange` rule still selects small manifests and manifests containing DELETE entries for merging. The check reuses decoded tuples across manifests and decodes the bounds once for the tuple loop. Its worst-case cost per candidate is proportional to the number of distinct deleted partitions times the number of partition columns. A dense workload where little can be pruned may see additional CPU cost. ### Tests - Added a real-manifest regression for both merger implementations. Each of the 38 base manifests spans two hours. The test counts base files opened and checks all surviving and replacement entries. - Added partition statistics tests covering range gaps, tuple correlation, inclusive bounds, nulls, missing statistics and predicate serialization. - Ran `PartitionPredicateTest`, `ManifestFileMetaTest`, `NoPartitionManifestFileMetaTest` and `FileStoreCommitTest` with JDK 8. - Replayed a five-column dynamic `INSERT OVERWRITE` with Spark 3.5.9 on a local filesystem. During the measured overwrite, OS file-open tracing counted 38 distinct base manifest bodies on master and 2 with the change. Both runs returned the same 76 expected rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
