Zouxxyy opened a new pull request, #11277:
URL: https://github.com/apache/paimon/pull/11277

   ### Purpose
   
   Addresses the full manifest compaction case in #5683.
   
   Suppose the deleted files belong to partitions `(day=1, event=A)` and 
`(day=19, event=B)`. A base manifest covering days 9 through 11 can pass the 
combined partition bounds even though neither deleted partition falls within 
it. Opening such manifests can dominate the cost of an overwrite commit.
   
   After the existing bounds check, test each distinct DELETE partition tuple 
against the manifest's per-field statistics. Reuse the base manifest when every 
tuple can be ruled out by at least one field. Stop at the first possible match 
and handle unknown bounds and null counts conservatively. Both the block and 
legacy full-compaction mergers opt into the check. The existing `mustChange` 
rule still selects small manifests and manifests containing DELETE entries for 
merging.
   
   The check reuses decoded tuples across manifests and decodes the bounds once 
for the tuple loop. Its worst-case cost per candidate is proportional to the 
number of distinct deleted partitions times the number of partition columns. A 
dense workload where little can be pruned may see additional CPU cost.
   
   ### Tests
   
   - Added a real-manifest regression for both merger implementations. Each of 
the 38 base manifests spans two hours. The test counts base files opened and 
checks all surviving and replacement entries.
   - Added partition statistics tests covering range gaps, tuple correlation, 
inclusive bounds, nulls, missing statistics and predicate serialization.
   - Ran `PartitionPredicateTest`, `ManifestFileMetaTest`, 
`NoPartitionManifestFileMetaTest` and `FileStoreCommitTest` with JDK 8.
   - Replayed a five-column dynamic `INSERT OVERWRITE` with Spark 3.5.9 on a 
local filesystem. During the measured overwrite, OS file-open tracing counted 
38 distinct base manifest bodies on master and 2 with the change. Both runs 
returned the same 76 expected rows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to