XiaoHongbo-Hope opened a new pull request, #9114:
URL: https://github.com/apache/paimon/pull/9114
## What changed
- retain value statistics until data-evolution row-id groups have been formed
- merge per-column statistics from the latest file that contains each field
- prune non-matching logical groups before split packing
- fail open for missing or corrupt statistics, incompatible schemas and
partial providers
- exclude BLOB and VECTOR files from ordinary predicate statistics
## Why
A data-evolution row can be stored across multiple files, so filtering each
file independently is unsafe. PyPaimon therefore skipped file-level statistics
pruning, but never applied the group-level pruning used by Java Paimon and
paimon-rust. In full scalar-index mode this could carry many non-matching
fallback files into the read plan.
This change keeps single-file pruning disabled and evaluates statistics only
after overlapping row-id ranges have been grouped within each partition and
bucket.
## Validation
- `146 passed, 3 deselected, 45 subtests passed` across data-evolution,
deletion-vector, rolling, global-index and schema-evolution suites
- flake8 and `git diff --check` pass
- local full-fallback test with 300 data files:
- before: 2 splits / 300 files, 290 ms read
- after: 1 split / 1 file, 1.5 ms read
- results are identical
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
