JingsongLi opened a new pull request, #548: URL: https://github.com/apache/paimon-rust/pull/548
## Summary Align Paimon scan statistics with DataFusion native Parquet semantics. This exposes conservative manifest-derived column statistics while avoiding compressed full-file bytes that do not represent projected scan bytes. ## Changes - decode per-file value statistics using the current schema and dense column mappings - aggregate manifest min/max bounds and publish exact null counts only for row-preserving scans - fail open on malformed, inconsistent, schema-mismatched, or ordering-incompatible statistics - suppress unsafe statistics for aggregation merge engines, filters, limits, row ranges, and deletion files - report `total_byte_size` as unknown because Paimon only stores compressed full-file sizes - add regression coverage for aggregation, unsafe null-count paths, malformed rows, and binary ordering ## Testing - `cargo test -p paimon --all-targets --features fulltext,vortex` - `cargo test -p paimon value_stats_for_field` - `cargo test -p paimon-datafusion --lib test_partition_statistics` - `cargo test -p paimon-datafusion --lib test_binary_manifest_bounds` - `cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` Runtime filter support remains a separate follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
