JingsongLi opened a new pull request, #839: URL: https://github.com/apache/paimon-rust/pull/839
DE stats pruning currently interprets historical files using the current schema layout. Dropping a column can make a predicate on the next column read the wrong statistics and discard matching rows. Aggregation writes also retain duplicate keys within a single data file, violating the merge-tree reader assumption shared with Java and Python. This aligns both paths with Java: - Resolve DE file schemas and match fields by ID, including dense statistics, renames, reordering and re-added columns. Changed types or unavailable schemas keep the affected data conservatively. - Use DE statistics only when the latest field provider is unique and covers the entire row-id group; reject inconsistent statistics. - Apply the existing aggregate merge function during write-buffer flush, preserving user-sequence order, partition primary keys, original input changelog rows and output sequence metadata. The merge reader and writer share the aggregation implementation. Regression tests cover schema changes with full/dense stats, missing schemas, provider ties and partial overlap, invalid stats, physical Parquet rows, null/default aggregators, sequence ties, partition keys, changelog preservation, retract rejection and aggregation across commits. Validation: - `cargo test --locked -p paimon --lib table::` — 1,278 passed, 1 ignored. - `cargo clippy --locked -p paimon --all-targets -- -D warnings` — passed. - `cargo fmt --all --check` — passed. - Built the Python wheel and verified real DE schema-evolution reads through Python/native planning. A Rust-written aggregation file containing inputs 10 and 20 now has one physical row with value 30, read identically by both Python planning paths and the Rust reader. Java references: [DE evolution stats](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java), [write-buffer merge](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/mergetree/SortBufferWriteBuffer.java), [output sequence metadata](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
