fresh-borzoni opened a new pull request, #7447: URL: https://github.com/apache/datafusion-comet/pull/7447
## Which issue does this PR close? Backport of #7419 to `branch-1.2`, for #7417. ## Rationale for this change The native Iceberg scan fails with `cannot skip miniblock of size N` when position deletes make it skip a long run of rows in a `DELTA_BINARY_PACKED` column with miniblocks wider than 64 values, which Snowflake and Databricks Photon write (#7417). `branch-1.2` locks parquet 59.3.0 and has the bug. arrow-rs 59.3.2 fixes it. Moving from 59.3.0 also picks up 59.3.1, a patch release of bug fixes that `main` has had since #7169. The fix also goes to `branch-1.1`, opened together with this one. `branch-1.0` locks parquet 58.4.0, whose decoder fails the same way, but no 58.x release has the fix (58.4.1 doesn't), so it gets no backport here. ## What changes are included in this PR? `git cherry-pick -x 644a870` of #7419, with one adaptation. The lockfiles conflicted, because this branch locks 59.3.0 rather than `main`'s 59.3.1. I kept the branch's lockfiles, reran `cargo update -p arrow --precise 59.3.2` and `cargo update -p parquet --precise 59.3.2` in `native/` and `contrib/delta/native/`, and restored itertools 0.14.0 for `prost-build` and `prost-derive`, as #7419 does. The lockfiles change only the arrow-*, parquet and parquet-variant* crates. The RSS frame-size test change applied cleanly. ## How are these changes tested? On this branch: - `./mvnw test-compile` - In `native/`: `cargo fmt --all -- --check`, `cargo check --workspace --all-targets --locked`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` (2637 passed) - `cargo check --locked --all-targets` in `contrib/delta/native` #7419 adds no Comet test, as arrow-rs covers the decoder. The reproduction in #7417 fails on 1.1.0, and returns Spark's results on 1.2.0-rc1 built with parquet 59.3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
