alamb opened a new pull request, #25335: URL: https://github.com/apache/datafusion/pull/25335
## Which issue does this PR close? - Related to apache/arrow-rs#9880 (arrow 60.0.0 release) - Replaces #24366 (pre-release testing against arrow-rs `main`) ## Rationale for this change Update to the latest arrow/parquet release, `60.0.0`. See the upstream [CHANGELOG](https://github.com/apache/arrow-rs/blob/60.0.0/CHANGELOG.md) for details. ## What changes are included in this PR? 1. Update to arrow/parquet `60.0.0` and `object_store` `0.14.2` 2. Adjust code for upstream API differences. The main ones are: - New `Metadata` type for `Field` / `Schema` metadata (replaces `HashMap<String, String>`) - Parquet page indexes via `ParquetMetaData::page_index()` / `PageIndexProvider` / `PageIndexBuilder` - `OffsetBuffer::first` / `last` return a value instead of an `Option` - `GenericByteViewArray::data_buffers` returns `&Arc<[Buffer]>` - `MapBuilder` default field names changed to `key` / `value` - `rand` `0.10` in arrow's test utilities - `flight-sql-experimental` feature renamed to `flight-sql` - New `INT96_TIMESTAMP` parquet sort order 3. Update test expectations for upstream behavior changes (interval arithmetic with numerics, decimal formatting, `RunEndEncoded` display, IEEE 754 total order statistics, memory / heap size accounting) I will comment inline where upstream changes required updates. ## What is the testing strategy for this PR? Existing CI. Note: `memory_limit::nested_key_spill_keeps_groups_unique` failed in the `cargo test (amd64)` job of #24366 with a `ResourcesExhausted` error, but passes locally on macOS (8/8 runs), so I am watching for it on this PR's CI. ## Are there any user-facing changes? New dependency versions. Following the upstream change, a few DataFusion APIs now use arrow's `Metadata` type instead of `HashMap<String, String>`: `DFSchema::metadata`, `ExprSchema::metadata`, the `SchemaFieldMetadata` alias, `CastExpr::target_metadata` / `TryCastExpr::target_metadata`, and `FieldMetadata::inner` / `into_inner` (`api change`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
