parthchandra opened a new issue, #2698: URL: https://github.com/apache/iceberg-rust/issues/2698
Follow up from https://github.com/apache/iceberg-rust/pull/2668 ## Summary iceberg-rust currently maps `PrimitiveType::Binary` to `DataType::LargeBinary` in the Arrow type conversion (`type_to_arrow_type`). This is inconsistent with: 1. **Java's implementation** — [ArrowSchemaUtil.java#L134](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java#L134) maps Iceberg Binary to Arrow Binary. 2. **parquet-rs behavior** — parquet-rs reads binary columns from Parquet files as `DataType::Binary`, not `LargeBinary`. This mismatch causes unnecessary casts when reading Binary columns and creates inconsistency with the Java ecosystem. ## Scope - Change `PrimitiveType::Binary` mapping from `DataType::LargeBinary` to `DataType::Binary` in `crates/iceberg/src/arrow/schema.rs` - Update `create_primitive_array_repeated` in `crates/iceberg/src/arrow/value.rs` — the existing `DataType::Binary` arm already handles this, but the `LargeBinary` arm added in #2668 can be removed or kept as a fallback - Audit all places that construct or expect `LargeBinary` arrays for Iceberg Binary columns - Update tests that assert `LargeBinary` output ## Context Raised in review of #2668 by @advancedxy. The `LargeBinary` arm in `create_primitive_array_repeated` was added to handle the current mapping for `_partition` columns partitioned by Binary. Once this mapping is corrected, that arm becomes unnecessary (but harmless to keep as a safety net). This is a broader change that affects all Binary column reads, not just `_partition`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
