jaideeppyne opened a new issue, #3119:
URL: https://github.com/apache/iceberg-rust/issues/3119
### Apache Iceberg Rust version
main (0.10.0)
### Describe the bug
The spec lists `binary` as a valid source type for `truncate[W]` ("Partition
Transforms": `truncate[W]` source types are `int`, `long`, `decimal`, `string`,
`binary`; "Truncate Transform Details": `binary` / `L`, length / `v.subarray(0,
L)`). `Transform::result_type` agrees and accepts `PrimitiveType::Binary`, so a
`truncate[W]` partition field on a binary column is a legal spec.
But both halves of `TransformFunction for Truncate` reject it:
1. `transform_literal` has no `PrimitiveLiteral::Binary` arm, so every
predicate projection on such a field returns `FeatureUnsupported`.
`InclusiveProjection` (used in `scan/cache.rs`) propagates that error, so any
scan with a filter on that column fails instead of pruning.
2. `transform` handles `DataType::Binary` but not `DataType::LargeBinary`,
and `schema_to_arrow_schema` maps Iceberg `binary` to `LargeBinary`
(`crates/iceberg/src/arrow/schema.rs`, `PrimitiveType::Binary =>
DataType::LargeBinary`). So the array path fails on exactly the arrow type this
crate produces for a binary column.
`Truncate::truncate_binary` already exists and is correct; it is just not
wired to either entry point. pyiceberg and iceberg-go both implement truncate
on binary.
### To Reproduce
```rust
let t = Transform::Truncate(3);
// 1. literal / projection path
let pred = /* binary column `b`, `b <= x'0102030405'` */;
t.project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform:
Binary)
t.strict_project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform:
Binary)
// 2. array path
let f = create_transform_function(&t).unwrap();
f.transform(Arc::new(BinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Ok([1, 2, 3])
f.transform(Arc::new(LargeBinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Err(FeatureUnsupported => Unsupported data type for truncate transform:
LargeBinary)
```
### Expected behavior
`truncate[W]` on a binary column truncates to the first `W` bytes, in both
the array and the literal path, so partition values can be computed and
predicates can be projected.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
