c-thiel opened a new issue, #3607:
URL: https://github.com/apache/iceberg-rust/issues/3607
### Apache Iceberg Rust version
main @ 1ee5c72a7
### Describe the bug
To build the delete predicate,
`CachingDeleteFileLoader::parse_equality_deletes_record_batch_stream` derives
the Iceberg types of the equality columns by converting the delete batch's
Arrow schema back with `arrow_schema_to_schema`. The conversiondepends on Arrow
extension metadata. A uuid column stored as `FIXED_LEN_BYTE_ARRAY(16)` without
the Parquet `UUID` annotation carries no `arrow.uuid`; released iceberg-rust
versions write uuid columns this way. Such a column converts to `fixed(16)`,
and the scan fails:
`DataInvalid => Can't convert datum from fixed(16) type to uuid type.`
The same delete file with the annotation works. The loader already has the
table schema; it passes that schema to `BasicDeleteFileLoader::evolve_schema`
just before.
### To Reproduce
1. Write a Parquet file with the default `ArrowWriter` and the Arrow field
`Field::new("u", DataType::FixedSizeBinary(16), false)` with `PARQUET:field_id
= 1` and no `arrow.uuid` extension. This matches an equality delete file with
`equality_ids =[1]`.
2. Attach it as an equality delete to a data file of a table with `required
1: u uuid`, and scan.
3. `ArrowReader::read` fails with `Can't convert datum from fixed(16) type
to uuid type`.
### Expected behavior
The row with that uuid is deleted. The equality columns' types come from the
table schema restricted to the equality ids, not from the delete file's Arrow
schema.
This is what iceberg-java does. `DeleteFilter` builds `deleteSchema =
TypeUtil.selectInIdOrder(requiredSchema, ids)` from the table schema and loads
the equality deletes with it.
More generally, Java never reads Iceberg types back from Arrow metadata:
- its Arrow module maps uuid to plain `FixedSizeBinary(16)` and does not use
`arrow.uuid` (`ArrowSchemaUtil`);
- Spark's Parquet reader and the Arrow vectorized reader choose the uuid
decoder from the Iceberg type (`SparkParquetReaders`, `VectorizedArrowReader`),
so an unannotated column still reads as uuid.
Related, lower priority. For an unannotated uuid data file, the read batch
keeps the file's schema: `compare_schemas` ignores field metadata, so the batch
is `Equivalent` and passed through. A scan over an annotated and an unannotated
file therefore returns batches with different Arrow schemas, one with
`arrow.uuid` and one without. A geometry column stored as unannotated
`BYTE_ARRAY` with a `LargeBinary` hint loses `geoarrow.wkb` in the same way.
Relabelling `Equivalent` batches with the target schema (no data copy, as
`ModifySchema` does) would make the output follow the table schema for every
file. #2880 doesthis for variant fields only (`with_variant_extensions`).
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
