ewoodbury opened a new issue, #3565: URL: https://github.com/apache/iceberg-rust/issues/3565
### Apache Iceberg Rust version `main` (6f23a2a77) ### Describe the bug Iceberg partition field names don't have to be valid Avro names, which must match `[A-Za-z_][A-Za-z0-9_]*`. When one isn't, Java writes the field to the manifest under a sanitized name and keeps the original in the `iceberg-field-name` property ([`TypeToSchema.java#L116-L128`](https://github.com/apache/iceberg/blob/781a30f3c9b5c74dc15ad5e16f7266919f2cbfeb/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java#L116-L128)). A partition on a nested column always hits this: Java and Spark name `bucket(16, user.id)` `user.id_bucket`, and the manifest stores it as `user_x2Eid_bucket`. Columns with `-`, a space, or a leading digit in their names do too. `Manifest::parse_avro` matches partition fields to the spec by name (`RawLiteral::project_by_name`). Partition fields are optional, so a field that doesn't match reads as null with no error. The Avro field also has the partition field's `field-id`, which the spec requires ([spec.md#L1750](https://github.com/apache/iceberg/blob/e5998eb8c147426397d726a29aa6d2f9185adfb1/format/spec.md?plain=1#L1750)), but the reader ignores it. The spec says manifests must use valid column projection ([spec.md#L700](https://github.com/apache/iceberg/blob/e5998eb8c147426397d726a29aa6d2f9185adfb1/format/spec.md?plain=1#L700)), and projection matches columns by field ID ([spec.md#L439](https://github.com/apache/iceberg/blob/e5998eb8c147426397d726a29aa6d2f9185adfb1/format/spec.md?plain=1#L439)). Java and PyIceberg ([`resolver.py`](https://github.com/apache/iceberg-python/blob/27a5162749bb670d00b23b8360869410b6e5b3a1/pyiceberg/avro/resolver.py#L405-L425)) both match these fields by ID. Scan planning then evaluates partition filters against null, so a filter on the partition's source column skips every file and the scan returns no rows. Unfiltered scans still return all rows. I hit this on a Spark table bucketed on a nested user ID field. This was raised in the #3354 review ([comment](https://github.com/apache/iceberg-rust/pull/3354#discussion_r4204959679)). It's older than that PR: the reader-schema resolution it replaced also matched by name. ### To Reproduce Write a table with iceberg-java 1.12.0: ```java Types.StructType userType = Types.StructType.of(Types.NestedField.optional(3, "id", Types.LongType.get())); Schema schema = new Schema( Types.NestedField.optional(1, "event", Types.StringType.get()), Types.NestedField.optional(2, "user", userType)); PartitionSpec spec = PartitionSpec.builderFor(schema).bucket("user.id", 16).build(); // Create the table with HadoopTables and append four rows with user.id = 1..4, one file each. ``` The manifest's partition field is: ```json {"name": "user_x2Eid_bucket", "type": ["null", "int"], "default": null, "iceberg-field-name": "user.id_bucket", "field-id": 1000} ``` Java returns the matching row for `user.id = 3`. Reading the same table with iceberg-rust: ```rust let table = StaticTable::from_metadata_file(&metadata, ident, FileIO::new_with_fs()).await?; let scan = table .scan() .with_filter(Reference::new("user.id").equal_to(Datum::long(3))) .build()?; ``` | Filter | Files planned | Rows | |---|---|---| | none | 4 | 4 | | `user.id = 3` | 0 | 0 | Every planned task's partition is `Struct { fields: [None] }`. ### Expected behavior Partition fields are matched by `field-id`, with the name as a fallback for writers that don't store field IDs. The partition values read as written, and `user.id = 3` returns one row. Writing these names is a separate bug, tracked in #2535: iceberg-rust writes `user.id_bucket` as the Avro name as is, and then can't read its own manifest (`Invalid field name`). ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
