zhangfengcdt opened a new issue, #3436: URL: https://github.com/apache/iceberg-rust/issues/3436
### Apache Iceberg Rust version main @ `b154603a710d730d2f43f8af4db257cc137237ef` ### Describe the bug When a Parquet data file has no field ids and the table has no name mapping (`schema.name-mapping.default`), `ArrowReader` returns the values stored in the file for every nested field. Iceberg Java returns NULL for them. This covers struct fields, including structs inside lists and maps. Iceberg Java's fallback, [`ParquetSchemaUtil.addFallbackIds`](https://github.com/apache/iceberg/blob/db43b88a6c0d470bd4f8acf08a18328750aeed20/parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java#L282-L292), assigns ids by position to the top-level fields only. Its readers then match nested struct fields by id. [`SparkParquetReaders.ReadBuilder.struct`](https://github.com/apache/iceberg/blob/db43b88a6c0d470bd4f8acf08a18328750aeed20/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java#L146-L172), for example, registers a reader only for a file field that has an id. A nested field of such a file has none, so every expected field reads as NULL. The spec's column projection rules also give null for a field id the file doesn't contain when there is no name mapping. iceberg-rust assigns the same top-level ids (`add_fallback_field_ids_to_arrow_schema`). Then two things differ: - [`get_arrow_projection_mask_fallback`](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/arrow/reader/projection.rs#L243) reads each projected top-level column whole, including its nested content. - `RecordBatchTransformer` then matches the file's id-less nested fields to the table's [by position](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/arrow/record_batch_transformer.rs#L315-L318) when the counts line up. This was kept in #3255 as "a compatibility fallback until recursive name mapping (#1845)". #3384 now applies a name mapping at every depth, so the positional match only matters for files with neither ids nor a mapping, and that is the case where Java returns NULL. So the values come back even when the file's nested field names differ from the table's. Found through Apache DataFusion Comet, whose native Iceberg scan uses `ArrowReader`: apache/datafusion-comet#6794. With Spark 4.1 and Iceberg 1.11, for files added with `SparkTableUtil.importSparkTable` (no name mapping): | Query | Spark (Iceberg Java) | iceberg-rust | | --- | --- | --- | | `SELECT id, s FROM t WHERE id < 3 ORDER BY id` | `[0,[null,null]] [1,[null,null]] [2,[null,null]]` | `[0,[0,0]] [1,[1,-1]] [2,[2,-2]]` | | `SELECT id, items, m FROM t WHERE id < 3 ORDER BY id` | `[0,[],{k -> [null]}] …` | `[0,[[0]],{k -> [0]}] …` | | `SELECT count(*) FROM t WHERE s.a IS NULL` | `[50]` | `[0]` | The table is `(id INT, s STRUCT<a: INT, b: INT>, items ARRAY<STRUCT<x: INT>>, m MAP<STRING, STRUCT<v: INT>>)`. With the file's struct fields named `p` and `q` instead of `a` and `b`, Spark still returns NULL for `s.a` and `s.b`, while the iceberg-rust read returns the values by position. ### To Reproduce The existing test [`test_read_parquet_without_field_ids_with_struct`](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/arrow/reader/projection.rs#L1696) shows it. It writes a file without field ids holding `person STRUCT<name, age>`, reads it with no name mapping, and asserts `person.name` is `"Alice"`/`"Bob"` and `person.age` is `30`/`25`. It passes on main. For the same file, Iceberg Java returns a `person` struct whose `name` and `age` are NULL. ``` cargo test -p iceberg --lib test_read_parquet_without_field_ids_with_struct ``` ### Expected behavior For a file without field ids and no name mapping, top-level columns keep their positional fallback ids, and nested fields match by id. Since such a file has no nested ids, struct fields read as NULL, as in Iceberg Java. In Java's readers a list or map is matched to its element, key and value by position, so a top-level `ARRAY<INT>` or `MAP<STRING, INT>` still reads its values. Only fields inside a struct come back NULL. A list whose element is a struct, with no nested leaf read, comes back empty in Spark. `test_read_parquet_without_field_ids_with_struct` encodes the current behavior and would change with a fix. ### Willingness to contribute I would be willing to contribute a fix for this bug with guidance from the Iceberg community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
