Hi all, The spec is ambiguous about Parquet data files that carry field IDs for some fields but not all, and no two of the four implementations I checked resolve such a file the same way. I'd like to agree on one reading and write it into the spec.
Column Projection [1] says a reader should "use schema.name-mapping.default metadata to map field id to columns without field id", which reads as per column. The paragraph that follows says the mappings "provide fallback field ids to be used when a data file does not contain field id information", which reads as per file. For a file with no IDs at all, both readings agree. They diverge when only some fields carry an ID. What the readers do today, on their main branches: - Java (ReadConf, ParquetSchemaUtil.HasIds): if any field in the file has an ID, the file is resolved by ID only and the name mapping is not consulted. - PyIceberg (pyarrow_to_schema, _HasIds): IDs are used only if every field, including list elements and map keys and values, has one; otherwise the whole file is resolved through the name mapping. - iceberg-rust (arrow/reader/pipeline.rs): the first top-level column decides; if it has an ID, the file is resolved by ID only. - iceberg-go (table/internal/parquet_files.go, pruneParquetSchema.fieldID): Parquet column pruning resolves per field; an embedded ID wins, and the mapping fills in the fields without one. This happens in practice. Delta tables with column mapping, written without IcebergCompatV2, store IDs for top-level columns but none for map keys and values or list elements. When such a table gets Iceberg metadata [2], PyIceberg reads the map columns through the name mapping, while the Java reader resolves by ID only and does not find the map's key and value. The same table gives different results depending on the reader. I propose the per-field reading, which matches the Column Projection rule: a field with an embedded ID is resolved by that ID, and a field without one falls back to the name mapping. It never overrides an ID a writer recorded, it changes nothing for files with full IDs or with none, and it makes the partial case readable. If there is agreement, I can open a spec PR that clarifies the paragraph, and contribute a fixture to iceberg-verification [3]: a file with IDs on its top-level fields only, as a companion to the base name-mapping table that issue covers. Is there a reason for the per-file behaviour that I'm missing, for example a correctness concern with mixing ID and name resolution within one file? [1] https://www.google.com/url?q=https://iceberg.apache.org/spec/%23column-projection&source=gmail&ust=1793746430284000&sa=E [2] https://www.google.com/url?q=https://github.com/apache/incubator-xtable/issues/911&source=gmail&ust=1793746430284000&sa=E [3] https://www.google.com/url?q=https://github.com/apache/iceberg-verification/issues/14&source=gmail&ust=1793746430284000&sa=E Thanks, Sylwester
