I see Gang’s point about not trying to cover every malformed file in the spec. Could we keep this scoped to missing IDs where the mapping is unambiguous, and handle conflicting IDs separately?I see Gang’s point about not trying to cover every malformed file in the spec. Could we keep this scoped to missing IDs where the mapping is unambiguous, and handle conflicting IDs separately?
One thing I’d like to clarify is whether collision detection happens before column pruning. In JB’s example, column a has embedded ID 10 and the mapping assigns the same ID to column b. Checking only the pruned schema could hide that conflict. Which physical column supplies field 10 shouldn’t depend on what else the query selects. For the verification fixture, I’d suggest testing both full and single-field projections, plus reordered physical columns, to check that resolution or rejection is consistent. For collisions, I’d lean toward failing the read, as JB suggested. Including both column paths and the conflicting ID in the error would help diagnose an outdated mapping. On Thu, Nov 12, 2026 at 12:22 AM Gang Wu <[email protected]> wrote: > IMHO, handling files with partial or broken field IDs should remain > implementation-specific. > > Those files are produced outside of Iceberg's control. We can never > anticipate every way dirty files might conflict or miss IDs, or have > mismatched column names, and we cannot keep patching the spec to cover > every edge case. > > Best, > Gang > > On Thu, Nov 12, 2026 at 1:05 AM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi guys >> >> I checked the Java side, and it matches what you both described: >> - ParqueSchemaUtil.hasIds() returns true if any field at any depth has an >> ID, so a file with only top level IDs never uses the name mapping. >> - It does the same per file check with ORCSchemaUtil.hasIds(). >> - Avro already works per field. AvroSchemaUtil.getFieldId() uses the ID >> in the file when there is one and falls back to the mapping when there is >> not. >> >> For the Java change, removing the hasIds() branch won't be enough. When a >> path matches, the Parquet and ORC ApplyNameMapping visitors set the ID from >> the mapping and overwrite any ID already in the file. They will need to >> keep an ID that's already there and only fill in the missing ones. >> Otherwise, "never overrides an ID a writer recorded" is not true. >> >> I think the spec should cover a mapped ID that collides with an embedded >> ID on another column of the same file. I believe that can happen with an >> outdated or hand written mapping, and then two columns resolve to the same >> field ID. This case only exists with the per field reading, because today >> readers use either IDs or the mapping for the whole file. Maybe we could >> make embedded ID wins and the conflicting mapped ID is ignored, so that >> column is treated as having no ID. Failing the read would also be fine. >> Either way, we should be explicit in the spec. >> >> Something like that in the spec: "These mappings provide fallback field >> ids for fields in a data file that do not have a field id. A field id >> stored in the data file always takes precedence over the name mapping." >> >> Manu proposed a PR for the spec, I think it makes sense. >> >> Regards >> JB >> >> On Wed, Nov 11, 2026 at 3:55 PM Manu Zhang <[email protected]> >> wrote: >> >>> Hi Sylwester, >>> >>> Thanks for raising this issue. I am also working to resolve the >>> inconsistency in iceberg-cpp [1][2]. >>> >>> Your proposal looks reasonable, and it aligns with what the Avro reader >>> does in the Java implementation (whereas ORC follows the Parquet rule). >>> Whichever rule we decide to use, we should make sure it aligns with the >>> specification. >>> >>> Could you please open a spec PR so we can have a concrete discussion >>> there? >>> >>> [1] https://github.com/apache/iceberg-cpp/pull/1108 >>> [2] https://github.com/apache/iceberg-cpp/pull/1109 >>> >>> Best regards, >>> Manu >>> >>> >>> On Tue, Nov 3, 2026 at 8:20 AM Sylwester Lachiewicz via dev < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> The spec is ambiguous about Parquet data files that carry field IDs for >>>> some fields but not all, and no two of the four implementations I checked >>>> resolve such a file the same way. I'd like to agree on one reading and >>>> write it into the spec. >>>> >>>> Column Projection [1] says a reader should "use >>>> schema.name-mapping.default metadata to map field id to columns without >>>> field id", which reads as per column. The paragraph that follows says the >>>> mappings "provide fallback field ids to be used when a data file does not >>>> contain field id information", which reads as per file. For a file with no >>>> IDs at all, both readings agree. They diverge when only some fields carry >>>> an ID. >>>> >>>> What the readers do today, on their main branches: >>>> >>>> - Java (ReadConf, ParquetSchemaUtil.HasIds): if any field in the file >>>> has an ID, the file is resolved by ID only and the name mapping is not >>>> consulted. >>>> - PyIceberg (pyarrow_to_schema, _HasIds): IDs are used only if every >>>> field, including list elements and map keys and values, has one; otherwise >>>> the whole file is resolved through the name mapping. >>>> - iceberg-rust (arrow/reader/pipeline.rs): the first top-level column >>>> decides; if it has an ID, the file is resolved by ID only. >>>> - iceberg-go (table/internal/parquet_files.go, >>>> pruneParquetSchema.fieldID): Parquet column pruning resolves per field; an >>>> embedded ID wins, and the mapping fills in the fields without one. >>>> >>>> This happens in practice. Delta tables with column mapping, written >>>> without IcebergCompatV2, store IDs for top-level columns but none for map >>>> keys and values or list elements. When such a table gets Iceberg metadata >>>> [2], PyIceberg reads the map columns through the name mapping, while the >>>> Java reader resolves by ID only and does not find the map's key and value. >>>> The same table gives different results depending on the reader. >>>> >>>> I propose the per-field reading, which matches the Column Projection >>>> rule: a field with an embedded ID is resolved by that ID, and a field >>>> without one falls back to the name mapping. It never overrides an ID a >>>> writer recorded, it changes nothing for files with full IDs or with none, >>>> and it makes the partial case readable. >>>> >>>> If there is agreement, I can open a spec PR that clarifies the >>>> paragraph, and contribute a fixture to iceberg-verification [3]: a file >>>> with IDs on its top-level fields only, as a companion to the base >>>> name-mapping table that issue covers. >>>> >>>> Is there a reason for the per-file behaviour that I'm missing, for >>>> example a correctness concern with mixing ID and name resolution within one >>>> file? >>>> >>>> [1] >>>> https://www.google.com/url?q=https://iceberg.apache.org/spec/%23column-projection&source=gmail&ust=1793746430284000&sa=E >>>> [2] >>>> https://www.google.com/url?q=https://github.com/apache/incubator-xtable/issues/911&source=gmail&ust=1793746430284000&sa=E >>>> [3] >>>> https://www.google.com/url?q=https://github.com/apache/iceberg-verification/issues/14&source=gmail&ust=1793746430284000&sa=E >>>> >>>> Thanks, >>>> Sylwester >>>> >>>
