slachiewicz commented on issue #911: URL: https://github.com/apache/incubator-xtable/issues/911#issuecomment-5388141527
A specification detail that bears on the refuse-versus-warn question, and on how much of this #912 can close. `delta.columnMapping.nested.ids` is defined in `PROTOCOL.md` under **"Writer Requirements for IcebergCompatV2"**, so it is written only when `delta.enableIcebergCompatV2` is set. Under plain column mapping — `mode=id` and `mode=name` alike — a list's `element` and a map's `key`/`value` have no identity mechanism at all, and those Parquet nodes carry no `field_id`. Two things follow. Writing `field_id` for every `StructField` is a general column-mapping requirement rather than a mode-gated one; the modes differ only in how a reader resolves them. That is consistent with the footer diff in the issue body: the 3.x file has ids on the struct fields, and Iceberg switches to id-based resolution as soon as any id is present. The two cases in this issue therefore need different answers. Where IcebergCompatV2 is on, the ids exist and carrying them is a complete fix. Where it is off, there is nothing to carry — so any id appearing in the generated Iceberg schema for a collection node would have to be invented, which is the condition that produces the unreadable table. No amount of carrying closes that second case. Worth noting for anyone searching: there is no `delta.columnMapping.nested.physicalNames` key. A collection node's physical path is its nearest ancestor `StructField`'s `physicalName` plus `.element`, `.key` or `.value`. If it helps the decision on `testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg`: refusing has the advantage that the failure is otherwise silent at sync time and only surfaces when someone reads a collection column, which is exactly what that test pins. *This comment was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
