ShivamSharma43 opened a new pull request, #2646:
URL: https://github.com/apache/iceberg-go/pull/2646

   `UpdateSchema` never updated the table's `schema.name-mapping.default` 
property unless the caller passed a mapping with `WithNameMapping`, and nothing 
in the repo does. After `AddColumn` or `RenameColumn` the mapping still 
described the old schema, so files without Parquet field IDs could not be 
resolved for the new or renamed column. Since `AddDataFilesByPath` and 
`AddDataFiles` set the mapping on first use, any table that iceberg-go has 
added files to hits this: after adding a column, adding a file that contains it 
fails with `invalid schema: cannot convert category: type=utf8, nullable 
metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing field_id` (see 
the issue for the reproduction).
   
   ### Cause
   
   `BuildUpdates` only ran `iceberg.UpdateNameMapping` when `u.nameMapping` was 
set, and that field is only set by the `WithNameMapping` option. It never read 
the mapping stored in the table's properties.
   
   ### Fix
   
   When no mapping is passed, `BuildUpdates` now uses the table's own mapping 
(`meta.NameMapping()`, read from the transaction's staged metadata) and writes 
the updated mapping back to `schema.name-mapping.default`. That matches Java's 
[`SchemaUpdate.applyChangesToMetadata`](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L484-L505),
 which updates the property on every schema change, and PyIceberg, whose 
[`Transaction.update_schema`](https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/__init__.py#L471)
 passes `table_metadata.name_mapping()` to 
[`UpdateSchema._commit`](https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/update/schema.py#L645-L649).
   
   - Added columns get a new entry, and renamed columns keep their old name and 
gain the new one, as `UpdateNameMapping` already does (same as Java's 
`MappingUtil.update`).
   - A table without the property is left alone: no mapping is added.
   - An explicit `WithNameMapping` still takes precedence.
   - If the property can't be parsed, `meta.NameMapping()` returns nil and the 
update goes ahead without touching it. Java does the same: it logs a warning 
and keeps the old value.
   
   ### Testing
   
   - New `table/update_schema_name_mapping_test.go`:
     - `TestUpdateSchemaUpdatesTableNameMapping` adds a column and renames 
another on a table with a mapping, then checks the committed mapping 
(`category` for the new field, `data` and `payload` for the renamed one).
     - `TestUpdateSchemaWithoutNameMappingLeavesPropertyUnset` checks that a 
table without the property doesn't get one.
     - `TestAddFilesAfterAddColumnReadsNewColumnByName` runs the reproduction 
from the issue end to end: `AddDataFilesByPath`, `AddColumn`, 
`AddDataFilesByPath` with a file that has the new column and no field IDs, then 
a scan that reads `books` from it.
   
     On main, the first and third fail (the mapping has no entry for field 3, 
and the second `AddDataFilesByPath` fails with the `missing field_id` error 
above). The second passes on main and with the fix.
   - `go vet ./table/` is clean. `go test ./table/...` passes except for tests 
that fail the same way on unmodified main on my Windows machine (path 
handling): `TestEmptyStringIdentityPartitionScans`, 
`TestScanImportedSignedZeroBounds`, `TestDeleteImportedSignedZerosWithIn`, 
`TestDeleteNormallyWrittenSignedZeroWithIn`, 
`TestScanLegacyEmptyStringPartitionDeleteBounds`, 
`TestScanWithIDLessNameMappingFields`, two 
`TestRewriteDataFiles_PartialProgress*` tests and `TestWriteMapType`. `go test 
./catalog/...` gives the same results as main.
   
   Closes #2645
   
   This PR was prepared with AI assistance (Claude Code).
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to