ShivamSharma43 opened a new pull request, #2646: URL: https://github.com/apache/iceberg-go/pull/2646
`UpdateSchema` never updated the table's `schema.name-mapping.default` property unless the caller passed a mapping with `WithNameMapping`, and nothing in the repo does. After `AddColumn` or `RenameColumn` the mapping still described the old schema, so files without Parquet field IDs could not be resolved for the new or renamed column. Since `AddDataFilesByPath` and `AddDataFiles` set the mapping on first use, any table that iceberg-go has added files to hits this: after adding a column, adding a file that contains it fails with `invalid schema: cannot convert category: type=utf8, nullable metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing field_id` (see the issue for the reproduction). ### Cause `BuildUpdates` only ran `iceberg.UpdateNameMapping` when `u.nameMapping` was set, and that field is only set by the `WithNameMapping` option. It never read the mapping stored in the table's properties. ### Fix When no mapping is passed, `BuildUpdates` now uses the table's own mapping (`meta.NameMapping()`, read from the transaction's staged metadata) and writes the updated mapping back to `schema.name-mapping.default`. That matches Java's [`SchemaUpdate.applyChangesToMetadata`](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L484-L505), which updates the property on every schema change, and PyIceberg, whose [`Transaction.update_schema`](https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/__init__.py#L471) passes `table_metadata.name_mapping()` to [`UpdateSchema._commit`](https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/update/schema.py#L645-L649). - Added columns get a new entry, and renamed columns keep their old name and gain the new one, as `UpdateNameMapping` already does (same as Java's `MappingUtil.update`). - A table without the property is left alone: no mapping is added. - An explicit `WithNameMapping` still takes precedence. - If the property can't be parsed, `meta.NameMapping()` returns nil and the update goes ahead without touching it. Java does the same: it logs a warning and keeps the old value. ### Testing - New `table/update_schema_name_mapping_test.go`: - `TestUpdateSchemaUpdatesTableNameMapping` adds a column and renames another on a table with a mapping, then checks the committed mapping (`category` for the new field, `data` and `payload` for the renamed one). - `TestUpdateSchemaWithoutNameMappingLeavesPropertyUnset` checks that a table without the property doesn't get one. - `TestAddFilesAfterAddColumnReadsNewColumnByName` runs the reproduction from the issue end to end: `AddDataFilesByPath`, `AddColumn`, `AddDataFilesByPath` with a file that has the new column and no field IDs, then a scan that reads `books` from it. On main, the first and third fail (the mapping has no entry for field 3, and the second `AddDataFilesByPath` fails with the `missing field_id` error above). The second passes on main and with the fix. - `go vet ./table/` is clean. `go test ./table/...` passes except for tests that fail the same way on unmodified main on my Windows machine (path handling): `TestEmptyStringIdentityPartitionScans`, `TestScanImportedSignedZeroBounds`, `TestDeleteImportedSignedZerosWithIn`, `TestDeleteNormallyWrittenSignedZeroWithIn`, `TestScanLegacyEmptyStringPartitionDeleteBounds`, `TestScanWithIDLessNameMappingFields`, two `TestRewriteDataFiles_PartialProgress*` tests and `TestWriteMapType`. `go test ./catalog/...` gives the same results as main. Closes #2645 This PR was prepared with AI assistance (Claude Code). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
