ShivamSharma43 opened a new issue, #2645:
URL: https://github.com/apache/iceberg-go/issues/2645
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
`UpdateSchema` doesn't update the table's `schema.name-mapping.default`
property. After a column is added or renamed, the mapping still describes the
old schema, so files without Parquet field IDs can't be resolved for that
column.
This affects every table that iceberg-go has added files to:
`AddDataFilesByPath` and `AddDataFiles` set the mapping automatically
(`ensureNameMapping`) when it is missing. After a later `AddColumn`, adding a
file that contains the new column fails.
Reproduced on main (e7a4d96):
```go
// Table with columns id (1) and data (2), unpartitioned, no name mapping
yet.
// first.parquet has columns id, data and no field IDs.
txn := tbl.NewTransaction()
_ = txn.AddDataFilesByPath(ctx, []string{first}, nil, false) // sets
schema.name-mapping.default
tbl, _ = txn.Commit(ctx)
txn = tbl.NewTransaction()
_ = txn.UpdateSchema(true, false).
AddColumn([]string{"category"}, iceberg.PrimitiveTypes.String, "",
false, nil).
Commit()
tbl, _ = txn.Commit(ctx)
fmt.Println(tbl.Metadata().Properties()[table.DefaultNameMappingKey])
// second.parquet has columns id, data, category and no field IDs.
txn = tbl.NewTransaction()
err := txn.AddDataFilesByPath(ctx, []string{second}, nil, false)
fmt.Println(err)
```
Output:
```
[{"field-id":1,"names":["id"]},{"field-id":2,"names":["data"]}]
invalid schema: cannot convert category: type=utf8, nullable
metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing
field_id
```
The mapping has no entry for `category` (field 3). A rename has the same
problem: `RenameColumn([]string{"data"}, "payload")` leaves field 2 mapped only
as `data`, so a file that names the column `payload` can't be read either.
Expected: the mapping becomes
`[{"field-id":1,"names":["id"]},{"field-id":2,"names":["data"]},{"field-id":3,"names":["category"]}]`
(and `["data","payload"]` for the rename), and the second file is added and
scanned normally.
**Cause:** `UpdateSchema.BuildUpdates` in `table/update_schema.go` only
updates the mapping when the caller passes one with `WithNameMapping`. Nothing
in the repo does that, so by default the property is never updated.
**Java:** `SchemaUpdate.applyChangesToMetadata` reads the table's mapping
property and updates it with `MappingUtil.update` on every schema change:
https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L484-L505
PyIceberg's `Transaction.update_schema` passes
`self.table_metadata.name_mapping()` to `UpdateSchema`, which updates it in
`_commit`:
https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/__init__.py#L471
https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/update/schema.py#L645-L649
I'd like to work on this. I have a fix and a unit test ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]