ShivamSharma43 opened a new issue, #2645:
URL: https://github.com/apache/iceberg-go/issues/2645

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `UpdateSchema` doesn't update the table's `schema.name-mapping.default` 
property. After a column is added or renamed, the mapping still describes the 
old schema, so files without Parquet field IDs can't be resolved for that 
column.
   
   This affects every table that iceberg-go has added files to: 
`AddDataFilesByPath` and `AddDataFiles` set the mapping automatically 
(`ensureNameMapping`) when it is missing. After a later `AddColumn`, adding a 
file that contains the new column fails.
   
   Reproduced on main (e7a4d96):
   
   ```go
   // Table with columns id (1) and data (2), unpartitioned, no name mapping 
yet.
   // first.parquet has columns id, data and no field IDs.
   txn := tbl.NewTransaction()
   _ = txn.AddDataFilesByPath(ctx, []string{first}, nil, false) // sets 
schema.name-mapping.default
   tbl, _ = txn.Commit(ctx)
   
   txn = tbl.NewTransaction()
   _ = txn.UpdateSchema(true, false).
        AddColumn([]string{"category"}, iceberg.PrimitiveTypes.String, "", 
false, nil).
        Commit()
   tbl, _ = txn.Commit(ctx)
   fmt.Println(tbl.Metadata().Properties()[table.DefaultNameMappingKey])
   
   // second.parquet has columns id, data, category and no field IDs.
   txn = tbl.NewTransaction()
   err := txn.AddDataFilesByPath(ctx, []string{second}, nil, false)
   fmt.Println(err)
   ```
   
   Output:
   
   ```
   [{"field-id":1,"names":["id"]},{"field-id":2,"names":["data"]}]
   invalid schema: cannot convert category: type=utf8, nullable
             metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing 
field_id
   ```
   
   The mapping has no entry for `category` (field 3). A rename has the same 
problem: `RenameColumn([]string{"data"}, "payload")` leaves field 2 mapped only 
as `data`, so a file that names the column `payload` can't be read either.
   
   Expected: the mapping becomes 
`[{"field-id":1,"names":["id"]},{"field-id":2,"names":["data"]},{"field-id":3,"names":["category"]}]`
 (and `["data","payload"]` for the rename), and the second file is added and 
scanned normally.
   
   **Cause:** `UpdateSchema.BuildUpdates` in `table/update_schema.go` only 
updates the mapping when the caller passes one with `WithNameMapping`. Nothing 
in the repo does that, so by default the property is never updated.
   
   **Java:** `SchemaUpdate.applyChangesToMetadata` reads the table's mapping 
property and updates it with `MappingUtil.update` on every schema change:
   
https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SchemaUpdate.java#L484-L505
   PyIceberg's `Transaction.update_schema` passes 
`self.table_metadata.name_mapping()` to `UpdateSchema`, which updates it in 
`_commit`:
   
https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/__init__.py#L471
   
https://github.com/apache/iceberg-python/blob/639a3bf505b6932945bba3f661395255bac872c2/pyiceberg/table/update/schema.py#L645-L649
   
   I'd like to work on this. I have a fix and a unit test ready.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to