Ankitkj1999 opened a new issue, #4362:
URL: https://github.com/apache/iceberg-python/issues/4362

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   ### Describe the bug
   When a table evolves its partition spec by renaming a partition field:
   1. `TableMetadata.specs_struct()` adopts the latest field name for that 
partition `field_id` (e.g. `1000: p` -> `1000: p_renamed`).
   2. However, `table.inspect.partitions()`, `files()`, `data_files()`, 
`all_files()`, `all_data_files()`, and `entries()` build partition dictionaries 
using `field.name` from each file's historical `PartitionSpec` directly (`{"p": 
"a"}`).
   3. When PyArrow converts these rows to the schema defined by 
`specs_struct()` (which has field `"p_renamed"`), the `"p"` key is ignored and 
`"p_renamed"` is filled with `None`. All pre-rename partition values are 
silently replaced with `None` (`{'p_renamed': None}`).
   
   Similarly, on format-version 1 tables, removing a partition field 
(`remove_field("p")`) and subsequently re-adding it (`add_identity("p")`) 
automatically renames the voided field (`1000: p_1000`) and creates `1001: p`. 
Because `inspect` keys by historical `field.name == "p"`, historical spec 0 
data files written under `field_id=1000` are silently misattributed to 
`field_id=1001` (`{'p_1000': None, 'p': 'a'}`).
   
   ### To Reproduce
   ```python
   import tempfile
   import pyarrow as pa
   from pyiceberg.catalog.memory import InMemoryCatalog
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.transforms import IdentityTransform
   from pyiceberg.types import LongType, NestedField, StringType
   
   with tempfile.TemporaryDirectory() as tmp:
       cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
       cat.create_namespace("ns")
       tbl = cat.create_table(
           "ns.t_rename",
           schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", 
StringType())),
           partition_spec=PartitionSpec(PartitionField(2, 1000, 
IdentityTransform(), "p")),
           properties={"format-version": "2"},
       )
       tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
       tbl.update_spec().rename_field("p", "p_renamed").commit()
       tbl.append(pa.table({"id": [3], "p": ["c"]}))
   
       # Historical values 'a' and 'b' silently become None:
       print(tbl.inspect.partitions().to_pydict()["partition"])
       # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, 
{'p_renamed': None}]
       # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, 
{'p_renamed': 'b'}]
   
       print([r["partition"] for r in tbl.inspect.files().to_pylist()])
       # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, 
{'p_renamed': None}]
       # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, 
{'p_renamed': 'b'}]
   ```
   
   ### Expected behavior
   `table.inspect` tables should map partition field values using the canonical 
field names from `specs_struct()` based on `field_id`, preserving historical 
partition values across spec renames and correctly separating partition entries 
per specification layout.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to