rahulsmahadev opened a new pull request, #4457:
URL: https://github.com/apache/iceberg-python/pull/4457

   # Rationale for this change
   
   Bug: `_determine_partitions` reads the partition source from the input by 
its full name in the current table schema, split on dots, while the writer 
follows the table's struct path with the input's own names (`task_schema`, from 
the Arrow field IDs or the name mapping). That gives two bugs:
   
   - A dot in a name on the path to a nested source fails the write. With a 
struct `a.b` and an identity partition on its child `c`, `append` raises 
`KeyError: 'Field "a" does not exist in schema'`, because the split (from 
#2204) looks for a column `a`.
   - When the input has field IDs from before a rename, the partition values 
come from another column than the one the writer stores, so filtered scans miss 
the rows:
   
   ```python
   schema = Schema(NestedField(1, "a", StringType()), NestedField(2, "b", 
StringType()))
   spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, 
transform=IdentityTransform(), name="a_part"))
   tbl = catalog.create_table("db.t", schema=schema, partition_spec=spec)
   df = pa.Table.from_pylist([{"a": "x", "b": "y"}], 
schema=tbl.schema().as_arrow())  # field IDs 1 and 2
   for old, new in [("a", "tmp"), ("b", "a"), ("tmp", "b")]:
       with tbl.update_schema() as update:
           update.rename_column(old, new)
   tbl.append(df)
   tbl.scan().to_arrow().to_pylist()  # [{'b': 'x', 'a': 'y'}]
   [task.file.partition for task in tbl.scan().plan_files()]  # [Record[y]], 
expected [Record[x]]
   tbl.scan(row_filter="b = 'x'").to_arrow().to_pylist()  # [], expected the row
   ```
   
   Fix: find the source the same way as the writer, following its struct path 
in the table schema with each name taken from the input schema, and pass the 
names to Arrow literally. How the writer matches columns is unchanged. An input 
without the partition source's field ID now raises a `ValueError`, instead of a 
`KeyError` or a wrong partition value.
   
   Java: `SparkWriteBuilder` aligns the input schema's IDs with the table, and 
`SparkWrite` reuses a `PartitionKey` whose `StructTransform` reads each row 
through `schema.accessorForField(sourceId)` of that input schema.
   
   Not covered here: dynamic partition overwrite still has #4196 (open fix 
#4204), and the row filter converter 
(`_ConvertToArrowExpression._get_field_name`) also splits names, so delete 
rewrites and filtered scans of `add_files` files with dotted struct names can 
still raise `ArrowInvalid`.
   
   ## Are these changes tested?
   
   Yes, `test_append_to_identity_partitioned_table_with_dotted_names`, 
`test_append_with_field_ids_after_swapping_column_names` and 
`test_partition_values_follow_writer_struct_path` in 
`tests/catalog/test_catalog_behaviors.py`, and 
`test_partition_for_nested_field_with_dotted_names` in 
`tests/io/test_pyarrow.py`. On main the dotted-name tests fail with the 
`KeyError` above and the swap test with partition `y` instead of `x`; all pass 
with this change.
   
   ## Are there any user-facing changes?
   
   Yes, writes to tables partitioned on a nested field with a dot in a name on 
its path work, and partition values match the written data when the input has 
field IDs from before a rename.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to