rahulsmahadev opened a new pull request, #4457:
URL: https://github.com/apache/iceberg-python/pull/4457
# Rationale for this change
Bug: `_determine_partitions` reads the partition source from the input by
its full name in the current table schema, split on dots, while the writer
follows the table's struct path with the input's own names (`task_schema`, from
the Arrow field IDs or the name mapping). That gives two bugs:
- A dot in a name on the path to a nested source fails the write. With a
struct `a.b` and an identity partition on its child `c`, `append` raises
`KeyError: 'Field "a" does not exist in schema'`, because the split (from
#2204) looks for a column `a`.
- When the input has field IDs from before a rename, the partition values
come from another column than the one the writer stores, so filtered scans miss
the rows:
```python
schema = Schema(NestedField(1, "a", StringType()), NestedField(2, "b",
StringType()))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000,
transform=IdentityTransform(), name="a_part"))
tbl = catalog.create_table("db.t", schema=schema, partition_spec=spec)
df = pa.Table.from_pylist([{"a": "x", "b": "y"}],
schema=tbl.schema().as_arrow()) # field IDs 1 and 2
for old, new in [("a", "tmp"), ("b", "a"), ("tmp", "b")]:
with tbl.update_schema() as update:
update.rename_column(old, new)
tbl.append(df)
tbl.scan().to_arrow().to_pylist() # [{'b': 'x', 'a': 'y'}]
[task.file.partition for task in tbl.scan().plan_files()] # [Record[y]],
expected [Record[x]]
tbl.scan(row_filter="b = 'x'").to_arrow().to_pylist() # [], expected the row
```
Fix: find the source the same way as the writer, following its struct path
in the table schema with each name taken from the input schema, and pass the
names to Arrow literally. How the writer matches columns is unchanged. An input
without the partition source's field ID now raises a `ValueError`, instead of a
`KeyError` or a wrong partition value.
Java: `SparkWriteBuilder` aligns the input schema's IDs with the table, and
`SparkWrite` reuses a `PartitionKey` whose `StructTransform` reads each row
through `schema.accessorForField(sourceId)` of that input schema.
Not covered here: dynamic partition overwrite still has #4196 (open fix
#4204), and the row filter converter
(`_ConvertToArrowExpression._get_field_name`) also splits names, so delete
rewrites and filtered scans of `add_files` files with dotted struct names can
still raise `ArrowInvalid`.
## Are these changes tested?
Yes, `test_append_to_identity_partitioned_table_with_dotted_names`,
`test_append_with_field_ids_after_swapping_column_names` and
`test_partition_values_follow_writer_struct_path` in
`tests/catalog/test_catalog_behaviors.py`, and
`test_partition_for_nested_field_with_dotted_names` in
`tests/io/test_pyarrow.py`. On main the dotted-name tests fail with the
`KeyError` above and the swap test with partition `y` instead of `x`; all pass
with this change.
## Are there any user-facing changes?
Yes, writes to tables partitioned on a nested field with a dot in a name on
its path work, and partition values match the written data when the input has
field IDs from before a rename.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]