rahulsmahadev opened a new pull request, #4344:
URL: https://github.com/apache/iceberg-python/pull/4344
# Rationale for this change
`_determine_partitions` adds the transformed partition values to the Arrow
table as temporary columns named `_partition_<partition field name>`. If the
data already has a column with that name, the lookup by name is ambiguous and
`append()` fails:
```python
schema = Schema(
NestedField(1, "id", LongType()),
NestedField(2, "_partition_region", StringType()),
NestedField(3, "region", StringType()),
)
spec = PartitionSpec(PartitionField(source_id=3, field_id=1000,
transform=IdentityTransform(), name="region"))
table = catalog.create_table("default.t", schema=schema, partition_spec=spec)
table.append(pa.table({"id": [1], "_partition_region": ["a"], "region":
["us"]}))
# KeyError: 'Field "_partition_region" exists 2 times in schema'
```
This change picks helper column names that are not already in the table, by
prefixing underscores until the name is free. The helper columns are still
dropped before the data files are written.
## Are these changes tested?
Yes. `test_determine_partitions_with_column_named_like_partition_helper` in
`tests/io/test_pyarrow.py` partitions a table that has a `_partition_region`
data column and checks that every row lands in the right partition with that
column unchanged. It fails without the fix.
## Are there any user-facing changes?
Bug fix only: appends to such tables now succeed. No API changes.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
