sohailshk opened a new pull request, #4421:
URL: https://github.com/apache/iceberg-python/pull/4421
# Rationale for this change
`Table.append` accepts any Arrow column that converts to the Iceberg column
type, but the partition transforms of pyiceberg-core only handle the Arrow type
that matches the Iceberg type exactly. On a partitioned table these appends
fail with a `ValueError` from pyiceberg-core (`FeatureUnsupported =>
Unsupported data type for bucket transform` or `Unexpected => Should not call
internally for unsupported data type`), although the same data appends fine to
an unpartitioned table or an identity-partitioned one.
This is easy to hit: `pyarrow.csv.read_csv` and `pyarrow.json.read_json`
infer `timestamp[s]`, which breaks the common spec of a table partitioned by
day on the event time. DuckDB `TIMESTAMP_S`, `TIMESTAMP_MS` and `SMALLINT` come
out as `timestamp[s]`, `timestamp[ms]` and `int16`, and polars produces
`large_binary`.
```python
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import DayTransform
from pyiceberg.types import NestedField, TimestampType
catalog = SqlCatalog("c", uri="sqlite:///:memory:",
warehouse="file:///tmp/wh")
catalog.create_namespace("n")
schema = Schema(NestedField(1, "ts", TimestampType(), required=False))
spec = PartitionSpec(PartitionField(1, 1000, DayTransform(), "ts_day"))
table = catalog.create_table("n.t", schema, partition_spec=spec)
table.append(pa.table({"ts": pa.array([1, 2], pa.timestamp("s"))}))
# ValueError: Unexpected => Should not call internally for unsupported data
type Timestamp(Second, None)
```
The other affected combinations of Arrow type and transform:
- `int8`, `int16`, `uint8`, `uint16` (and `uint32` or in-range `uint64` on a
long column) for `bucket` and `truncate`
- `timestamp[s]` and `timestamp[ms]`, with or without a time zone, for
`bucket`, `day` and `hour`, and for `identity` with time zone `+00:00`
- `large_binary` and `binary_view` for `truncate`
## The fix
`_determine_partitions` now casts the source column of each partition field
to the Arrow type of its Iceberg type (int32/int64, microsecond timestamps,
binary) before applying the transform. Only the partition computation changes;
the data that is written is unchanged (narrow integers are already widened on
write since #2799). `large_binary` is cast to `binary` for `truncate` only
(apache/iceberg-rust#3119 tracks `LargeBinary` support in the Rust truncate
kernel), so `bucket` and `identity` on `large_binary` behave as before.
Nanosecond timestamps and dictionary-encoded columns are left alone (#4340
covers dictionary).
Partition values are computed on the Iceberg-typed value (the spec hashes
`int` as `long`, and timestamps are microseconds), which are the values Java
computes; the unit test checks every value against the Python transforms.
## Are these changes tested?
Yes, in tests/io/test_pyarrow.py:
-
`test_determine_partitions_with_arrow_types_narrower_than_the_iceberg_type`
checks the partition values of each type and transform pair against the Python
transforms
- `test_determine_partitions_keeps_large_binary_for_bucket` checks that
`bucket` on `large_binary` is not cast
-
`test_append_arrow_types_narrower_than_the_iceberg_type_to_partitioned_table`
appends to a table partitioned by bucket, day and truncate and checks the scan
round trip
The first and the last fail on main (`Unsupported data type for bucket
transform: Int16`) and pass now. `pytest tests/io/test_pyarrow.py
tests/table/test_partitioning.py tests/test_transforms.py` (without the
network-bound S3/GCS/ADLS/retry/ssl tests): 557 passed, 1 skipped.
## Are there any user-facing changes?
Partitioned tables accept these Arrow column types on append and overwrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]