sohailshk opened a new pull request, #4421:
URL: https://github.com/apache/iceberg-python/pull/4421

   # Rationale for this change
   
   `Table.append` accepts any Arrow column that converts to the Iceberg column 
type, but the partition transforms of pyiceberg-core only handle the Arrow type 
that matches the Iceberg type exactly. On a partitioned table these appends 
fail with a `ValueError` from pyiceberg-core (`FeatureUnsupported => 
Unsupported data type for bucket transform` or `Unexpected => Should not call 
internally for unsupported data type`), although the same data appends fine to 
an unpartitioned table or an identity-partitioned one.
   
   This is easy to hit: `pyarrow.csv.read_csv` and `pyarrow.json.read_json` 
infer `timestamp[s]`, which breaks the common spec of a table partitioned by 
day on the event time. DuckDB `TIMESTAMP_S`, `TIMESTAMP_MS` and `SMALLINT` come 
out as `timestamp[s]`, `timestamp[ms]` and `int16`, and polars produces 
`large_binary`.
   
   ```python
   import pyarrow as pa
   from pyiceberg.catalog.sql import SqlCatalog
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.transforms import DayTransform
   from pyiceberg.types import NestedField, TimestampType
   
   catalog = SqlCatalog("c", uri="sqlite:///:memory:", 
warehouse="file:///tmp/wh")
   catalog.create_namespace("n")
   schema = Schema(NestedField(1, "ts", TimestampType(), required=False))
   spec = PartitionSpec(PartitionField(1, 1000, DayTransform(), "ts_day"))
   table = catalog.create_table("n.t", schema, partition_spec=spec)
   table.append(pa.table({"ts": pa.array([1, 2], pa.timestamp("s"))}))
   # ValueError: Unexpected => Should not call internally for unsupported data 
type Timestamp(Second, None)
   ```
   
   The other affected combinations of Arrow type and transform:
   
   - `int8`, `int16`, `uint8`, `uint16` (and `uint32` or in-range `uint64` on a 
long column) for `bucket` and `truncate`
   - `timestamp[s]` and `timestamp[ms]`, with or without a time zone, for 
`bucket`, `day` and `hour`, and for `identity` with time zone `+00:00`
   - `large_binary` and `binary_view` for `truncate`
   
   ## The fix
   
   `_determine_partitions` now casts the source column of each partition field 
to the Arrow type of its Iceberg type (int32/int64, microsecond timestamps, 
binary) before applying the transform. Only the partition computation changes; 
the data that is written is unchanged (narrow integers are already widened on 
write since #2799). `large_binary` is cast to `binary` for `truncate` only 
(apache/iceberg-rust#3119 tracks `LargeBinary` support in the Rust truncate 
kernel), so `bucket` and `identity` on `large_binary` behave as before. 
Nanosecond timestamps and dictionary-encoded columns are left alone (#4340 
covers dictionary).
   
   Partition values are computed on the Iceberg-typed value (the spec hashes 
`int` as `long`, and timestamps are microseconds), which are the values Java 
computes; the unit test checks every value against the Python transforms.
   
   ## Are these changes tested?
   
   Yes, in tests/io/test_pyarrow.py:
   
   - 
`test_determine_partitions_with_arrow_types_narrower_than_the_iceberg_type` 
checks the partition values of each type and transform pair against the Python 
transforms
   - `test_determine_partitions_keeps_large_binary_for_bucket` checks that 
`bucket` on `large_binary` is not cast
   - 
`test_append_arrow_types_narrower_than_the_iceberg_type_to_partitioned_table` 
appends to a table partitioned by bucket, day and truncate and checks the scan 
round trip
   
   The first and the last fail on main (`Unsupported data type for bucket 
transform: Int16`) and pass now. `pytest tests/io/test_pyarrow.py 
tests/table/test_partitioning.py tests/test_transforms.py` (without the 
network-bound S3/GCS/ADLS/retry/ssl tests): 557 passed, 1 skipped.
   
   ## Are there any user-facing changes?
   
   Partitioned tables accept these Arrow column types on append and overwrite.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to