vishnuprakaz opened a new issue, #3652:
URL: https://github.com/apache/iceberg-python/issues/3652
### Apache Iceberg version
None
### Please describe the bug 🐞
While looking at the partitioning code I noticed that
`_to_partition_representation` (`pyiceberg/partitioning.py`) doesn't handle
the v3 nano timestamp types.
For a normal `TimestampType` column, a `datetime` partition value gets
converted to microseconds since epoch. For
`TimestampNanoType` / `TimestamptzNanoType` there is no handler, so the
value just passes through unchanged:
```python
from datetime import datetime
from pyiceberg.partitioning import _to_partition_representation
from pyiceberg.types import TimestampType, TimestampNanoType
dt = datetime(2020, 1, 1)
_to_partition_representation(TimestampType(), dt) # 1577836800000000
_to_partition_representation(TimestampNanoType(), dt) #
datetime.datetime(2020, 1, 1, 0, 0) — not converted
```
That `datetime` object then ends up in the `DataFile.partition` record,
and the manifest writer expects an int there
(`TimestampNanoWriter` calls `write_int`), so writing to a table with an
identity partition on a `timestamp_ns` column would fail.
Users can't hit this today because writing v3 is still blocked (#1551),
but it will surface once that lands. It looks like an
oversight #1632 added nano support to transforms, readers/writers and
conversions, but `partitioning.py` was never updated.
Expected: `datetime` values for nano timestamp columns should convert to
nanoseconds since epoch, the same way the microsecond
types convert to micros.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
