anxkhn opened a new pull request, #3659:
URL: https://github.com/apache/iceberg-python/pull/3659
# Rationale for this change
An identity transform on a `timestamp_ns` / `timestamptz_ns` column renders
the
raw integer value in its human string, so identity partition directory paths
come out as `ts=1740342104375612001` instead of an ISO-8601 timestamp. The
microsecond `timestamp` / `timestamptz` types render correctly (e.g.
`ts=2025-02-23T20:21:44.375612`), so this is an inconsistency in the
nanosecond
timestamp types added for the v3 spec.
The cause is in `_int_to_human_string` (`pyiceberg/transforms.py`): it
dispatches
`DateType`, `TimeType`, `TimestampType`, and `TimestamptzType`, but has no
registration for `TimestampNanoType` / `TimestamptzNanoType`, so those values
fall through to the base handler `str(value)`. Because
`IdentityTransform.to_human_string` and `PartitionSpec.partition_to_path`
(`pyiceberg/partitioning.py`) both route through `_int_to_human_string`, an
identity-partitioned nanosecond timestamp column produces a raw-integer
partition path.
This change adds `to_human_timestamp_ns` / `to_human_timestamptz_ns` helpers
in
`pyiceberg/utils/datetime.py` and registers them for the two nanosecond
timestamp types. Python's `datetime` only supports microsecond precision, so
the
helpers render the microsecond instant at a fixed 6 fractional digits and
append
the remaining 3 sub-microsecond digits, yielding a full 9 fractional-digit
ISO-8601 string with no trailing-zero trimming. This matches the Java
`DateTimeUtil.nanosToIsoTimestamp` / `nanosToIsoTimestamptz` behaviour used
by
`Transforms.identity().toHumanString()` (for example
`1510871468000001001 -> 2017-11-16T22:31:08.000001001`, with the zone offset
kept
after the fraction for the tz variant).
## Are these changes tested?
Yes.
- `tests/utils/test_datetime.py`: new `test_to_human_timestamp_ns` and
`test_to_human_timestamptz_ns` cover the epoch boundary and the
sub-microsecond digit boundaries (0, 1, 999, 1000), a whole-second-plus-1ns
case (where the microsecond fraction is zero), and the Java reference
value.
- `tests/test_transforms.py::test_identity_human_string`: adds `timestamp_ns`
and `timestamptz_ns` rows alongside the existing type rows.
- `tests/table/test_partitioning.py::test_partition_spec_to_path_timestamp`:
an
end-to-end `PartitionSpec.partition_to_path` check for the microsecond and
nanosecond timestamp/timestamptz types (the user-facing symptom).
The targeted subset passes (14 passed). Running the full
`tests/test_transforms.py`,
`tests/utils/test_datetime.py`, and `tests/table/test_partitioning.py` files
shows only the pre-existing failures that require the optional
`pyiceberg_core`
extension; these new tests do not touch that path. Integration tests
(Docker + Spark) were not run in this environment; the behaviour is covered
by
the unit tests end-to-end through `partition_to_path`.
## Are there any user-facing changes?
Yes. Identity partition paths for `timestamp_ns` and `timestamptz_ns`
columns are
now ISO-8601 strings (matching the microsecond timestamp types and the Java
implementation) instead of raw integers. `IdentityTransform.to_human_string`
for
these types changes correspondingly.
<!-- This is a user-facing behaviour fix; maintainers may want the changelog
label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
