takayoshi-makabe opened a new pull request, #3855:
URL: https://github.com/apache/iceberg-python/pull/3855
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
`TimestampNanoType` and `TimestamptzNanoType` are registered for byte
conversion but not for `to_json` / `from_json`, so the JSON single-value
serialization path raises for them. A nanosecond timestamp field with a default
cannot even be constructed:
```python
from pyiceberg.types import NestedField, TimestampNanoType
NestedField(1, "ts", TimestampNanoType(), required=False,
initial_default=1510871468123456789)
```
```shell
TypeError: Cannot deserialize bytes, type timestamp_ns not supported:
1510871468123456789
```
`NestedField` runs `from_json` on `initial-default` / `write-default` during
validation, so this fails on construction as well as on `model_dump_json()` and
`model_validate_json()`.
## Implementation notes
Per [Appendix
D](https://iceberg.apache.org/spec/#json-single-value-serialization) both types
serialize as ISO-8601 strings with nanosecond precision, matching Java's
`SingleValueParser`:
```
timestamp_ns "2017-11-16T22:31:08.123456789"
timestamptz_ns "2017-11-16T22:31:08.123456789+00:00"
```
`from_json` returns an `int` (epoch nanoseconds) rather than a `datetime`,
because Python's `datetime` only holds microseconds and would silently drop the
last three digits. `from_bytes` already returns an `int` for these two types.
`to_human_timestamp_ns` / `to_human_timestamptz_ns` are new — no existing
helper formats at nanosecond precision.
Two things left out of scope: the `TypeError` above says "Cannot deserialize
bytes" even from `to_json`, which is a pre-existing copy/paste from `from_json`
and unrelated to this fix. And this is independent of #3853, which refactors
the `*_to_nanos` parsers while this adds the reverse direction plus the JSON
dispatch — they touch different functions.
## Are these changes tested?
Yes.
- `tests/test_conversions.py`: both types added to
`test_json_single_serialization` (int and `datetime` inputs) and
`test_json_serialize_roundtrip`.
- `tests/utils/test_datetime.py`: the new formatters, covering the epoch,
zero-padded sub-second digits, and a pre-epoch value — each asserted to
round-trip back through `timestamp_to_nanos` / `timestamptz_to_nanos`.
- `tests/test_types.py`: end-to-end `NestedField` round-trip for
`initial-default` / `write-default`, which is the path in the report above.
## Are there any user-facing changes?
Yes. Nanosecond timestamp fields with an `initial-default` or
`write-default` now serialize and deserialize instead of raising `TypeError`.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
