jaychia opened a new issue, #37580:
URL: https://github.com/apache/arrow/issues/37580
### Describe the bug, including details regarding any error messages,
version, and platform.
PyArrow version: 13.0.0
Python version: 3.10.9
OS: MacOSX, arm64
Some Timestamp values cannot be successfully round-tripped through
PyArrow-Parquet[INT96 types]-PyArrow, even though they are within the
expressible range of the timeunit ("milliseconds" in the example below).
---
To reproduce:
```
In [1]: import datetime
In [2]: import pyarrow as pa
In [3]: import pyarrow.parquet as papq
In [4]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 59),
datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)],
pa.timestamp("ms"))
In [5]: table = pa.table({"ts": ts})
In [6]: papq.write_table(table, "test.pq",
use_deprecated_int96_timestamps=True)
In [7]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[7]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1784-07-20 23:35:32.709,2000-01-01 00:00:00.000,2400-01-01
00:00:00.000]]
```
Note that the first timestamp is received as `1784-07-20 23:35:32.709`
instead of the expected `1200-01-01 00:00:59.000`.
However, if we change the timestamp from `datetime.datetime(1200, 1, 1, 0,
0, 59)` to `datetime.datetime(1200, 1, 1, 0, 0, 00)` it now seems to work.
```
In [9]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 00),
datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)],
pa.timestamp("ms"))
In [10]: table = pa.table({"ts": ts})
In [11]: papq.write_table(table, "test.pq",
use_deprecated_int96_timestamps=True)
In [12]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[12]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1200-01-01 00:00:00.000,2000-01-01 00:00:00.000,2400-01-01
00:00:00.000]]
```
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]