danepitkin commented on PR #35656:
URL: https://github.com/apache/arrow/pull/35656#issuecomment-1611753259
The pandas integration test failure is:
```
> table.column('a').to_pandas(
safe=False, coerce_temporal_nanoseconds=True)
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_pandas.py:1535:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
pyarrow/array.pxi:868: in pyarrow.lib._PandasConvertible.to_pandas
???
pyarrow/table.pxi:472: in pyarrow.lib.ChunkedArray._to_pandas
???
opt/conda/envs/arrow/lib/python3.10/site-packages/pandas/core/dtypes/dtypes.py:866:
in __from_arrow__
array = array.cast(pyarrow.timestamp(unit=self._unit), safe=True)
pyarrow/table.pxi:556: in pyarrow.lib.ChunkedArray.cast
???
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/compute.py:402: in
cast
return call_function("cast", [arr], options, memory_pool)
pyarrow/_compute.pyx:572: in pyarrow._compute.call_function
???
pyarrow/_compute.pyx:367: in pyarrow._compute.Function.call
???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
> ???
E pyarrow.lib.ArrowInvalid: Casting from timestamp[s, tz=America/New_York]
to timestamp[ns] would result in out of bounds timestamp: -62135596800
```
The issue here is that the pyarrow user can specify the `safe=False` option,
but pyarrow internally can't pass that option to the pandas
`DatetimeTZDtype.__from_arrow_` method:
https://github.com/pandas-dev/pandas/blob/main/pandas/core/dtypes/dtypes.py#L897.
The only option is to either disallow usage of options in pyarrow's
`to_pandas()` methods for pandas ExtensionDType objects or ignore the
__from_arrow__ method and do the conversion in pyarrow instead.
Another approach is to explicitly add option parameters to `__from_arrow__`,
but his adds a higher level of complexity to package integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
