[
https://issues.apache.org/jira/browse/ARROW-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361491#comment-17361491
]
Joris Van den Bossche commented on ARROW-13040:
-----------------------------------------------
bq. Most of them mistakenly assume nanoseconds,
The problem is that pandas supports _only_ nanoseconds, so from that point of
view, this is correct.
So the result you get with {{to_pandas_dtype}} is consistent with the dtype of
the result of {{to_pandas}}:
{code:python}
>>> pa.timestamp("s").to_pandas_dtype()
dtype('<M8[ns]')
>>> pa.array([1, 2], pa.timestamp("s")).to_pandas().dtype
dtype('<M8[ns]')
{code}
We should maybe have a "to_numpy_dtype" equivalent, that can then preserve the
resolution (for numpy dtypes, we only seem to have the reverse mapping with
{{pyarrow.from_numpy_dtype}}).
> [Python] to_pandas_dtype values are wrong or unimplemented for date-time types
> ------------------------------------------------------------------------------
>
> Key: ARROW-13040
> URL: https://issues.apache.org/jira/browse/ARROW-13040
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 4.0.0
> Reporter: Jim Pivarski
> Priority: Major
>
> Most of them mistakenly assume nanoseconds, but some are not implemented.
> Here's the complete run-down:
> {{date32/date64/time32/time64}}
> {{---------------------------}}
> {{>>> pyarrow.date32()}}
> {{DataType(date32[day])}}
> {{>>> pyarrow.date32().to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{>>> pyarrow.date64()}}
> {{DataType(date64[ms])}}
> {{>>> pyarrow.date64().to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{>>> pyarrow.time32("s")}}
> {{Time32Type(time32[s])}}
> {{>>> pyarrow.time32("s").to_pandas_dtype()}}
> {{Traceback (most recent call last):}}
> {{ File "<stdin>", line 1, in <module>}}
> {{ File "pyarrow/types.pxi", line 200, in
> pyarrow.lib.DataType.to_pandas_dtype}}
> {{NotImplementedError: time32[s]}}
> {{>>> pyarrow.time32("ms")}}
> {{Time32Type(time32[ms])}}
> {{>>> pyarrow.time32("ms").to_pandas_dtype()}}
> {{Traceback (most recent call last):}}
> {{ File "<stdin>", line 1, in <module>}}
> {{ File "pyarrow/types.pxi", line 200, in
> pyarrow.lib.DataType.to_pandas_dtype}}
> {{NotImplementedError: time32[ms]}}
> {{>>> pyarrow.time64("us")}}
> {{Time64Type(time64[us])}}
> {{>>> pyarrow.time64("us").to_pandas_dtype()}}
> {{Traceback (most recent call last):}}
> {{ File "<stdin>", line 1, in <module>}}
> {{ File "pyarrow/types.pxi", line 200, in
> pyarrow.lib.DataType.to_pandas_dtype}}
> {{NotImplementedError: time64[us]}}
> {{>>> pyarrow.time64("ns")}}
> {{Time64Type(time64[ns])}}
> {{>>> pyarrow.time64("ns").to_pandas_dtype()}}
> {{Traceback (most recent call last):}}
> {{ File "<stdin>", line 1, in <module>}}
> {{ File "pyarrow/types.pxi", line 200, in
> pyarrow.lib.DataType.to_pandas_dtype}}
> {{NotImplementedError: time64[ns]}}
> {{timestamp}}
> {{---------}}
> {{>>> pyarrow.timestamp("s")}}
> {{TimestampType(timestamp[s])}}
> {{>>> pyarrow.timestamp("s").to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{>>> pyarrow.timestamp("ms")}}
> {{TimestampType(timestamp[ms])}}
> {{>>> pyarrow.timestamp("ms").to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{>>> pyarrow.timestamp("us")}}
> {{TimestampType(timestamp[us])}}
> {{>>> pyarrow.timestamp("us").to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{>>> pyarrow.timestamp("ns")}}
> {{TimestampType(timestamp[ns])}}
> {{>>> pyarrow.timestamp("ns").to_pandas_dtype()}}
> {{dtype('<M8[ns]')}}
> {{duration}}
> {{--------}}
> {{>>> pyarrow.duration("s")}}
> {{DurationType(duration[s])}}
> {{>>> pyarrow.duration("s").to_pandas_dtype()}}
> {{dtype('<m8[ns]')}}
> {{>>> pyarrow.duration("ms")}}
> {{DurationType(duration[ms])}}
> {{>>> pyarrow.duration("ms").to_pandas_dtype()}}
> {{dtype('<m8[ns]')}}
> {{>>> pyarrow.duration("us")}}
> {{DurationType(duration[us])}}
> {{>>> pyarrow.duration("us").to_pandas_dtype()}}
> {{dtype('<m8[ns]')}}
> {{>>> pyarrow.duration("ns")}}
> {{DurationType(duration[ns])}}
> {{>>> pyarrow.duration("ns").to_pandas_dtype()}}
> {{dtype('<m8[ns]')}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
