[
https://issues.apache.org/jira/browse/ARROW-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109983#comment-17109983
]
Joris Van den Bossche edited comment on ARROW-8816 at 5/18/20, 7:46 AM:
------------------------------------------------------------------------
{quote}the extra field gives "datetime" for pandas_type and "object" for
numpy_type;
{quote}
Ah, if there is pandas metadata present and it indicates object dtype, we could
indeed use that to avoid conversion to datetime64[ns], but keep datetime
objects. That sounds as it should be possible in principle.
{quote}IMHO the important thing is to always be able to read back in what one
wrote (possibly with wider types) if the write was successful, provided that
one uses the same pyarrow version and the default options for both reading and
writing.
{quote}
Yes, but again: we need to distinguish writing parquet and pandas<->pyarrow
conversion. Just for writing parquet from a pyarrow table, fully correct
roundtrip works perfectly fine. It's only the pandas<->pyarrow conversion that
gives problems.
And note that in general, roundtrip is always tricky when a single pyarrow
type can map to multiple pandas types (like dates that can be converted to
datetime64[D] or datetime.date, or ListArray that can be converted to a column
of tuples, a column of lists or a column of numpy arrays)
But I agree a roundtrip should be possible: a {{timestamp_as_object}} keyword
should at least help (while still needing to specify a keyword). And with the
pandas metadata we could maybe try to automatically choose the good default.
was (Author: jorisvandenbossche):
> the extra field gives "datetime" for pandas_type and "object" for numpy_type;
Ah, if there is pandas metadata present and it indicates object dtype, we could
indeed use that to avoid conversion to datetime64[ns], but keep datetime
objects. That sounds as it should be possible in principle.
> IMHO the important thing is to always be able to read back in what one wrote
> (possibly with wider types) if the write was successful, provided that one
> uses the same pyarrow version and the default options for both reading and
> writing.
Yes, but again: we need to distinguish writing parquet and pandas<->pyarrow
conversion. Just for writing parquet from a pyarrow table, fully correct
roundtrip works perfectly fine. It's only the pandas<\->pyarrow conversion that
gives problems.
And note that in general, roundtrip is always tricky when a single pyarrow type
can map to multiple pandas types (like dates that can be converted to
datetime64[D] or datetime.date, or ListArray that can be converted to a column
of tuples, a column of lists or a column of numpy arrays)
But I agree a roundtrip should be possible: a {{timestamp_as_object}} keyword
should at least help (while still needing to specify a keyword). And with the
pandas metadata we could maybe try to automatically choose the good default.
> [Python] Year 2263 or later datetimes get mangled when written using pandas
> ---------------------------------------------------------------------------
>
> Key: ARROW-8816
> URL: https://issues.apache.org/jira/browse/ARROW-8816
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.16.0, 0.17.0
> Environment: Tested using pyarrow 0.17.0 and 0.16.0, pandas 1.0.3,
> python 3.7.5, mojave (macos). Also tested using pyarrow 0.16.0, pandas 1.0.3,
> python 3.8.2, ubuntu 20.04 (linux).
> Reporter: Rauli Ruohonen
> Priority: Major
>
> Using pyarrow 0.17.0, this
>
> {code:java}
> import datetime
> import pandas as pd
> def try_with_year(year):
> print(f'Year {year:_}:')
> df = pd.DataFrame({'x': [datetime.datetime(year, 1, 1)]})
> df.to_parquet('foo.parquet', engine='pyarrow', compression=None)
> try:
> print(pd.read_parquet('foo.parquet', engine='pyarrow'))
> except Exception as exc:
> print(repr(exc))
> print()
> try_with_year(2_263)
> try_with_year(2_262)
> {code}
>
> prints
>
> {noformat}
> Year 2_263:
> ArrowInvalid('Casting from timestamp[ms] to timestamp[ns] would result in out
> of bounds timestamp: 9246182400000')
> Year 2_262:
> x
> 0 2262-01-01{noformat}
> and using pyarrow 0.16.0, it prints
>
>
> {noformat}
> Year 2_263:
> x
> 0 1678-06-12 00:25:26.290448384
> Year 2_262:
> x
> 0 2262-01-01{noformat}
> The issue is that 2263-01-01 is out of bounds for a timestamp stored using
> epoch nanoseconds, but not out of bounds for a Python datetime.
> While pyarrow 0.17.0 refuses to read the erroneous output, it is still
> possible to read it using other parquet readers (e.g. pyarrow 0.16.0 or
> fastparquet), yielding the same result as with 0.16.0 above (i.e. only
> reading has changed in 0.17.0, not writing). It would be better if an error
> was raised when attempting to write the file instead of silently producing
> erroneous output.
> The reason I suspect this is a pyarrow issue instead of a pandas issue is
> this modified example:
>
> {code:java}
> import datetime
> import pandas as pd
> import pyarrow as pa
> df = pd.DataFrame({'x': [datetime.datetime(2_263, 1, 1)]})
> table = pa.Table.from_pandas(df)
> print(table[0])
> try:
> print(table.to_pandas())
> except Exception as exc:
> print(repr(exc))
> {code}
> which prints
>
>
> {noformat}
> [
> [
> 2263-01-01 00:00:00.000000
> ]
> ]
> ArrowInvalid('Casting from timestamp[us] to timestamp[ns] would result in out
> of bounds timestamp: 9246182400000000'){noformat}
> on pyarrow 0.17.0 and
>
>
> {noformat}
> [
> [
> 2263-01-01 00:00:00.000000
> ]
> ]
> x
> 0 1678-06-12 00:25:26.290448384{noformat}
> on pyarrow 0.16.0. Both from_pandas() and to_pandas() are pyarrow methods,
> pyarrow prints the correct timestamp when asked to produce it as a string (so
> it was not lost inside pandas), but the pa.Table.from_pandas(df).to_pandas()
> round-trip fails.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
