[
https://issues.apache.org/jira/browse/ARROW-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Shieh updated ARROW-1958:
--------------------------------
Description:
The pandas conversion of a datetimetz row index in a Table fails with non-UTC
time zones because the values are stored as datetime64\[ns\] and interpreted as
datetime64\[ns, tz\], rather than interpreted as datetime64\[ns, UTC\] and
converted to datetime64\[ns, tz\]. There's correct handling for time zones for
columns in Column.to_pandas, but not for the row index in table_to_blockmanager.
This is a minimal example demonstrating the failure of a roundtrip between a
DataFrame and a Table:
{code}
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({
'a': pd.date_range(
start='2017-01-01', periods=3, tz='America/New_York'
)
})
df = df.set_index('a')
df_pa = pa.Table.from_pandas(df).to_pandas()
print(df)
print(df_pa)
{code}
The output is:
{noformat}
Empty DataFrame
Columns: []
Index: [2017-01-01 00:00:00-05:00, 2017-01-02 00:00:00-05:00, 2017-01-03
00:00:00-05:00]
Empty DataFrame
Columns: []
Index: [2017-01-01 05:00:00-05:00, 2017-01-02 05:00:00-05:00, 2017-01-03
05:00:00-05:00]
{noformat}
was:
The pandas conversion of a datetimetz row index in a Table fails with non-UTC
time zones because the values are stored as datetime64\[ns\] and interpreted as
datetime64\[ns, tz\], rather than interpreted as datetime64\[ns, UTC\] and
converted to datetime64\[ns, tz\]. There's correct handling for time zones for
columns in Column.to_pandas, but not for the row index in table_to_blockmanager.
This is a minimal example demonstrating the failure of a roundtrip between a
DataFrame and a Table:
{code}
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({
'a': pd.date_range(
start='2017-01-01', periods=3, tz='America/New_York'
)
})
df.set_index('a')
df_pa = pa.Table.from_pandas(df).to_pandas()
print(df)
print(df_pa)
{code}
The output is:
{noformat}
Empty DataFrame
Columns: []
Index: [2017-01-01 00:00:00-05:00, 2017-01-02 00:00:00-05:00, 2017-01-03
00:00:00-05:00]
Empty DataFrame
Columns: []
Index: [2017-01-01 05:00:00-05:00, 2017-01-02 05:00:00-05:00, 2017-01-03
05:00:00-05:00]
{noformat}
> [Python] Error in pandas conversion for datetimetz row index
> ------------------------------------------------------------
>
> Key: ARROW-1958
> URL: https://issues.apache.org/jira/browse/ARROW-1958
> Project: Apache Arrow
> Issue Type: Bug
> Components: C
> Affects Versions: 0.8.0
> Environment: Ubuntu 16.04
> Reporter: Albert Shieh
>
> The pandas conversion of a datetimetz row index in a Table fails with non-UTC
> time zones because the values are stored as datetime64\[ns\] and interpreted
> as datetime64\[ns, tz\], rather than interpreted as datetime64\[ns, UTC\] and
> converted to datetime64\[ns, tz\]. There's correct handling for time zones
> for columns in Column.to_pandas, but not for the row index in
> table_to_blockmanager.
> This is a minimal example demonstrating the failure of a roundtrip between a
> DataFrame and a Table:
> {code}
> import pandas as pd
> import pyarrow as pa
> df = pd.DataFrame({
> 'a': pd.date_range(
> start='2017-01-01', periods=3, tz='America/New_York'
> )
> })
> df = df.set_index('a')
> df_pa = pa.Table.from_pandas(df).to_pandas()
> print(df)
> print(df_pa)
> {code}
> The output is:
> {noformat}
> Empty DataFrame
> Columns: []
> Index: [2017-01-01 00:00:00-05:00, 2017-01-02 00:00:00-05:00, 2017-01-03
> 00:00:00-05:00]
> Empty DataFrame
> Columns: []
> Index: [2017-01-01 05:00:00-05:00, 2017-01-02 05:00:00-05:00, 2017-01-03
> 05:00:00-05:00]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
