[
https://issues.apache.org/jira/browse/ARROW-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724695#comment-16724695
]
Wes McKinney commented on ARROW-2592:
-------------------------------------
It will look like
{code}
In [6]: t.to_pandas(ignore_metadata=True).head()
Out[6]:
Row ID Order ID Order Date ... Product Base Margin Ship
Date __index_level_0__
0 1 3 2010-10-13 ... 0.8
2010-10-20 0
1 49 293 2012-10-01 ... 0.58
2012-10-02 1
2 50 293 2012-10-01 ... 0.39
2012-10-03 2
3 80 483 2011-07-10 ... 0.58
2011-07-12 3
4 85 515 2010-08-28 ... 0.5
2010-08-30 4
[5 rows x 22 columns]
{code}
I'll put up a PR once I write some tests.
If others really think this is a bad idea, backwards compatibility with this
old metadata _might_ be possible, but it would be a bit hacky
> [Python] Error reading old Parquet file due to metadata backwards
> compatibility issue
> -------------------------------------------------------------------------------------
>
> Key: ARROW-2592
> URL: https://issues.apache.org/jira/browse/ARROW-2592
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.11.1
> Reporter: Dima Ryazanov
> Assignee: Wes McKinney
> Priority: Major
> Labels: parquet
> Fix For: 0.12.0
>
>
> Pyarrow 0.8 and 0.9 raises an AssertionError for one of the datasets I have
> (created using an older version of pyarrow). Repro steps:
> {{In [1]: from pyarrow.parquet import ParquetDataset}}
> {{In [2]: d = ParquetDataset(['bug.parq'])}}
> {{In [3]: t = d.read()}}
> {{In [4]: t.to_pandas()}}
> {{---------------------------------------------------------------------------}}
> {{AssertionError Traceback (most recent call
> last)}}
> {{<ipython-input-4-d17c9e2818f1> in <module>()}}
> {{----> 1 t.to_pandas()}}
> {{table.pxi in pyarrow.lib.Table.to_pandas()}}
> {{~/envs/cli3/lib/python3.6/site-packages/pyarrow/pandas_compat.py in
> table_to_blockmanager(options, table, memory_pool, nthreads, categories)}}
> {{ 529 # There must be the same number of field names and physical
> names}}
> {{ 530 # (fields in the arrow Table)}}
> {{--> 531 assert len(logical_index_names) == len(index_columns_set)}}
> {{ 532 }}
> {{ 533 # It can never be the case in a released version of pyarrow
> that}}
> {{AssertionError: }}
>
> Here's the file: [https://www.dropbox.com/s/oja3khjsc5tycfh/bug.parq]
> (I was not able to attach it here due to a "missing token", whatever that
> means.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
