[
https://issues.apache.org/jira/browse/ARROW-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169745#comment-17169745
]
Tonnam Balankura edited comment on ARROW-9573 at 8/3/20, 6:07 AM:
------------------------------------------------------------------
Thank you [~bkietz] for taking a look. Seems like [legacy
pq.read_table|https://github.com/apache/arrow/blob/master/python/pyarrow/parquet.py#L970]
pushes filtered files if no filtered directories are found.
If I understand correctly, with the new dataset API,
[discovery.cc|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/discovery.cc#L184]
handles the logic of ignoring directories that starts with "." and "_". I
presume that modifying the C++ code to behave like the legacy {{pq.read_table}}
is not the preferred approach.
One partial solution I can think of is to add the {{ignore_prefixes}} option to
{{read_table}} and {{_ParquetDatasetV2}} (and passing the option to
{{ds.dataset}}) to allow users to set it when calling {{read_table}}. Maybe
that would help users with the loading parquet with a partitioned column that
starts with an underscore. Do you think that could work?
was (Author: tonnamb):
Thank you [~bkietz] for taking a look. Seems like [legacy
pq.read_table|https://github.com/apache/arrow/blob/master/python/pyarrow/parquet.py#L970]
pushes filtered files if no filtered directories are found.
If I understand correctly, with the new dataset API,
[discovery.cc|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/discovery.cc#L184]
handles the logic of ignoring directories that starts with "." and "_". I
presume that modifying the C++ code to behave like the legacy {{pq.read_table}}
is not the preferred approach.
One partial solution I can think of is to add the {{ignore_prefixes}} option to
{{read_table}} and {{_ParquetDatasetV2}} to allow users to set it when calling
{{read_table}}. Maybe that would help users with the loading parquet with a
partitioned column that starts with an underscore. Do you think that could work?
> [Python] Parquet doesn't load when partitioned column starts with '_'
> ---------------------------------------------------------------------
>
> Key: ARROW-9573
> URL: https://issues.apache.org/jira/browse/ARROW-9573
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 1.0.0
> Reporter: Tonnam Balankura
> Assignee: Ben Kietzman
> Priority: Major
>
> When the loading parquet with partitioned column that starts with an
> underscore '_', nothing is loaded. No exceptions are raised either. Loading
> this parquet have worked for me in pyarrow 0.17.1, but not working anymore in
> pyarrow 1.0.0.
> On the other hand, loading parquet with a partitioned column starting with
> '_' is possible by using the `use_legacy_dataset` option. Also, when the
> column that starts with an underscore is not a partitioned column, loading
> parquet seems to work as expected.
> {code:python}
> >>> import pyarrow as pa
> >>> import pyarrow.parquet as pq
> >>> import pandas as pd
> >>> df1 = pd.DataFrame(data={'_COL_1': [1, 2], 'COL_2': [3, 4], 'COL_3': [5,
> >>> 6]})
> >>> table1 = pa.Table.from_pandas(df1)
> >>> pq.write_to_dataset(table1, partition_cols=['_COL_1', 'COL_2'],
> >>> root_path='test_parquet1')
> >>> df_pq1 = pq.read_table('test_parquet1')
> >>> df_pq1
> pyarrow.Table
> >>> len(df_pq1)
> 0
> >>> df_pq1_legacy = pq.read_table('test_parquet1', use_legacy_dataset=True)
> pyarrow.Table
> COL_3: int64
> _COL_1: dictionary<values=int64, indices=int32, ordered=0>
> COL_2: dictionary<values=int64, indices=int32, ordered=0>
> >>> len(df_pq1_legacy)
> 2
> >>> df2 = pd.DataFrame(data={'COL_1': [1, 2], 'COL_2': [3, 4], '_COL_3': [5,
> >>> 6]})
> >>> table2 = pa.Table.from_pandas(df2)
> >>> pq.write_to_dataset(table2, partition_cols=['COL_1', 'COL_2'],
> >>> root_path='test_parquet2')
> >>> df_pq2 = pq.read_table('test_parquet2')
> >>> df_pq2
> pyarrow.Table
> _COL_3: int64
> COL_1: int32
> COL_2: int32
> >>> len(df_pq2)
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
