[
https://issues.apache.org/jira/browse/ARROW-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joris Van den Bossche resolved ARROW-17483.
-------------------------------------------
Fix Version/s: 10.0.0
Resolution: Fixed
Issue resolved by pull request 14011
[https://github.com/apache/arrow/pull/14011]
> [Python] Support for 'pa.compute.Expression' in filter argument to
> 'pa.read_table'
> ----------------------------------------------------------------------------------
>
> Key: ARROW-17483
> URL: https://issues.apache.org/jira/browse/ARROW-17483
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Patrik Kjærran
> Assignee: Miles Granger
> Priority: Major
> Labels: pull-request-available
> Fix For: 10.0.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently, the _filters_ argument supports {{{}List{}}}[{{{}Tuple{}}}] or
> {{{}List{}}}[{{{}List{}}}[{{{}Tuple{}}}]] or None as its input types. I was
> suprised to see that Expressions were not supported, considering that filters
> are converted to expressions internally when using use_legacy_dataset=False.
> The check on
> [L150-L153|https://github.com/apache/arrow/blob/28cf3f9f769dda11ddfe52bd316c96aecb656522/python/pyarrow/parquet/core.py#L150-L153]
> short-circuits and succeeds when encountering an expression, but later fails
> on
> [L2343|https://github.com/apache/arrow/blob/28cf3f9f769dda11ddfe52bd316c96aecb656522/python/pyarrow/parquet/core.py#L2343]
> as the expression is evaluated as part of a boolean expression.
> I think declaring filters using pa.compute.Expressions more pythonic and less
> error-prone, and ill-formed filters will be detected much earlier than when
> using list-of-tuple-of-string equivalents.
> *Example:*
> {code:java}
> import pyarrow as pa
> import pyarrow.compute as pc
> import pyarrow.parquet as pq
> # Creating a dummy table
> table = pa.table({
> 'year': [2020, 2022, 2021, 2022, 2019, 2021],
> 'n_legs': [2, 2, 4, 4, 5, 100],
> 'animal': ["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars",
> "Centipede"]
> })
> pq.write_to_dataset(table, root_path='dataset_name_2',
> partition_cols=['year'])
> # Reading using 'pyarrow.compute.Expression'
> pq.read_table('dataset_name_2', columns=["n_legs", "animal"],
> filters=pc.field("n_legs") < 4)
> # Reading using List[Tuple]
> pq.read_table('dataset_name_2', columns=["n_legs", "animal"],
> filters=[('n_legs', '<', 4)]) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
