[
https://issues.apache.org/jira/browse/ARROW-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630175#comment-17630175
]
Lance Dacey commented on ARROW-15716:
-------------------------------------
Yes, if I could easily retrieve a list of the unique partitions which were
written to that would be helpful. If I could then parse the list of partitions
into a dataset expression (used for table(filter=expression)), that would be
even better.
Right now I can get a list of the fragments, parse them into expressions, and
from there I can determine the partitions using ds._get_partition_keys()
Full example below. I am essentially just looking for a potential shortcut,
convenience method, or better approach.
Say these are the fragments which were written during dataset write:
{code:python}
['path/to/data/month_id=202105/v1-manual__2022-11-06T22:50:20.parquet',
'path/to/data/month_id=202106/v1-manual__2022-11-06T22:50:20.parquet',
'path/to/data/month_id=202107/v1-manual__2022-11-06T22:50:20..parquet']
{code}
My ultimate goal is for a downstream task to filter the dataset for those three
partitions (not just the fragments since other files might exist).
{code:python}
partitioning = dataset.partitioning
#parse each fragment path to get a list of expressions
expressions = [partitioning.parse(file) for file in paths]
#get the partitions
filters = [ds._get_partition_keys(expression) for expression in expressions]
[{'month_id': 202105}, {'month_id': 202106}, {'month_id': 202107}]
#Convert to an expression
from pyarrow.parquet import filters_to_expression
filters_to_expression(filters)
<pyarrow.compute.Expression (((month_id == 202105) or (month_id == 202106)) or
(month_id == 202107))>
{code}
> [Dataset][Python] Parse a list of fragment paths to gather filters
> ------------------------------------------------------------------
>
> Key: ARROW-15716
> URL: https://issues.apache.org/jira/browse/ARROW-15716
> Project: Apache Arrow
> Issue Type: Wish
> Components: Python
> Affects Versions: 7.0.0
> Reporter: Lance Dacey
> Priority: Minor
>
> Is it possible for partitioning.parse() to be updated to parse a list of
> paths instead of just a single path?
> I am passing the .paths from file_visitor to downstream tasks to process data
> which was recently saved, but I can run into problems with this if I
> overwrite data with delete_matching in order to consolidate small files since
> the paths won't exist.
> Here is the output of my current approach to use filters instead of reading
> the paths directly:
> {code:python}
> # Fragments saved during write_dataset
> ['dev/dataset/fragments/date_id=20210813/data-0.parquet',
> 'dev/dataset/fragments/date_id=20210114/data-2.parquet',
> 'dev/dataset/fragments/date_id=20210114/data-1.parquet',
> 'dev/dataset/fragments/date_id=20210114/data-0.parquet']
> # Run partitioning.parse() on each fragment
> [<pyarrow.compute.Expression (date_id == 20210813)>,
> <pyarrow.compute.Expression (date_id == 20210114)>,
> <pyarrow.compute.Expression (date_id == 20210114)>,
> <pyarrow.compute.Expression (date_id == 20210114)>]
> # Format those expressions into a list of tuples
> [('date_id', 'in', [20210114, 20210813])]
> # Convert to an expression which is used as a filter in .to_table()
> is_in(date_id, {value_set=int64:[
> 20210114,
> 20210813
> ], skip_nulls=false})
> {code}
> My hope would be to do something like filt_exp = partitioning.parse(paths)
> which would return a dataset expression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
