[
https://issues.apache.org/jira/browse/ARROW-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298181#comment-17298181
]
Micah Kornfield commented on ARROW-7224:
----------------------------------------
{quote}Also worth noting: the case discussed in this thread (of a filter which
references each partition field exactly once and specifies an equality
condition for each) corresponds to a single subdirectory which needs to be
scanned. This is not the case for all filters, but it would be possible to add
a special case when such a prefix can be extracted. This would require the
{{Partitioning}} be explicitly constructed (so that we know without inspection
of paths what partition fields are in play), but that's fairly straightforward.
{quote}
Agreed. I'll note that there is actually an interaction with the file system
here as well, but specialization for equality is a starting point. For
instance >= partition keys is achievable for S3, as well as a range lb <=
column <= ub. But strict less then/less than or equal would not achieve the
same efficiencies. FWIW, Spark as has APIs for push-down predicates that allow
a source to tell it which predicates it can be pushed down effectively and
which need to be done as part of the engine (i.e. using compute kernels). A
similar abstraction might be useful here.
> [C++][Dataset] Partition level filters should be able to provide filtering to
> file systems
> ------------------------------------------------------------------------------------------
>
> Key: ARROW-7224
> URL: https://issues.apache.org/jira/browse/ARROW-7224
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Micah Kornfield
> Priority: Major
> Labels: dataset
>
> When providing a filter for partitions, it should be possible in some cases
> to use it to optimize file system list calls. This can greatly improve the
> speed for reading data from partitions because fewer number of
> directories/files need to be explored/expanded. I've fallen behind on the
> dataset code, but I want to make sure this issue is tracked someplace. This
> came up in SO question linked below (feel free to correct my analysis if I
> missed the functionality someplace).
> Reference:
> [https://stackoverflow.com/questions/58868584/pyarrow-parquetdataset-read-is-slow-on-a-hive-partitioned-s3-dataset-despite-u/58951477#58951477]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
