[
https://issues.apache.org/jira/browse/ARROW-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Kietzman resolved ARROW-8658.
---------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 9670
[https://github.com/apache/arrow/pull/9670]
> [C++][Dataset] Implement subtree pruning for FileSystemDataset::GetFragments
> ----------------------------------------------------------------------------
>
> Key: ARROW-8658
> URL: https://issues.apache.org/jira/browse/ARROW-8658
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 0.17.0
> Reporter: Ben Kietzman
> Assignee: Ben Kietzman
> Priority: Major
> Labels: dataset, pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> This is a very handy optimization for large datasets with multiple partition
> fields. For example, given a hive-style directory {{$base_dir/a=3/}} and a
> filter {{"a"_ == 2}} none of its files or subdirectories need be examined.
> After ARROW-8318 FileSystemDataset stores only files so subtree pruning
> (whose implementation depended on the presence of directories to represent
> subtrees) was disabled. It should be possible to reintroduce this without
> reference to directories by examining partition expressions directly and
> extracting a tree structure from their subexpressions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
