[ https://issues.apache.org/jira/browse/ARROW-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben Kietzman updated ARROW-8065: -------------------------------- Description: Currently: a fragment is a product of a scan; it is a lazy collection of scan tasks corresponding to a data source which is logically singular (like a single file, a single row group, ...). It would be more useful if instead a fragment were the direct object of a scan; one scans a fragment (or a collection of fragments): # Remove {{ScanOptions}} from Fragment's properties and move it into {{Fragment::Scan}} parameters. # Remove {{ScanOptions}} from {{Dataset::GetFragments}}. We can provide an overload to support predicate pushdown in FileSystemDataset and UnionDataset {{Dataset::GetFragments(std::shared_ptr<Expression> predicate)}}. # {{Fragment::schema}} property should be set at construction time, usually extracted from the fragment's Dataset. This will lessen the cognitive dissonance between fragments and files since fragments will no longer include references to scan properties. was: We should be able to list fragments without going through the Scanner/ScanOptions hoops. This exposes a flaw with the current API where it require a ScanOptions to create Fragment, this is also a problem for ARROW-7824, i.e. why do we need a ScanOptions (read manifest) to write record batches to a given path. # Remove {{ScanOptions}} from Fragment's properties and move it into {{Fragment::Scan}} parameters. # Remove {{ScanOptions}} from {{Dataset::GetFragments}}, if required, we can still provide an alternate signature, e.g. {{Dataset::GetFragments(std::shared_ptr<Expression> predicate)}} for sub-tree pruning in FileSystemDataset. # Fragment constructor should take a schema (and store it as a property), usually extracted from the Dataset schema. Update the schema() method accordingly. > [C++][Dataset] Untangle Dataset, Fragment and ScanOptions > --------------------------------------------------------- > > Key: ARROW-8065 > URL: https://issues.apache.org/jira/browse/ARROW-8065 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ - Dataset > Reporter: Francois Saint-Jacques > Priority: Major > > Currently: a fragment is a product of a scan; it is a lazy collection of scan > tasks corresponding to a data source which is logically singular (like a > single file, a single row group, ...). It would be more useful if instead a > fragment were the direct object of a scan; one scans a fragment (or a > collection of fragments): > # Remove {{ScanOptions}} from Fragment's properties and move it into > {{Fragment::Scan}} parameters. > # Remove {{ScanOptions}} from {{Dataset::GetFragments}}. We can provide an > overload to support predicate pushdown in FileSystemDataset and UnionDataset > {{Dataset::GetFragments(std::shared_ptr<Expression> predicate)}}. > # {{Fragment::schema}} property should be set at construction time, usually > extracted from the fragment's Dataset. > This will lessen the cognitive dissonance between fragments and files since > fragments will no longer include references to scan properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)