[
https://issues.apache.org/jira/browse/ARROW-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317934#comment-17317934
]
Joris Van den Bossche edited comment on ARROW-12311 at 4/9/21, 1:52 PM:
------------------------------------------------------------------------
Python is a bit inconsistent IMO: there are methods on Dataset that create a
scanner for you and call to_table/scan_batches (this is fine). The scanner
itself is also exposed (but not the builder), getting a scanner itself is less
obvious (have to manually construct a scanner). The better thing would be to
have {{dataset.scan(**kwargs)}} exposed. And then I would also lean towards #3.
Particularly the projected schema doesn't really feel like an 'option' anyways.
was (Author: lidavidm):
Python is a bit inconsistent IMO: there are methods on Dataset that create a
scanner for you and call to_table/scan_batches (this is fine). The scanner
itself is also exposed (but not the builder), getting a scanner itself is less
obvious (have to manually construct a scanner). The better thing would be to
have {{dataset.scan(**kwargs)}}exposed. And then I would also lean towards #3.
Particularly the projected schema doesn't really feel like an 'option' anyways.
> [Python][R] Expose (hide?) ScanOptions
> --------------------------------------
>
> Key: ARROW-12311
> URL: https://issues.apache.org/jira/browse/ARROW-12311
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python, R
> Reporter: Weston Pace
> Assignee: Weston Pace
> Priority: Major
> Fix For: 5.0.0
>
>
> Currently R completely hides the `ScanOptions` class.
> In python the class is exposed but the documentation prefers `dataset.scan`
> (which hides both the scanner and the scan options).
> However, there is some useful information in the `ScanOptions`.
> Specifically, the projected schema (which is a product of the dataset schema
> and the projection expression and not easily recreated) and the materialized
> fields (the list of fields referenced by either the filter or the projection)
> which might be useful for reporting purposes.
> Currently R uses the projected schema to convert a list of column names into
> a partition schema. Python does not rely on either field.
>
> Options:
> - Keep the status quo
> - Expose the ScanOptions object (which itself is exposed via the Scanner)
> - Expose the interesting fields via the Scanner
>
> Currently the C++ design is halfway between the latter two (projected schema
> is exposed and options). My preference would be the third option. It raises
> a further question about how to expose the scanner itself in Python? Should
> the user be using ScannerBuilder? Should they use NewScan? Should they use
> the scanner directly at all or should it be hidden?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
