[
https://issues.apache.org/jira/browse/ARROW-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205456#comment-17205456
]
Ben Kietzman commented on ARROW-10134:
--------------------------------------
[~jorisvandenbossche] Maybe in Python ParquetFileFragment could have a property
named {{whole_file_fragment}} or so, which would be set on child/subset
fragments by {{split_by_row_group}} and other subsetting methods.
{{f.whole_file_fragment.num_row_groups}} is then unambiguously the row groups
of the whole file. This would also allow using {{f.whole_file_fragment == f}}
to check for subselection
> [C++][Dataset] Add ParquetFileFragment::num_row_groups property
> ---------------------------------------------------------------
>
> Key: ARROW-10134
> URL: https://issues.apache.org/jira/browse/ARROW-10134
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Joris Van den Bossche
> Assignee: Ben Kietzman
> Priority: Major
> Labels: dataset, dataset-dask-integration
> Fix For: 2.0.0
>
>
> From https://github.com/dask/dask/pull/6534#issuecomment-699512602, comment
> by [~rjzamora]:
> bq. it would be great to have access the total row-group count for the
> fragment from a {{num_row_groups}} attribute (which pyarrow should be able to
> get without parsing all row-group metadata/statistics - I think?).
> One question is: does this attribute correspond to the row groups in the
> parquet file, or the (subset of) row groups represented by the fragment?
> I expect the second (so if you do SplitByRowGroup, you would get a fragment
> with num_row_groups==1), but this might be a potential confusing aspect of
> the attribute.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
