[
https://issues.apache.org/jira/browse/ARROW-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254643#comment-17254643
]
Neville Dipale commented on ARROW-11016:
----------------------------------------
I'm not yet sure about what it would take to read the same file across threads.
Would that be like sharing the file handle, as that might not work on all OSes.
>From working with the arrow -> parquet IO, I think that if the batch size is
>aligned with record group size, it might make it easier to read partial data
>from 1 or more groups.
I'll have a look in the coming days, and update this ticket with more info. It
might likely be a case of exposing some method to do partial reads.
> [Rust] Parquet ArrayReader should allow reading a subset of row groups
> ----------------------------------------------------------------------
>
> Key: ARROW-11016
> URL: https://issues.apache.org/jira/browse/ARROW-11016
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Rust
> Reporter: Andy Grove
> Priority: Major
>
> Parquet ArrayReader currently only supports reading an entire file from start
> to finish and does not allow selectively reading a subset of row groups. This
> prevents us from parallelizing work across threads when processing a single
> parquet file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
