tustvold opened a new issue, #5522:
URL: https://github.com/apache/arrow-rs/issues/5522
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always
frustrated when [...]
(This section helps Arrow developers understand the context and *why* for
this feature, in addition to the *what*)
-->
Currently we provide `ParquetRecordBatchReader` and
`ParquetRecordBatchStream` as interfaces to read parquet data.
These interfaces are relatively straightforward to use, but have limitations:
* Cannot easily support concurrent decode (#5120)
* Limited flexibility w.r.t how data is fetched or decoded (#5141)
* Avoid decoding columns multiple times in the presence of predicates (#4864)
There is the experimental `ArrayReader` interface, however, this is very
hard to use correctly and exposes a lot of what should probably remain
implementation details.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I would like an interface, perhaps similar in spirit to that added to the
write side by #4871, that achieves the following:
* Makes it easy to parallelise both:
* The decoding of the parquet leaf columns
* The re-assembly of the arrow data from the dremel encodings
* Facilitates overriding the data source, e.g. by exposing the RowGroups
trait
* Avoids exposing too many internal implementation details
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features
you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
