[
https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204955#comment-17204955
]
Andrew Lamb commented on ARROW-10135:
-------------------------------------
I also agree with keeping as much IO logic in the generic traits.
However, I think you could implement an S3 reader that implements
`ParquetReader` and avoid fetching the entire file at once without changing how
the parquet reader is implemented.
Despite the trait and struct names, which I find confusing, I don't think
`SerializedFileReader`
[link|https://github.com/apache/arrow/blob/master/rust/parquet/src/file/reader.rs#L158]
is tied to files per-se -- it is implemented in terms of a `ParquetReader`
ParquetReader is defined as {{ParquetReader: Read + Seek + Length + TryClone {}
}} -- in other words it is defined in terms of the standard rust IO traits Read
and Seek and also requires knowledge of the input length as well as the ability
to clone itself (so multiple read streams can be read in parallel).
If you implemented ParquetReader for your S3 reader you should be able to read
parquet files in a streaming fashion just fine.
> [Rust] [Parquet] Refactor file module to help adding sources
> ------------------------------------------------------------
>
> Key: ARROW-10135
> URL: https://issues.apache.org/jira/browse/ARROW-10135
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust
> Affects Versions: 1.0.1
> Reporter: Remi Dettai
> Priority: Major
> Labels: parquet, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, the Parquet reader is very strongly tied to file system reads.
> This makes it hard to add other sources. For instance, to implement S3, we
> would need a reader that loads entire columns at once rather than buffered
> reads of a few Ko.
> To improve modularity, we could try to move as much logic as possible to the
> generic traits (FileReader, RowGroupReader...) and reduce the code in the
> implementing structs (SerializedFileReader, SerializedRowGroupReader...) to
> the part that is specific to file/buffered reads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
