[
https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17206012#comment-17206012
]
Remi Dettai commented on ARROW-10135:
-------------------------------------
[~alamb] thanks again for you response, very interesting elements !
I agree that you might need another mechanism to keep the number of calls to S3
under control. For instance the C++ implementation uses read coalescing. This
is not easy to balance though because you will want to:
* increase the number of GETs to parallelize downloads and speed up bandwidth
* limit the total number of GETs to control overhead and price
* avoid reading data from columns that you are not going to use
The way I see it, ChunckReader is nothing more than a way to write the
"read_hint(chunk_start, chunk_len)" and the required indirection between the
SerializedReader and the BufferedReader. I'm going to give the implementation a
try, it might not be that intrusive if I manage to use traits correctly :)
> [Rust] [Parquet] Refactor file module to help adding sources
> ------------------------------------------------------------
>
> Key: ARROW-10135
> URL: https://issues.apache.org/jira/browse/ARROW-10135
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust
> Affects Versions: 1.0.1
> Reporter: Remi Dettai
> Priority: Major
> Labels: parquet, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, the Parquet reader is very strongly tied to file system reads.
> This makes it hard to add other sources. For instance, to implement S3, we
> would need a reader that loads entire columns at once rather than buffered
> reads of a few Ko.
> To improve modularity, we could try to move as much logic as possible to the
> generic traits (FileReader, RowGroupReader...) and reduce the code in the
> implementing structs (SerializedFileReader, SerializedRowGroupReader...) to
> the part that is specific to file/buffered reads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
