[
https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205405#comment-17205405
]
Andrew Lamb commented on ARROW-10135:
-------------------------------------
[~rdettai] -- It seems to me that buffering (either in memory or on the local
disk) is going to be required in any general purpose solution when reading from
object store.
Even if you implement a ChunkReader that reads entire column chunks, for many
parquet files that will result in a substantial number of calls to S3 (e.g. a
100 column parquet file in 10 row groups will be at least 1001 S3 calls, in a
naive implementation).
Another way to achieve similar outcome that might result in a less intrusive
code change could be to extend SerializedReader to pass "hints" somehow to the
underlying ParquetReader of what I/O will be necessary soon (e.g. call
"read_hint(chunk_start, chunk_len)" so you could start the S3 I/O for the
entire chunk)
Adding some way to avoid the SerializedReader's direct use of BufferedReader
definitely seems like a good idea (perhaps factor out into
`RawSerializedReader` that doesn't doesn't use BufferedReader and then have
`SerializedReader` create a RawSerializedReader after first wrapping the input
in a BufferedReader.
> [Rust] [Parquet] Refactor file module to help adding sources
> ------------------------------------------------------------
>
> Key: ARROW-10135
> URL: https://issues.apache.org/jira/browse/ARROW-10135
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust
> Affects Versions: 1.0.1
> Reporter: Remi Dettai
> Priority: Major
> Labels: parquet, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, the Parquet reader is very strongly tied to file system reads.
> This makes it hard to add other sources. For instance, to implement S3, we
> would need a reader that loads entire columns at once rather than buffered
> reads of a few Ko.
> To improve modularity, we could try to move as much logic as possible to the
> generic traits (FileReader, RowGroupReader...) and reduce the code in the
> implementing structs (SerializedFileReader, SerializedRowGroupReader...) to
> the part that is specific to file/buffered reads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
