[
https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205356#comment-17205356
]
Remi Dettai commented on ARROW-10135:
-------------------------------------
Hi [~alamb] ! Thanks for your insight. The problem when you use S3 (or any http
blob storage) is that you typically want to query large chunks at once because
each call is costly and has high latency. I am not talking about optimization
here, calling tiny ranges from a blob storage is just baaaaad ;) (and using a
buffered reader is also very sub-optimal).
In the parquet from S3 use case, you'll want to download the footer at once
then entire columns at each call. This is not compatible with the standard IO
traits that are typically tailored to read only the amount of data that is
needed for the current operation: read footer length+magic bytes, deserialize
row group header, deserialize page... The problem is the same in the C++
implem, but it was left to the `ParquetFileReader` implementation to fetch
those full chunks at once when calling `ReadAt(position,nbytes)` IO interface.
This is not the case in Rust, and I don't believe it is a good solution: it is
hidden in the implementation how the IO API will be called and nothing guards
from updates in the `FileReader` to break it again.
What I'm working on is introducing an intermediate `ChunckReader` trait object
that generates "sliced readers" implementing the standard IO traits. These
"sliced readers" will be of the maximum size (e.g. entire footer, entire
columns) but the rest of the implementation can keep using normal "small
reads". These brings together the best of two worlds: the standard FS reader
can keep working with its buffered reader with barely no overhead and blob
storage clients can get large ranges at once then read the response stream
progressively.
[~andygrove] It is not entirely clear to me how to introduce async efficiently
into this, but I will try to figure it out as I go.
> [Rust] [Parquet] Refactor file module to help adding sources
> ------------------------------------------------------------
>
> Key: ARROW-10135
> URL: https://issues.apache.org/jira/browse/ARROW-10135
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust
> Affects Versions: 1.0.1
> Reporter: Remi Dettai
> Priority: Major
> Labels: parquet, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, the Parquet reader is very strongly tied to file system reads.
> This makes it hard to add other sources. For instance, to implement S3, we
> would need a reader that loads entire columns at once rather than buffered
> reads of a few Ko.
> To improve modularity, we could try to move as much logic as possible to the
> generic traits (FileReader, RowGroupReader...) and reduce the code in the
> implementing structs (SerializedFileReader, SerializedRowGroupReader...) to
> the part that is specific to file/buffered reads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
