Hor911 opened a new issue, #34460:
URL: https://github.com/apache/arrow/issues/34460
### Describe the enhancement requested
Current implementation of arrow::FileReader::ReadRowGroups() does 2 things:
1. Triggers pre-buffering if enabled
2. Blocks until all data is received and decodes it
Sync interface complicates use in environments where additional threads are
undesirable. I suggest to break this method:
```
Status FileReader::ReadRowGroups(const std::vector<int>& row_groups,
const std::vector<int>& column_indices,
std::shared_ptr<Table>* out);
```
To these two:
```
Status FileReader::WillNeedRowGroups(const std::vector<int>& row_groups,
const std::vector<int>& column_indices);
Status FileReader::DecodeRowGroups(const std::vector<int>& row_groups,
const std::vector<int>& column_indices,
std::shared_ptr<::arrow::Table>* out);
```
WillNeedRowGroups() will trigger pre-buffering while DecodeRowGroups() will
blocks and read the data.
Functionality of ReadRowGroups() will remain the same and equals for calling
of 2 new methods sequentially.
New methods allow read data in completely async way with custom
arrow::io::RandomAccessFile implementation as follows
1. Lazy pre-buffering must be enabled:
```
parquet::ArrowReaderProperties properties;
properties.set_cache_options(arrow::io::CacheOptions::LazyDefaults());
properties.set_pre_buffer(true);
```
2. Application code calls arrow::FileReader::WillNeedRowGroups(...) to start
pre-buffering
3. Custom arrow::io::RandomAccessFile receives WillNeed(...) call, starts
async IO and returns instantly
4. Application continue to run until all data are loaded into
RandomAccessFile instance
5. Application calls arrow::FileReader::DecodeRowGroups(...) to get actual
data. It will use Futures for waiting, but since data is ready, all
arrow::io::RandomAccessFile::ReadAt/AsyncRead calls will be replied instantly
and all Futures will be completed without waiting.
Another great technique to increase performance in high latency systems
(yes, it's S3) is read ahead:
1. Create N instances of arrow::FileReader
2. Start pre-buffering of N row groups, one per each reader with
WillNeedRowGroups() call
3. As soon as data are loaded (into RandomAccessFile) get it with
DecodeRowGroups()
4. and call WillNeedRowGroups() again to receive into this reader next row
group data
It makes possible fast reading of large parquet files in stream-like mode
with performance compared to single ReadRowGroups call of current API with
complete list of row groups, but without complete caching it in the memory.
In both scenarios application code will never locks on future provided that
correct custom RandomAccessFile implementation will cache data from (any) async
IO library and activate data load in any way after data is ready.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]