Hor911 opened a new issue, #34460:
URL: https://github.com/apache/arrow/issues/34460

   ### Describe the enhancement requested
   
   Current implementation of arrow::FileReader::ReadRowGroups() does 2 things:
   
   1. Triggers pre-buffering if enabled
   2. Blocks until all data is received and decodes it
   
   Sync interface complicates use in environments where additional threads are 
undesirable. I suggest to break this method:
   
   ```
   Status FileReader::ReadRowGroups(const std::vector<int>& row_groups,
                                        const std::vector<int>& column_indices,
                                        std::shared_ptr<Table>* out);
   ```
   
   To these two:
   
   ```
   Status FileReader::WillNeedRowGroups(const std::vector<int>& row_groups,
                                       const std::vector<int>& column_indices);
   Status FileReader::DecodeRowGroups(const std::vector<int>& row_groups,
                                       const std::vector<int>& column_indices,
                                       std::shared_ptr<::arrow::Table>* out);
   ```
   
   WillNeedRowGroups() will trigger pre-buffering while DecodeRowGroups() will 
blocks and read the data.
   Functionality of ReadRowGroups() will remain the same and equals for calling 
of 2 new methods sequentially. 
   
   New methods allow read data in completely async way with custom 
arrow::io::RandomAccessFile implementation as follows
   
   1. Lazy pre-buffering must be enabled:
   ```
   parquet::ArrowReaderProperties properties;
   properties.set_cache_options(arrow::io::CacheOptions::LazyDefaults());
   properties.set_pre_buffer(true);
   ```
   2. Application code calls arrow::FileReader::WillNeedRowGroups(...) to start 
pre-buffering 
   3. Custom arrow::io::RandomAccessFile receives WillNeed(...) call, starts 
async IO and returns instantly
   4. Application continue to run until all data are loaded into 
RandomAccessFile instance
   5. Application calls arrow::FileReader::DecodeRowGroups(...) to get actual 
data. It will use Futures for waiting, but since data is ready, all 
arrow::io::RandomAccessFile::ReadAt/AsyncRead calls will be replied instantly 
and all Futures will be completed without waiting.
   
   Another great technique to increase performance in high latency systems 
(yes, it's S3) is read ahead:
   
   1. Create N instances of arrow::FileReader
   2. Start pre-buffering of N row groups, one per each reader with 
WillNeedRowGroups() call
   3. As soon as data are loaded (into RandomAccessFile) get it with 
DecodeRowGroups() 
   4. and call WillNeedRowGroups() again to receive into this reader next row 
group data
   
   It makes possible fast reading of large parquet files in stream-like mode 
with performance compared to single ReadRowGroups call of current API with 
complete list of row groups, but without complete caching it in the memory.
   
   In both scenarios application code will never locks on future provided that 
correct custom RandomAccessFile implementation will cache data from (any) async 
IO library and activate data load in any way after data is ready.
   
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to