wgtmac commented on code in PR #49855:
URL: https://github.com/apache/arrow/pull/49855#discussion_r3671970164
##########
cpp/src/parquet/arrow/reader_internal.h:
##########
@@ -133,5 +135,50 @@ Status
TransferColumnData(::parquet::internal::RecordReader* reader,
const ColumnDescriptor* descr, const ReaderContext*
ctx,
std::shared_ptr<::arrow::ChunkedArray>* out);
+// ----------------------------------------------------------------------
+// Pre-buffer eviction
+
+// GH-39808: as row groups finish decoding (possibly out of order under
+// readahead), advance a watermark over the leading run of completed ones and
+// evict cache entries ending before the lowest byte any remaining one needs.
+class ReadCacheEvictionState {
Review Comment:
I think we can remove `ReadCacheEvictionState` if eviction is performed in
row-group output order instead of decode-completion order.
`GetRecordBatchGenerator()` can keep the existing suffix-min calculation,
but move the resulting vector directly into `RowGroupGenerator`:
```cpp
RowGroupGenerator(..., std::vector<int64_t> evict_before_offsets)
: ...,
evict_before_offsets_(std::move(evict_before_offsets)) {}
std::vector<int64_t> evict_before_offsets_;
```
FetchNext() would only create and enqueue row_group_read. Eviction can
instead be attached when operator() pops the corresponding FIFO request:
```cpp
Future<RecordBatchGenerator> operator()() {
if (index_ >= row_groups_.size()) {
return AsyncGeneratorEnd<RecordBatchGenerator>();
}
const size_t request_index = index_++;
FillReadahead();
DCHECK(!in_flight_reads_.empty());
ReadRequest next = std::move(in_flight_reads_.front());
in_flight_reads_.pop();
rows_in_flight_ -= next.num_rows;
if (evict_before_offsets_.empty()) {
return next.read;
}
auto reader = arrow_reader_;
return next.read.Then(
[reader, offset = evict_before_offsets_[request_index + 1]](
RecordBatchGenerator generator) {
reader->parquet_reader()->EvictPreBufferedDataBefore(offset);
return generator;
});
}
```
The queue already preserves row-group order, and `MakeConcatenatedGenerator`
consumes these generators sequentially. A read completed early by readahead
would therefore retain its compressed buffer until its output turn, but
retention remains bounded by the readahead window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
