bharath-techie commented on issue #10655:
URL: https://github.com/apache/arrow-rs/issues/10655#issuecomment-5605021819
> Would an API to apply a new RowSelection be useful ? As in apply a
Rowselection to the remaining rows 🤔
I think even this works @alamb
> It seems like for an OpenSearch like thing you would want "get()` where
the row-ids are supplied by some external indexes? That would be the so called
"random row access' pattern
Yes for row by row use case, we have a parquet reader over a file that and
we can advance to a particular doc at any given time and read N rows
This is the high level API we use
```
// One per (file, row group, consumer). Consumers never share a cursor.
struct ParquetConsumerCursor {
reader: ParquetRecordBatchReader, // per row group, retained
pos: usize, // physical rows consumed
window: usize, // adaptive: AIMD on hit density
}
impl ParquetCursor {
/// Lucene advanceExact(doc): row >= pos always holds.
fn advance(&mut self, row: usize) -> Result<Option<RecordBatch>> {
// today (fork):
reader.skip_rows(row - pos); reader.read_next_batch(window)
// with your suggestion :
reader.apply_selection([skip(row - pos), select(window)]);
reader.next();
pos += (row - pos) + rows_returned
}
}
```
is this what you're suggesting ? just to be in same page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
