alamb commented on issue #10655: URL: https://github.com/apache/arrow-rs/issues/10655#issuecomment-5714977845
Here are some notes that I took on the call: The key thing @bharath-techie sorted me out is that the OpenSearch usecase is being driven by Lucene, and thus they don't know / can get the list of row ids up front -- they have to provide them row by row as Lucene calls them. The API from https://github.com/bharath-techie/arrow-rs/commit/38b2f7a6a7f58d0dde9f932be1497444f7646a4c to skip some number of rows in the Parquet reader seems like a reasonable thing to add ```rust impl ParquetRecordBatchReader { /// Skip up to `num_rows` rows from the current reader position. /// /// This advances the existing column readers without materializing a /// [`RecordBatch`]. When an offset index is available, complete pages in /// the skipped range are discarded by the underlying page readers without /// fetching or decoding their data. /// /// This method is only supported for readers without a [`RowSelection`]. /// A selection has its own logical cursor, which cannot be kept in sync /// with an independent physical skip. pub fn skip_rows(&mut self, num_rows: usize) -> Result<usize> { ... } ``` I can imagine it could be useful for other dynamic filtering usecases, such as [dynamic page pruning as mentioned](https://github.com/apache/arrow-rs/issues/10655#issuecomment-5603199107) by @zhuqi-lucas above. Specifically I think the idea there would be that the ParquetPushDecoder would be constructed with a RowSelection, but then at some point during the scan the caller would get additional information like "the next 1000 rows can be skipped". At that point the caller would use `skip_rows` In terms of the `window` / read_next_batch, one idea we had on the call is that we could use the existing `RowSelection` machinery that @hhhizzz and @haohuaijin have been working on, along with [`ParquetPushDecoder::into_builder`] to adjust the selection during the scan. I think @bharath-techie said he may look into this. [`ParquetPushDecoder::into_builder`]: https://docs.rs/parquet/latest/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.into_builder -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
