alamb commented on issue #8157: URL: https://github.com/apache/arrow-rs/issues/8157#issuecomment-3192084633
Copying my response: > we could ask the decoder to provide us a "reason" or a "trace" on the individual read requests. This is a neat idea. I definitely don't want to make `AsyncFileReader` and more complicated than it currently is 🤮 However, this could be fairly easily added to the Push decoder API here: - https://github.com/apache/arrow-rs/pull/8080 Maybe instead of ```rust /// This is used to communicate between the decoder and the caller /// to indicate what data is needed next, or what the result of decoding is. #[derive(Debug)] pub enum DecodeResult<T: Debug> { /// The ranges of data necessary to proceed // TODO: distinguish between minimim needed to make progress and what could be used? NeedsData(Vec<Range<u64>>), /// The decoder produced an output item Data(T), /// The decoder finished processing Finished, } ``` We could add something like ```rust pub enum DataRequest { /// last 8 bytes Footer, /// Metadata at the eend Metadata, PageIndex, DataPage { row_group_index, page_index}, Unknown, ... } #[derive(Debug)] pub enum DecodeResult<T: Debug> { /// The ranges of data necessary to proceed // TODO: distinguish between minimim needed to make progress and what could be used? NeedsData { trace: DataRequest ranges: Vec<Range>, }, /// The decoder produced an output item Data(T), /// The decoder finished processing Finished, } ``` 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
