alamb commented on PR #11238: URL: https://github.com/apache/arrow-rs/pull/11238#issuecomment-6249562292
> @alamb I asked an agent to implement your idea: #11613. In general it looks much better. The one loss I see is that `RowFilter`s are not eagerly emitted once their pages are buffered and we have a whole batch of data. They require the entire row group's worth of pages to be buffered. Maybe you see a way to address that when you do your implementation? Yeah I think we'll need to handle the state of "evaluating predicates" as its own separate state -- basically if we have enough data to evaluate a row filter partially then if we have the pages to produce data immediately we would have to switch to a state that produced output before evaluating more data pages One thing I have been thinking is that maybe I can make some more I/O pattern tests so it is celarer what we are triyng to accomplish. I'll try and do that ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
