ozankabak commented on PR #4694: URL: https://github.com/apache/arrow-datafusion/pull/4694#issuecomment-1361562009
As we all agreed when discussing [the streaming roadmap](https://github.com/apache/arrow-datafusion/issues/4285) a few weeks ago, our initial focus was to identify the small-but-powerful infrastructural improvements and hooks we can introduce in Datafusion so that more complex streaming use case can be supported either outside or inside Datafusion. IMO, this PR makes a huge step towards this goal. It enables Datafusion to process infinite files like FIFOs, to present an API for factoring in boundedness during planning and optimization, and it even gives Datafusion power to deduce whether it can run a given query with the given finite/infinite inputs. As @metesynnada mentions, the PR looks big, but the LOC comes mostly from tests. Other than those, changes are mostly localized to the file defining the `PipelineChecker` rule. Looking forward to your comments and feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
