2010YOUY01 commented on issue #24265: URL: https://github.com/apache/datafusion/issues/24265#issuecomment-5383969323
Here is a follows up discussion for - https://github.com/apache/datafusion/pull/24353#issuecomment-5380741644 My suggestion is to completely separate the streaming variant from the existing implementation—using different Streams/state machines—instead of adding internal control flags and reusing the same implementation. The reason is that adding a new flag for "streaming vs. batch" does not just add a small amount of complexity. It adds a new dimension to the operator's state space, and the complexity grows combinatorially as more such dimensions are introduced. This is somewhat counterintuitive. What #24353 is doing may look like a small, linear increase in implementation complexity, but it is actually adding another independent dimension to the operator, and the extra complexity will continue to grow on that newly added dimension. There are already other dimensions, such as ordered vs. unordered input, spilling vs. non-spilling, etc. 10 innocent-looking binary dimensions can already produce 1,024 combinations. We have suffered from the same problem before; the following write-up is a case study: - https://github.com/apache/datafusion/issues/22710 The alternative I propose is to represent these as hard separated stream types, such as `StreamingFilterStream` and `FilterStream`, each with its own state machine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
