viirya commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-766292712
> Also, when they can do arbitrary aggregation in forked process per partition, the aggregation is only bound to the micro-batch, even only bound to the single partition, which is no longer matched the concept of aggregation in SS which should be across the micro-batches. (That said, the output can be different between batch and streaming even if the input is same.) Hmm, I'd say not to think it as aggregation, they are different things. Pipe is widely used in Unix/Linux command line and I don't think we should mix it with aggregation. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
