jose-torres commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode URL: https://github.com/apache/spark/pull/23576#issuecomment-461530739 I'd agree that min is the only reasonable way to compute an operator watermark. What I think we need a design for is operator watermarks in general, and how they slot into the rest of Spark. Questions I worry can't be addressed by a PR include: * I have a plan tree A: EventTimeExec -> B: StatefulOperator -> C: StatefulOperator. Can C use the watermark in A? If so, is it safe to do that when B transforms or projects away the watermarked column - if not, what are the rules for how watermarks must be provided with multiple aggregates? * Do all of our optimization and execution rules respect the semantics of operator watermarks? * We can currently call `withWatermark` at any point in the query plan. Is this consistent with operator watermarks? Even if we can support the two of them together, do we want to?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
