jose-torres commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode URL: https://github.com/apache/spark/pull/23576#issuecomment-467678468 I would want a proposal that's a commit rather than a diff, if that makes sense. Something in the form of: 1. Here's the definition of what a watermark means in Spark. When an operator asks the streaming engine "what is my watermark?", it's calculated in suchandsuch way. 2. This definition of watermark supports use cases A, B, C. We want to support these for suchandsuch reason. 3. This definition of watermark does not support use cases X, Y, Z. For suchandsuch reason, we're confident that we do not want to support them and will not want to support them in the near future. What I'm worried about is possibilities like this. Suppose we decide that we want to support multiple aggregates in complete mode in 3.1, and realize that we need a separate watermark channel in that case. Then we'll be stuck; we will be forced to either break the semantic we just added, or establish a weird piecemeal semantic where you specify watermarks differently depending on the shape of your query.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
