HeartSaVioR commented on PR #58562: URL: https://github.com/apache/spark/pull/58562#issuecomment-5597831035
It isn't "impossible" to implement EXCEPT DISTINCT - it's basically LeftAnti + DISTINCT. I'd still question about the semantic though. The first issue is the rewrite itself. I don't personally think rewriting DISTINCT to AGGREGATE works in streaming. (That said, I don't agree with what we are doing today on streaming SQL statement.) IMHO, for streaming, dropDuplicates is always the right answer and I don't think updating/producing the same row every single time whenever there is a duplicate is what users really want. It's a bonus that Aggregate requires watermark to be set in append mode (default) and users would blindly change the output mode to update mode rather than understanding what's going on, and see the same row appearing again while it's a DISTINCT. The second issue is the fact that static (right input) is changing over time. This could bring the tricky semantic question - for example, in batch 0, the operator produces the row A since static does not provide the row A, but in batch 1, static has updated and it could have row A which we already produce the output and can't correct it. Would user tolerate that behavior? That's indeed what happens with stream-static join so it's not a new thing, though enrichment is less concerned about this while expecting the behavior of LeftAnti is probably more concerned with this. That said, I'd rather say we should be really careful with this. INTERSECT can be implemented for the same if we loose the semantic but we explicitly blocked it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
