xccui commented on pull request #28523: URL: https://github.com/apache/spark/pull/28523#issuecomment-628826865
Hi all, I am not quite familiar with the version history of the streaming sink, but would like to share pieces of my thought here. Please correct me if I misunderstood. 1. I think `SupportsStreamingUpdate` should NOT be a sink specific feature. We concentrate on sink now because the current SS implementation doesn't allow chaining operators (other than the sink) which produce updating results. 2. The only reason we provide update mode should be to produce the "correct" result table, i.e., to make the result table identical with the one produced by applying the same query on the materialized input rows. 3. The semantics of update mode we provided lacks deleting support, which makes it unreliable sometimes. I suppose we all agree on offering a better design in the future. But for now, it's better to keep it unchanged with previous versions (and also with the lowest risk and efforts). I'll try to make some improvements to the related issues. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
