cloud-fan commented on issue #23859: [SPARK-26956][SQL] remove streaming output mode from data source v2 APIs URL: https://github.com/apache/spark/pull/23859#issuecomment-466267552 @HeartSaVioR this proposal was first discussed in a data source v2 community meeting, and then in [this doc](https://docs.google.com/document/d/1vI26UEuDpVuOjWw4WPoH2T6y8WAekwtI7qoowhOFnI4/edit#heading=h.5g08se9m4r6o). The recent discussion happens in [this PR](https://github.com/apache/spark/pull/23702#discussion_r258113277). In general, Update mode is not supported right now. The streaming framework is not able to propagate the "update key" to the sink. But users can look at the query manually and hardcode the "update key" in their custom sink, so we leave it for streaming data source v1, but not v2, as it's too hacky. @arunmahadevan you are right that other modes except Append are not very useful and are mostly used for debugging. However, Complete mode has a very clear semantic(truncate all the old data and write new data), and we already have the `SupportsTruncate` trait that matches this semantic, so we still support it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
