HeartSaVioR commented on pull request #29715: URL: https://github.com/apache/spark/pull/29715#issuecomment-690254043
cc. @rdblue @cloud-fan @brkyvz Let me leave some background about the PR in current status... I focused on adding the API similar to `writeTo` in Structured Streaming, as it looks to be a missing piece on SS for DSv2. At first I planned to go deep and touch logical plans as well, but unlike batch, SS is a bit more complicated as there're two different execution modes (micro-batch, continuous) as well as `Sink` interface on DSv1 should be still supported in the code path. So the PR in current state doesn't touch the logical plan and try to address the surface, so that the necessary changes for logical plan can be discussed and handled later without changing the API introducing here (hopefully). If we want to require the PR to include the changes for logical plan, I may need to have some more time to look into the details of logical plan and propose some approach (or it should be even better if we discussed something already). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
