rdblue commented on a change in pull request #23702: [SPARK-26785][SQL] data
source v2 API refactor: streaming write
URL: https://github.com/apache/spark/pull/23702#discussion_r258593059
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -513,13 +514,16 @@ class MicroBatchExecution(
val triggerLogicalPlan = sink match {
case _: Sink => newAttributePlan
- case s: StreamingWriteSupportProvider =>
- val writer = s.createStreamingWriteSupport(
- s"$runId",
- newAttributePlan.schema,
- outputMode,
- new DataSourceOptions(extraOptions.asJava))
- WriteToDataSourceV2(new MicroBatchWrite(currentBatchId, writer),
newAttributePlan)
+ case s: SupportsStreamingWrite =>
+ // TODO: we should translate OutputMode to concrete write actions like
truncate, but
+ // the truncate action is being developed in SPARK-26666.
Review comment:
@cloud-fan, I don't want to remove the update *feature*. I want the v2 API
to correctly communicate how to update records. That requires passing the
update key.
Consequently, configuring a sink using `OutputMode.Update` is unsafe. It
doesn't communicate the key. My position is (and has been) that we cannot use
`OutputMode.Update` to configure v2 sinks because the update behavior -- which
rows should be replaced -- is undefined.
Instead, I proposed a [mix-in to the
`WriteBuilder`](https://docs.google.com/document/d/1vI26UEuDpVuOjWw4WPoH2T6y8WAekwtI7qoowhOFnI4/edit?disco=AAAACidsdOM):
```java
public interface SupportsUpdate extends WriteBuilder {
WriteBuilder updateByKey(String... column)
}
```
Clearly, we might not want to use `String...` but this shows a simple way to
communicate the key columns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
