HeartSaVioR opened a new pull request #29767:
URL: https://github.com/apache/spark/pull/29767
### What changes were proposed in this pull request?
This PR proposes to add `DataStreamWriter.table` to specify the output
"table" to write from the streaming query.
### Why are the changes needed?
For now, there's no way to write to the table (especially catalog table)
even the table is capable to handle streaming write, so even with Spark 3,
writing to the catalog table via SS should go through the
`DataStreamWriter.format(provider)` and wish the provider can handle it as same
as we do with catalog table.
With the new API, we can directly point to the catalog table which supports
streaming write. Some of usages are covered with tests - simply saying, end
users can do the following:
```
// assuming `testcat` is a custom catalog, and `ns` is a namespace in the
catalog
spark.sql("CREATE TABLE testcat.ns.table1 (id bigint, data string) USING
foo")
val query = inputDF
.writeStream
.table("testcat.ns.table1")
.option(...)
.start()
```
### Does this PR introduce _any_ user-facing change?
Yes, as this adds a new public API in DataStreamWriter. This doesn't bring
backward incompatible change.
### How was this patch tested?
New unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
