[
https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242898#comment-17242898
]
Yuanjian Li commented on SPARK-33638:
-------------------------------------
There is already some discussion in the PR comments:
[https://github.com/apache/spark/pull/30521]
Since the file format in streaming doesn't support DSv2 for now, the current
implementation mainly focuses on V1 support(per the
[comment|https://github.com/apache/spark/pull/30521#issuecomment-737488404]).
Also, for providing a convenient API, we added the support for creating a table
if not exists in the API, which also focuses on the V1 table(per the
[comment|https://github.com/apache/spark/pull/30521#issuecomment-736948942]).
But as our initial rationalization of the stream table API described in the
[comment|[https://github.com/apache/spark/pull/30521#issuecomment-737515816|https://github.com/apache/spark/pull/30521#issuecomment-737515816,]],
the table API should support DSv2 table, which supports streaming writes.
To unblock the current V1 support and add the new API, we create this ticket to
let the V2 support on track.
> Full support of V2 table creation in DataStreamWriter.toTable API
> -----------------------------------------------------------------
>
> Key: SPARK-33638
> URL: https://issues.apache.org/jira/browse/SPARK-33638
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.0
> Reporter: Yuanjian Li
> Priority: Major
>
> Currently, we add support of creating if not exists in
> DataStreamWriter.toTable API. Since the file format in streaming doesn't
> support DSv2 for now, the current implementation mainly focuses on V1
> support. We need more work to do for the full support of V2 table creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
