[
https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243070#comment-17243070
]
Yuanjian Li commented on SPARK-33638:
-------------------------------------
[https://github.com/apache/spark/pull/30521#discussion_r533869095]
For the partition column checking, different with V1 API, we can't pass the
partition info directly in the V2 table now. To make the behavior consistent
between V1 and V2, seems two ways we can consider:
* Make it possible to pass the partition column info to DSv2
* Check the partition column conflict in API directly for both V1/V2
> Full support of V2 table creation in DataStreamWriter.toTable API
> -----------------------------------------------------------------
>
> Key: SPARK-33638
> URL: https://issues.apache.org/jira/browse/SPARK-33638
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.0
> Reporter: Yuanjian Li
> Priority: Major
>
> Currently, we add support of creating if not exists in
> DataStreamWriter.toTable API. Since the file format in streaming doesn't
> support DSv2 for now, the current implementation mainly focuses on V1
> support. We need more work to do for the full support of V2 table creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
