[
https://issues.apache.org/jira/browse/SPARK-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-25709:
---------------------------------
Target Version/s: (was: 2.4.0)
> Can't append to a data source V2 unless target already has schema information
> or fixed schema matching dataframe
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25709
> URL: https://issues.apache.org/jira/browse/SPARK-25709
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Assaf Mendelson
> Priority: Blocker
>
> Let's assume we have a writer data source which writes to a dynamic schema
> (e.g. writing to a database table). If we have mode append but the table is
> missing then writing will fail.
>
> The reason for this is that in the dataframeWriter save method we have val
> relation = DataSourceV2Relation.create(source, options) which in turn
> contains: val reader = source.createReader(options, userSpecifiedSchema)
> The problem is that this sets the schema from the reader portion of the data
> source. If the data source can't infer the schema then we will fail to write
> (even thought the schema is actually part of the dataframe).
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
