gengliangwang opened a new pull request #23836: [SPARK-26915][SQL]File source should write without schema validation in DataFrameWriter.save() URL: https://github.com/apache/spark/pull/23836 ## What changes were proposed in this pull request? Spark supports writing to file data sources without getting and validation with the table schema. For example, ``` spark.range(10).write.orc(path) val newDF = spark.range(20).map(id => (id.toDouble, id.toString)).toDF("double", "string") newDF.write.mode("overwrite").orc(path) ``` 1. There is no need to get/infer the schema from the table/path 2. The schema of `newDF` can be different with the original table schema. However, the behavior is changed since https://github.com/apache/spark/pull/23606 . Currently, data source V2 always validates the output query with the table schema. Even after the catalog support of DS V2 is implemented, I think it is hard to support both behaviors with the current API/framework. To me, `DataFrameWriter.save` is more like a simple IO API(e.g. file IO). It doesn't have to be involved with the `table` concept, otherwise: 1. Overwrite: can be `Drop table + CTAS`, can be `Insert overwrite`, can be `CTAS`... 2. Append: can be `Insert`, can be `Alter table add column + Insert`, can be `CTAS`.. Things can be too complex if we decide to allow both with/without schema validation in one API `DataFrameWriter.save`. That is to say, let's remove the expression `AppendData` and `OverwriteByExpression` in `DataFrameWriter.save`, since their behaviors are different with the API's. The expressions are still useful. We can use `AppendData` and `OverwriteByExpression` in `DataFrameWriter.saveAsTable`, which is more appropriate. This PR proposes to remove the new expressions in `DataFrameWriter.save`, and reenable ORC V2. I am aware that the interface `SupportsSaveMode` might be removed in the future. But in the current stage, we should prevent the regression, and make sure the behavior is not changed or become unpredictable in the furture development. ## How was this patch tested? Unit test
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
