cloud-fan commented on a change in pull request #24233: [SPARK-26356][SQL]
remove SaveMode from data source v2
URL: https://github.com/apache/spark/pull/24233#discussion_r271110151
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -376,14 +376,20 @@ trait V2WriteCommand extends Command {
override lazy val resolved: Boolean = outputResolved
def outputResolved: Boolean = {
- table.resolved && query.resolved && query.output.size == table.output.size
&&
- query.output.zip(table.output).forall {
- case (inAttr, outAttr) =>
- // names and types must match, nullability must be compatible
- inAttr.name == outAttr.name &&
+ table.resolved && query.resolved && {
+ // When the table schema is empty, skip the schema check.
+ // TODO: introduce a table capability to disable schema check.
+ table.schema.isEmpty || {
Review comment:
The noop data source reports its schema as empty, but can accept input data
set of any schema.
Previously, `SupportsSaveMode` was a special case, and we skip schema check
for it. Now, only file source is a special case, and schema check is applied
to noop data source as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
