johanl-db commented on code in PR #58831:
URL: https://github.com/apache/spark/pull/58831#discussion_r4047770918
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInsertionBase.scala:
##########
@@ -64,25 +65,27 @@ abstract class ResolveInsertionBase extends
Rule[LogicalPlan] {
}
private def renameFieldsInStruct(input: StructType, expected: StructType):
StructType = {
- if (input.length == expected.length) {
+ if (input.length == expected.length || resolveNestedFieldsByPosition) {
val newFields = input.zip(expected).map { case (f1, f2) =>
f1.copy(name = f2.name, dataType = renameFieldsInType(f1.dataType,
f2.dataType))
}
- StructType(newFields)
+ StructType(newFields ++ input.drop(expected.length))
Review Comment:
What happens here if we have target `{ x: 1 }` and input `{ y: 2, x: 3 }`?
`renamedFields` is `[x]`, and `input.drop(1)` is `[x]` also, so we get `[x,
x]`
You should add a test for this case
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala:
##########
@@ -736,6 +736,31 @@ trait InsertIntoSchemaEvolutionTests { this:
InsertIntoTests =>
}
}
+ Seq(
+ true -> Row(1, 2, 3),
+ false -> Row(2, 1, 3)
+ ).foreach { case (resolveNestedFieldsByPosition, expected) =>
+ test("Insert schema evolution: column list resolves nested fields by
position - " +
Review Comment:
Add a test for maps and for arrays also, since that's the other case this PR
fixes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
