ericm-db commented on code in PR #47880:
URL: https://github.com/apache/spark/pull/47880#discussion_r1733454140
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala:
##########
@@ -176,11 +179,18 @@ class StateSchemaCompatibilityChecker(
true
} else {
// validate if the new schema is compatible with the existing schema
- existingStateSchemaList.lazyZip(newStateSchemaList).foreach {
- case (existingStateSchema, newStateSchema) =>
- check(existingStateSchema, newStateSchema, ignoreValueSchema)
+ val existingSchemaMap = existingStateSchemaList.map { schema =>
Review Comment:
It's because the deleteIfExists test case will fail because lazyZip compares
the schema by position in the list, so what ends up happening is [countState,
mostRecent] gets compared with [mostRecent] after countState is deleted, and
this is returned as invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
