jingz-db commented on code in PR #44927:
URL: https://github.com/apache/spark/pull/44927#discussion_r1488336417
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -184,6 +185,41 @@ class IncrementalExecution(
}
}
+ /**
Review Comment:
Thanks for the review Jungtaek! I also like the idea of adding a map.
> So the check can be also done after executing physical planning rules,
maybe at the end of state.apply()
I tried to add the check after `WriteStatefulOperatorMetadataRule` but this
will miss detecting for adding an operator after restart (because the
additional operator is already written to metadata). So I keep the check before
`WriteStatefulOperatorMetadataRule` and will omit the check if metadata is
empty.
It is also worth noting that if we do not perform the check before writing
to metadata and fail the query, untruthful info will be written to state
metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
