uros-b commented on code in PR #56692:
URL: https://github.com/apache/spark/pull/56692#discussion_r3461451693
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala:
##########
@@ -252,6 +253,15 @@ class AsyncProgressTrackingMicroBatchExecution(
override protected def getTrigger(): TriggerExecutor =
validateAndGetTrigger()
private def validateAndGetTrigger(): TriggerExecutor = {
+ // Sink evolution persists per-sink metadata via the V3 commit log written
in the base
+ // MicroBatchExecution.markMicroBatchEnd, which this class overrides with
an async write that
+ // only emits V1 commit metadata. The sink metadata would therefore never
be persisted, so
+ // reject the combination explicitly instead of silently dropping it.
+ if (sparkSession.sessionState.conf.enableStreamingSinkEvolution) {
+ throw new IllegalArgumentException(
Review Comment:
Are we sure we want to throw a raw IllegalArgumentException, rather than a
structured error class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
