anishshri-db commented on code in PR #50123:
URL: https://github.com/apache/spark/pull/50123#discussion_r2022132623
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2291,6 +2291,70 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STATE_STORE_COORDINATOR_MULTIPLIER_FOR_MIN_VERSION_DIFF_TO_LOG =
+
buildConf("spark.sql.streaming.stateStore.multiplierForMinVersionDiffToLog")
+ .internal()
+ .doc(
+ "Determines the version threshold for logging warnings when a state
store falls behind. " +
+ "The coordinator logs a warning when the store's uploaded snapshot
version trails the " +
+ "query's latest version by the configured number of deltas needed to
create a snapshot, " +
+ "times this multiplier."
+ )
+ .version("4.1.0")
+ .longConf
+ .checkValue(k => k >= 1L, "Must be greater than or equal to 1")
+ .createWithDefault(5L)
+
+ val STATE_STORE_COORDINATOR_MULTIPLIER_FOR_MIN_TIME_DIFF_TO_LOG =
+ buildConf("spark.sql.streaming.stateStore.multiplierForMinTimeDiffToLog")
+ .internal()
+ .doc(
+ "Determines the time threshold for logging warnings when a state store
falls behind. " +
+ "The coordinator logs a warning when the store's uploaded snapshot
timestamp trails the " +
Review Comment:
Would we also detect the case where its likely that maintenance runs very
infrequently ? for eg - if the user is running an availNow query with small
batch durations ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
