jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119688075
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self:
SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most
stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold
of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most
built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of
the event time value
+ * for the output so far (including output from eviction). Operators which
behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] =
Some(inputWatermarkMs)
Review Comment:
So given the current logic:
wm1 = input watermark for 5 minute window
wm2 = input watermark for 10 minute window
Batch 0: wm1 = 0 wm2 = 0
batch 1: wm1 = 3 wm2 = 3
While this strategy may work in the current implementation and for most use
cases, is it comprehensive? This works in situations where you have have
windowed operators with increasing window sizes but as you pointed out what if
that is not true. Like in the example you provided source -> 5 minutes window
-> 3 minutes window, the 3 minute window would output a result before the 5
minute window would which is incorrect. The example you provided if the 5
minute window operator outputted a window of 0-5, this window record should be
buffered as an entry in 3-6 window range for the 3 minute window. I don't have
a specific use case in mind of why someone would do this but why shouldn't it
be possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
