val aggMetric = AggregatedMetric(metricKey, sum)
state.update(aggMetric)
state.setTimeoutTimestamp(metricKey.substring(metricKey.lastIndexOf(".") +
1).toLong + (30 * 6))
Iterator(aggMetric)
}
}
From: Tathagata Das
Date: Friday, August 10, 2018 at 4:16 PM
To: subramgr
Cc:
Structured Streaming internally maintains one global watermark by taking a
min of the two watermarks. Thats why one gets reported. In Spark 2.4, there
will be the option of choosing max instead of min.
Just curious. Why do you have to two watermarks? Whats the query like.
TD
On Thu, Aug 9, 2018
Hi,
We have two *flatMapGroupWithState* in our job and we have two
*withWatermark*
We are getting the event max time, event time and watermarks from
*QueryProgressEvent*.
Right now it just returns one *watermark* value.
Are two watermarks maintained by Spark or just one.
If one which one
