HeartSaVioR opened a new pull request #24936: [SPARK-24634][SS] Add a new metric regarding number of rows later than watermark URL: https://github.com/apache/spark/pull/24936 ## What changes were proposed in this pull request? Please refer https://issues.apache.org/jira/browse/SPARK-24634 to see rationalization of the issue. This patch adds a new metric to count the number of rows arrived later than watermark. To count the number of rows correctly, this patch adds a new physical node before stateful operator node(s) which filter out late rows and count the number. The metric will be exposed to two places: * streaming query listener - numLateInputRows in stateOperators * SQL tab in UI - `number of late rows being discarded` in DiscardLateRowsExec This is a revised version of #21617. ## How was this patch tested? Modified UT.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
