HeartSaVioR commented on code in PR #58096: URL: https://github.com/apache/spark/pull/58096#discussion_r3819277518
########## sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala: ########## @@ -42,6 +42,31 @@ import org.apache.spark.sql.streaming.SinkProgress.DEFAULT_NUM_OUTPUT_ROWS /** * Information about updates made to stateful operators in a [[StreamingQuery]] during a trigger. + * + * @param operatorName + * Name of the stateful operator this progress describes. + * @param numRowsTotal + * Number of state rows held by the operator after this trigger. + * @param numRowsUpdated + * Number of state rows updated during this trigger. + * @param allUpdatesTimeMs + * Time taken, in milliseconds, to apply all state updates in this trigger. + * @param numRowsRemoved + * Number of state rows removed during this trigger. + * @param allRemovalsTimeMs + * Time taken, in milliseconds, to remove all evicted state rows in this trigger. + * @param commitTimeMs + * Time taken, in milliseconds, to commit the state changes of this trigger. + * @param memoryUsedBytes + * Memory used, in bytes, by the operator's state store. + * @param numRowsDroppedByWatermark + * Number of input rows dropped because they were later than the watermark. Review Comment: Here "later" is interpreted in opposite meaning as well - it can also be interpreted as `the timestamp is later than the watermark`. We should clarified as `because their event time was older than the watermark.`. ########## sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala: ########## @@ -42,6 +42,31 @@ import org.apache.spark.sql.streaming.SinkProgress.DEFAULT_NUM_OUTPUT_ROWS /** * Information about updates made to stateful operators in a [[StreamingQuery]] during a trigger. + * + * @param operatorName + * Name of the stateful operator this progress describes. + * @param numRowsTotal + * Number of state rows held by the operator after this trigger. + * @param numRowsUpdated + * Number of state rows updated during this trigger. + * @param allUpdatesTimeMs + * Time taken, in milliseconds, to apply all state updates in this trigger. + * @param numRowsRemoved + * Number of state rows removed during this trigger. + * @param allRemovalsTimeMs + * Time taken, in milliseconds, to remove all evicted state rows in this trigger. + * @param commitTimeMs + * Time taken, in milliseconds, to commit the state changes of this trigger. + * @param memoryUsedBytes + * Memory used, in bytes, by the operator's state store. + * @param numRowsDroppedByWatermark + * Number of input rows dropped because they were later than the watermark. + * @param numShufflePartitions + * Number of shuffle partitions the operator ran with. + * @param numStateStoreInstances + * Number of state store instances backing the operator. + * @param customMetrics + * State store implementation specific metrics, keyed by metric name. Review Comment: `Custom metrics specific to the stateful operator or state store implementation, keyed by metric name.` The custom metrics can be from both state store and stateful operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
