uros-b commented on code in PR #58096: URL: https://github.com/apache/spark/pull/58096#discussion_r3820141423
########## sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala: ########## @@ -42,6 +42,31 @@ import org.apache.spark.sql.streaming.SinkProgress.DEFAULT_NUM_OUTPUT_ROWS /** * Information about updates made to stateful operators in a [[StreamingQuery]] during a trigger. + * + * @param operatorName + * Name of the stateful operator this progress describes. + * @param numRowsTotal + * Number of state rows held by the operator after this trigger. + * @param numRowsUpdated + * Number of state rows updated during this trigger. + * @param allUpdatesTimeMs + * Time taken, in milliseconds, to apply all state updates in this trigger. + * @param numRowsRemoved + * Number of state rows removed during this trigger. + * @param allRemovalsTimeMs + * Time taken, in milliseconds, to remove all evicted state rows in this trigger. + * @param commitTimeMs + * Time taken, in milliseconds, to commit the state changes of this trigger. + * @param memoryUsedBytes + * Memory used, in bytes, by the operator's state store. + * @param numRowsDroppedByWatermark + * Number of input rows dropped because they were later than the watermark. Review Comment: Good catch, done! ########## sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala: ########## @@ -42,6 +42,31 @@ import org.apache.spark.sql.streaming.SinkProgress.DEFAULT_NUM_OUTPUT_ROWS /** * Information about updates made to stateful operators in a [[StreamingQuery]] during a trigger. + * + * @param operatorName + * Name of the stateful operator this progress describes. + * @param numRowsTotal + * Number of state rows held by the operator after this trigger. + * @param numRowsUpdated + * Number of state rows updated during this trigger. + * @param allUpdatesTimeMs + * Time taken, in milliseconds, to apply all state updates in this trigger. + * @param numRowsRemoved + * Number of state rows removed during this trigger. + * @param allRemovalsTimeMs + * Time taken, in milliseconds, to remove all evicted state rows in this trigger. + * @param commitTimeMs + * Time taken, in milliseconds, to commit the state changes of this trigger. + * @param memoryUsedBytes + * Memory used, in bytes, by the operator's state store. + * @param numRowsDroppedByWatermark + * Number of input rows dropped because they were later than the watermark. + * @param numShufflePartitions + * Number of shuffle partitions the operator ran with. + * @param numStateStoreInstances + * Number of state store instances backing the operator. + * @param customMetrics + * State store implementation specific metrics, keyed by metric name. Review Comment: Done, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
