This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b36fc0ebc5f0 [SPARK-58851][SS] Add missing scaladoc for streaming
progress parameters
b36fc0ebc5f0 is described below
commit b36fc0ebc5f07cb29a3208bb5846a3f31f60cf0d
Author: Uros <[email protected]>
AuthorDate: Fri Aug 21 12:38:14 2026 +0900
[SPARK-58851][SS] Add missing scaladoc for streaming progress parameters
### What changes were proposed in this pull request?
This PR adds missing scaladoc `param` documentation to the public
Structured Streaming progress classes in `sql/api`
(`org/apache/spark/sql/streaming/progress.scala`):
- `StreamingQueryProgress`: adds `param sink`.
- `SinkProgress`: adds `param metrics`.
- `StateOperatorProgress`: adds `param` tags for all of its constructor
parameters (`operatorName`, `numRowsTotal`, `numRowsUpdated`,
`allUpdatesTimeMs`, `numRowsRemoved`, `allRemovalsTimeMs`, `commitTimeMs`,
`memoryUsedBytes`, `numRowsDroppedByWatermark`, `numShufflePartitions`,
`numStateStoreInstances`, `customMetrics`).
This is a documentation-only change; no code is modified.
### Why are the changes needed?
These are public, `Evolving` API classes surfaced through the streaming
query listener and `StreamingQueryProgress`. Their scaladoc omitted several
`param` entries, so the generated API docs were incomplete for fields users
read when consuming progress events.
### Does this PR introduce _any_ user-facing change?
No. Documentation-only change.
### How was this patch tested?
No tests are required; the change adds scaladoc only. Each added `param`
name was verified to match a constructor parameter of its class, and each
description was checked against how the field is populated (`ProgressReporter`,
`statefulOperators`). Changed lines stay within 100 characters and introduce no
non-ASCII characters.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #58096 from uros-b/apidoc-streaming-progress-params.
Authored-by: Uros <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
(cherry picked from commit 1fc4b96cfd52652c9d66a19a390ca5a463dae85e)
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../org/apache/spark/sql/streaming/progress.scala | 30 ++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala
b/sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala
index 3411e962a571..aabb09c5741f 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/streaming/progress.scala
@@ -42,6 +42,32 @@ import
org.apache.spark.sql.streaming.SinkProgress.DEFAULT_NUM_OUTPUT_ROWS
/**
* Information about updates made to stateful operators in a
[[StreamingQuery]] during a trigger.
+ *
+ * @param operatorName
+ * Name of the stateful operator this progress describes.
+ * @param numRowsTotal
+ * Number of state rows held by the operator after this trigger.
+ * @param numRowsUpdated
+ * Number of state rows updated during this trigger.
+ * @param allUpdatesTimeMs
+ * Time taken, in milliseconds, to apply all state updates in this trigger.
+ * @param numRowsRemoved
+ * Number of state rows removed during this trigger.
+ * @param allRemovalsTimeMs
+ * Time taken, in milliseconds, to remove all evicted state rows in this
trigger.
+ * @param commitTimeMs
+ * Time taken, in milliseconds, to commit the state changes of this trigger.
+ * @param memoryUsedBytes
+ * Memory used, in bytes, by the operator's state store.
+ * @param numRowsDroppedByWatermark
+ * Number of input rows dropped because their event time was older than the
watermark.
+ * @param numShufflePartitions
+ * Number of shuffle partitions the operator ran with.
+ * @param numStateStoreInstances
+ * Number of state store instances backing the operator.
+ * @param customMetrics
+ * Custom metrics specific to the stateful operator or state store
implementation, keyed by
+ * metric name.
*/
@Evolving
class StateOperatorProgress private[spark] (
@@ -172,6 +198,8 @@ class StateOperatorProgress private[spark] (
* Information about operators in the query that store state.
* @param sources
* detailed statistics on data being read from each of the streaming sources.
+ * @param sink
+ * detailed statistics on data being written to the sink.
* @since 2.1.0
*/
@Evolving
@@ -354,6 +382,8 @@ class SourceProgress protected[spark] (
* @param numOutputRows
* Number of rows written to the sink or -1 for Continuous Mode
(temporarily) or Sink V1 (until
* decommissioned).
+ * @param metrics
+ * Sink-specific metrics reported for this trigger, keyed by metric name.
* @since 2.1.0
*/
@Evolving
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
