[
https://issues.apache.org/jira/browse/SPARK-59986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123582#comment-18123582
]
Alyssa Gorbaneva commented on SPARK-59986:
------------------------------------------
Could a Spark committer assign this issue to me? I am the reporter and author
of apache/spark#59247 and am working on the implementation.
> Expose trigger metadata in StreamingQueryProgress JSON
> ------------------------------------------------------
>
> Key: SPARK-59986
> URL: https://issues.apache.org/jira/browse/SPARK-59986
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Alyssa Gorbaneva
> Priority: Major
> Labels: pull-request-available
>
> h2. Motivation
> StreamingQueryProgress JSON reports query identifiers, batch timings, source,
> sink, state operator, and observed metrics, but it does not report the
> Trigger selected for the query. Consumers of progress events cannot determine
> whether a query uses ProcessingTime, Continuous, RealTime, AvailableNow, or
> OneTime from the progress payload.
> h2. Proposed change
> Add an optional top-level trigger object to StreamingQueryProgress JSON and
> preserve it when QueryProgressEvent JSON and status-store records are
> serialized and deserialized.
> The object contains the selected trigger type and any relevant configuration:
> {code:json}
> "trigger": {"type": "ProcessingTime", "intervalMs": 1000}
> "trigger": {"type": "RealTime", "batchDurationMs": 300000}
> "trigger": {"type": "AvailableNow"}
> {code}
> The type field is the discriminator for the remaining fields. The
> configuration names are intentionally type-specific because the values have
> different meanings:
> * ProcessingTime uses intervalMs to schedule batch starts. A batch's actual
> execution duration can be shorter or longer than this interval, so naming it
> batchDurationMs would be inaccurate.
> * Continuous uses intervalMs to schedule checkpoints and does not execute
> micro-batches.
> * RealTime uses batchDurationMs as each real-time scan's deadline.
> Consecutive real-time batches start immediately, so this duration also
> influences batch cadence.
> * AvailableNow and OneTime do not have a duration field.
> Consumers should read type first and then the configuration field defined for
> that trigger type.
> The field should:
> * support all built-in trigger types;
> * provide stable, queryable type and configuration fields;
> * remain backward-compatible with progress and event records that do not
> contain it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
