[
https://issues.apache.org/jira/browse/SPARK-59986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alyssa Gorbaneva updated SPARK-59986:
-------------------------------------
Description:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger object to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The object contains the selected trigger type and any relevant configuration:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "RealTime", "batchDurationMs": 300000}
"trigger": {"type": "AvailableNow"}
{code}
The type field is the discriminator for the remaining fields. The configuration
names are intentionally type-specific because the values have different
meanings:
* ProcessingTime uses intervalMs to schedule batch starts. A batch's actual
execution duration can be shorter or longer than this interval, so naming it
batchDurationMs would be inaccurate.
* Continuous uses intervalMs to schedule checkpoints and does not execute
micro-batches.
* RealTime uses batchDurationMs as each real-time scan's deadline. Consecutive
real-time batches start immediately, so this duration also influences batch
cadence.
* AvailableNow and OneTime do not have a duration field.
Consumers should read type first and then the configuration field defined for
that trigger type.
The field should:
* support all built-in trigger types;
* provide stable, queryable type and configuration fields;
* remain backward-compatible with progress and event records that do not
contain it.
was:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger object to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The object contains the selected trigger type and any relevant configuration:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "RealTime", "batchDurationMs": 300000}
"trigger": {"type": "AvailableNow"}
{code}
The type field is the discriminator for the remaining fields. The configuration
names match the existing Trigger APIs and execution semantics. ProcessingTime
uses intervalMs to schedule batch starts, and Continuous uses intervalMs to
schedule checkpoints. RealTime uses batchDurationMs as each real-time scan's
deadline; because consecutive real-time batches start immediately, this
duration also influences batch cadence. AvailableNow and OneTime do not have a
duration field. Consumers should read type first and then the configuration
field defined for that trigger type.
The field should:
* support all built-in trigger types;
* provide stable, queryable type and configuration fields;
* remain backward-compatible with progress and event records that do not
contain it.
> Expose trigger metadata in StreamingQueryProgress JSON
> ------------------------------------------------------
>
> Key: SPARK-59986
> URL: https://issues.apache.org/jira/browse/SPARK-59986
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Alyssa Gorbaneva
> Priority: Major
>
> h2. Motivation
> StreamingQueryProgress JSON reports query identifiers, batch timings, source,
> sink, state operator, and observed metrics, but it does not report the
> Trigger selected for the query. Consumers of progress events cannot determine
> whether a query uses ProcessingTime, Continuous, RealTime, AvailableNow, or
> OneTime from the progress payload.
> h2. Proposed change
> Add an optional top-level trigger object to StreamingQueryProgress JSON and
> preserve it when QueryProgressEvent JSON and status-store records are
> serialized and deserialized.
> The object contains the selected trigger type and any relevant configuration:
> {code:json}
> "trigger": {"type": "ProcessingTime", "intervalMs": 1000}
> "trigger": {"type": "RealTime", "batchDurationMs": 300000}
> "trigger": {"type": "AvailableNow"}
> {code}
> The type field is the discriminator for the remaining fields. The
> configuration names are intentionally type-specific because the values have
> different meanings:
> * ProcessingTime uses intervalMs to schedule batch starts. A batch's actual
> execution duration can be shorter or longer than this interval, so naming it
> batchDurationMs would be inaccurate.
> * Continuous uses intervalMs to schedule checkpoints and does not execute
> micro-batches.
> * RealTime uses batchDurationMs as each real-time scan's deadline.
> Consecutive real-time batches start immediately, so this duration also
> influences batch cadence.
> * AvailableNow and OneTime do not have a duration field.
> Consumers should read type first and then the configuration field defined for
> that trigger type.
> The field should:
> * support all built-in trigger types;
> * provide stable, queryable type and configuration fields;
> * remain backward-compatible with progress and event records that do not
> contain it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
