[
https://issues.apache.org/jira/browse/SPARK-59986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alyssa Gorbaneva updated SPARK-59986:
-------------------------------------
Description:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger object to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The object contains the selected trigger type and any relevant configuration:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "RealTime", "batchDurationMs": 300000}
"trigger": {"type": "AvailableNow"}
{code}
The field should:
* support all built-in trigger types;
* provide stable, queryable type and configuration fields;
* remain backward-compatible with progress and event records that do not
contain it.
was:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger field to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The field should:
* include the selected trigger type and relevant configuration;
* support all built-in trigger types;
* remain backward-compatible with progress and event records that do not
contain it.
Two representations are under consideration:
Structured:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "AvailableNow"}
{code}
Compact scalar:
{code:json}
"trigger": "ProcessingTime(1000)"
"trigger": "AvailableNow"
{code}
The final representation should provide a stable contract for programmatic
consumers while remaining consistent with existing StreamingQueryProgress JSON.
> Expose trigger metadata in StreamingQueryProgress JSON
> ------------------------------------------------------
>
> Key: SPARK-59986
> URL: https://issues.apache.org/jira/browse/SPARK-59986
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Alyssa Gorbaneva
> Priority: Major
>
> h2. Motivation
> StreamingQueryProgress JSON reports query identifiers, batch timings, source,
> sink, state operator, and observed metrics, but it does not report the
> Trigger selected for the query. Consumers of progress events cannot determine
> whether a query uses ProcessingTime, Continuous, RealTime, AvailableNow, or
> OneTime from the progress payload.
> h2. Proposed change
> Add an optional top-level trigger object to StreamingQueryProgress JSON and
> preserve it when QueryProgressEvent JSON and status-store records are
> serialized and deserialized.
> The object contains the selected trigger type and any relevant configuration:
> {code:json}
> "trigger": {"type": "ProcessingTime", "intervalMs": 1000}
> "trigger": {"type": "RealTime", "batchDurationMs": 300000}
> "trigger": {"type": "AvailableNow"}
> {code}
> The field should:
> * support all built-in trigger types;
> * provide stable, queryable type and configuration fields;
> * remain backward-compatible with progress and event records that do not
> contain it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
