[
https://issues.apache.org/jira/browse/SPARK-38234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Subramanian updated SPARK-38234:
----------------------------------------
Attachment: StreamingAPI-SS1.jpg
StreamingAPI-SS2.jpg
> Provide monitoring REST API for Structured Streaming
> ----------------------------------------------------
>
> Key: SPARK-38234
> URL: https://issues.apache.org/jira/browse/SPARK-38234
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.3.0
> Reporter: Karthik Subramanian
> Priority: Major
> Attachments: StreamingAPI-SS1.jpg, StreamingAPI-SS2.jpg,
> StreamingAPIsSampleIO.txt
>
>
> In [SPARK-31953|https://issues.apache.org/jira/browse/SPARK-31953] Structured
> Streaming is added to the history server and a "Structured Streaming" tab
> appears in the history UI when a streaming query is present. However, even
> though a store exists for it and the data is presented in the UI, this data
> is not exposed as a REST API. This data can be used for monitoring, detecting
> streaming and to build custom dashboards. This monitoring API will be similar
> to the monitoring APIs that are present for DStreams - refer
> [SPARK-18470|https://issues.apache.org/jira/browse/SPARK-18470].
> In this change, we plan to add two simple APIs that expose the data in the
> store and can be used to monitor streaming queries.
> h3. *Summary API*
> To list the summary of all existing streaming queries.
> GET {{/\{appId}/sql/streamingqueries}}
> Response is list of {_}StreamingQueryData{_}.
> h3. *Progress API*
> To list the progress events of a specific streaming query by {_}runId{_}.
> User can also specify how many of the most recent events needs to be
> retrieved by using the _last_ query parameter. By default, we can return the
> most recent progress event i.e. last is set to 1.
> GET {{{}/\{appId}/sql/streamingqueries/\{runId}/progress?last={N{}}}}
> Response is list of {_}StreamingQueryProgress{_}.
> *Note:* We are not introducing new objects for the response since we are just
> returning the data from the store without aggregation, these are existing
> event structures.
> Will attach sample I/O.
> {{{{}}{}}}{{{{}}{}}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
