[
https://issues.apache.org/jira/browse/SPARK-38234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493542#comment-17493542
]
Apache Spark commented on SPARK-38234:
--------------------------------------
User 'yeskarthik' has created a pull request for this issue:
https://github.com/apache/spark/pull/35547
> Provide monitoring REST API for Structured Streaming
> ----------------------------------------------------
>
> Key: SPARK-38234
> URL: https://issues.apache.org/jira/browse/SPARK-38234
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.3.0
> Reporter: Karthik Subramanian
> Priority: Major
> Attachments: StreamingAPI-SS1.jpg, StreamingAPI-SS2.jpg,
> StreamingAPIsSampleIO.txt
>
>
> In SPARK-31953 Structured Streaming is added to the history server and a
> "Structured Streaming" tab appears in the history UI when a streaming query
> is present. However, even though a store exists for it and the data is
> presented in the UI, this data is not exposed as a REST API. This data can be
> used for monitoring, detecting streaming and to build custom dashboards. This
> monitoring API will be similar to the monitoring APIs that are present for
> DStreams - refer SPARK-18470.
> In this change, we plan to add two simple APIs that expose the data in the
> store and can be used to monitor streaming queries.
> h3. *Summary API*
> To list the summary of all existing streaming queries.
> GET {{/\{appId}/sql/streamingqueries}}
> Response is list of {_}StreamingQueryData{_}.
> h3. *Progress API*
> To list the progress events of a specific streaming query by {_}runId{_}.
> User can also specify how many of the most recent events needs to be
> retrieved by using the _last_ query parameter. By default, we can return the
> most recent progress event i.e. last is set to 1.
> GET {{{}/\{appId}/sql/streamingqueries/\{runId}/progress?last={N{}}}}
> Response is list of {_}StreamingQueryProgress{_}.
> *Note:* We are not introducing new objects for the response since we are just
> returning the data from the store without aggregation, these are existing
> event structures.
> Attached sample I/O and screenshots.{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
