[ https://issues.apache.org/jira/browse/SPARK-38234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493542#comment-17493542 ]
Apache Spark commented on SPARK-38234: -------------------------------------- User 'yeskarthik' has created a pull request for this issue: https://github.com/apache/spark/pull/35547 > Provide monitoring REST API for Structured Streaming > ---------------------------------------------------- > > Key: SPARK-38234 > URL: https://issues.apache.org/jira/browse/SPARK-38234 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.3.0 > Reporter: Karthik Subramanian > Priority: Major > Attachments: StreamingAPI-SS1.jpg, StreamingAPI-SS2.jpg, > StreamingAPIsSampleIO.txt > > > In SPARK-31953 Structured Streaming is added to the history server and a > "Structured Streaming" tab appears in the history UI when a streaming query > is present. However, even though a store exists for it and the data is > presented in the UI, this data is not exposed as a REST API. This data can be > used for monitoring, detecting streaming and to build custom dashboards. This > monitoring API will be similar to the monitoring APIs that are present for > DStreams - refer SPARK-18470. > In this change, we plan to add two simple APIs that expose the data in the > store and can be used to monitor streaming queries. > h3. *Summary API* > To list the summary of all existing streaming queries. > GET {{/\{appId}/sql/streamingqueries}} > Response is list of {_}StreamingQueryData{_}. > h3. *Progress API* > To list the progress events of a specific streaming query by {_}runId{_}. > User can also specify how many of the most recent events needs to be > retrieved by using the _last_ query parameter. By default, we can return the > most recent progress event i.e. last is set to 1. > GET {{{}/\{appId}/sql/streamingqueries/\{runId}/progress?last={N{}}}} > Response is list of {_}StreamingQueryProgress{_}. > *Note:* We are not introducing new objects for the response since we are just > returning the data from the store without aggregation, these are existing > event structures. > Attached sample I/O and screenshots.{{{}{}}} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org