[ https://issues.apache.org/jira/browse/SPARK-38234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-38234: --------------------------------- Affects Version/s: 3.3.0 (was: 3.2.0) (was: 3.1.2) > Provide monitoring REST API for Structured Streaming > ---------------------------------------------------- > > Key: SPARK-38234 > URL: https://issues.apache.org/jira/browse/SPARK-38234 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.3.0 > Reporter: Karthik Subramanian > Priority: Major > > In [SPARK-31953] Add Spark Structured Streaming History Server Support - ASF > JIRA (apache.org) Structured Streaming is added to the history server and a > "Structured Streaming" tab appears in the history UI when a streaming query > is present. However, even though a store exists for it and the data is > presented in the UI, this data is not exposed as a REST API. This data can be > used for monitoring, detecting streaming and to build custom dashboards. This > monitoring API will be similar to the monitoring APIs that are present for > DStreams - refer [SPARK-18470] Provide Spark Streaming Monitor Rest Api - ASF > JIRA (apache.org). > In this change, we plan to add two simple APIs that expose the data in the > store and can be used to monitor streaming queries. > h3. *Summary API* > To list the summary of all existing streaming queries. > GET {{/\{appId}/sql/streamingqueries}} > Response is list of {_}StreamingQueryData{_}. > h3. *Progress API* > To list the progress events of a specific streaming query by {_}runId{_}. > User can also specify how many of the most recent events needs to be > retrieved by using the _last_ query parameter. By default, we can return the > most recent progress event i.e. last is set to 1. > GET {{{}/\{appId}/sql/streamingqueries/\{runId}/progress?last={N{}}}} > Response is list of {_}StreamingQueryProgress{_}. > *Note:* We are not introducing new objects for the response since we are just > returning the data from the store without aggregation, these are existing > event structures. > Will attach sample I/O. > {{{{}}{}}}{{{{}}{}}} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org