[
https://issues.apache.org/jira/browse/SPARK-33249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiyi Kong updated SPARK-33249:
-------------------------------
Remaining Estimate: (was: 24h)
Original Estimate: (was: 24h)
> Add status plugin for live application
> --------------------------------------
>
> Key: SPARK-33249
> URL: https://issues.apache.org/jira/browse/SPARK-33249
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core, Web UI
> Affects Versions: 2.4.7, 3.0.1
> Reporter: Weiyi Kong
> Priority: Minor
>
> There are cases that developer may want to extend the current REST API of Web
> UI. In most cases, adding external module is a better option than directly
> editing the original Spark code.
> For an external module, to extend the REST API of the Web UI, 2 things may
> need to be done:
> * Add extra API to provide extra status info. This can be simply done by
> implementing another ApiRequestContext which will be automatically loaded.
> * If the info can not be calculated from the original data in the store, add
> extra listeners to generate them.
> For history server, there is an interface called AppHistoryServerPlugin,
> which is loaded based on SPI, providing a method to create listeners. In live
> application, the only way is spark.extraListeners based on
> Utils.loadExtensions. But this is not enough for the cases.
> To let the API get the status info, the data need to be written to the
> AppStatusStore, which is the only store that an API can get by accessing
> "ui.store" or "ui.sc.statusStore". But listeners created by
> Utils.loadExtensions only get a SparkConf in construction, and are unable to
> write the AppStatusStore.
> So I think we still need plugin like AppHistorySever for live UI. For
> concerns like SPARK-22786, the plugin for live app can be separated from the
> history server one, and also loaded using Utils.loadExtensions with an extra
> configurations. So by default, nothing will be loaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
