[
https://issues.apache.org/jira/browse/SPARK-33249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiyi Kong updated SPARK-33249:
-------------------------------
Description:
There are cases that developer may want to extend the current REST API of Web
UI. In most cases, adding external module is a better option than directly
editing the original Spark code.
For an external module, to extend the REST API of the Web UI, 2 things may need
to be done:
* Add extra API to provide extra status info. This can be simply done by
implementing another ApiRequestContext which will be automatically loaded.
* If the info can not be calculated from the original data in the store, add
extra listeners to generate them.
For history server, there is an interface called AppHistoryServerPlugin, which
is loaded based on SPI, providing a method to create listeners. In live
application, the only way is spark.extraListeners based on
Utils.loadExtensions. But this is not enough for the cases.
To let the API get the status info, the data need to be written to the
AppStatusStore, which is the only store that an API can get by accessing
"ui.store" or "ui.sc.statusStore". But listeners created by
Utils.loadExtensions only get a SparkConf in construction, and are unable to
write the AppStatusStore.
So I think we still need plugin like AppHistorySever for live UI. For concerns
like SPARK-22786, the plugin for live app can be separated from the history
server one, and also loaded using Utils.loadExtensions with an extra
configurations. So by default, nothing will be loaded.
was:
There are cases that developer may want to extend the current REST API of Web
UI. In most cases, adding external module is a better option than directly
editing the original Spark code.
For an external module, to extend the REST API of the Web UI, 2 things may need
to be done:
* Add extra API to provide extra status info. This can be simply done by
implementing another ApiRequestContext which will be automatically loaded.
* If the info can not be calculated from the original data in the store, add
extra listeners to generate them.
For history server, there is an interface called AppHistoryServerPlugin, which
is loaded based on SPI, providing a method to create listeners. In live
application, the only way is spark.extraListeners based on
Utils.loadExtensions. But this is not enough for the cases.
To let the API get the status info, the data need to be written to the
AppStatusStore, which is the only store that an API can get by accessing
"ui.store" or "ui.sc.statusStore". But listeners created by
Utils.loadExtensions only get a SparkConf in construction, and are unable to
write the AppStatusStore.
So I think we still need plugin like AppHistorySever for live UI. For concerns
like [#SPARK-22786], the plugin for live app can be separated from the history
server one, and also loaded using Utils.loadExtensions with an extra
configurations. So by default, nothing will be loaded.
> Add status plugin for live application
> --------------------------------------
>
> Key: SPARK-33249
> URL: https://issues.apache.org/jira/browse/SPARK-33249
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core, Web UI
> Affects Versions: 2.4.7, 3.0.1
> Reporter: Weiyi Kong
> Priority: Minor
>
> There are cases that developer may want to extend the current REST API of Web
> UI. In most cases, adding external module is a better option than directly
> editing the original Spark code.
> For an external module, to extend the REST API of the Web UI, 2 things may
> need to be done:
> * Add extra API to provide extra status info. This can be simply done by
> implementing another ApiRequestContext which will be automatically loaded.
> * If the info can not be calculated from the original data in the store, add
> extra listeners to generate them.
> For history server, there is an interface called AppHistoryServerPlugin,
> which is loaded based on SPI, providing a method to create listeners. In live
> application, the only way is spark.extraListeners based on
> Utils.loadExtensions. But this is not enough for the cases.
> To let the API get the status info, the data need to be written to the
> AppStatusStore, which is the only store that an API can get by accessing
> "ui.store" or "ui.sc.statusStore". But listeners created by
> Utils.loadExtensions only get a SparkConf in construction, and are unable to
> write the AppStatusStore.
> So I think we still need plugin like AppHistorySever for live UI. For
> concerns like SPARK-22786, the plugin for live app can be separated from the
> history server one, and also loaded using Utils.loadExtensions with an extra
> configurations. So by default, nothing will be loaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
