Weiyi Kong created SPARK-33249:
----------------------------------
Summary: Add status plugin for live application
Key: SPARK-33249
URL: https://issues.apache.org/jira/browse/SPARK-33249
Project: Spark
Issue Type: New Feature
Components: Spark Core, Web UI
Affects Versions: 3.0.1, 2.4.7
Reporter: Weiyi Kong
There are cases that developer may want to extend the current REST API of Web
UI. In most cases, adding external module is a better option than directly
editing the original Spark code.
For an external module, to extend the REST API of the Web UI, 2 things may need
to be done:
* Add extra API to provide extra status info. This can be simply done by
implementing another ApiRequestContext which will be automatically loaded.
Add extra listeners to generate the status info if it can not be calculated
from the original data. This brings the issue.
For history server, there is an interface called AppHistoryServerPlugin, which
is loaded based on SPI, providing a method to create listeners. In live
application, the only way is spark.extraListeners based on
Utils.loadExtensions. But this is not enough for the cases.
To let the API get the status info, the data need to be written to the
AppStatusStore, which is the only store that an API can get by accessing
"ui.store" or "ui.sc.statusStore". But listeners created by
Utils.loadExtensions only get a SparkConf in construction, and are unable to
write the AppStatusStore.
So I think we still need plugin like AppHistorySever for live UI. For concerns
like [#SPARK-22786], the plugin for live app can be separated from the history
server one, and also loaded using Utils.loadExtensions with an extra
configurations. So by default, nothing will be loaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
