yaooqinn commented on code in PR #3178:
URL: https://github.com/apache/incubator-kyuubi/pull/3178#discussion_r937283867
##########
docs/extensions/server/applications.rst:
##########
@@ -20,5 +20,137 @@ Manage Applications against Extra Cluster Managers
.. caution:: unstable
-.. warning::
- This page is still in-progress.
+Inside Kyuubi, the Kyuubi server uses the ``ApplicationManager`` module to
manage all applications launched by itself, including different kinds of Kyuubi
engines and self-contained applications.
+
+The ``ApplicationManager`` leverages methods provided by application operation
implementations derived from ``org.apache.kyuubi.engine.ApplicationOperation``
to monitor the status of those applications and kill abnormal applications in
case they get orphaned and may introduce more methods in the future.
+
+An ``ApplicationOperation`` implementation is usually built upon clients or
APIs provided by cluster managers, such as Hadoop YARN, Kubernetes, etc.
+
+For now, Kyuubi has already supported serval built-in application operations:
+
+- ``JpsApplicationOperation``: an operation that can manage apps with a local
process, e.g. a local mode spark application
+- ``YarnApplicationOperation``: an operation that can manage apps with a
Hadoop Yarn cluster, e.g. a spark on yarn application
+- ``KubernetesApplicationOperation``: an operation that can manage apps with a
k8s cluster, e.g. a spark on k8s application
+
+Besides those built-in ones, Kyuubi also supports loading custom
``ApplicationOperation``s through the Java `ServiceLoader`(SPI) for extra
cluster managers.
+
+The rest of this article will show you the specifications and steps to build
and enable a custom operation.
+
+ .. code-block:: scala
+
+ trait ApplicationOperation {
+
+ /**
+ * Step for initializing the instance.
+ */
+ def initialize(conf: KyuubiConf): Unit
+
+ /**
+ * Step to clean up the instance
+ */
+ def stop(): Unit
+
+ /**
+ * Called before other method to do a quick skip
+ *
+ * @param clusterManager the underlying cluster manager or just local
instance
+ */
+ def isSupported(clusterManager: Option[String]): Boolean
+
+ /**
+ * Kill the app/engine by the unique application tag
+ *
+ * @param tag the unique application tag for engine instance.
+ * For example,
+ * if the Hadoop Yarn is used, for spark applications,
+ * the tag will be preset via spark.yarn.tags
+ * @return a message contains response describing how the kill process.
+ *
+ * @note For implementations, please suppress exceptions and always
return KillResponse
+ */
+ def killApplicationByTag(tag: String): KillResponse
+
+ /**
+ * Get the engine/application status by the unique application tag
+ *
+ * @param tag the unique application tag for engine instance.
+ * @return [[ApplicationInfo]]
+ */
+ def getApplicationInfoByTag(tag: String): ApplicationInfo
+ }
+
+ .. code-block:: scala
+
+ /**
+ * (killed or not, hint message)
+ */
+ type KillResponse = (Boolean, String)
+
+An ``ApplicationInfo`` is used to represented the application information,
including application id, name, state, url address and error message.
+
+ .. code-block:: scala
+
+ object ApplicationState extends Enumeration {
+ type ApplicationState = Value
+ val PENDING, RUNNING, FINISHED, KILLED, FAILED, ZOMBIE, NOT_FOUND,
UNKNOWN = Value
+ }
+
+ case class ApplicationInfo(
+ id: String,
+ name: String,
+ state: ApplicationState,
+ url: Option[String] = None,
+ error: Option[String] = None)
+
+For application state mapping, you can reference the implementation of yarn:
+
+ .. code-block:: scala
+
+ def toApplicationState(state: YarnApplicationState): ApplicationState =
state match {
+ case YarnApplicationState.NEW => ApplicationState.PENDING
+ case YarnApplicationState.NEW_SAVING => ApplicationState.PENDING
+ case YarnApplicationState.SUBMITTED => ApplicationState.PENDING
+ case YarnApplicationState.ACCEPTED => ApplicationState.PENDING
+ case YarnApplicationState.RUNNING => ApplicationState.RUNNING
+ case YarnApplicationState.FINISHED => ApplicationState.FINISHED
+ case YarnApplicationState.FAILED => ApplicationState.FAILED
+ case YarnApplicationState.KILLED => ApplicationState.KILLED
+ case _ =>
+ warn(s"The yarn driver state: $state is not supported, " +
+ "mark the application state as UNKNOWN.")
+ ApplicationState.UNKNOWN
+ }
+
+Build A Custom Application Operation
+------------------------------------
+
+- reference kyuubi-server
+
+ .. code-block:: xml
+
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-server_2.12</artifactId>
+ <version>1.5.2-incubating</version>
+ <scope>provided</scope>
+ </dependency>
+
+- create a custom class which implements the
``org.apache.kyuubi.engine.ApplicationOperation``.
+
+- create a directory META-INF.services and a file with
``org.apache.kyuubi.engine.ApplicationOperation``:
+
+ .. code-block:: java
+
+ META-INF.services/org.apache.kyuubi.engine.ApplicationOperation
+
+ then add your fully-qualified name of custom application operation into the
file.
+
+
+Enable Custom Application Operation
+-----------------------------------
+
+.. note:: Kyuubi uses Java SPI to load the custom Application Operation
+
+- compile and put the jar into ``$KYUUBI_HOME/jars``
+
+For now, Kyuubi has already supported three built-in application operations:
``JpsApplicationOperation``, ``YarnApplicationOperation`` and
``KubernetesApplicationOperation``.
Review Comment:
can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
