pan3793 commented on code in PR #5662:
URL: https://github.com/apache/kyuubi/pull/5662#discussion_r1400071344
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala:
##########
@@ -141,6 +156,42 @@ case class SparkSQLEngine(spark: SparkSession) extends
Serverable("SparkSQLEngin
}
}
+ private[kyuubi] def startMetricsReporter(): Unit = {
+ val interval = conf.get(ENGINE_REPORT_INTERVAL)
+ val engineSpace = conf.get(HA_NAMESPACE)
+ val statusTracker = spark.sparkContext.statusTracker
+ val metricsSpace = s"/metrics$engineSpace"
+ val report: Runnable = () => {
+ if (!shutdown.get) {
+ val openSessionCount =
backendService.sessionManager.getOpenSessionCount
+ val activeTask = statusTracker.getActiveStageIds()
+ .flatMap { stage =>
+ statusTracker.getStageInfo(stage).map(_.numActiveTasks)
+ }.sum
+ val engineMetrics = Map(
+ "openSessionCount" -> openSessionCount,
+ "activeTask" -> activeTask,
+ "poolId" -> engineSpace.split("-").last)
+ info(s"Spark engine has $openSessionCount open sessions and
$activeTask active tasks.")
+ DiscoveryClientProvider.withDiscoveryClient(conf) { client =>
+ if (client.pathNonExists(metricsSpace)) {
+ client.create(metricsSpace, "PERSISTENT")
+ }
+ client.setData(
+ s"/metrics$engineSpace",
Review Comment:
is it feasible to append the "load" value in engine's info zonde, and update
in place?
if not, and decide to use a new path, we should consider deleting it after
engine dead
@yaooqinn how do you think of this path to store engine metrics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
