maropu commented on a change in pull request #28544:
URL: https://github.com/apache/spark/pull/28544#discussion_r426997661
##########
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
##########
@@ -131,60 +132,81 @@ private[thriftserver] class HiveThriftServer2Listener(
updateLiveStore(session)
}
- private def onSessionClosed(e: SparkListenerThriftServerSessionClosed): Unit
= {
- val session = sessionList.get(e.sessionId)
- session.finishTimestamp = e.finishTime
- updateStoreWithTriggerEnabled(session)
- sessionList.remove(e.sessionId)
- }
+ private def onSessionClosed(e: SparkListenerThriftServerSessionClosed): Unit
=
+ Option(sessionList.get(e.sessionId)) match {
+ case None => logWarning(s"onSessionClosed called with unknown session
id: ${e.sessionId}")
+ case Some(sessionData) =>
+ val session = sessionData
+ session.finishTimestamp = e.finishTime
+ updateStoreWithTriggerEnabled(session)
+ sessionList.remove(e.sessionId)
+ }
- private def onOperationStart(e: SparkListenerThriftServerOperationStart):
Unit = {
- val info = getOrCreateExecution(
- e.id,
- e.statement,
- e.sessionId,
- e.startTime,
- e.userName)
-
- info.state = ExecutionState.STARTED
- executionList.put(e.id, info)
- sessionList.get(e.sessionId).totalExecution += 1
- executionList.get(e.id).groupId = e.groupId
- updateLiveStore(executionList.get(e.id))
- updateLiveStore(sessionList.get(e.sessionId))
- }
+ private def onOperationStart(e: SparkListenerThriftServerOperationStart):
Unit =
+ Option(sessionList.get(e.sessionId)) match {
+ case None => logWarning(s"onOperationStart called with unknown session
id: ${e.sessionId}")
Review comment:
> @maropu this code is in a listener that executes on the listener bus.
The listener bus can drop events if it becomes to busy.
So it might happen that events are dropped, and id is unavailable. It will
not affect query execution.
Does a server get back to a normal state by just ignoring the case? What I a
bit worry about is that users can get the same warning message repeatedly if a
session id gets lost (I'm not sure the case can happen in event sequences).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
