juliuszsompolski commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r343170296
##########
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
##########
@@ -62,16 +59,25 @@ object HiveThriftServer2 extends Logging {
server.init(executionHive.conf)
server.start()
- listener = new HiveThriftServer2Listener(server, sqlContext.conf)
- sqlContext.sparkContext.addSparkListener(listener)
- uiTab = if (sqlContext.sparkContext.getConf.get(UI_ENABLED)) {
- Some(new ThriftServerTab(sqlContext.sparkContext))
+ val sc = sqlContext.sparkContext
+ val kvStore = sc.statusStore.store.asInstanceOf[ElementTrackingStore]
+ listener = new HiveThriftServer2Listener(kvStore, Some(server),
Some(sqlContext), Some(sc))
+ sc.addSparkListener(listener)
Review comment:
It would help for dropping events, if instead we did
```
val liveListenerBus = sc.liveListener
liveListenerBus.addToStatusQueue(listener)
```
These separate queues were implemented in
https://github.com/apache/spark/pull/19211 to minimize that risk of dropping
events.
Then `spark.scheduler.listenerbus.eventqueue.${name}.capacity` can be set to
set the statusQueue capacity to something very large (default is 10000), to
make sure events won't get dropped.
I think we could fix it more systematically in a followup, e.g. add a
`spark.scheduler.listenerbus.eventqueue.${name}.blocking` setting for critical
events queue that we don't want events to ever be dropped from.
cc @vanzin who implemented #19211. WDYT?
We could also unprivate `addToQueue` in `LiveListenerBus`, to be able to
define more queues (and thus more thread processing the events) as we want, and
have the Thrift listener have it's own queue (or maybe also configurable what
queue it should go to).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
