yaooqinn commented on pull request #1023:
URL: https://github.com/apache/incubator-kyuubi/pull/1023#issuecomment-913501350
> hi @yaooqinn, I can not understand why kyuubi need SparkHistoryEventLogger?
SparkHistoryEventLogger is used to log all Kyuubi's events to the same file
of spark's event log. We also support other destinations, like Kyuubi's own
JSON log store, JDBC(planned), etc..
> it seems duplicated to spark.eventLog.enabled=true.
It's not. it follows spark.eventLog.enabled when logging. We just don't push
the events to the listener bus but call the event log listener directly.
> And, the current design of logger service is not clear to me. As we know a
service means that it should listen to some port and can start / stop.
Hmm... do not go too far with the spark's listeners and the listener bus if
you are not understanding what you are doing here.
IIUC, you want everything to work with spark's listeners and the listener
bus, this is not what we want. We are a Spark application **BUT** not only a
spark application that is limited to spark's functionalities.
What we want is only a size-limit buffer (using `ElementTrackingStore` is
just an option) that holds all the current Kyuubi events in memory. Then render
one or more Spark UI pages/tables based on this buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
