Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9297#issuecomment-151857883
Hi @carsonwang,
This is a really useful addition, but I'm afraid this approach is not
really scalable. For example, you have the same problem with the Streaming tab
in the UI, and trying to fix that with your approach would just make things
worse.
I actually think the underlying code needs some lower level changes before
this can be implemented more cleanly. For example, a few things that I thought
about in the past:
- `SparkListenerEvent` should not be a sealed trait; that way streaming
events and sql events can extend it without having to exist in core
- `SparkListener` should not be the root of the listener hierarchy;
`SparkFirehoseListener` should, and `SparkListener` should be a specialization
of that trait that provides more type-specific method.
- events should be written to the logs using a library like Jackson instead
of having to modify `JsonProtocol` for each new event. The current approach is
just not scalable. The main issue here is backwards compatibility, but I think
this is a much needed change in this part of the code.
With those in place, the only thing left is a protocol so that the history
server can feed events to backend-specific listeners; that could be done by
having the history server expose a new trait that is loaded at runtime using
something like `java.util.ServiceLoader`; then there could be an implementation
of that trait in sql (and another one in streaming) to process events read from
the logs and add stuff to the UI.
What do you think? I really would like to avoid moving more stuff to the
core; it feels like this would be a pretty good opportunity to work on some of
the outstanding issues with this whole part of the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
