Github user carsonwang commented on the pull request:
https://github.com/apache/spark/pull/9297#issuecomment-152536961
@vanzin Thanks a lot for the comment. This sounds great and is very
helpful. I agree it is not a good idea to move more stuff to the core. The
underlying code change you mentioned sounds a lot of works to do, especially
when we consider backwards compatibility. Can we apply part of them so that we
can support SQL UI on the history server in this PR and also avoid moving the
SQL classes to the core? I was thinking adding a `toJson` method to the
`SparkListenerEvent`. As you mentioned when it is not a sealed trait, we can
define sql events in the sql module. For existing events in the core, we still
use the `JsonProtocol` in the `toJson` method to convert the event to json. For
sql events, we can define `toJson` method in the sql module. This way we will
be able to write the sql events to the log. Then we can use something you
mentioned to feed events from the history server to the sql listener.
I know this might still not be very scalable. But we don't make things
worse once we are able to avoid moving sql classes to the core and also support
SQL UI on the history server. Do you think this is doable for this ticket?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
