cloud-fan commented on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-810849844
@viirya I think this is hard to fix. The history server is a different JVM and may not have the data source implementation classes loaded, we can't run the aggregate code in the history server. One idea is to provide some builtin implementations of `CustomMetric`, so that it's in the Spark codebase and will be available in the history server JVM as well. In the event logs we can record the class name, and the history server can instantiate the class and call the aggregate method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
