Ngone51 commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-532213518 Hey @HeartSaVioR , sorry for the late reply. I'd like to try recover live entities part to support our following work in recent days. I'd give an initial version as soon as possible. And sorry I can't provide a detail design doc to you. As I think the idea in this PR is quite simple and I've already elaborated it in PR desc. In my way, it does not have much careful design on snapshot, but simply using Java serialization. And I just realize a big difference between our two issues from high level, that is, in my issue, I aways need to accurately track the proceeded events number in AppStatusListener why you needn't. That's because, in my issue, in SHS, events comes from AppStatusListener(snapshot/checkpoint of InMemoryStore) and EventLoggingListener(event log file) while in your issue events only comes from EventLoggingListener(snapshot also generated from event log file). However, tracking the proceeded events number accurately in AppStatusListener is difficult, at least, under current framework. For example, `SparkListenerStageExecutorMetrics` only generated in EventLoggingListener and log to file, but never shows in a living AppStatusListener. And this would result in inconsistent status between AppStatusListener and EventLoggingListener. Anyway, I'd try to implement live entities recovering part firstly.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
