Ngone51 commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-525367646 Hi @HeartSaVioR , thanks for sharing your work here. I've gone through your design doc of some main parts and it's quite inspiring, especially for the part of leveraging `KVStoreScalaSerializer` to dump objects. While in this PR, using JavaSerialization is quite simple and brute and has already revealed the disadvantages(StackOverError) while deserializing deep nested objects. And I can see snapshot part could be used in both issues. One thing I want to point out is that we still the piece of "recover live entities" in AppStatusListener to make incremental replay work. I've elaborated details in the doc, please check it. Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
