HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-525153007 Hi @Ngone51 , I came across this PR as I've been working on similar things, [SPARK-28594](https://issues.apache.org/jira/browse/SPARK-28594). While the goals of both issues seem orthogonal, the basic approach looks similar, snapshotting ("checkpoint" in here). I've experimented various approaches of snapshotting in POC (https://github.com/HeartSaVioR/spark/commit/d2f4c8970eb87218dfc9f426ec8e611248ef8f33) and came up with the snapshot approach which is cross-compatible with any KVStore implementations. Could you please go through the [design doc](https://docs.google.com/document/d/12bdCC4nA58uveRxpeo8k7kGOI2NRTXmXyBOweSi4YcY/edit?usp=sharing), especially how to snapshot the KVStore instance and the format of the snapshot file? The design doc is in reviewing, but given the piece is common thing across both goals, I'll work on snapshot part first if you think it looks good. It would be also great if you could provide design doc and the plan for this issue so that we can evaluate this as well. Thanks in advance!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
