Ngone51 commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-533806214 > That means, we should provide a way to roll the event log file within deterministic size, which is not applicable to roll file per lines. All right. We may could attach both size and lines to the log file(a.g. `[sequeneceId, size, evetsNum]`), then, it does no harm to SPARK-28594. > consolidate the approach later (assuming the snapshot file is compatible). I think SPARK-28594 and SPARK-28867 are both going to leverage #25811 to dump KVStore. So, the snapshot file should be compatible. > if we have both sync/async snapshot for KVStore, that should be reflected to the KVStore API. As for "sync/async snapshot", do you mean the blocking way and "two maps" way for snapshotting of `InMemoryStore` ? If so, I'd say yes, we need KVStore level API to let user do snapshot, e.g. dump(), since `InMemoryStore` would have different behavior compares to `LevelDB` in dump(). But, I think whatever which way we have for `InMemoryStore`, the output snapshot file should always have the same format. And, very possibly, leverage #25811, too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
