HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-534003329 > Oh, sorry. I was just discussing the possible integrate way in the following design and not ask you to make the change right now. Yeah. I feel I was over-reacted. Sorry about that. Just wanted to not going back and spend another time to wait for new thing. > Sorry, I don't know this part well. Why do we need to implement sync and async dump separately ? It's not always easy to implement async dump for all possible implementations of KVStore (though KVStore is private API so for now we only have InMemoryStore and LevelDB...). We only discussed an idea of async snapshot for InMemoryStore - LevelDB is not considered here. So for now even in happy case InMemoryStore will support asynchronous dump and LevelDB will not. Suppose a case, LevelDB is provided to the AppStatusListener as underlying KVStore. (That wouldn't be true for live application but let's see it as interface level of view - as it receives ElementTrackingStore.) If you call dump() it should run as synchronous, and listener would be stuck until dump() has been finished and open the chance to drop events due to heavy latency. That's what we want to avoid and the reason to discuss asynchronous approach, right? For live application of AppStatusListener, underlying KVStore must support asynchronous dump. Otherwise it should just fail or give up dumping KVStore which caller should indicate and make decision.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
