HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-533835863 Sorry, but we need to remember that my overall design passed the review in both internal and community. I think I can go on and revisit this if you succeed to design something based on new idea and being reviewed. It's not even verified via POC, so I can't see any reason to adopt it right now. > If so, I'd say yes, we need KVStore level API to let user do snapshot, e.g. dump(), since InMemoryStore would have different behavior compares to LevelDB in dump(). The thing is, caller logic should be different based on the support of sync/async snapshot. We'll have dump()/dumpAsync() and supportAsyncDump() and let caller deal with this. If caller needs asynchronous dump to respect some SLA (like AppStatusListener in driver) caller should fail if KVStore implementation doesn't support async dump.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
