squito commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-543284232 >> If you really wanted to do that, I'd have the driver just write out the leveldb kvstore when the application terminates. > Actually, this is the original plan when we try to do this. But as @gengliangwang points out that application may crash unexpectedly and LevelDB could be corrupt. yes, that is true. However, now we're considering quite a bit of additional complexity, to help only when the SHS isn't running (nor some separate "pre-processor" independent of the SHS which produces the leveldb store), and your application crashes. And even then, without this change, things would still work correctly, just not as efficiently. I think with all that in mind, I'm leaning against a change like this, it doesn't seem worth the complexity.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
