vanzin commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-525880969 Sorry guys but I don't have unlimited bandwidth, and I have to be picky about the stuff I spend my time on. Right now I don't have time to go into details about yet another approach to handle this, so you'll have to reach a consensus among yourselves. What I'll say is that snapshotting live entities is a bad idea, and that the snapshot and the source of the events should be in agreement about what data lives in the snapshot and how new events are applied to it afterwards (and I have no idea about whether that comment applies to this approach). AppStatusListener has code to "flush" live entities to the kvstore, so you should never need to snapshot any live data.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
