HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-532398507 > But, it actually creates snapshot in a live AppStatusListener basing on InMemoryStore, not from that single event log file in SHS. So you're planning to create snapshot from driver, not SHS. Got it. For snapshotting in driver, I also considered both sides (driver/SHS) in design phase but took SHS because listener is the thing which must not be blocked by long operation. Either you'll end up with cloning InMemoryStore and snapshotting with cloned one (requires more memory on driver, but unless deep copy is required, then maybe applicable), or block listener to process next event until snapshot has been taken. You can't take snapshot asynchronously with live AppStatusListener. I got such feedback from reviewing phase - around 3 weeks were needed to reach current design - that's why I suggested providing design doc. It doesn't only mean writing design to the doc. It does mean the design is reviewed earlier than PR by Spark community(, and ideally you're ensuring your design is already reviewed internally in your team, but that's optional). Going back to co-work, thanks for taking up the work 2). With 2), we will get over our concerns about mismatch between *StatusListener instances and underlying KVStore, and then we can just take snapshot from underlying KVStore without concerning on listeners (assuming we ensure KVStore doesn't change while snapshotting).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
