Ngone51 commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-532273297 > In SPARK-28867, it creates snapshot (checkpoint) for specific line because the event log file is single. Not exactly. SPARK-28867 really intends to creates snapshot for specific line in that single event log file. But, it actually creates snapshot in a *live* AppStatusListener basing on InMemoryStore, not from that single event log file in SHS. And I think, in SPARK-28594, snapshot is aways created from event log files in SHS, right ? In SPARK-28867, it tries to speed up replay for a huge log file at the first time. And it includes a case that SHS may not running before the application finished. That's why SPARK-28867 creates snapshot from InMemoryStore rather than log file. For work 2), I'll show you an initial version as soon as possible.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
