HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-532872003 I guess we are talking based on different things, what I referred is SPARK-28594, whereas you seem to refer your suggestion. I agree your suggestion doesn't bring considerable consumption of memory (and maybe also CPU). I'm just trying not to let both approaches be diverged (approach A for single event log, approach B for rolling event log). My point for your suggestion was that when there's an update on object, even freezing the map, elements in map could be changed which would be out of sync with handled count of events for freeze map. It should be prevented. Btw, I still feel this represents the needs of design doc for SPARK-28867. We're talking based on WIP and adding ideas here which is on top of uncertain thing.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
