HeartSaVioR commented on issue #26821: [SPARK-20656][CORE]Support Incremental parsing of event logs in SHS URL: https://github.com/apache/spark/pull/26821#issuecomment-563780160 > Instead, in this PR, I am not closing the store, whenever there are changes in event logs or invalidate cache. This patch is simpler because this doesn't take "restarting SHS" into account. Restarting SHS will lose the information. And for now we may not want to tracking line offset in SHS's KV store (`listing`) since the line offset is only effective during single run of SHS. When you consider restoring KV store & state of listeners during restarting of SHS, you will have to store the snapshot of KV store into somewhere (that's why SPARK-29111 came in) and then you have to concern about compatibility of snapshot (entities in KV store including live entities on listeners) across Spark versions. That's why I had to change the design and introduce SPARK-29779 instead of snapshotting. We've already gone through bunch of discussions because it is not that simple in reality as it seems; so please go through these PRs as well as design docs. I guess the patch can be reviewed right now if the community prefers to have a solution which works within single SHS run first (though this may conflict with compaction #26416 which needs some arrangement), but if the community wants to have a solution which covers more cases, SPARK-28870 seems to be the way to go. (It doesn't mean this patch will not be valid - this patch could cover SPARK-29261 with some modification.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
