baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638515964
> Thanks for the contribution. The concept looks great. > > I've skimmed the implementation, and got the feeling that this is a bit complicated because of dual writes on the foreground & background. The complication is even exposed to the FsHistoryProvider. > > Given the HybridKVStore is only used for loading event log and no further exposed for modification (AFAIK - please correct me if I'm missing here), I'm seeing a chance to simplify the logic - dump in-memory KVStore back to LevelDB (in background) once the loading is done to the in-memory KVStore. (KVStore should be read-only and reject the writes.) We won't need to deal with dual writes on concurrent threads, only need to switch the KVStore correctly. > > This wouldn't bring latency on serving content - needs more seconds to write to LevelDB, in other words, needs more seconds to keep up the memory usage. (Not sure how long it would be.) > > What do you think? @HyukjinKwon KVStore is not read-only after the rebuildAppStore() is done. In core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#357, kvstore writes CachedQuantile. AFAIK, this is the only type of objects that can be written to kvstore after the loading is done. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
