redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436146548
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED =
ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a
background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is
completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
Review comment:
Do we need to still use this with caution as we are managing memory now?
Do you still see any corner cases with it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
