HeartSaVioR commented on a change in pull request #34942:
URL: https://github.com/apache/spark/pull/34942#discussion_r774225422
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -211,4 +211,11 @@ private[spark] object History {
.version("3.1.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("2g")
+
+ val HYBRID_STORE_DISK_BACKEND =
ConfigBuilder("spark.history.store.hybridStore.diskBackend")
+ .doc("Specifies a disk-based store used in hybrid store; 'leveldb' or
'rocksdb'.")
+ .version("3.3.0")
+ .stringConf
+ .checkValues(Set("leveldb", "rocksdb"))
Review comment:
> I'm going to propose the removal of LevelDB dependency completely. Do
you have any concern about the removal of LevelDB JNI, @HeartSaVioR ?
Two points I can think of:
1. How I interpret the benchmark in PR description of #34913? Since it takes
histogram and uses timer, I'd guess it is about latency, and then looks like
RocksDB is "slower" than LevelDB. It's good if I'm mistaken, but if I'm not
mistaken, pretty sure I don't support the plan to bring performance regression.
"It doesn't work with Apple Silicon" is not a sufficient fact to drop it out.
2. How we provide smooth migration from LevelDB to RocksDB? End users
already loaded their old applications via LevelDB. This applies to LevelDB
KVStore and current Hybrid KVStore backed by LevelDB KVStore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
