Siying Dong created SPARK-45503:
-----------------------------------
Summary: RocksDB State Store to Use LZ4 Compression
Key: SPARK-45503
URL: https://issues.apache.org/jira/browse/SPARK-45503
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 3.5.1
Reporter: Siying Dong
LZ4 is generally faster than Snappy. That's probably we use LZ4 in changelogs
and other places by default. However, we don't change RocksDB's default of
Snappy compression style. The RocksDB Team recommend LZ4 or ZSTD and the
default is kept to Snappy only for backward compatible reason. We should use
LZ4 instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
