Neil Ramaswamy created SPARK-48707:
--------------------------------------
Summary: RocksDB setLevelCompactionDynamicLevelBytes should be set
to false
Key: SPARK-48707
URL: https://issues.apache.org/jira/browse/SPARK-48707
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Neil Ramaswamy
In RocksDB 8.4.0, they changed the default of
`level_compaction_dynamic_level_bytes` to true. This means that any query that
runs with this enabled will have an L6 in the LSM tree, so if a user rolls back
to a previous version of Spark that doesn't have RocksDB 8.4.0, then their L6
of the LSM tree will not be compacted. We should set this conf to be false so
that this rolling back works in Spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
