[
https://issues.apache.org/jira/browse/SPARK-54072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
B. Micheal Okutubo updated SPARK-54072:
---------------------------------------
Description:
This change is to make sure the RocksDB snapshot zip file doesn't contain empty
files (except RocksDB log file). Other files such as OPTION, manifest, metadata
are not expected to be empty in any situation.
Introduced a conf {{verifyNonEmptyFilesInZip}} to make sure we can turn this
off if needed.
was:Integrity verification to verify that there are no empty files in the
rocksdb snapshot file.
> [State store integrity] Make sure we don't upload empty files in RocksDB
> snapshot
> ---------------------------------------------------------------------------------
>
> Key: SPARK-54072
> URL: https://issues.apache.org/jira/browse/SPARK-54072
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.1.0
> Reporter: B. Micheal Okutubo
> Priority: Major
>
> This change is to make sure the RocksDB snapshot zip file doesn't contain
> empty files (except RocksDB log file). Other files such as OPTION, manifest,
> metadata are not expected to be empty in any situation.
> Introduced a conf {{verifyNonEmptyFilesInZip}} to make sure we can turn this
> off if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
