viirya commented on a change in pull request #33683: URL: https://github.com/apache/spark/pull/33683#discussion_r689246669
########## File path: docs/structured-streaming-programming-guide.md ########## @@ -1792,7 +1792,85 @@ hence the number is not same as the number of original input rows. You'd like to There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional. -### State Store and task locality +### State Store + +State store is a versioned key-value store which provides both read and write operations. In +structured streaming, we use the state store provider to handle the stateful operations across +batches. There are two built-in state store provider implementations. End users can also implement +their own state store provider by extending StateStoreProvider interface. + +#### HDFS state store provider + +The HDFS backend state store provider is the default implementation of [[StateStoreProvider]] and +[[StateStore]] in which all the data is stated in memory map in the first stage, and then backed +by files in an HDFS-compatible file system. All updates to the store have to be done in sets +transactionally, and each set of updates increments the store's version. These versions can be +used to re-execute the updates (by retries in RDD operations) on the correct version of the store, +and regenerate the store version. + +#### RocksDB state store implementation + +As of Spark 3.2, we add a new built-in state store implementation, RocksDB state store provider. + +If you have stateful operations in your streaming query (for example, streaming aggregation, +streaming dropDuplicates, stream-stream joins, mapGroupsWithState, or flatMapGroupsWithState) +and you want to maintain millions of keys in the state, then you may face issues related to large +JVM garbage collection (GC) pauses causing high variations in the micro-batch processing times. +This occurs because, by the implementation of HDFSBackedStateStore, the state data is maintained +in the JVM memory of the executors and large number of state objects puts memory pressure on the +JVM causing high GC pauses. + +In such cases, you can choose to use a more optimized state management solution based on +[RocksDB](https://rocksdb.org/). Rather than keeping the state in the JVM memory, this solution +uses RocksDB to efficiently manage the state in the native memory and the local disk. Furthermore, +any changes to this state are automatically saved by Structured Streaming to the checkpoint +location you have provided, thus providing full fault-tolerance guarantees (the same as default +state management). + +To enable the new build-in state store implementation, set `spark.sql.streaming.stateStore.providerClass` +to `org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider`. + +Here are the configs regarding to RocksDB instance of the state store provider: + +<table class="table"> + <tr> + <th>Config Name</th> + <th>Description</th> + <th>Default Value</th> + </tr> + <tr> + <td>spark.sql.streaming.stateStore.rocksdb.compactOnCommit</td> + <td>Whether we perform a range compaction of RocksDB instance for commit operation</td> + <td>False</td> + </tr> + <tr> + <td>spark.sql.streaming.stateStore.rocksdb.blockSizeKB</td> + <td>Approximate size in KB of user data packed per block for a RocksDB BlockBasedTable, which is a RocksDB's default SST file format.</td> + <td>4</td> + </tr> + <tr> + <td>spark.sql.streaming.stateStore.rocksdb.blockCacheSizeMB</td> + <td>The size capacity in MB for a cache of blocks.</td> + <td>8</td> + </tr> + <tr> + <td>spark.sql.streaming.stateStore.rocksdb.blockCacheSizeMB</td> + <td>The size capacity in MB for a cache of blocks.</td> + <td>8</td> Review comment: Is `spark.sql.streaming.stateStore.rocksdb.blockCacheSizeMB` duplicate? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
