viirya commented on a change in pull request #33683:
URL: https://github.com/apache/spark/pull/33683#discussion_r689118183
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1814,6 +1814,23 @@ Specifically for built-in HDFS state store provider,
users can check the state s
it is best if cache missing count is minimized that means Spark won't waste
too much time on loading checkpointed state.
User can increase Spark locality waiting configurations to avoid loading state
store providers in different executors across batches.
+### RocksDB state store implementation
Review comment:
Actually we don't mention state store clearly in this doc before. The
doc only mentions `spark.sql.streaming.stateStore.providerClass` could be used
to set up state store implementation.
How about introducing both state stores? E.g.,
``` markdown
### State Store
#### HDFS state store provider
.....
#### RocksDB state store provider
.....
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
