[
https://issues.apache.org/jira/browse/SPARK-28120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133112#comment-17133112
]
Vikram Agrawal commented on SPARK-28120:
----------------------------------------
The implementation is available here
(https://github.com/qubole/spark-state-store). I have published it in mvn. It
can be downloaded from here
(https://mvnrepository.com/artifact/com.qubole.spark/spark-rocksdb-state-store)
> RocksDB state storage
> ---------------------
>
> Key: SPARK-28120
> URL: https://issues.apache.org/jira/browse/SPARK-28120
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Vikram Agrawal
> Priority: Major
>
> SPARK-13809 introduced a framework for state management for computing
> Streaming Aggregates. The default implementation was in-memory hashmap which
> was backed up in HDFS complaint file system at the end of every micro-batch.
> Current implementation suffers from Performance and Latency Issues. It uses
> Executor JVM memory to store the states. State store size is limited by the
> size of the executor memory. Also
> Executor JVM memory is shared by state storage and other tasks operations.
> State storage size will impact the performance of task execution
> Moreover, GC pauses, executor failures, OOM issues are common when the size
> of state storage increases which increases overall latency of a micro-batch
> RocksDb is an embedded DB which can provide major performance improvements.
> Other major streaming frameworks have rocksdb as default state storage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
