[
https://issues.apache.org/jira/browse/KAFKA-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on KAFKA-4484 started by Damian Guy.
-----------------------------------------
> Set more conservative default values on RocksDB for memory usage
> ----------------------------------------------------------------
>
> Key: KAFKA-4484
> URL: https://issues.apache.org/jira/browse/KAFKA-4484
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Damian Guy
> Labels: performance
>
> Quoting from email thread:
> {code}
> The block cache size defaults to a whopping 100Mb per store, and that gets
> expensive
> fast. I reduced it to a few megabytes. My data size is so big that I doubt
> it is very effective
> anyway. Now it seems more stable.
> I'd say that a smaller default makes sense, especially because the failure
> case is
> so opaque (running all tests just fine but with a serious dataset it dies
> slowly)
> {code}
> {code}
> Before we have the a single-knob memory management feature, I'd like to
> propose reducing the Streams' default config values for RocksDB caching and
> memory block size. For example, I remember Henry has done some fine tuning on
> the RocksDB config for his use case:
> https://github.com/HenryCaiHaiying/kafka/commit/b297f7c585f5a883ee068277e5f0f1224c347bd4
> https://github.com/HenryCaiHaiying/kafka/commit/eed1726d16e528d813755a6e66b49d0bf14e8803
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
> We could check if some of those changes are appropriate in general and if yes
> change the default settings accordingly.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
