[
https://issues.apache.org/jira/browse/KAFKA-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464791#comment-17464791
]
A. Sophie Blee-Goldman commented on KAFKA-12559:
------------------------------------------------
Nope, you can just write up the KIP and send it out to the mailing list for
discussion – everyone will help get it into good shape before you call for a
vote, no need to wait for me specifically to review it :)
What are your questions on that section? Note that because we are adding an
entirely new config here, there's not going to be much to say in that section –
the main thing to decide is whether to keep the old behavior (unbounded memory)
by default, or to start bounding the memory to some default value. The latter
is a bit questionable to do in a minor release, I would personally propose in
the KIP to continue not bounding the memory at all by default (eg with a
sentinel value like "-1" for the config to mean "don't bound", and then use -1
as the default) for the time being. Then in the next major version bump, ie
when 4.0 comes out, we could switch over to a new default value and start
bounding the memory.
Just my personal take – in the end, i recommend just proposing whatever makes
sense to you in the KIP. The discussion thread is the place to hash out all the
options and arguments to figure out the final proposal
> Add a top-level Streams config for bounding off-heap memory
> -----------------------------------------------------------
>
> Key: KAFKA-12559
> URL: https://issues.apache.org/jira/browse/KAFKA-12559
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Assignee: Aditya Upadhyaya
> Priority: Major
> Labels: needs-kip, newbie, newbie++
>
> At the moment we provide an example of how to bound the memory usage of
> rocskdb in the [Memory
> Management|https://kafka.apache.org/27/documentation/streams/developer-guide/memory-mgmt.html#rocksdb]
> section of the docs. This requires implementing a custom RocksDBConfigSetter
> class and setting a number of rocksdb options for relatively advanced
> concepts and configurations. It seems a fair number of users either fail to
> find this or consider it to be for more advanced use cases/users. But RocksDB
> can eat up a lot of off-heap memory and it's not uncommon for users to come
> across a {{RocksDBException: Cannot allocate memory}}
> It would probably be a much better user experience if we implemented this
> memory bound out-of-the-box and just gave users a top-level StreamsConfig to
> tune the off-heap memory given to rocksdb, like we have for on-heap cache
> memory with cache.max.bytes.buffering. More advanced users can continue to
> fine-tune their memory bounding and apply other configs with a custom config
> setter, while new or more casual users can cap on the off-heap memory without
> getting their hands dirty with rocksdb.
> I would propose to add the following top-level config:
> rocksdb.max.bytes.off.heap: medium priority, default to -1 (unbounded), valid
> values are [0, inf]
> I'd also want to consider adding a second, lower priority top-level config to
> give users a knob for adjusting how much of that total off-heap memory goes
> to the block cache + index/filter blocks, and how much of it is afforded to
> the write buffers. I'm struggling to come up with a good name for this
> config, but it would be something like
> rocksdb.memtable.to.block.cache.off.heap.memory.ratio: low priority, default
> to 0.5, valid values are [0, 1]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
