Hi, I've noticed that similar to these already mentioned settings also segment.index.bytes has a minimum value of 4. This conflicts with OffsetIndex, which throws `java.lang.IllegalArgumentException: Invalid max index size: 4` with such settings, because hard-coded entry size in OffsetIndex is 8. Setting segment.index.bytes to less than 8 leads to errors.
On Mon, Mar 11, 2024 at 7:33 PM Divij Vaidya <[email protected]> wrote: > Hey folks > > Before I file a KIP to change this in 4.0, I wanted to understand the > historical context for the value of the following setting. > > Currently, segment.ms minimum threshold is set to 1ms [1]. > > Segments are expensive. Every segment uses multiple file descriptors and > it's easy to run out of OS limits when creating a large number of segments. > Large number of segments also delays log loading on startup because of > expensive operations such as iterating through all directories & > conditionally loading all producer state. > > I am currently not aware of a reason as to why someone might want to work > with a segment.ms of less than ~10s (number chosen arbitrary that looks > sane) > > What was the historical context of setting the minimum threshold to 1ms for > this setting? > > [1] https://kafka.apache.org/documentation.html#topicconfigs_segment.ms > > -- > Divij Vaidya >
