[ 
https://issues.apache.org/jira/browse/KAFKA-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hudeqi updated KAFKA-15086:
---------------------------
    Attachment: WechatIMG364.jpeg

> The unreasonable segment size setting of the internal topics in MM2 may cause 
> the worker startup time to be too long
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-15086
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15086
>             Project: Kafka
>          Issue Type: Improvement
>          Components: mirrormaker
>    Affects Versions: 3.4.1
>            Reporter: hudeqi
>            Assignee: hudeqi
>            Priority: Major
>         Attachments: WechatIMG364.jpeg, WechatIMG365.jpeg
>
>
> As the config 'segment.bytes' for topics related MM2(such as 
> offset.storage.topic, config.storage.topic,status.storage.topic), if 
> following the default configuration of the broker or set it larger, then when 
> the MM cluster runs many and complicated tasks, especially the log volume of 
> the topic 'offset.storage.topic' is very large, it will affect the restart 
> speed of the MM workers.
> After investigation, the reason is that a consumer needs to be started to 
> read the data of ‘offset.storage.topic’ at startup. Although this topic is 
> set to compact, if the 'segment size' is set to a large value, such as the 
> default value of 1G, then this topic may have tens of gigabytes of data that 
> cannot be compacted and has to be read from the earliest (because the active 
> segment cannot be cleaned), which will consume a lot of time (in our online 
> environment, we found that this topic stores 13G of data, it took nearly half 
> an hour for all the data to be consumed), which caused the worker to be 
> unable to start and execute tasks for a long time.
> Of course, the number of consumer threads can also be adjusted, but I think 
> it may be easier to reduce the 'segment size', for example, refer to the 
> default value of __consumer_offsets: 100MB



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to