*Hi everyone!*
*In our company we use Kafka in hundreds of clusters in PRO. When the log cleaner triggers it “could” create some resource issues (disk I/O).* *For example when we are running a deployment it is possible one or several of the brokers run a log cleaner. As we know there is a throttle config (50 MiB/s) which is cool. As we know in a deploy * *We have consumers that maybe are reading from disk, the streaming data to be sent to the restarted broker and if we are not lucky a heavy log-cleaner running.* *As far as I understand a log-cleaner that runs in 3 seconds or 30 is exactly the same, I mean as long as we don’t fall “behind” and start accumulating logs to clean we are fine.* *So I was thinking it would be nice to avoid the resource competition between user pain (produce/consume) and the log cleaner, it is evident producers and consumers should have higher priority to get the disk access.* *For that reason I want to propose 2 possible KIPs* 1. *Can we somehow give less priority on log-cleaner reads when the read gets to the I/O scheduler? (I have to investigate) if that is possible this would be a clean solution.* 1. *Monitor the “pending” logs that are candidates for compaction, with this we could set a very low throttle in log-cleaner (for example 10 MiB/s) and increase it when the metric starts reporting the log cleaner candidates are increasing. So we could adjust the best throttle with the minimal impact in the cluster leaving the disk access to the producers/consumers* *What do you think? If you think this could be a solution I could start working on one of them .* *Thanks a lot!*
