FrankChen021 edited a comment on issue #10566: URL: https://github.com/apache/druid/issues/10566#issuecomment-725189863
We don't manually delete kafka records by offsets as you did but I think I can share you something we do to avoid such a problem. 1st, it's better to set shorter log retention period for such a large topic to avoid disk shortage. For example, the `log.retention.hours` in our kafka clusters are usually set to 12 hours for high traffic topics however the default configuration is 168 hours(7 days). 2nd, it's also better to enable `LZ4` compression and a reasonable `linger.ms` at producer side which would both greatly benefit the network traffic, and disk storage at kafka broker side. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
