[
https://issues.apache.org/jira/browse/KAFKA-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-2623.
------------------------------
Resolution: Fixed
Time-based log retention is enforced in KIP-33. Pl reopen if you think the
issue still exists
> Kakfa broker not deleting logs after configured retention time properly
> -----------------------------------------------------------------------
>
> Key: KAFKA-2623
> URL: https://issues.apache.org/jira/browse/KAFKA-2623
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.8.2.0
> Environment: DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=12.04
> DISTRIB_CODENAME=precise
> DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
> NAME="Ubuntu"
> VERSION="12.04.5 LTS, Precise Pangolin"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
> VERSION_ID="12.04"
> Reporter: Hao Zhang
> Assignee: Jay Kreps
>
> Context:
> To get an accurate estimate on how much retention we have for each
> topic/partition, we have a cron job iterating each topic/partition folder on
> each node of a cluster, measuring the timestamp difference between the newest
> and oldest log files.
> Problem:
> We notice that it's very common that between leaders and followers, the time
> differences are vastly different. On the leader the timestamp differences are
> normally about a week (our retention policy), but on the follower the
> timestamp differences can sometimes range between just a few hours to 2-3
> days.
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log
> -rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log
> Observation:
> We noticed that there are a lot of index files with size 200. The modified
> timestamp of these index files are pretty much identical.
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index
> Theory:
> Our guess is that the broker is trying to delete these log and index files,
> thus the timestamps are all modified at pretty much the same time. However,
> for some reason the deletion doesn't succeed and thus leave all the log/index
> files with the same timestamp. But there is no log deletion error in the log
> whatsoever.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
