[
https://issues.apache.org/jira/browse/KAFKA-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802030#comment-15802030
]
Michael Andre Pearce (IG) edited comment on KAFKA-4497 at 1/6/17 8:35 AM:
--------------------------------------------------------------------------
When we got the issue on 0.10.1.1 we were just as surprised and as such that
was our very first thing we checked.
our internal mail at the time when this occured in UAT (first env we saw it
post 0.10.1.1 upgrade):
---->
Date: Tuesday, 3 January 2017 at 17:07
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
Yeah I already double checked that, all the cp1 jars are gone and the
0.10.1.1’s exist, pasted below, hope ive missed something..?
Class path:
:/opt/projects/confluent/bin/../share/java/kafka/jetty-continuation-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/aopalliance-repackaged-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-file-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/argparse4j-0.5.0.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-json-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/avro-1.7.7.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-beanutils-1.8.3.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-codec-1.9.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-http-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-collections-3.2.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-compress-1.4.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-digester-1.8.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-lang3-3.1.jar:/opt/projects/confluent/bin/../share/java/kafka/log4j-1.2.17.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-logging-1.2.jar:/opt/projects/confluent/bin/../share/java/kafka/lz4-1.3.0.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-validator-1.4.1.jar:/opt/projects/confluent/bin/../share/java/kafka/metrics-core-2.2.0.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-io-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-util-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jopt-simple-4.9.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-runtime-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/guava-18.0.jar:/opt/projects/confluent/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-api-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/paranamer-2.3.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-locator-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/reflections-0.9.10.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-utils-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-clients-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpclient-4.5.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-streams-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpcore-4.4.3.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-tools-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpmime-4.5.1.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-annotations-2.6.0.jar:/opt/projects/confluent/bin/../share/java/kafka/rocksdbjni-4.9.0.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-core-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/scala-library-2.11.8.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-core-asl-1.9.13.jar:/opt/projects/confluent/bin/../share/java/kafka/slf4j-api-1.7.21.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-databind-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/slf4j-log4j12-1.7.21.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-jaxrs-base-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-mapper-asl-1.9.13.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-media-jaxb-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/snappy-java-1.1.2.6.jar:/opt/projects/confluent/bin/../share/java/kafka/javassist-3.18.2-GA.jar:/opt/projects/confluent/bin/../share/java/kafka/support-metrics-client-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.annotation-api-1.2.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-test.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.inject-1.jar:/opt/projects/confluent/bin/../share/java/kafka/support-metrics-common-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.inject-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/validation-api-1.1.0.Final.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/xz-1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.ws.rs-api-2.0.1.jar:/opt/projects/confluent/bin/../share/java/kafka/zkclient-0.9.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-client-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/zookeeper-3.4.8.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-common-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-container-servlet-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-server-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-container-servlet-core-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-api-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-guava-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-security-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-server-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-servlet-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-servlets-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/scala-parser-combinators_2.11-1.0.4.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-log4j-appender-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-streams-examples-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-javadoc.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-scaladoc.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-sources.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-test-sources.jar:/opt/projects/confluent/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/*:/opt/projects/jolokia/jolokia-jvm-agent.jar:/opt/projects/jHiccup/jHiccup-latest/jHiccup.jar
Date: Tuesday, 3 January 2017 at 17:05
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
Hi Andy,
Can you confirm the jar classpath for us. Just want to make sure all correct
versions, and at 0.10.1.1
Cheers
Mike
Date: Tuesday, 3 January 2017 at 16:43
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
ROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner)
kafka.common.InvalidOffsetException: Attempt to append an offset (-1) to slot
461 no larger than the last offset appended (880354) to
/var/kafka/logs/com_ig_trade_v1_position_event--live--compacted-2/00000000000000000000.timeindex.cleaned.
at
kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:117)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
at kafka.log.LogSegment.append(LogSegment.scala:106)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:518)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400)
at scala.collection.immutable.List.foreach(List.scala:381)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.fore
was (Author: michael.andre.pearce):
When we got the issue on 0.10.1.1 we were just as surprised and as such that
was our very first thing we checked.
our internal mail at the time when this occured in UAT (first env we saw it
post 0.10.1.1 upgrade):
---->
From: Andrew Holford <andrew.holf...@ig.com>
Date: Tuesday, 3 January 2017 at 17:07
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
Yeah I already double checked that, all the cp1 jars are gone and the
0.10.1.1’s exist, pasted below, hope ive missed something..?
Class path:
:/opt/projects/confluent/bin/../share/java/kafka/jetty-continuation-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/aopalliance-repackaged-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-file-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/argparse4j-0.5.0.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-json-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/avro-1.7.7.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-beanutils-1.8.3.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-codec-1.9.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-http-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-collections-3.2.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-compress-1.4.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-digester-1.8.1.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-lang3-3.1.jar:/opt/projects/confluent/bin/../share/java/kafka/log4j-1.2.17.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-logging-1.2.jar:/opt/projects/confluent/bin/../share/java/kafka/lz4-1.3.0.jar:/opt/projects/confluent/bin/../share/java/kafka/commons-validator-1.4.1.jar:/opt/projects/confluent/bin/../share/java/kafka/metrics-core-2.2.0.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-io-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-util-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jopt-simple-4.9.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-runtime-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/guava-18.0.jar:/opt/projects/confluent/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-api-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/paranamer-2.3.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-locator-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/reflections-0.9.10.jar:/opt/projects/confluent/bin/../share/java/kafka/hk2-utils-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-clients-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpclient-4.5.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-streams-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpcore-4.4.3.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-tools-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/httpmime-4.5.1.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-annotations-2.6.0.jar:/opt/projects/confluent/bin/../share/java/kafka/rocksdbjni-4.9.0.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-core-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/scala-library-2.11.8.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-core-asl-1.9.13.jar:/opt/projects/confluent/bin/../share/java/kafka/slf4j-api-1.7.21.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-databind-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/slf4j-log4j12-1.7.21.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-jaxrs-base-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-mapper-asl-1.9.13.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-media-jaxb-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.6.3.jar:/opt/projects/confluent/bin/../share/java/kafka/snappy-java-1.1.2.6.jar:/opt/projects/confluent/bin/../share/java/kafka/javassist-3.18.2-GA.jar:/opt/projects/confluent/bin/../share/java/kafka/support-metrics-client-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.annotation-api-1.2.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-test.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.inject-1.jar:/opt/projects/confluent/bin/../share/java/kafka/support-metrics-common-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.inject-2.4.0-b34.jar:/opt/projects/confluent/bin/../share/java/kafka/validation-api-1.1.0.Final.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/xz-1.0.jar:/opt/projects/confluent/bin/../share/java/kafka/javax.ws.rs-api-2.0.1.jar:/opt/projects/confluent/bin/../share/java/kafka/zkclient-0.9.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-client-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/zookeeper-3.4.8.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-common-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-container-servlet-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-server-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-container-servlet-core-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/connect-api-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/jersey-guava-2.22.2.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-security-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-server-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-servlet-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka.jar:/opt/projects/confluent/bin/../share/java/kafka/jetty-servlets-9.2.15.v20160210.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/scala-parser-combinators_2.11-1.0.4.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-log4j-appender-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka-streams-examples-0.10.1.1.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-javadoc.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-scaladoc.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-sources.jar:/opt/projects/confluent/bin/../share/java/kafka/kafka_2.11-0.10.1.1-test-sources.jar:/opt/projects/confluent/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/*:/opt/projects/jolokia/jolokia-jvm-agent.jar:/opt/projects/jHiccup/jHiccup-latest/jHiccup.jar
Date: Tuesday, 3 January 2017 at 17:05
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
Hi Andy,
Can you confirm the jar classpath for us. Just want to make sure all correct
versions, and at 0.10.1.1
Cheers
Mike
Date: Tuesday, 3 January 2017 at 16:43
Subject: Re: Kafka Issues fixed in 0.10.1.1 how we can patch until confluent
release.
ROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner)
kafka.common.InvalidOffsetException: Attempt to append an offset (-1) to slot
461 no larger than the last offset appended (880354) to
/var/kafka/logs/com_ig_trade_v1_position_event--live--compacted-2/00000000000000000000.timeindex.cleaned.
at
kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:117)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
at kafka.log.LogSegment.append(LogSegment.scala:106)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:518)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400)
at scala.collection.immutable.List.foreach(List.scala:381)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.fore
> log cleaner breaks on timeindex
> -------------------------------
>
> Key: KAFKA-4497
> URL: https://issues.apache.org/jira/browse/KAFKA-4497
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.10.1.0
> Environment: Debian Jessie, Oracle Java 8u92, kafka_2.11-0.10.1.0
> Reporter: Robert Schumann
> Assignee: Jiangjie Qin
> Priority: Critical
> Labels: reliability
> Fix For: 0.10.1.1
>
> Attachments: com_ig_trade_v1_order_event--demo--compacted-7.tar.bz2,
> com_ig_trade_v1_position_event--demo--compacted-14.tar.gz,
> log-cleaner.log.1.zip, vrtstokf005_4thJan
>
>
> _created from ML entry by request of [~ijuma]_
> Hi all,
> we are facing an issue with latest kafka 0.10.1 and the log cleaner thread
> with regards to the timeindex files. From the log of the log-cleaner we see
> after startup that it tries to cleanup a topic called xdc_listing-status-v2
> [1]. The topic is setup with log compaction [2] and the kafka cluster
> configuration has log.cleaner enabled [3]. Looking at the log and the newly
> created file [4], the cleaner seems to refer to tombstones prior to
> epoch_time=0 - maybe because he finds messages, which don’t have a timestamp
> at all (?). All producers and consumers are using 0.10.1 and the topics have
> been created completely new, so I’m not sure, where this issue would come
> from. The original timeindex file [5] seems to show only valid timestamps for
> the mentioned offsets. I would also like to mention that the issue happened
> in two independent datacenters at the same time, so I would rather expect an
> application/producer issue instead of random disk failures. We didn’t have
> the problem with 0.10.0 for around half a year, it appeared shortly after the
> upgrade to 0.10.1.
> The puzzling message from the cleaner “cleaning prior to Fri Dec 02 16:35:50
> CET 2016, discarding tombstones prior to Thu Jan 01 01:00:00 CET 1970” also
> confuses me a bit. Does that mean, it does not find any log segments which
> can be cleaned up or the last timestamp of the last log segment is somehow
> broken/missing?
> I’m also a bit wondering, why the log cleaner thread stops completely after
> an error with one topic. I would at least expect that it keeps on cleaning up
> other topics, but apparently it doesn’t do that, e.g. it’s not even cleaning
> the __consumer_offsets anymore.
> Does anybody have the same issues or can explain, what’s going on? Thanks for
> any help or suggestions.
> Cheers
> Robert
> [1]
> {noformat}
> [2016-12-06 12:49:17,885] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,895] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,947] INFO Cleaner 0: Beginning cleaning of log
> xdc_listing-status-v2-1. (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,948] INFO Cleaner 0: Building offset map for
> xdc_listing-status-v2-1... (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,989] INFO Cleaner 0: Building offset map for log
> xdc_listing-status-v2-1 for 1 segments in offset range [0, 194991).
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,572] INFO Cleaner 0: Offset map for log
> xdc_listing-status-v2-1 complete. (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,577] INFO Cleaner 0: Cleaning log
> xdc_listing-status-v2-1 (cleaning prior to Fri Dec 02 16:35:50 CET 2016,
> discarding tombstones prior to Thu Jan 01 01:00:00 CET 1970)...
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,580] INFO Cleaner 0: Cleaning segment 0 in log
> xdc_listing-status-v2-1 (largest timestamp Fri Dec 02 16:35:50 CET 2016) into
> 0, retaining deletes. (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,968] ERROR [kafka-log-cleaner-thread-0], Error due to
> (kafka.log.LogCleaner)
> kafka.common.InvalidOffsetException: Attempt to append an offset (-1) to slot
> 9 no larger than the last offset appended (11832) to
> /var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned.
> at
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:117)
> at
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
> at
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
> at kafka.log.LogSegment.append(LogSegment.scala:106)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:518)
> at
> kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404)
> at
> kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at kafka.log.Cleaner.clean(LogCleaner.scala:363)
> at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-12-06 12:49:24,971] INFO [kafka-log-cleaner-thread-0], Stopped
> (kafka.log.LogCleaner)
> {noformat}
> [2]
> {noformat}
> $ kafka-configs-topics xdc_listing-status-v2 --describe
> Configs for topic 'xdc_listing-status-v2' are cleanup.policy=compact
> {noformat}
> [3]
> {noformat}
> broker.id=74
> listeners = PLAINTEXT://0.0.0.0:9092,TRACE://localhost:9093
> advertised.listeners = PLAINTEXT://broker2:9092
> num.network.threads=2
> num.io.threads=2
> inter.broker.protocol.version=0.10.1.0
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> log.dirs=/var/lib/kafka
> num.partitions=3
> default.replication.factor=2
> log.flush.interval.messages=10000
> log.flush.interval.ms=1000
> log.retention.hours=168
> log.segment.bytes=536870912
> log.cleanup.interval.mins=1
> zookeeper.connect=broker1:2181,broker2:2181,broker3:2181
> zookeeper.connection.timeout.ms=300000
> kafka.metrics.reporters=com.airbnb.kafka.KafkaStatsdMetricsReporter
> external.kafka.statsd.reporter.enabled=true
> external.kafka.statsd.host=localhost
> external.kafka.statsd.port=8125
> external.kafka.statsd.tag.enabled=false
> external.kafka.statsd.metrics.prefix=collectd.broker2
> kafka.metrics.polling.interval.secs=60
> external.kafka.statsd.dimension.enabled.meanRate=false
> external.kafka.statsd.dimension.enabled.rate15m=false
> external.kafka.statsd.dimension.enabled.min=false
> external.kafka.statsd.dimension.enabled.max=false
> external.kafka.statsd.dimension.enabled.median=false
> external.kafka.statsd.dimension.enabled.p75=false
> external.kafka.statsd.dimension.enabled.p98=false
> external.kafka.statsd.dimension.enabled.p999=false
> delete.topic.enable=true
> controlled.shutdown.enable=true
> auto.leader.rebalance.enable=true
> auto.create.topics.enable=false
> log.cleaner.enable=true
> offsets.topic.compression.codec=2
> offsets.topic.num.partitions=20
> {noformat}
> [4]
> {noformat}
> #
> ti=/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned;
> for e in $(xxd -c 12 -p $ti); do echo -n "file:[$ti] raw:[$e] "; d="$(date
> -d @$(echo -n $((16#${e:0:16})) | sed 's/.\{10\}/&./g'))"; echo -n
> "timestamp:[$d] "; echo "offset:[$((16#${e:16:24}))]";done | head -n 20
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589c490ae1ffffffff] timestamp:[Fri Nov 25 17:19:42 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589c81f249ffffffff] timestamp:[Fri Nov 25 18:21:51 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589d4b4ea7ffffffff] timestamp:[Fri Nov 25 22:01:48 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589e04a783ffffffff] timestamp:[Sat Nov 26 01:24:14 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589ede5af6ffffffff] timestamp:[Sat Nov 26 05:22:02 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589f8ccc16ffffffff] timestamp:[Sat Nov 26 08:32:34 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589ffb82bcffffffff] timestamp:[Sat Nov 26 10:33:30 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[00000158a04fef7fffffffff] timestamp:[Sat Nov 26 12:05:43 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[00000158a0b506eb00002e38] timestamp:[Sat Nov 26 13:56:08 CET 2016]
> offset:[11832]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> {noformat}
> [5]
> {noformat}
> /var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex
> ...
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06cbddc00002a21] timestamp:[Sat Nov 26 12:37:10 CET 2016]
> offset:[10785]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06cded500002a29] timestamp:[Sat Nov 26 12:37:19 CET 2016]
> offset:[10793]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06d25d400002a31] timestamp:[Sat Nov 26 12:37:37 CET 2016]
> offset:[10801]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06dc34700002a39] timestamp:[Sat Nov 26 12:38:17 CET 2016]
> offset:[10809]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06e3ccf00002a41] timestamp:[Sat Nov 26 12:38:48 CET 2016]
> offset:[10817]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07051ae00002a4a] timestamp:[Sat Nov 26 12:41:05 CET 2016]
> offset:[10826]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a071168100002a52] timestamp:[Sat Nov 26 12:41:55 CET 2016]
> offset:[10834]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a071603400002a5a] timestamp:[Sat Nov 26 12:42:14 CET 2016]
> offset:[10842]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07196c600002a62] timestamp:[Sat Nov 26 12:42:28 CET 2016]
> offset:[10850]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a072531500002a6a] timestamp:[Sat Nov 26 12:43:16 CET 2016]
> offset:[10858]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a072c5f700002a73] timestamp:[Sat Nov 26 12:43:46 CET 2016]
> offset:[10867]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07340a500002a7b] timestamp:[Sat Nov 26 12:44:17 CET 2016]
> offset:[10875]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0743e5a00002a85] timestamp:[Sat Nov 26 12:45:22 CET 2016]
> offset:[10885]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a074d37b00002a8d] timestamp:[Sat Nov 26 12:46:00 CET 2016]
> offset:[10893]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07551fe00002a95] timestamp:[Sat Nov 26 12:46:33 CET 2016]
> offset:[10901]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a075e21900002a9e] timestamp:[Sat Nov 26 12:47:09 CET 2016]
> offset:[10910]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a075faa900002aa6] timestamp:[Sat Nov 26 12:47:16 CET 2016]
> offset:[10918]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07619f900002aae] timestamp:[Sat Nov 26 12:47:24 CET 2016]
> offset:[10926]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076447500002ab6] timestamp:[Sat Nov 26 12:47:35 CET 2016]
> offset:[10934]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076728000002abf] timestamp:[Sat Nov 26 12:47:46 CET 2016]
> offset:[10943]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0768e7d00002ac8] timestamp:[Sat Nov 26 12:47:54 CET 2016]
> offset:[10952]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076af9300002ad1] timestamp:[Sat Nov 26 12:48:02 CET 2016]
> offset:[10961]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076d81500002ad9] timestamp:[Sat Nov 26 12:48:12 CET 2016]
> offset:[10969]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0773f2200002ae1] timestamp:[Sat Nov 26 12:48:39 CET 2016]
> offset:[10977]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a077babf00002ae9] timestamp:[Sat Nov 26 12:49:10 CET 2016]
> offset:[10985]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0792cee00002af1] timestamp:[Sat Nov 26 12:50:45 CET 2016]
> offset:[10993]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a079cb9b00002af9] timestamp:[Sat Nov 26 12:51:26 CET 2016]
> offset:[11001]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07a28a800002b02] timestamp:[Sat Nov 26 12:51:50 CET 2016]
> offset:[11010]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07a651500002b0a] timestamp:[Sat Nov 26 12:52:05 CET 2016]
> offset:[11018]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07c0d0800002b14] timestamp:[Sat Nov 26 12:53:54 CET 2016]
> offset:[11028]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07cc5cd00002b1c] timestamp:[Sat Nov 26 12:54:41 CET 2016]
> offset:[11036]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07d807300002b25] timestamp:[Sat Nov 26 12:55:29 CET 2016]
> offset:[11045]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07de02c00002b2d] timestamp:[Sat Nov 26 12:55:53 CET 2016]
> offset:[11053]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e505c00002b35] timestamp:[Sat Nov 26 12:56:22 CET 2016]
> offset:[11061]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e6c6e00002b3d] timestamp:[Sat Nov 26 12:56:29 CET 2016]
> offset:[11069]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e8a2600002b45] timestamp:[Sat Nov 26 12:56:37 CET 2016]
> offset:[11077]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07ed26b00002b4d] timestamp:[Sat Nov 26 12:56:55 CET 2016]
> offset:[11085]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07f729600002b55] timestamp:[Sat Nov 26 12:57:36 CET 2016]
> offset:[11093]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a080278400002b5e] timestamp:[Sat Nov 26 12:58:23 CET 2016]
> offset:[11102]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a08168e300002b67] timestamp:[Sat Nov 26 12:59:45 CET 2016]
> offset:[11111]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a08257c900002b6f] timestamp:[Sat Nov 26 13:00:46 CET 2016]
> offset:[11119]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a083f4b900002b77] timestamp:[Sat Nov 26 13:02:32 CET 2016]
> offset:[11127]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a084d01c00002b7f] timestamp:[Sat Nov 26 13:03:28 CET 2016]
> offset:[11135]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a086450500002b87] timestamp:[Sat Nov 26 13:05:03 CET 2016]
> offset:[11143]
> ...
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09d4a7700002cff] timestamp:[Sat Nov 26 13:30:12 CET 2016]
> offset:[11519]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09d783100002d07] timestamp:[Sat Nov 26 13:30:24 CET 2016]
> offset:[11527]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09da2a100002d0f] timestamp:[Sat Nov 26 13:30:35 CET 2016]
> offset:[11535]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09dcdf100002d17] timestamp:[Sat Nov 26 13:30:46 CET 2016]
> offset:[11543]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09dfdb700002d1f] timestamp:[Sat Nov 26 13:30:58 CET 2016]
> offset:[11551]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09e311700002d28] timestamp:[Sat Nov 26 13:31:11 CET 2016]
> offset:[11560]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09eccaa00002d31] timestamp:[Sat Nov 26 13:31:51 CET 2016]
> offset:[11569]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09f542300002d3a] timestamp:[Sat Nov 26 13:32:26 CET 2016]
> offset:[11578]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a026c100002d43] timestamp:[Sat Nov 26 13:33:20 CET 2016]
> offset:[11587]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a0d8c900002d4b] timestamp:[Sat Nov 26 13:34:05 CET 2016]
> offset:[11595]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a163d400002d53] timestamp:[Sat Nov 26 13:34:41 CET 2016]
> offset:[11603]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a310fe00002d5c] timestamp:[Sat Nov 26 13:36:31 CET 2016]
> offset:[11612]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a3b0a300002d65] timestamp:[Sat Nov 26 13:37:11 CET 2016]
> offset:[11621]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a3cdf100002d6d] timestamp:[Sat Nov 26 13:37:19 CET 2016]
> offset:[11629]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a42ba200002d75] timestamp:[Sat Nov 26 13:37:43 CET 2016]
> offset:[11637]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a4864900002d7e] timestamp:[Sat Nov 26 13:38:06 CET 2016]
> offset:[11646]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a500e700002d86] timestamp:[Sat Nov 26 13:38:38 CET 2016]
> offset:[11654]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a5a38200002d8e] timestamp:[Sat Nov 26 13:39:19 CET 2016]
> offset:[11662]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a5f5a400002d96] timestamp:[Sat Nov 26 13:39:40 CET 2016]
> offset:[11670]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a6e30400002d9e] timestamp:[Sat Nov 26 13:40:41 CET 2016]
> offset:[11678]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a7e6b600002da7] timestamp:[Sat Nov 26 13:41:47 CET 2016]
> offset:[11687]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aa276300002db1] timestamp:[Sat Nov 26 13:44:15 CET 2016]
> offset:[11697]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ab040c00002dbb] timestamp:[Sat Nov 26 13:45:12 CET 2016]
> offset:[11707]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ab6d2c00002dc3] timestamp:[Sat Nov 26 13:45:38 CET 2016]
> offset:[11715]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aba59e00002dcb] timestamp:[Sat Nov 26 13:45:53 CET 2016]
> offset:[11723]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0acb9b200002dd4] timestamp:[Sat Nov 26 13:47:04 CET 2016]
> offset:[11732]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae188e00002ddc] timestamp:[Sat Nov 26 13:48:33 CET 2016]
> offset:[11740]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae682b00002de4] timestamp:[Sat Nov 26 13:48:54 CET 2016]
> offset:[11748]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae7a9b00002ded] timestamp:[Sat Nov 26 13:48:59 CET 2016]
> offset:[11757]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae9bed00002df5] timestamp:[Sat Nov 26 13:49:07 CET 2016]
> offset:[11765]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aeb82200002dfd] timestamp:[Sat Nov 26 13:49:14 CET 2016]
> offset:[11773]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aed16a00002e05] timestamp:[Sat Nov 26 13:49:21 CET 2016]
> offset:[11781]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aedd8d00002e0d] timestamp:[Sat Nov 26 13:49:24 CET 2016]
> offset:[11789]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aff17200002e16] timestamp:[Sat Nov 26 13:50:34 CET 2016]
> offset:[11798]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b0d65e00002e1e] timestamp:[Sat Nov 26 13:51:33 CET 2016]
> offset:[11806]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b1e31f00002e26] timestamp:[Sat Nov 26 13:52:42 CET 2016]
> offset:[11814]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b3736300002e2e] timestamp:[Sat Nov 26 13:54:24 CET 2016]
> offset:[11822]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b4648a00002e36] timestamp:[Sat Nov 26 13:55:26 CET 2016]
> offset:[11830]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b7ad3d00002e40] timestamp:[Sat Nov 26 13:59:01 CET 2016]
> offset:[11840]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b7efe700002e49] timestamp:[Sat Nov 26 13:59:18 CET 2016]
> offset:[11849]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
