Re: Logs truncated at o'clock

mosto...@gmail.com Fri, 14 Jul 2017 04:32:58 -0700

we are using a local ZFS


On 14/07/17 13:31, Tom Crayford wrote:

Hi,

Which folder are you storing kafka's data in? By default that's /tmp, which
might be getting wiped by your OS.

Thanks

Tom Crayford
Heroku Kafka

On Fri, Jul 14, 2017 at 11:50 AM, mosto...@gmail.com <mosto...@gmail.com>
wrote:

anyone?



On 13/07/17 17:09, mosto...@gmail.com wrote:

Hi

With swiss precission, our kafka test environment seems to truncate
topics at o'clock hours.

This might be confirmed with the following trace, which states
"Truncating log ... to offset 0"

We are still using Kafka 0.10.2.1, but I was wondering if this is
resolved in recent versions, it's a know bug or just a misconfiguration (I
think it is shown below).

Thanks

     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,986]
     INFO [TopicChangeListener on Controller 1002]: New topics:
     [Set(mytopic.2017-07-13-16)], deleted topics: [Set()], new
     partition replica assignment [Map([mytopic.2017-07-13-16,0] ->
     List(1001, 1002), [mytopic.2017-07-13-16,2] -> List(1003, 1001),
     [mytopic.2017-07-13-16,1] -> List(1002, 1003))]
     (kafka.controller.PartitionStateMachine$TopicChangeListener)
     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,987]
     INFO [Controller 1002]: New topic creation callback for
     [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
2017-07-13-16,1]
     (kafka.controller.KafkaController)
     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
     INFO [Controller 1002]: New partition creation callback for
     [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
2017-07-13-16,1]
     (kafka.controller.KafkaController)
     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
     INFO [Partition state machine on Controller 1002]: Invoking state
     change to NewPartition for partitions
     [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
2017-07-13-16,1]
     (kafka.controller.PartitionStateMachine)
     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
     INFO [Replica state machine on controller 1002]: Invoking state
     change to NewReplica for replicas
     [Topic=mytopic.2017-07-13-16,Partition=1,Replica=1003],[
Topic=mytopic.2017-07-13-16,Partition=0,Replica=1001],[Topic=mytopic.
2017-07-13-16,Partition=2,Replica=1001],[Topic=mytopic.2017-07-13-16,Pa
rtition=1,Replica=1002],[Topic=mytopic.2017-07-13-16,
Partition=2,Replica=1003],[Topic=mytopic.2017-07-13-16,
Partition=0,Replica=1002]
     (kafka.controller.ReplicaStateMachine)
     Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,993]
     INFO [Partition state machine on Controller 1002]: Invoking state
     change to OnlinePartition for partitions
     [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
2017-07-13-16,1]
     (kafka.controller.PartitionStateMachine)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,036]
     INFO [Replica state machine on controller 1002]: Invoking state
     change to OnlineReplica for replicas
     [Topic=mytopic.2017-07-13-16,Partition=1,Replica=1003],[
Topic=mytopic.2017-07-13-16,Partition=0,Replica=1001],[Topic=mytopic.
2017-07-13-16,Partition=2,Replica=1001],[Topic=mytopic.2017-07-13-16,Pa
rtition=1,Replica=1002],[Topic=mytopic.2017-07-13-16,
Partition=2,Replica=1003],[Topic=mytopic.2017-07-13-16,
Partition=0,Replica=1002]
     (kafka.controller.ReplicaStateMachine)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,037]
     INFO [ReplicaFetcherManager on broker 1002] Removed fetcher for
     partitions mytopic.2017-07-13-16-1
     (kafka.server.ReplicaFetcherManager)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,039]
     INFO Completed load of log mytopic.2017-07-13-16-1 with 1 log
     segments and log end offset 0 in 0 ms (kafka.log.Log)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,040]
     INFO Created log for partition [mytopic.2017-07-13-16,1] in
     /data/kafka-1 with properties {compression.type -> producer,
     message.format.version -> 0.10.2-IV0, file.delete.delay.ms ->
     60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
     message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
     segment.jitter.ms -> 0, preallocate -> false,
     min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
     unclean.leader.election.enable -> true, retention.bytes -> -1,
     delete.retention.ms -> 3600000, cleanup.policy -> [delete],
     flush.ms -> 300000, segment.ms -> 600000, segment.bytes ->
     1073741824, retention.ms -> 14400000,
     message.timestamp.difference.max.ms -> 9223372036854775807,
     segment.index.bytes -> 10485760, flush.messages -> 10000}.
     (kafka.log.LogManager)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,045]
     INFO Partition [mytopic.2017-07-13-16,1] on broker 1002: No
     checkpointed highwatermark is found for partition
     mytopic.2017-07-13-16-1 (kafka.cluster.Partition)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,047]
     INFO Completed load of log mytopic.2017-07-13-16-0 with 1 log
     segments and log end offset 0 in 1 ms (kafka.log.Log)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,047]
     INFO Created log for partition [mytopic.2017-07-13-16,0] in
     /data/kafka-1 with properties {compression.type -> producer,
     message.format.version -> 0.10.2-IV0, file.delete.delay.ms ->
     60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
     message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
     segment.jitter.ms -> 0, preallocate -> false,
     min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
     unclean.leader.election.enable -> true, retention.bytes -> -1,
     delete.retention.ms -> 3600000, cleanup.policy -> [delete],
     flush.ms -> 300000, segment.ms -> 600000, segment.bytes ->
     1073741824, retention.ms -> 14400000,
     message.timestamp.difference.max.ms -> 9223372036854775807,
     segment.index.bytes -> 10485760, flush.messages -> 10000}.
     (kafka.log.LogManager)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
     INFO Partition [mytopic.2017-07-13-16,0] on broker 1002: No
     checkpointed highwatermark is found for partition
     mytopic.2017-07-13-16-0 (kafka.cluster.Partition)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
     INFO [ReplicaFetcherManager on broker 1002] Removed fetcher for
     partitions mytopic.2017-07-13-16-0
     (kafka.server.ReplicaFetcherManager)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
     INFO Truncating log mytopic.2017-07-13-16-0 to offset 0.
     (kafka.log.Log)
     Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,059]
     INFO [ReplicaFetcherManager on broker 1002] Added fetcher for
     partitions List([mytopic.2017-07-13-16-0, initOffset 0 to broker
     BrokerEndPoint(1001,10.0.0.24,9092)] )
     (kafka.server.ReplicaFetcherManager)
     Jul 13 16:00:03 computer kafka[28511]: [2017-07-13 16:00:03,614]
     ERROR [ReplicaFetcherThread-0-1001], Error for partition
     [mytopic.2017-07-13-16,0] to broker
     1001:org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
     This server does not host this topic-partition.
     (kafka.server.ReplicaFetcherThread)

Re: Logs truncated at o'clock

Reply via email to