looks like these logs coming immediately after topic creation. did you see
any data loss?
otherwise, these should be normal.
On Fri, Jul 14, 2017 at 5:02 PM, mosto...@gmail.com <mosto...@gmail.com>
wrote:
> we are using a local ZFS
>
>
>
> On 14/07/17 13:31, Tom Crayford wrote:
>
>> Hi,
>>
>> Which folder are you storing kafka's data in? By default that's /tmp,
>> which
>> might be getting wiped by your OS.
>>
>> Thanks
>>
>> Tom Crayford
>> Heroku Kafka
>>
>> On Fri, Jul 14, 2017 at 11:50 AM, mosto...@gmail.com <mosto...@gmail.com>
>> wrote:
>>
>> anyone?
>>>
>>>
>>>
>>> On 13/07/17 17:09, mosto...@gmail.com wrote:
>>>
>>> Hi
>>>>
>>>> With swiss precission, our kafka test environment seems to truncate
>>>> topics at o'clock hours.
>>>>
>>>> This might be confirmed with the following trace, which states
>>>> "Truncating log ... to offset 0"
>>>>
>>>> We are still using Kafka 0.10.2.1, but I was wondering if this is
>>>> resolved in recent versions, it's a know bug or just a misconfiguration
>>>> (I
>>>> think it is shown below).
>>>>
>>>> Thanks
>>>>
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,986]
>>>> INFO [TopicChangeListener on Controller 1002]: New topics:
>>>> [Set(mytopic.2017-07-13-16)], deleted topics: [Set()], new
>>>> partition replica assignment [Map([mytopic.2017-07-13-16,0] ->
>>>> List(1001, 1002), [mytopic.2017-07-13-16,2] -> List(1003, 1001),
>>>> [mytopic.2017-07-13-16,1] -> List(1002, 1003))]
>>>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,987]
>>>> INFO [Controller 1002]: New topic creation callback for
>>>> [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
>>>> 2017-07-13-16,1]
>>>> (kafka.controller.KafkaController)
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
>>>> INFO [Controller 1002]: New partition creation callback for
>>>> [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
>>>> 2017-07-13-16,1]
>>>> (kafka.controller.KafkaController)
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
>>>> INFO [Partition state machine on Controller 1002]: Invoking state
>>>> change to NewPartition for partitions
>>>> [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
>>>> 2017-07-13-16,1]
>>>> (kafka.controller.PartitionStateMachine)
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,988]
>>>> INFO [Replica state machine on controller 1002]: Invoking state
>>>> change to NewReplica for replicas
>>>> [Topic=mytopic.2017-07-13-16,Partition=1,Replica=1003],[
>>>> Topic=mytopic.2017-07-13-16,Partition=0,Replica=1001],[Topic=mytopic.
>>>> 2017-07-13-16,Partition=2,Replica=1001],[Topic=mytopic.2017-07-13-16,Pa
>>>> rtition=1,Replica=1002],[Topic=mytopic.2017-07-13-16,
>>>> Partition=2,Replica=1003],[Topic=mytopic.2017-07-13-16,
>>>> Partition=0,Replica=1002]
>>>> (kafka.controller.ReplicaStateMachine)
>>>> Jul 13 16:00:00 computer kafka[28511]: [2017-07-13 16:00:00,993]
>>>> INFO [Partition state machine on Controller 1002]: Invoking state
>>>> change to OnlinePartition for partitions
>>>> [mytopic.2017-07-13-16,0],[mytopic.2017-07-13-16,2],[mytopic.
>>>> 2017-07-13-16,1]
>>>> (kafka.controller.PartitionStateMachine)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,036]
>>>> INFO [Replica state machine on controller 1002]: Invoking state
>>>> change to OnlineReplica for replicas
>>>> [Topic=mytopic.2017-07-13-16,Partition=1,Replica=1003],[
>>>> Topic=mytopic.2017-07-13-16,Partition=0,Replica=1001],[Topic=mytopic.
>>>> 2017-07-13-16,Partition=2,Replica=1001],[Topic=mytopic.2017-07-13-16,Pa
>>>> rtition=1,Replica=1002],[Topic=mytopic.2017-07-13-16,
>>>> Partition=2,Replica=1003],[Topic=mytopic.2017-07-13-16,
>>>> Partition=0,Replica=1002]
>>>> (kafka.controller.ReplicaStateMachine)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,037]
>>>> INFO [ReplicaFetcherManager on broker 1002] Removed fetcher for
>>>> partitions mytopic.2017-07-13-16-1
>>>> (kafka.server.ReplicaFetcherManager)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,039]
>>>> INFO Completed load of log mytopic.2017-07-13-16-1 with 1 log
>>>> segments and log end offset 0 in 0 ms (kafka.log.Log)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,040]
>>>> INFO Created log for partition [mytopic.2017-07-13-16,1] in
>>>> /data/kafka-1 with properties {compression.type -> producer,
>>>> message.format.version -> 0.10.2-IV0, file.delete.delay.ms ->
>>>> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
>>>> message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
>>>> segment.jitter.ms -> 0, preallocate -> false,
>>>> min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
>>>> unclean.leader.election.enable -> true, retention.bytes -> -1,
>>>> delete.retention.ms -> 3600000, cleanup.policy -> [delete],
>>>> flush.ms -> 300000, segment.ms -> 600000, segment.bytes ->
>>>> 1073741824, retention.ms -> 14400000,
>>>> message.timestamp.difference.max.ms -> 9223372036854775807,
>>>> segment.index.bytes -> 10485760, flush.messages -> 10000}.
>>>> (kafka.log.LogManager)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,045]
>>>> INFO Partition [mytopic.2017-07-13-16,1] on broker 1002: No
>>>> checkpointed highwatermark is found for partition
>>>> mytopic.2017-07-13-16-1 (kafka.cluster.Partition)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,047]
>>>> INFO Completed load of log mytopic.2017-07-13-16-0 with 1 log
>>>> segments and log end offset 0 in 1 ms (kafka.log.Log)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,047]
>>>> INFO Created log for partition [mytopic.2017-07-13-16,0] in
>>>> /data/kafka-1 with properties {compression.type -> producer,
>>>> message.format.version -> 0.10.2-IV0, file.delete.delay.ms ->
>>>> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0,
>>>> message.timestamp.type -> CreateTime, min.insync.replicas -> 1,
>>>> segment.jitter.ms -> 0, preallocate -> false,
>>>> min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
>>>> unclean.leader.election.enable -> true, retention.bytes -> -1,
>>>> delete.retention.ms -> 3600000, cleanup.policy -> [delete],
>>>> flush.ms -> 300000, segment.ms -> 600000, segment.bytes ->
>>>> 1073741824, retention.ms -> 14400000,
>>>> message.timestamp.difference.max.ms -> 9223372036854775807,
>>>> segment.index.bytes -> 10485760, flush.messages -> 10000}.
>>>> (kafka.log.LogManager)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
>>>> INFO Partition [mytopic.2017-07-13-16,0] on broker 1002: No
>>>> checkpointed highwatermark is found for partition
>>>> mytopic.2017-07-13-16-0 (kafka.cluster.Partition)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
>>>> INFO [ReplicaFetcherManager on broker 1002] Removed fetcher for
>>>> partitions mytopic.2017-07-13-16-0
>>>> (kafka.server.ReplicaFetcherManager)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,052]
>>>> INFO Truncating log mytopic.2017-07-13-16-0 to offset 0.
>>>> (kafka.log.Log)
>>>> Jul 13 16:00:01 computer kafka[28511]: [2017-07-13 16:00:01,059]
>>>> INFO [ReplicaFetcherManager on broker 1002] Added fetcher for
>>>> partitions List([mytopic.2017-07-13-16-0, initOffset 0 to broker
>>>> BrokerEndPoint(1001,10.0.0.24,9092)] )
>>>> (kafka.server.ReplicaFetcherManager)
>>>> Jul 13 16:00:03 computer kafka[28511]: [2017-07-13 16:00:03,614]
>>>> ERROR [ReplicaFetcherThread-0-1001], Error for partition
>>>> [mytopic.2017-07-13-16,0] to broker
>>>> 1001:org.apache.kafka.common.errors.UnknownTopicOrPartition
>>>> Exception:
>>>> This server does not host this topic-partition.
>>>> (kafka.server.ReplicaFetcherThread)
>>>>
>>>>
>>>>
>>>>
>
