[ 
https://issues.apache.org/jira/browse/KAFKA-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387859#comment-16387859
 ] 

dongyan li commented on KAFKA-6610:
-----------------------------------

[~huxi_2b] I think your suspect is correct, all other brokers other than the 
leader get the highwatermark of -1 for this topic. Can you guide me some ways 
to query the highwatermark of the leader and how can it be set to -1, what this 
value is based of?

[~hachikuji] some related logs:
 Timeline:
{quote}3/5/2018 11:28:23 AM[2018-03-05 17:28:23,084] INFO Replica loaded for 
TOPICNAME-0 with initial high watermark 0 (kafka.cluster.Replica) 
 3/5/2018 11:28:23 AM[2018-03-05 17:28:23,086] INFO Replica loaded for 
partition TOPICNAME-0 with initial high watermark -1 (kafka.cluster.Replica) 
 3/5/2018 11:28:23 AM[2018-03-05 17:28:23,086] INFO Replica loaded for 
partition TOPICNAME-0 with initial high watermark 0 (kafka.cluster.Replica)
{quote}
Later:
{quote}3/5/2018 11:28:23 AM[2018-03-05 17:28:23,305] WARN 
[ReplicaFetcherThread-0-1]: Based on follower's leader epoch, leader replied 
with an unknown offset in TOPICNAME-0. High watermark -1 will be used for 
truncation. (kafka.server.ReplicaFetcherThread)
{quote}
Then, error: (timestamp is not right, but the content is the same)
{quote}3/5/2018 10:32:26 AMjava.lang.IllegalArgumentException: Cannot truncate 
to a negative offset (-1). 
 3/5/2018 10:32:26 AM at kafka.log.Log.truncateTo(Log.scala:1377) 
 3/5/2018 10:32:26 AM at 
kafka.log.LogManager.$anonfun$truncateTo$2(LogManager.scala:330) 
 3/5/2018 10:32:26 AM at 
kafka.log.LogManager.$anonfun$truncateTo$2$adapted(LogManager.scala:321) 
 3/5/2018 10:32:26 AM at 
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
 
 3/5/2018 10:32:26 AM at 
scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:138) 
 3/5/2018 10:32:26 AM at 
scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:236) 
 3/5/2018 10:32:26 AM at 
scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:229) 
 3/5/2018 10:32:26 AM at 
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) 
 3/5/2018 10:32:26 AM at 
scala.collection.mutable.HashMap.foreach(HashMap.scala:138) 
 3/5/2018 10:32:26 AM at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788) 
 3/5/2018 10:32:26 AM at kafka.log.LogManager.truncateTo(LogManager.scala:321) 
 3/5/2018 10:32:26 AM at 
kafka.server.ReplicaFetcherThread.maybeTruncate(ReplicaFetcherThread.scala:279) 
 3/5/2018 10:32:26 AM at 
kafka.server.AbstractFetcherThread.$anonfun$maybeTruncate$2(AbstractFetcherThread.scala:133)
 3/5/2018 10:32:26 AM at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) 
 3/5/2018 10:32:26 AM at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213) 
 3/5/2018 10:32:26 AM at 
kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:130)
 
 3/5/2018 10:32:26 AM at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102) 
 3/5/2018 10:32:26 AM at 
kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
{quote}

> initial high watermark -1, used for truncation, cause "Cannot truncate to a 
> negative offset"
> --------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6610
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6610
>             Project: Kafka
>          Issue Type: Bug
>          Components: log, replication
>    Affects Versions: 0.11.0.2
>            Reporter: dongyan li
>            Priority: Major
>
> Hello,
> I got issue that cause the Kafka broker not in-sync with topics. When I check 
> the log, found one of the topic has "-1" highwatermark, then used as the 
> offset for truncating and later caused exception: 
> {quote}3/5/2018 10:32:26 AM[2018-03-05 16:32:26,576] ERROR 
> [ReplicaFetcherThread-0-1]: Error due to (kafka.server.ReplicaFetcherThread)
>  3/5/2018 10:32:26 AMjava.lang.IllegalArgumentException: Cannot truncate to a 
> negative offset (-1).
>  3/5/2018 10:32:26 AM at kafka.log.Log.truncateTo(Log.scala:1377)
>  3/5/2018 10:32:26 AM at 
> kafka.log.LogManager.$anonfun$truncateTo$2(LogManager.scala:330)
>  3/5/2018 10:32:26 AM at 
> kafka.log.LogManager.$anonfun$truncateTo$2$adapted(LogManager.scala:321)
>  3/5/2018 10:32:26 AM at 
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
>  3/5/2018 10:32:26 AM at 
> scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:138)
>  3/5/2018 10:32:26 AM at 
> scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:236)
>  3/5/2018 10:32:26 AM at 
> scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:229)
>  3/5/2018 10:32:26 AM at 
> scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>  3/5/2018 10:32:26 AM at 
> scala.collection.mutable.HashMap.foreach(HashMap.scala:138)
>  3/5/2018 10:32:26 AM at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
>  3/5/2018 10:32:26 AM at kafka.log.LogManager.truncateTo(LogManager.scala:321)
>  3/5/2018 10:32:26 AM at 
> kafka.server.ReplicaFetcherThread.maybeTruncate(ReplicaFetcherThread.scala:279)
>  3/5/2018 10:32:26 AM at 
> kafka.server.AbstractFetcherThread.$anonfun$maybeTruncate$2(AbstractFetcherThread.scala:133)
>  3/5/2018 10:32:26 AM at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
>  3/5/2018 10:32:26 AM at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
>  3/5/2018 10:32:26 AM at 
> kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:130)
>  3/5/2018 10:32:26 AM at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)
>  3/5/2018 10:32:26 AM at 
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> {quote}
> I deleted the log directory of that topic but topic partition got re-created 
> after restart, with same error after. I'm wondering how the `highwatermark` 
> get set to -1. I failed to trace the code flow from the `Replica` 
> constructor. Thank you.
>  
> Dongyan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to