Harish Seshadri created KAFKA-2427: -------------------------------------- Summary: Error writing to highwatermark file Key: KAFKA-2427 URL: https://issues.apache.org/jira/browse/KAFKA-2427 Project: Kafka Issue Type: Bug Components: replication Affects Versions: 0.8.2.1 Environment: Ubuntu 14.04 Reporter: Harish Seshadri Assignee: Neha Narkhede Priority: Critical
Periodically one instance of the kafka broker crashes (process exits) with the following error. Note: The persistence of files makes use of NFS mount [2015-08-12 08:42:12,480] FATAL [Replica Manager on Broker 1]: Error writing to highwatermark file: (kafka.server.ReplicaManager) java.io.IOException: File rename from /nfs/data/kafka1-logs/replication-offset-checkpoint.tmp to /nfs/data/kafka1-logs/replication-offset-checkpoint failed. at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:66) at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:596) at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:593) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:593) at kafka.server.ReplicaManager$$anonfun$1.apply$mcV$sp(ReplicaManager.scala:99) at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:99) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) (END)packet_write_wait: Connection to 10.23.2.110: Broken pipe -- This message was sent by Atlassian JIRA (v6.3.4#6332)