Ramnatthan Alagappan created KAFKA-4127:
-------------------------------------------
Summary: Possible data loss
Key: KAFKA-4127
URL: https://issues.apache.org/jira/browse/KAFKA-4127
Project: Kafka
Issue Type: Bug
Environment: Normal three node Kafka cluster. All machines running
linux.
Reporter: Ramnatthan Alagappan
I am running a three node Kakfa cluster. ZooKeeper runs in a standalone mode.
When I create a new message topic, I see the following sequence of system calls:
mkdir("/appdir/my-topic1-0")
creat("/appdir/my-topic1-0/00000000000000000000.log")
I have configured Kafka to write the messages persistently to the disk before
acknowledging the client. Specifically, I have set flush.interval_messages to
1, min_insync_replicas to 3, and disabled dirty election. Now, I insert a new
message into the created topic.
I see that Kafka writes the message to the log file and flushes the data down
to disk by carefully fsync'ing the log file. I get an acknowledgment back from
the cluster after the message is safely persisted on all three replicas and
written to disk.
Unfortunately, Kafka can still lose data since it does not explicitly fsync the
directory to persist the directory entries of the topic directory and the log
file. If a crash happens after acknowledging the client, it is possible for
Kafka lose the directory entry for the topic directory or the log file. Many
systems carefully issue fsync to the parent directory when a new file or
directory is created. This is required for the file to be completely persisted
to the disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
