[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733902#comment-14733902 ] Jiangjie Qin commented on KAFKA-2510: - Gwen, do you mean we should have some cross check of the configurations? e.g. store some config in ZK as well? Currently the broker config file is the only configuration source. It seems hard to tell whether it is misconfigured or not when you only have one source. How can we distinguish between 1) Sysadmin wants to configure a new directory to store log; and 2) Sysadmin misconfigured a new directory to store the logs? > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733917#comment-14733917 ] Flavio Junqueira commented on KAFKA-2510: - yeah, it is true, my discussion focused on a single broker, but the dbid idea still works, since brokers would refuse to start if the dbid isn't the one they are expecting. you have this scenario when all brokers replicating a partition have been directed to a different directory. as for controlled shutdown, I think you're referring to shutting down brokers one by one, changing the configuration of the broker before restarting. I haven't really thought too much about a rolling upgrade in the context of kafka, but in general it's a fragile mechanism for changing configuration/versions in replicated systems. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733923#comment-14733923 ] Flavio Junqueira commented on KAFKA-2510: - my proposal was to use zk metadata (e.g., dbid), and one could override it via command line, which in principle should prevent data loss due to misconfiguration. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733939#comment-14733939 ] Flavio Junqueira commented on KAFKA-2510: - using a controlled shutdown in principle solves the problem because of the following argument. let's assume that you have only two brokers to make it simple. the directory is initially set to /a and the misconfiguration now points each to /b. if you shutdown broker 1 and restarts it before doing it with broker 2, then broker 1 will copy the data from broker 2. the issue for me is that this is very fragile because there are all sort of corner cases that you could have while doing this that could cause you to lose data. say for example that you do not wait enough time and shutdown broker 2 before it has time to copy the data. in this case, you lost data. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732096#comment-14732096 ] Jay Kreps commented on KAFKA-2510: -- @gwen I think maybe I'm confused. The discussion in the ticket is all about losing data or losing the cluster. But isn't this prevented by controlled shutdown? > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14730832#comment-14730832 ] Flavio Junqueira commented on KAFKA-2510: - The key problem is the one of a broker waking up and not finding data on disk. In such a scenario, is the broker faulty and lost disk state (via misconfiguration maybe), or is it starting from scratch? The solution is to write on some persistent store that the broker has written something to disk once it does. It could for example add to the partition metadata a mark once it creates the directory. Another way is to use some form of dbid, a number that reflects the instance of the disk state. The broker writes the dbid to the drive and immediately after to ZK. Upon restarting, the dbid must match. Note that the broker can't simply write the dbid to the registration znode, which is ephemeral. It must be a persistent znode. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731403#comment-14731403 ] Gwen Shapira commented on KAFKA-2510: - the old data directory will be untouched, but because there are no real errors and everything will appear to be working. New data coming in starting with offset 0, consumers happily consuming this data, and a new __offsets topic. Few hours or days later, when you figured what happened - you are kind of stuck - what are you going to do with the old data directory? > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731393#comment-14731393 ] Jiangjie Qin commented on KAFKA-2510: - Hey Gwen, can you elaborate a bit why the consumer offsets would be lost? Do you mean Zookeeper based offset storage? My understanding is that if the log directory was configured to be some wrong new path, the old data directory will still have all the data untouched. So it seems no actual data loss will be there. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731497#comment-14731497 ] Jay Kreps commented on KAFKA-2510: -- I actually think we shouldn't prevent this. In our replication model the data on disk is basically a cache. If it's there the broker uses it to help make its own recovery faster and just pulls the diff from replicas. If its not there it recreates it. You are allowed to lose what is on disk at any time. For the chef case, yes, if you botch the directory you will replicate data, but the same is true of the old node id as well as things like the zk URL, etc. Replicating will be slow but not fatal. The case of rolling restart is not correct if you use controlled shutdown. If you don't use controlled shutdown you will lose data no matter what. So today if you want you can wipe your data and restart and the broker happily re-replicates. If you have a disk failure you can repair it and restart. And if your AWS instance disappears you can move it over to another and it re-replicates. We actually intended to exploit this for running in AWS and Mesos more elastically when we do automated data balancing. For example in Mesos the mesos guys are going to add a feature in Marathon where tasks will be semi-sticky to nodes. So if a Kafka node is restarted or dies mesos will prefer to restart it on the node it was on (if that node is still around and has free slots). If not it will start it elsewhere (where it will, of course, have no data). > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731629#comment-14731629 ] Gwen Shapira commented on KAFKA-2510: - [~jkreps] I agree that this capability is important to some uses cases. I also think that preventing sysadmins from accidentally losing data on an entire cluster can be an important thing to support. Which is why I think it should be configurable (like delete.topic.enable and unsafe.leader.election) - there are legit tradeoffs. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729413#comment-14729413 ] Gwen Shapira commented on KAFKA-2510: - Note that if you accidentally manage to set an entire cluster to the wrong directory (easy when Chef or similar manages your configuration), you also lose the consumer offsets - so only clients that use external offset store will even notice that the data is gone. Losing ALL data in the cluster without any errors is a huge problem. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
