John, Can you give more detail as in how the data is inconsistent and post the logs somewhere. Are the log and data directories on different mountpoints?
To recover immediately, you should stop zookeeper on the divergent nodes. Backup then delete the log and snap directories on those nodes and then restart zookeeper on those nodes. Asad From: jlindwall <[email protected]> Sent: Apr 22, 2015 6:25 PM To: [email protected] Subject: Inconsistent data across 3.4.6 ensemble We somehow are seeing inconsistent data across our 3-node prod ensemble. Never saw anything like it in dev or qa. We are running on Solaris. The dataDirs for the nodes were recently involved in a situation in which the nfs disk they live on was dismounted and remounted, while zk was running. Not sure if it is related. Regardless, this seems like it should never happe n with zookeeper. Any ideas for correcting the situation? I have 2 ideas, please critique: 1. Bring down follower 1, delete it's logDataDir and dataDir contents, restart; do same with follower 2 2. Bring down the whole thing; delete all logDataDir and dataDir contents; restart I'd prefer not to do option #2, but I will if I must. Thanks, John -- View this message in context: http://zookeeper-user.578899.n2.nabble.com/Inconsistent-data-across-3-4-6-ensemble-tp7581007.html Sent from the zookeeper-user mailing list archive at Nabble.com.
