ivankelly opened a new issue #544: Bootup cookie validation considers an empty journal to signify a new bookie URL: https://github.com/apache/bookkeeper/issues/544 This is wrong, and can lead to split brain. From the dev list: We check all ledger, journal and index directories for a cookie. If it doesn't exist, it gets added to a missingCookieDirs list. We then iterate over this directory. If any directory in missingCookieDirs isn't listed as a ledger directory in the journal dir cookies, or isn't empty, we fail to start. The issue is that a journal dir could be emptied and we wouldn't detect it. It would be great if someone else could eyeball the code and tell me I'm wrong. The code is in Bookie#checkEnvironment. This breaks correctness. Imagine we have a ledger on b1, b2, b3. Writer w1 is writing to the ledger. The state of the ledger on the bookies is: b1: e0 Fenced: false, LAC: - b2: e0 Fenced: false, LAC: - b3: e0 Fenced: false, LAC: - w1 gets partitioned from network. w2 tries to recover the ledger, it tries to fence on all bookies. The message to b3 gets lost. b1 and b2 acknowledge the fencing, so w2 continues to recover and close the ledger with e0 as the last entry. b1: e0 Fenced: true, LAC: e0 b2: e0 Fenced: true, LAC: e0 b3: e0 Fenced: false, LAC: - If w1 became unpartitioned at this point, it wouldn't be able to add a new entry to the ledger as any quorum would see fenced on b1 or b2. However, imagine that the fenced message is only in the journal on b2, b2 crashes, something wipes the journal directory and then b2 comes back up. The new state of the ledger on the bookies will be. b1: e0 Fenced: true, LAC: e0 b2: e0 Fenced: false, LAC: - b3: e0 Fenced: false, LAC: - Now w1 can write a new entry, e1, and b2 & b3 would both acknowledge it, even though the end of the ledger is e0. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
With regards, Apache Git Services
