[Bug 1312156] Re: [Precise] Potential for data corruption
This bug was fixed in the package pacemaker - 1.1.6-2ubuntu3.3 --- pacemaker (1.1.6-2ubuntu3.3) precise; urgency=high * Removed buggy logic that tried to prevent rare secondary dc fencing (LP: #1312156) -- Rafael David Tinoco rafael.tin...@canonical.com Fri, 02 May 2014 15:47:36 -0500 ** Changed in: pacemaker (Ubuntu Precise) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Changed in: pacemaker (Ubuntu Precise) Assignee: Rafael David Tinoco (inaddy) = (unassigned) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
Brian, I've made several tests on this and everything works like expected. Changing tag. Thanks ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
Hello Peter, or anyone else affected, Accepted pacemaker into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/pacemaker/1.1.6-2ubuntu3.3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: pacemaker (Ubuntu Precise) Status: In Progress = Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Branch linked: lp:ubuntu/precise-proposed/pacemaker -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
Uploaded -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
Here is the patch fixing corosync misbehavior described above. Description: Remove buggy logic to prevent secondary dc fencing On logic before commit 82aa2d8d17 the node responsible for fencing (executioner) the dc was responsible also for updating cib. If this update failed (due to a executioner fail, for ex) the dc would be fenced a second time because the cluster would not know about fencing result. On upstream commit 82aa2d8d17, a logic trying to avoid this second dc fencing was introduced. If this node was not the dc fence executioner it would keep its name. With its name, in the case executioner node died and this node became the new dc it would be able to update cib telling the result of last dc fencing. Problem is that this list is never cleaned and there might be cases wrong cib update is given (when a dc takeover has to run) resulting in a bad, bad thing: same resource running on different nodes. It is much more acceptable for SRU to restore old behavior, known to be safe even if it implies killing dc twice, than to backport several pieces of code to implement a logic that was not there on the stable version release. ** Patch added: precise_pacemaker_1.1.6-2ubuntu3.3.diff https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+attachment/4103593/+files/precise_pacemaker_1.1.6-2ubuntu3.3.diff -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Also affects: pacemaker (Ubuntu Precise) Importance: Undecided Status: New ** Changed in: pacemaker (Ubuntu Precise) Assignee: (unassigned) = Rafael David Tinoco (inaddy) ** Changed in: pacemaker (Ubuntu) Assignee: Rafael David Tinoco (inaddy) = (unassigned) ** Changed in: pacemaker (Ubuntu) Status: In Progress = Fix Released ** Changed in: pacemaker (Ubuntu Precise) Status: New = In Progress ** Changed in: pacemaker (Ubuntu Precise) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Description changed: + [Impact] + + * Pacemaker designated controller can make wrong decisions based on + uncleared node status on a rare specific situation. This situation can + make the same resource starts on two nodes at the same time, resulting + in data corruption. + + [Test Case] + + * The bug trigger is very hard hard to achieve: + + 1) If stonith was successful on fencing a node (any node was fenced). + 2) If the target and origin are the same (node killed itself). + 3) If we do not have a dc or the fenced node is our dc (our dc killed itself). + 4) If the executor is not this node (at least 3 nodes). + 5) If this node is elected new DC anytime in the future. + 7) If a policy engine was not yet scheduled. + 8) If takeover runs before policy engine. + + * The bug couldn't be reproduced so far: the patch was made based on a + community report (https://www.mail- + archive.com/pacema...@oss.clusterlabs.org/msg19509.html) analyzed by + upstream code developer (Andrew Beekhof). + + [Regression Potential] + + * On logic before commit 82aa2d8d17 the node responsible for fencing + (executioner) the dc was responsible also for updating cib. If this + update failed (due to a executioner fail, for ex) the dc would be fenced + a second time because the cluster would not know about fencing result. + On upstream commit 82aa2d8d17, a logic trying to avoid this second dc + fencing was introduced. This logic by itself is buggy. + + * To minimize any kind of regression, instead of going forward on + pacemaker versions, it was decided to go backwards removing only this + piece of code. + + * It is much more acceptable for SRU to restore old behavior, known to + be safe even if it implies killing dc twice, than to backport several + pieces of code to implement a logic that was not there on the stable + version release. + + [Other Info / Original Description] + Under certain conditions there is faulty logic in function tengine_stonith_notify() which can incorrectly add successfully fenced nodes to a list, causing Pacemaker to subsequently erase that node’s status section when the next DC (Designated Controller) election occurs. With the status section erased, the cluster considers that node is down and starts corresponding services on other nodes. Multiple instances of the same service can cause data corruption. Conditions: 1. fenced node must have been the previous DC and been sufficiently functional to request its own fencing 2. fencing notification must arrive after the new DC has been elected but before it invokes the policy engine Pacemaker versions affected: 1.1.6 - 1.1.9 Stable Ubuntu releases affected: Ubuntu 12.04 LTS Ubuntu 12.10 (EOL?) Fix: https://github.com/ClusterLabs/pacemaker/commit/f30e1e43 References: https://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg19509.html http://blog.clusterlabs.org/blog/2014/potential-for-data-corruption-in-pacemaker-1-dot-1-6-through-1-dot-1-9/ -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Changed in: pacemaker (Ubuntu) Assignee: (unassigned) = Rafael David Tinoco (inaddy) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1312156] Re: [Precise] Potential for data corruption
** Changed in: pacemaker (Ubuntu) Status: New = In Progress -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1312156 Title: [Precise] Potential for data corruption To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1312156/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
