[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
This bug was fixed in the package fence-agents - 4.0.25-2ubuntu1.2 --- fence-agents (4.0.25-2ubuntu1.2) bionic; urgency=medium * fence_aws backport from Focal (LP: #1894323): + d/p/lp1894323-01-fence_aws-new-agent.patch -- Rafael David Tinoco Thu, 22 Oct 2020 04:47:00 + ** Changed in: fence-agents (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
I verified the test case using the package available in bionic-proposed and I confirm it is working as expected. I set up a 3 nodes cluster on AWS to test this. Note: When installing fence-agents also install the Suggested dependencies, otherwise the 'fence_aws' command will not work. ubuntu@node1:~$ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic ubuntu@node1:~$ dpkg -l | grep fence-agents ii fence-agents 4.0.25-2ubuntu1.2 amd64 Fence Agents for Red Hat Cluster ubuntu@node1:~$ sudo crm configure show node 1: node1 node 2: node2 node 3: node3 primitive fence-node1 stonith:fence_aws \ params access_key= secret_key="" region=us-east-2 plug=i-093f875f9f2ffa1db pcmk_host_map="node1:i-093f875f9f2ffa1db;node2:i-08649fdfb0a74bc9f;node3:i-0394f790feeba28b0" primitive fence-node2 stonith:fence_aws \ params access_key= secret_key="" region=us-east-2 plug=i-08649fdfb0a74bc9f pcmk_host_map="node1:i-093f875f9f2ffa1db;node2:i-08649fdfb0a74bc9f;node3:i-0394f790feeba28b0" primitive fence-node3 stonith:fence_aws \ params access_key= secret_key="" region=us-east-2 plug=i-0394f790feeba28b0 pcmk_host_map="node1:i-093f875f9f2ffa1db;node2:i-08649fdfb0a74bc9f;node3:i-0394f790feeba28b0" location l-fence-node1 fence-node1 -inf: node1 location l-fence-node2 fence-node2 -inf: node2 location l-fence-node3 fence-node3 -inf: node3 property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ cluster-name=clubionic \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop If I go to node2 and run the following command to reject connections from the network interface in use the node is properly fenced (in this case rebooted): ubuntu@node2:~$ sudo iptables -A INPUT -i eth0 -j REJECT After some minutes the node2 gets back online. I also tested it without pacemaker in a standalone mode. I ran the following command to do that: ubunt@node3:~$ sudo fence_aws --plug= --action=reboot --region=us-east-2 --access-key="xxx" --secret-key="xxx" --verbose ** Tags removed: verification-needed verification-needed-bionic ** Tags added: verification-done verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
Hello Rafael, or anyone else affected, Accepted fence-agents into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fence- agents/4.0.25-2ubuntu1.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: fence-agents (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
fence-agents (4.0.25-2ubuntu1.2) bionic; urgency=medium * fence_aws backport from Focal (LP: #1894323): + d/p/lp1894323-01-fence_aws-new-agent.patch -- Rafael David Tinoco Thu, 22 Oct 2020 04:47:00 + [rafaeldtinoco@bionic fence-agents]$ git ubuntu tag --upload [rafaeldtinoco@bionic fence-agents]$ git describe upload/4.0.25-2ubuntu1.2 [rafaeldtinoco@bionic fence-agents]$ git push pkg upload/4.0.25-2ubuntu1.2 Counting objects: 15, done. Delta compression using up to 24 threads. Compressing objects: 100% (15/15), done. Writing objects: 100% (15/15), 4.49 KiB | 460.00 KiB/s, done. Total 15 (delta 10), reused 0 (delta 0) To ssh://git.launchpad.net/ubuntu/+source/fence-agents * [new tag] upload/4.0.25-2ubuntu1.2 -> upload/4.0.25-2ubuntu1.2 [rafaeldtinoco@bionic ubuntu]$ debdiff fence-agents_4.0.25-2ubuntu1.1.dsc fence-agents_4.0.25-2ubuntu1.2.dsc | diffstat changelog |7 + control|5 - patches/lp1894323-01-fence_aws-new-agent.patch | 286 + patches/series |1 4 files changed, 298 insertions(+), 1 deletion(-) [rafaeldtinoco@bionic ubuntu]$ dput ubuntu fence-agents_4.0.25-2ubuntu1.2_source.changes Uploading to ubuntu (via ftp to upload.ubuntu.com): Uploading fence-agents_4.0.25-2ubuntu1.2.dsc: done. Uploading fence-agents_4.0.25-2ubuntu1.2.debian.tar.xz: done. Uploading fence-agents_4.0.25-2ubuntu1.2_source.buildinfo: done. Uploading fence-agents_4.0.25-2ubuntu1.2_source.changes: done. Successfully uploaded packages. Note: the fence_aws agent primitive should be declared as: primitive fence-bionic stonith:fence_aws \ params access_key= secret_key="" region=us-east-1 pcmk_host_map="bionic01:i-068e134de1beddc7f;bionic02:i-0136eddd045ceb7e2;bionic03:i-0de279ab4e6d642c8" and cluster properties as: have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=bionic crm configure might complain about you not specifying the "plug" argument, you can safely ignore that as this fence_agent does not require the plug argument (and this pacemaker version has an issue when plug is given, see comment #11 for more information). ** Changed in: fence-agents (Ubuntu Focal) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Description changed: + SRU reviewer: I got a complex backport for Focal thinking about + introducing a better version of fence_aws in Bionic but I abandoned the + idea and created a simple 1 patch fence_aws backport to Bionic (so it is + like Focal, and not both be like Groovy). + [Impact] * Currently Ubuntu Bionic does not have fence_aws available and that is needed in order to have a fully working HA solution in AWS environment. * fence_aws from Focal fence-agents (4.5.2-1) is missing some fixes that happened in between Focal and Groovy versions. With that, I initially opted to fully bring all the fixes from version 4.6.0 to Focal and backport this same version to Bionic (this way Bionic and Focal were in the same level). * After MR reviews, thinking about the SRU review, I agreed to minimize this change making Ubuntu Bionic fence_aws agent just like focal and working in any needed Focal fix for fence_aws (as long as there was a test case for it). [Test Case] * Provision 3 nodes in AWS with Ubuntu Focal (and Ubuntu Bionic) and configure it adding the following primitive as a fencing resource: primitive fence-focal stonith:fence_aws params access_key="" secret_key="" region="us-east-1" pcmk_host_map="focal01:i-034dc89cca4310b03;focal02:i-0a160b14b40f1330a;focal03:i-03b6976ab0a7f377c" and the cluster cib options: property cib-bootstrap-options: \ have-watchdog=false \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=bionic * After that you can remove the interconnect of one of the nodes and watch the cluster to shutdown the node that was disconnected from the cluster ring. [Regression Potential] * Bionic wise: - Same as focal, but here it is even better situation as there isn't an existing fence_aws agent. Biggest problem here could be introduce something that does not fully work (which is technically not a regression). [Other Info] This is a request from AWS to backport existing fence_aws agent into Bionic - Currently fence-agents Ubuntu Bionic version is at: 4.0.25-2ubuntu1 and the fence_aws new agent started at: $ git tag --contains a3f45322 | head -1 v4.1.0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Description changed: [Impact] - * Currently Ubuntu Bionic does not have fence_aws available and that is + * Currently Ubuntu Bionic does not have fence_aws available and that is needed in order to have a fully working HA solution in AWS environment. - * fence_aws from Focal fence-agents (4.5.2-1) is missing some important - fixes that happened in between Focal and Groovy versions. With that, I - opted to fully bring all the fixes from version 4.6.0 to Focal and - backport this same version to Bionic (this way Bionic and Focal are in - the same level also). + * fence_aws from Focal fence-agents (4.5.2-1) is missing some fixes + that happened in between Focal and Groovy versions. With that, I + initially opted to fully bring all the fixes from version 4.6.0 to Focal + and backport this same version to Bionic (this way Bionic and Focal were + in the same level). + + * After MR reviews, thinking about the SRU review, I agreed to minimize + this change making Ubuntu Bionic fence_aws agent just like focal and + working in any needed Focal fix for fence_aws (as long as there was a + test case for it). [Test Case] - * Provision 3 nodes in AWS with Ubuntu Focal (and Ubuntu Bionic) and + * Provision 3 nodes in AWS with Ubuntu Focal (and Ubuntu Bionic) and configure it adding the following primitive as a fencing resource: primitive fence-focal stonith:fence_aws params access_key="" secret_key="" region="us-east-1" pcmk_host_map="focal01:i-034dc89cca4310b03;focal02:i-0a160b14b40f1330a;focal03:i-03b6976ab0a7f377c" and the cluster cib options: property cib-bootstrap-options: \ - have-watchdog=false \ - cluster-infrastructure=corosync \ - stonith-enabled=on \ - stonith-action=reboot \ - no-quorum-policy=stop \ - cluster-name=bionic + have-watchdog=false \ + cluster-infrastructure=corosync \ + stonith-enabled=on \ + stonith-action=reboot \ + no-quorum-policy=stop \ + cluster-name=bionic - * After that you can remove the interconnect of one of the nodes and + * After that you can remove the interconnect of one of the nodes and watch the cluster to shutdown the node that was disconnected from the cluster ring. [Regression Potential] - * Focal wise: + * Bionic wise: -- We're only touching agents/aws/fence_aws.py and - tests/data/metadata/fence_aws.xml files. This reduces *a lot* the scope - of the code change and helps in the SRU approval (as the change is - confined into the aws fence agent only). - -- Biggest problem here could be related to rebuilds, as the fence- - agents are directly called from pacemaker as "exec'ed scripts with right - arguments", and there is no change to any part of the fence-agents core - package. - - * Bionic wise: - -- Same as focal, but here it is even better situation as there isn't + - Same as focal, but here it is even better situation as there isn't an existing fence_aws agent. Biggest problem here could be introduce something that does not fully work (which is technically not a regression). [Other Info] - This is a request from AWS to backport existing fence_aws agent into - Bionic: - - - commit 50772024 - Author: Oyvind Albrigtsen - Date: Mon May 25 12:07:14 2020 - - fence_aws: improve boto3_debug boolean handling - - commit be206158 - Author: Oyvind Albrigtsen - Date: Mon May 25 12:03:53 2020 - - fence_aws: catch ConnectionError and suppress traceback for caught - exceptions - - commit 9758f8c8 - Author: Oyvind Albrigtsen - Date: Tue Mar 24 14:31:13 2020 - - fence_aws: fix Python 3 encoding issue - - commit 3f5676a7 - Author: gguifelixamz <45173771+gguifelix...@users.noreply.github.com> - Date: Fri Mar 20 14:41:07 2020 - - fence_aws: Fix fence race, logging improvement and new debug option - (#323) - - * fence_aws: Fix fence race condition by checking local instance status - * fence_aws: Use local logger and improve logging experience - * fence_aws: Decouple boto3 and botocore debug logging from local logging - - commit 1c2f791b - Author: Oyvind Albrigtsen + This is a request from AWS to backport existing fence_aws agent into Bionic: + @redhat.com> Date: Thu Mar 5 13:10:29 2020 fence_aws: improve connect parameter logic, so region can be specified as parameter, while using role or keys from ~/.aws/config commit 7ac16fb2 Author: Oyvind Albrigtsen Date: Fri Jan 31 14:53:47 2020 fence_aws: improve logging and metadata/usage text commit a3f45322 Author: Oyvind Albrigtsen Date: Tue Mar 7 10:04:49 2017 fence_aws: new agent Currently fence-agents Ubuntu Bionic version is at: 4.0.25-2ubuntu1 and the fence_aws new agent started at: $ git tag --contains a3f45322 | head -1 v4.1.0 ** Description changed: [Impact] * Currently Ubuntu Bionic does not have fence_aws available and that is
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
To help with SRU review, I'm adding here the discussion taken from the merge review.. basically I would like to give it a try backporting all fixes for fence_aws into Focal and adding that same agent version in Bionic (like a minor SRU exception). Instead of relying in the test case results, I would like to rely in my functional/regression tests for a pacemaker cluster configured in AWS with this new agent (if possible). Discussion with @bryce from the Ubuntu Server Team: > I also read through each of the patches to understand what they do, and make sure the changes look safe, which they indeed do. One thought I had though is that the common theme in the patches is improvements to debug/logging output; the SRU team sometimes demurs over debug/logging changes as less important than actual bug fixes. At least you'll want to include good justification on this in the SRU text. My justification to that is that Bionic does not have anything and I really would like Focal to be "as good as Focal", instead of adding something better in Bionic just because it did not have anything. Or even adding something not as good as Groovy just because of formal reasons. > Commit 1c2f791b changes the cli option behavior, which is akin to an API change. I.e. before if you passed --region but not --access-key or --secret-key it would ignore --region and use configured values, with this change you can specify just --region and the keys will come from the config file. This feels more like a behavioral change than a bug fix, so I might anticipate some pushback from the SRU team on this. Yes, this was per AWS request... and follows the same idea as the previous justification. This change allows one not to explicitly put the access or secret keys in the cluster CIB file (so its more secure also). > In terms of SRU, I notice there are not (upstream|downstream) bug reports associated with the patches, which may make one wonder if these fix actual defects encountered in the wild, or are more like clarification/refactoring. > I understand the logic of since the scripts don't exist in bionic to bring the current versions rather so as to have the most up to date code. But as you mention this then leaves a weird situation and having to pull delta into focal that otherwise might not be needed. > Did you consider pulling the fence_aws from focal rather than the one in groovy? (And then cherrypicking the most relevant bug fixes from groovy, like the encoding fix and/or the race fix?) Yes I did.. unfortunately its a SRU philosophical question. I'm considering fence_aws here as a confined code that is mostly supported by AWS themselves. I can go on that direction but I feel it is not the best for our user base. > Alternatively, if you definitely do want to backport the whole stack, did you consider filing for an SRU exception for this package? If it really is important to keep the scripts identical on all LTS's that might be a better long term approach. That would be a no-go because of agents metadata and pacemaker. Pacemaker should be able to handler older and newer fence-agents packages.. but it is not as good as "compatible with all further versions". I would like this to be considered a small SRU exception as it is for fence_aws only. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Description changed: - This is a request to backport existing fence_aws agent into Bionic: + [Impact] + + * Currently Ubuntu Bionic does not have fence_aws available and that is + needed in order to have a fully working HA solution in AWS environment. + + * fence_aws from Focal fence-agents (4.5.2-1) is missing some important + fixes that happened in between Focal and Groovy versions. With that, I + opted to fully bring all the fixes from version 4.6.0 to Focal and + backport this same version to Bionic (this way Bionic and Focal are in + the same level also). + + [Test Case] + + * Provision 3 nodes in AWS with Ubuntu Focal (and Ubuntu Bionic) and + configure it adding the following primitive as a fencing resource: + + primitive fence-focal stonith:fence_aws params access_key="" secret_key="" + region="us-east-1" pcmk_host_map="focal01:i-034dc89cca4310b03;focal02:i-0a160b14b40f1330a;focal03:i-03b6976ab0a7f377c" + + and the cluster cib options: + + property cib-bootstrap-options: \ + have-watchdog=false \ + cluster-infrastructure=corosync \ + stonith-enabled=on \ + stonith-action=reboot \ + no-quorum-policy=stop \ + cluster-name=bionic + + * After that you can remove the interconnect of one of the nodes and + watch the cluster to shutdown the node that was disconnected from the + cluster ring. + + [Regression Potential] + + * Focal wise: + +- We're only touching agents/aws/fence_aws.py and + tests/data/metadata/fence_aws.xml files. This reduces *a lot* the scope + of the code change and helps in the SRU approval (as the change is + confined into the aws fence agent only). + +- Biggest problem here could be related to rebuilds, as the fence- + agents are directly called from pacemaker as "exec'ed scripts with right + arguments", and there is no change to any part of the fence-agents core + package. + + * Bionic wise: + +- Same as focal, but here it is even better situation as there isn't + an existing fence_aws agent. Biggest problem here could be introduce + something that does not fully work (which is technically not a + regression). + + [Other Info] + + This is a request from AWS to backport existing fence_aws agent into + Bionic: commit 50772024 Author: Oyvind Albrigtsen Date: Mon May 25 12:07:14 2020 - fence_aws: improve boto3_debug boolean handling + fence_aws: improve boto3_debug boolean handling commit be206158 Author: Oyvind Albrigtsen Date: Mon May 25 12:03:53 2020 - fence_aws: catch ConnectionError and suppress traceback for caught + fence_aws: catch ConnectionError and suppress traceback for caught exceptions commit 9758f8c8 Author: Oyvind Albrigtsen Date: Tue Mar 24 14:31:13 2020 - fence_aws: fix Python 3 encoding issue + fence_aws: fix Python 3 encoding issue commit 3f5676a7 Author: gguifelixamz <45173771+gguifelix...@users.noreply.github.com> Date: Fri Mar 20 14:41:07 2020 - fence_aws: Fix fence race, logging improvement and new debug option (#323) - - * fence_aws: Fix fence race condition by checking local instance status - * fence_aws: Use local logger and improve logging experience - * fence_aws: Decouple boto3 and botocore debug logging from local logging + fence_aws: Fix fence race, logging improvement and new debug option + (#323) + + * fence_aws: Fix fence race condition by checking local instance status + * fence_aws: Use local logger and improve logging experience + * fence_aws: Decouple boto3 and botocore debug logging from local logging commit 1c2f791b Author: Oyvind Albrigtsen Date: Thu Mar 5 13:10:29 2020 - fence_aws: improve connect parameter logic, so region can be specified - as parameter, while using role or keys from ~/.aws/config + fence_aws: improve connect parameter logic, so region can be specified + as parameter, while using role or keys from ~/.aws/config commit 7ac16fb2 Author: Oyvind Albrigtsen Date: Fri Jan 31 14:53:47 2020 - fence_aws: improve logging and metadata/usage text + fence_aws: improve logging and metadata/usage text commit a3f45322 Author: Oyvind Albrigtsen Date: Tue Mar 7 10:04:49 2017 - fence_aws: new agent + fence_aws: new agent Currently fence-agents Ubuntu Bionic version is at: 4.0.25-2ubuntu1 and the fence_aws new agent started at: $ git tag --contains a3f45322 | head -1 v4.1.0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Merge proposal unlinked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence-agents/+git/fence-agents/+merge/392432 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence-agents/+git/fence-agents/+merge/392432 ** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence-agents/+git/fence-agents/+merge/392433 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
Everything is good and ready but I have discovered a small issue in the Bionic Backport... and relates to: https://access.redhat.com/solutions/4642491 The story is this: when declaring the fence_aws primitive, you can either declare it as a single resource and describe the pcmk_host_map... OR you can declare one fence resource PER NODE doing the exact same thing BUT using the "plug/port" resource argument. The thing is... in Focal, both methods work but in Bionic, the second method does not work. It is not a big deal as there are some fence agents designed to work with "pcmk_host_map" only, and some others are designed to work with "plug/port" argument... but I have opened the bug: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1900374 to deal with this in a later moment (might require pacemaker bisecting, etc) For this SRU.. after it is complete, the correct way of declaring the fence_aws resource is: # focal node 1: focal01 node 2: focal02 node 3: focal03 primitive fence-focal stonith:fence_aws \ params access_key= secret_key="" region=us-east-1 pcmk_host_map="focal01:i-abcdefgh;focal02:i-ijlmnop;focal03:i-qrstuvxz" property cib-bootstrap-options: \ have-watchdog=false \ dc-version=2.0.3-4b1f869f0f \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=focal # bionic node 1: bionic01 node 2: bionic02 node 3: bionic03 primitive fence-bionic stonith:fence_aws \ params access_key= secret_key="" region=us-east-1 pcmk_host_map="bionic01:i-abcdefgh;bionic02:i-ijlmnop;bionic03:i-qrstuvxz" property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
(how to get fence_aws working with bionic and aws <- keywords for google) # Bionic fence_aws tests with the backport (from the ppa): [rafaeldtinoco@bionic01 ~]$ crm status Stack: corosync Current DC: bionic01 (version 1.1.18-2b07d5c5a9) - partition with quorum Last updated: Thu Oct 15 01:25:05 2020 Last change: Thu Oct 15 01:25:01 2020 by root via cibadmin on bionic01 3 nodes configured 1 resource configured Online: [ bionic01 bionic02 bionic03 ] Full list of resources: fence-bionic (stonith:fence_aws):Started bionic01 and then I fail on purpose the interconnect on node03 (bionic03): [rafaeldtinoco@bionic01 ~]$ crm status Stack: corosync Current DC: bionic01 (version 1.1.18-2b07d5c5a9) - partition with quorum Last updated: Thu Oct 15 01:26:57 2020 Last change: Thu Oct 15 01:25:01 2020 by root via cibadmin on bionic01 3 nodes configured 1 resource configured Node bionic03: UNCLEAN (offline) Online: [ bionic01 bionic02 ] Full list of resources: fence-bionic (stonith:fence_aws):Started bionic01 and the AWS console I could see machine being restarted. After bionic03 was restarted... [rafaeldtinoco@bionic01 ~]$ crm status Stack: corosync Current DC: bionic01 (version 1.1.18-2b07d5c5a9) - partition with quorum Last updated: Thu Oct 15 01:27:01 2020 Last change: Thu Oct 15 01:25:01 2020 by root via cibadmin on bionic01 3 nodes configured 1 resource configured Node bionic03: UNCLEAN (offline) Online: [ bionic01 bionic02 ] Full list of resources: fence-bionic (stonith:fence_aws):Started bionic01 I need to fix something: fence_aws agent requires python-request and python-boto3 libraries to be installed. I will have to put those as fence-agents Recommends. I have also to check a requirement on "plug" argument (as it is not really required for fence_aws to work). The way I declared the primitive was: primitive fence-bionic stonith:fence_aws \ params \ access_key= \ secret_key="" \ region=us-east-1 \ pcmk_host_map="bionic01:i-068e134de1bed;bionic02:i-0136eddd045ce;bionic03:i-0de279ab4e6d6" \ power_timeout=240 \ pcmk_reboot_timeout=480 \ pcmk_reboot_retries=4 property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
Moving on to Bionic backport now... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence-agents/+git/fence-agents/+merge/392225 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
Using my proposed new version: $ dpkg -l | grep -i fence ii fence-agents 4.5.2-1ubuntu1~202010091535 amd64 Fence Agents for Red Hat Cluster [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal03 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 12:11:18 2020 * Last change: Wed Oct 14 04:38:10 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Online: [ focal01 focal02 focal03 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal01 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 12:11:24 2020 * Last change: Wed Oct 14 04:38:10 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Node focal03: UNCLEAN (offline) * Online: [ focal01 focal02 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 Pending Fencing Actions: * reboot of focal03 pending: client=pacemaker-controld.444, origin=focal01 [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal01 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 12:13:59 2020 * Last change: Wed Oct 14 04:38:10 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Online: [ focal01 focal02 focal03 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
(how to get fence_aws working with focal and aws <- keywords for google) # Focal fence_aws tests without any backport (current version): node 1: focal01 node 2: focal02 node 3: focal03 primitive fence-focal stonith:fence_aws \ params access_key= secret_key="" region=us-east-1 pcmk_host_map="focal01:i-034dc89cca431;focal02:i-0a160b14b40f1;focal03:i-03b6976ab0a7f" power_timeout=240 pcmk_reboot_timeout=480 pcmk_reboot_retries=4 property cib-bootstrap-options: \ have-watchdog=false \ dc-version=2.0.3-4b1f869f0f \ cluster-infrastructure=corosync \ stonith-enabled=on \ stonith-action=reboot \ no-quorum-policy=stop \ cluster-name=focal Fencing worked as expected: [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal01 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 04:29:34 2020 * Last change: Wed Oct 14 04:28:57 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Online: [ focal01 focal02 focal03 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal01 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 04:31:16 2020 * Last change: Wed Oct 14 04:28:57 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Node focal03: UNCLEAN (offline) * Online: [ focal01 focal02 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 Pending Fencing Actions: * reboot of focal03 pending: client=pacemaker-controld.3585, origin=focal01 [rafaeldtinoco@focal01 ~]$ crm status Cluster Summary: * Stack: corosync * Current DC: focal01 (version 2.0.3-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 14 04:32:14 2020 * Last change: Wed Oct 14 04:28:57 2020 by root via cibadmin on focal01 * 3 nodes configured * 1 resource instance configured Node List: * Online: [ focal01 focal02 focal03 ] Full List of Resources: * fence-focal (stonith:fence_aws): Started focal01 And EC2 AWS console showed instance being shutdown and restarting appropriately. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Changed in: fence-agents (Ubuntu Bionic) Status: Confirmed => In Progress ** Changed in: fence-agents (Ubuntu Focal) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
I'm also providing a PPA containing a full v4.6.0 fence_aws backport to Ubuntu Bionic fence-agents package at: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1894323-bionic with source at: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence- agents/+git/fence-agents/+ref/lp1894323-bionic-v4.6.0-backport/ Like explained in the previous comment, Bionic fence-agents package does not have fence_aws... so I have backported not only the fence_aws agent inclusion from Focal (just 1 commit), but also all the fixes up to Groovy (several fixes were provided from Focal to Groovy). I'm going to test both and as for a SRU exception considering this as an "enablement". -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
I'm providing a PPA containing a full v4.6.0 fence_aws backport to Ubuntu Focal fence-agents package at: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1894323-focal with source at: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence- agents/+git/fence-agents/+ref/lp1894323-focal-v4.6.0-backport/ Because of the nature of fence-agents, where they're practically isolated scripts in the form of a metadata <-> python script pair, I think it would be okay to SRU a particular agent to the latest Ubuntu version. All regression risk is confined to the agent itself and it would be easy to fix/revert if ever needed (without jeopardizing those not relying in the agent). For example, Groovy has fence-agents v4.6.0... containing all the patches needed for a good fence_aws support. I'm backporting all fixes from Groovy to Focal in this PPA. Focal had only the initial fence_aws patch, but missing all the fixes from v4.5.2 to v4.6.0. Bionic is missing all commits, including the fence_aws agent. I'm going to backport everything - the agent and fixes - to Bionic.. but it would be weird to have a more updated agent in Bionic (since it does not have the agent, it would be able to have the agent introduction with all fixes) than in Focal (that is why this SRU tries to keep both, Focal and Bionic, in the same codelevel for fence_aws). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
I see that AWS has 2 fencing mechanisms: - fence_aws using boto3 library (in fence-agents) - fence_ec2 (in cluster-glue) Bug: https://bugs.launchpad.net/ubuntu/+source/cluster-glue/+bug/1895355 has brought fence_ec2 support to our cluster-glue package by backporting needed patches. Perhaps that should also be checked for backport (backporting fence_ec2 to Focal and Bionic if possible). I opened the following bug for this: https://bugs.launchpad.net/ubuntu/+source/cluster-glue/+bug/1896696 ** Changed in: fence-agents (Ubuntu Bionic) Assignee: Rafael David Tinoco (rafaeldtinoco) => (unassigned) ** Changed in: fence-agents (Ubuntu Bionic) Importance: Undecided => Wishlist ** Changed in: fence-agents (Ubuntu Focal) Importance: Undecided => Wishlist -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
In bug: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894325 I have synced v4.6.0-1 recently from Debian, and it includes fence_aws and fence_ibmz: - fence_aws agent being backported to Bionic (LP: #1894323) - Add LPAR fence agent to Pacemaker (LP: #1889070) So in this bug we should backport fence_aws to Focal and Bionic, if possible. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
** Merge proposal unlinked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/fence-agents/+git/fence-agents/+merge/390322 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1894323] Re: Add fence_aws fencing from v4.6.0 agent to Bionic
I'm marking Focal as confirmed because it can catch up with fixes made for version in v4.6.0 so Bionic and Focal can have the same version/features for fence_aws. ** Summary changed: - Add fence_aws fencing agent to Bionic (4.0.25-2ubuntu1) + Add fence_aws fencing from v4.6.0 agent to Bionic ** Changed in: fence-agents (Ubuntu Focal) Status: Fix Released => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894323 Title: Add fence_aws fencing from v4.6.0 agent to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894323/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
