> -----Original Message----- > From: Users <users-boun...@clusterlabs.org> On Behalf Of Andrei > Borzenkov > Sent: Saturday, November 5, 2022 1:17 AM > To: users@clusterlabs.org > Subject: [External] : Re: [ClusterLabs] Fence Agent tests > > On 04.11.2022 23:46, Robert Hayden wrote: > > I am working on a Fencing agent for the Oracle Cloud Infrastructure (OCI) > environment to complete power fencing of compute instances. The only > fencing setups I have seen for OCI are using SBD, but that is insufficient > with > full network interruptions since OCI uses iSCSI to write/read to the SBD disk. > > > > Out of curiosity - why is it insufficient? If cluster node is completely > isolated, it should commit suicide. If host where cluster node is > running is completely isolated, then you cannot do anything with this > host anyway.
Personally, this was my first attempt with SBD, so I may be missing some core protections. I am more familiar with IPMILAN power fencing. In my testing with full network hang (firewall-cmd panic-on), I was not getting the expected fencing results with SBD like I would with power fencing. Hence, my long overdue learning of python to then attempt taking a crack at writing a fencing agent. In my configuration, I am using HA-LVM (vg tags) to protect XFS file systems. When resources fail over, the file system moves to another node. The OCI compute instances don't have a hardware watchdog, only the software watchdog. So, when the network goes completely hung (e.g. firewall-cmd panic-on), all network traffic stops which implies that IO to the SBD device also stops. I do not see the software watchdog take any action in response to the network hang. The remote node will see the network issue and write out the reset message in the SBD device slot for the hung node to suicide. But the impacted node cannot read that SBD device, so it never gets the message. It just sits. Applications can still run, but they don't have access to the disks either (which is good). In the full network hang, the remote node will wait until 2x SBD msg-timeout and then assumes fencing was successful. It then will attempt move the XFS file systems over. If the network-hung node wakes up, then I now have the XFS file systems mounted on both nodes leading to corruption. This may be eliminated if I moved the HA-LVM setup from the vg_tags to system_id. With vg_tags, pacemaker adds a "pacemaker" tag to all controlled volume groups regardless of the node that has the vg activated. With system_id, the nodes uname is added to the vg metadata so each node knows who officially has the vg activated. I have not played with that scenario in OCI just yet. I am not sure if pacemaker would simply remove the other node's uname and add its own when it attempts to move the resource. It is on my list to test because we moved to uname setup with Linux 8. Again, this was my first attempt with SBD, so I may have it setup completely wrong. > > I am not familiar with OCI architecture so I may be missing something > obvious here. > > > _______________________________________________ > Manage your subscription: > https://urldefense.com/v3/__https://lists.clusterlabs.org/mailman/listinfo/u > sers__;!!ACWV5N9M2RV99hQ!P-TvBc3_Pt- > EGjuAuWw7Fa8vFbMYbE3gi73KUfdyxDBCXFuCWXcbdHNm63_AkgmJ5vhcNX > mcIkMgXSBGaphrfZQ$ > > ClusterLabs home: > https://urldefense.com/v3/__https://www.clusterlabs.org/__;!!ACWV5N9 > M2RV99hQ!P-TvBc3_Pt- > EGjuAuWw7Fa8vFbMYbE3gi73KUfdyxDBCXFuCWXcbdHNm63_AkgmJ5vhcNX > mcIkMgXSBGT2ncT5M$ _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
