You need the "pcmk_reboot_action=off" for the fencing device ? Also, keep in mind that the fenced node will loose any reservations and pacemaker will shut down . You can use a workaround if you want to convert it from storage-based fencing to power fencing (there is a script that integrates with a watchdog service to reboot the node that was fenced).
Best Regards, Strahil Nikolov В сряда, 28 октомври 2020 г., 14:18:05 Гринуич+2, Patrick Vranckx <[email protected]> написа: Hi, I try yo setup an HA cluster for ZFS. I think fence_scsi is not working properly. I can reproduce the problem on two kind of hardware: iSCSI and SAS storage. Here is what I did: - set up a storage server with 3 iscsi targets to be accessed by my 2-nodes cluster - set up two nodes with centos 8.1. iscsi initiators let me see three scsi disks on each node: /dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca -> ../../sda /dev/disk/by-id/wwn-0x6001405a38892587b3f4a739046b9ff4 -> ../../sdb /dev/disk/by-id/wwn-0x6001405b249db0e617b41b19f3147af5 -> ../../sdc - configure zpool "vol1" on one node - defined resources on the cluster: pcs resource create vol1 ZFS pool="vol1" op start timeout="90" op stop timeout="90" --group=group-vol1 pcs resource create vol1-ip IPaddr2 ip=10.30.0.100 cidr_netmask=16 --group group-vol1 pcs stonith create fence-vol1 fence_scsi pcmk_monitor_action="metadata" pcmk_host_list="node1,node2" devices="/dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca,/dev/disk/by-id/wwn-0x6001405a38892587b3f4a739046b9ff4,/dev/disk/by-id/wwn-0x6001405b249db0e617b41b19f3147af5" meta provides=unfencing Everything seems ok: [root@node1 /]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.3-5.el8_2.1-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 28 12:55:52 2020 * Last change: Wed Oct 28 12:32:36 2020 by root via crm_resource on node2 * 2 nodes configured * 3 resource instances configured Node List: * Online: [ node1 node2 ] Full List of Resources: * fence-vol1 (stonith:fence_scsi): Started node1 * Resource Group: group-vol1: * vol1-ip (ocf::heartbeat:IPaddr2): Started node2 * vol1 (ocf::heartbeat:ZFS): Started node2 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Moving the resource to the other node runs fine and the zpool is only imported on the active node. SCSI reservations for scsi disks are as follows: LIO-ORG disk1 4.0 Peripheral device type: disk PR generation=0x2 Key=0x6c9d0000 All target ports bit clear Relative port address: 0x1 << Reservation holder >> scope: LU_SCOPE, type: Write Exclusive, registrants only Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node1 Key=0x6c9d0001 All target ports bit clear Relative port address: 0x1 not reservation holder Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node2 QUESTION : when I move the resource to the other node, SCSI reservations do not change. I would expect that the Write Exclusive reservation would follow the active node. Is it normal ? If I fence manually the active node for group-vol1: [root@node1 /]# pcs stonith fence node1 Node: node1 fenced the resource is restarted on the other node BUT THE ZFS FILESTYSTEMS ARE MOUNTED AND WRITEABLE ON BOTH NODES even if the scsi reservations seem ok: [root@node2 by-id]# sg_persist -s /dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca LIO-ORG disk1 4.0 Peripheral device type: disk PR generation=0x3 Key=0x6c9d0001 All target ports bit clear Relative port address: 0x1 << Reservation holder >> scope: LU_SCOPE, type: Write Exclusive, registrants only Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node2 I can reproduce the same thing with two node cluster with SAS attached storage running centos 7.8. In my understanding, if node2 has write exclusive reservation, node1 shouldn't be able to write on those disks. What's the problem ? Thanks for your help, Patrick _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
