Re: [ClusterLabs] PostgreSQL HA on EL9
Ah, good catch. FYI, we created a hook for situations like this a while back: resource-agents-deps.target. Which reminds me we really need to document it ... To use it, put a drop-in unit under /etc/systemd/system/resource- agents-deps.target.d/ (any name ending in .conf) with: [Unit] Requires= After= Pacemaker is ordered after resource-agents-deps, so you can use it to start any non-clustered depedencies. On Thu, 2023-09-14 at 15:43 +, Larry G. Mills via Users wrote: > I found my issue with reboots - and it wasn't pacemaker-related at > all. My EL9 test system was different from the EL7 system in that it > hosted the DB on a iSCSI-attached array. During OS shutdown, the > array was being unmounted concurrently with pacemaker shutdown, so it > was not able to cleanly shut down the pgsql resource. I added a > systemd override to make corosync dependent upon, and require, > "remote-fs.target". Everything shuts down cleanly now, as expected. > > Thanks for the suggestions, > > Larry > > > -Original Message- > > From: Users On Behalf Of Oyvind > > Albrigtsen > > Sent: Thursday, September 14, 2023 5:43 AM > > To: Cluster Labs - All topics related to open-source clustering > > welcomed > > > > Subject: Re: [ClusterLabs] PostgreSQL HA on EL9 > > > > If you're using network filesystems with the Filesystem agent this > > patch might solve your issue: > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__github.com_ClusterLabs_resource- > > 2Dagents_pull_1869=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=- > > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=vEhk79BWO > > NaF8zrTI3oGbq7xqEYdQUICm-2H3Wal0J8= > > > > > > Oyvind > > > > On 13/09/23 17:56 +, Larry G. Mills via Users wrote: > > > > On my RHEL 9 test cluster, both "reboot" and "systemctl reboot" > > > > wait > > > > for the cluster to stop everything. > > > > > > > > I think in some environments "reboot" is equivalent to > > > > "systemctl > > > > reboot --force" (kill all processes immediately), so maybe see > > > > if > > > > "systemctl reboot" is better. > > > > > > > > > On EL7, this scenario caused the cluster to shut itself down > > > > > on the > > > > > node before the OS shutdown completed, and the DB resource > > > > > was > > > > > stopped/shutdown before the OS stopped. On EL9, this is not > > > > > the > > > > > case, the DB resource is not stopped before the OS shutdown > > > > > completes. This leads to errors being thrown when the > > > > > cluster is > > > > > started back up on the rebooted node similar to the > > > > > following: > > > > > > > > > > > Ken, > > > > > > Thanks for the reply - and that's interesting that RHEL9 behaves > > > as expected > > and AL9 seemingly doesn't. I did try shutting down via "systemctl > > reboot", > > but the cluster and resources were still not stopped cleanly before > > the OS > > stopped. In fact, the commands "shutdown" and "reboot" are just > > symlinks > > to systemctl on AL9.2, so that make sense why the behavior is the > > same. > > > Just as a point of reference, my systemd version is: > > > systemd.x86_64 > > 252-14.el9_2.3 > > > Larry > > > ___ > > > Manage your subscription: > > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__lists.clusterlabs.org_mailman_listinfo_users=DwICAg=gRgGjJ3 > > BkIsb > > 5y6s49QqsA=- > > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=2Rx_74MVv > > kAWfZLyMhZw5GCY_37uyRffB2HV4_zkvOY= > > > ClusterLabs home: > > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__www.clusterlabs.org_=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=- > > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=lofFF14IrTG > > 21epUbKbV0oUl-IrXZDSuNcaM1GM7FvU= > > > > ___ > > Manage your subscription: > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__lists.clusterlabs.org_mailman_listinfo_users=DwICAg=gRgGjJ3 > > BkIsb > > 5y6s49QqsA=
Re: [ClusterLabs] PostgreSQL HA on EL9
I found my issue with reboots - and it wasn't pacemaker-related at all. My EL9 test system was different from the EL7 system in that it hosted the DB on a iSCSI-attached array. During OS shutdown, the array was being unmounted concurrently with pacemaker shutdown, so it was not able to cleanly shut down the pgsql resource. I added a systemd override to make corosync dependent upon, and require, "remote-fs.target". Everything shuts down cleanly now, as expected. Thanks for the suggestions, Larry > -Original Message- > From: Users On Behalf Of Oyvind Albrigtsen > Sent: Thursday, September 14, 2023 5:43 AM > To: Cluster Labs - All topics related to open-source clustering welcomed > > Subject: Re: [ClusterLabs] PostgreSQL HA on EL9 > > If you're using network filesystems with the Filesystem agent this > patch might solve your issue: > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_ClusterLabs_resource- > 2Dagents_pull_1869=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=- > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=vEhk79BWO > NaF8zrTI3oGbq7xqEYdQUICm-2H3Wal0J8= > > > Oyvind > > On 13/09/23 17:56 +, Larry G. Mills via Users wrote: > >> > >> On my RHEL 9 test cluster, both "reboot" and "systemctl reboot" wait > >> for the cluster to stop everything. > >> > >> I think in some environments "reboot" is equivalent to "systemctl > >> reboot --force" (kill all processes immediately), so maybe see if > >> "systemctl reboot" is better. > >> > >> > > >> > On EL7, this scenario caused the cluster to shut itself down on the > >> > node before the OS shutdown completed, and the DB resource was > >> > stopped/shutdown before the OS stopped. On EL9, this is not the > >> > case, the DB resource is not stopped before the OS shutdown > >> > completes. This leads to errors being thrown when the cluster is > >> > started back up on the rebooted node similar to the following: > >> > > > > >Ken, > > > >Thanks for the reply - and that's interesting that RHEL9 behaves as expected > and AL9 seemingly doesn't. I did try shutting down via "systemctl reboot", > but the cluster and resources were still not stopped cleanly before the OS > stopped. In fact, the commands "shutdown" and "reboot" are just symlinks > to systemctl on AL9.2, so that make sense why the behavior is the same. > > > >Just as a point of reference, my systemd version is: systemd.x86_64 > 252-14.el9_2.3 > > > >Larry > >___ > >Manage your subscription: > >https://urldefense.proofpoint.com/v2/url?u=https- > 3A__lists.clusterlabs.org_mailman_listinfo_users=DwICAg=gRgGjJ3BkIsb > 5y6s49QqsA=- > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=2Rx_74MVv > kAWfZLyMhZw5GCY_37uyRffB2HV4_zkvOY= > > > >ClusterLabs home: https://urldefense.proofpoint.com/v2/url?u=https- > 3A__www.clusterlabs.org_=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=- > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=lofFF14IrTG > 21epUbKbV0oUl-IrXZDSuNcaM1GM7FvU= > > > > ___ > Manage your subscription: > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__lists.clusterlabs.org_mailman_listinfo_users=DwICAg=gRgGjJ3BkIsb > 5y6s49QqsA=- > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=2Rx_74MVv > kAWfZLyMhZw5GCY_37uyRffB2HV4_zkvOY= > > ClusterLabs home: https://urldefense.proofpoint.com/v2/url?u=https- > 3A__www.clusterlabs.org_=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=- > 46XreMySVoZzxM8t8YcpIX4ayXVWYLvAe0EnGHidNE=VO4147YbENDjp3d > xoJeWclZ_EfLrehCht5CgW4_stkgPmryQN0kBA6G12wBwYztD=lofFF14IrTG > 21epUbKbV0oUl-IrXZDSuNcaM1GM7FvU= ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] PostgreSQL HA on EL9
If you're using network filesystems with the Filesystem agent this patch might solve your issue: https://github.com/ClusterLabs/resource-agents/pull/1869 Oyvind On 13/09/23 17:56 +, Larry G. Mills via Users wrote: On my RHEL 9 test cluster, both "reboot" and "systemctl reboot" wait for the cluster to stop everything. I think in some environments "reboot" is equivalent to "systemctl reboot --force" (kill all processes immediately), so maybe see if "systemctl reboot" is better. > > On EL7, this scenario caused the cluster to shut itself down on the > node before the OS shutdown completed, and the DB resource was > stopped/shutdown before the OS stopped. On EL9, this is not the > case, the DB resource is not stopped before the OS shutdown > completes. This leads to errors being thrown when the cluster is > started back up on the rebooted node similar to the following: > Ken, Thanks for the reply - and that's interesting that RHEL9 behaves as expected and AL9 seemingly doesn't. I did try shutting down via "systemctl reboot", but the cluster and resources were still not stopped cleanly before the OS stopped. In fact, the commands "shutdown" and "reboot" are just symlinks to systemctl on AL9.2, so that make sense why the behavior is the same. Just as a point of reference, my systemd version is: systemd.x86_64 252-14.el9_2.3 Larry ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] PostgreSQL HA on EL9
> > On my RHEL 9 test cluster, both "reboot" and "systemctl reboot" wait > for the cluster to stop everything. > > I think in some environments "reboot" is equivalent to "systemctl > reboot --force" (kill all processes immediately), so maybe see if > "systemctl reboot" is better. > > > > > On EL7, this scenario caused the cluster to shut itself down on the > > node before the OS shutdown completed, and the DB resource was > > stopped/shutdown before the OS stopped. On EL9, this is not the > > case, the DB resource is not stopped before the OS shutdown > > completes. This leads to errors being thrown when the cluster is > > started back up on the rebooted node similar to the following: > > Ken, Thanks for the reply - and that's interesting that RHEL9 behaves as expected and AL9 seemingly doesn't. I did try shutting down via "systemctl reboot", but the cluster and resources were still not stopped cleanly before the OS stopped. In fact, the commands "shutdown" and "reboot" are just symlinks to systemctl on AL9.2, so that make sense why the behavior is the same. Just as a point of reference, my systemd version is: systemd.x86_64 252-14.el9_2.3 Larry ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] PostgreSQL HA on EL9
On Wed, 2023-09-13 at 16:45 +, Larry G. Mills via Users wrote: > Hello Pacemaker community, > > I have several two-node postgres 14 clusters that I am migrating from > EL7 (Scientific Linux 7) to EL9 (AlmaLinux 9.2). > > My configuration: > > Cluster size: two nodes > Postgres version: 14 > Corosync version: 3.1.7-1.el9 > Pacemaker version: 2.1.5-9.el9_2 > pcs version: 0.11.4-7.el9_2 > > The migration has mostly gone smoothly, but I did notice one non- > trivial change in recovery behavior between EL7 and EL9. The > recovery scenario is: > > With the cluster running normally with one primary DB (i.e. Promoted) > and one standby (i.e. Unpromoted), reboot one of the cluster nodes > without first shutting down the cluster on that node. The reboot is > a “clean” system shutdown done via either the “reboot” or “shutdown” > OS commands. On my RHEL 9 test cluster, both "reboot" and "systemctl reboot" wait for the cluster to stop everything. I think in some environments "reboot" is equivalent to "systemctl reboot --force" (kill all processes immediately), so maybe see if "systemctl reboot" is better. > > On EL7, this scenario caused the cluster to shut itself down on the > node before the OS shutdown completed, and the DB resource was > stopped/shutdown before the OS stopped. On EL9, this is not the > case, the DB resource is not stopped before the OS shutdown > completes. This leads to errors being thrown when the cluster is > started back up on the rebooted node similar to the following: > > * pgsql probe on mynode returned 'error' (Instance "pgsql" > controldata indicates a running secondary instance, the instance has > probably crashed) > > While this is not too serious for a standby DB instance, as the > cluster is able to recover it back to the standby/Unpromoted state, > if you reboot the Primary/Promoted DB node, the cluster is not able > to recover it (because that DB still thinks it’s a primary), and the > node is fenced. > > Is this an intended behavior for the versions of pacemaker/corosync > that I’m running, or a regression? It may be possible to put an > override into the systemd unit file for corosync to force the cluster > to shutdown before the OS stops, but I’d rather not do that if > there’s a better way to handle this recovery scenario. > > Thanks for any advice, > > Larry -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] PostgreSQL HA on EL9
Hello Pacemaker community, I have several two-node postgres 14 clusters that I am migrating from EL7 (Scientific Linux 7) to EL9 (AlmaLinux 9.2). My configuration: Cluster size: two nodes Postgres version: 14 Corosync version: 3.1.7-1.el9 Pacemaker version: 2.1.5-9.el9_2 pcs version: 0.11.4-7.el9_2 The migration has mostly gone smoothly, but I did notice one non-trivial change in recovery behavior between EL7 and EL9. The recovery scenario is: With the cluster running normally with one primary DB (i.e. Promoted) and one standby (i.e. Unpromoted), reboot one of the cluster nodes without first shutting down the cluster on that node. The reboot is a "clean" system shutdown done via either the "reboot" or "shutdown" OS commands. On EL7, this scenario caused the cluster to shut itself down on the node before the OS shutdown completed, and the DB resource was stopped/shutdown before the OS stopped. On EL9, this is not the case, the DB resource is not stopped before the OS shutdown completes. This leads to errors being thrown when the cluster is started back up on the rebooted node similar to the following: * pgsql probe on mynode returned 'error' (Instance "pgsql" controldata indicates a running secondary instance, the instance has probably crashed) While this is not too serious for a standby DB instance, as the cluster is able to recover it back to the standby/Unpromoted state, if you reboot the Primary/Promoted DB node, the cluster is not able to recover it (because that DB still thinks it's a primary), and the node is fenced. Is this an intended behavior for the versions of pacemaker/corosync that I'm running, or a regression? It may be possible to put an override into the systemd unit file for corosync to force the cluster to shutdown before the OS stops, but I'd rather not do that if there's a better way to handle this recovery scenario. Thanks for any advice, Larry ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
