Re: [Linux-HA] ocf::LVM monitor needs excessive time to complete
05.08.2011 14:55, Ulrich Windl wrote: Hi, we run a cluster that has about 30 LVM VGs that are monitored every minute with a timeout interval of 90s. Surprisingly even if the system is in nominal state, the LVM monitor times out. I suspect this has to do with multiple LVM commands being run in parallel like this: # ps ax |grep vg 2014 pts/0D+ 0:00 vgs 2580 ?D 0:00 vgdisplay -v NFS_C11_IO 2638 ?D 0:00 vgck CBW_DB_BTD 2992 ?D 0:00 vgdisplay -v C11_DB_Exe 3002 ?D 0:00 vgdisplay -v C11_DB_15k 4564 pts/2S+ 0:00 grep vg # ps ax |grep vg 8095 ?D 0:00 vgck CBW_DB_Exe 8119 ?D 0:00 vgdisplay -v C11_DB_FATA 8194 ?D 0:00 vgdisplay -v NFS_SAP_Exe When I tried a vgs manually, it could not be suspended or killed, and it took more than 30 seconds to complete. You just need to filter unneeded block devices (or leave only needed) from LVM suspects to be PV. Otherwise LVM tries to open every LV to look if it is PV. Look at /etc/lvm/lvm.conf for filter line. BTW under very high CPU/IO load I found that chrt -r 99 helps LVM utils to work much faster. Combining this with timeout utility (to prevent that LVM utils from never-finish) does some more magic. Best, Vladislav ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] ocf::LVM monitor needs excessive time to complete
Hi, we run a cluster that has about 30 LVM VGs that are monitored every minute with a timeout interval of 90s. Surprisingly even if the system is in nominal state, the LVM monitor times out. I suspect this has to do with multiple LVM commands being run in parallel like this: # ps ax |grep vg 2014 pts/0D+ 0:00 vgs 2580 ?D 0:00 vgdisplay -v NFS_C11_IO 2638 ?D 0:00 vgck CBW_DB_BTD 2992 ?D 0:00 vgdisplay -v C11_DB_Exe 3002 ?D 0:00 vgdisplay -v C11_DB_15k 4564 pts/2S+ 0:00 grep vg # ps ax |grep vg 8095 ?D 0:00 vgck CBW_DB_Exe 8119 ?D 0:00 vgdisplay -v C11_DB_FATA 8194 ?D 0:00 vgdisplay -v NFS_SAP_Exe When I tried a vgs manually, it could not be suspended or killed, and it took more than 30 seconds to complete. Thus the LVM monitoring is quite useless as it is now (SLES 11 SP1 x86_64 on a machine with lots of disks, RAM and CPUs). As I had changed all the timeouts via crm configure edit, I suspect the LRM starts all these monitors at the same time, creating massive parallelism. Maybe a random star delay would be more useful than having the user specify a variable start delay for the monitor. Possibly those stuck monitor operations also affect monitors that would finish in time. Here's a part of the mess on one node: Aug 5 13:50:55 h03 lrmd: [14526]: WARN: operation monitor[360] on ocf::LVM::prm_cbw_ci_mnt_lvm for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[3] CRM_meta_interval=[1] volgrpname=[CBW_CI] : pid [29910] timed out Aug 5 13:50:55 h03 crmd: [14529]: ERROR: process_lrm_event: LRM operation prm_cbw_ci_mnt_lvm_monitor_1 (360) Timed Out (timeout=3ms) Aug 5 13:50:55 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[154] on ocf::IPaddr2::prm_a20_ip_1 for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[2] CRM_meta_interval=[1] iflabel=[a20] ip=[172.20.17.54] stayed in operation list for 24020 ms (longer than 1 ms) Aug 5 13:50:56 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[179] on ocf::Raid1::prm_nfs_cbw_trans_raid1 for client 14529, its parameters: CRM_meta_record_pending=[true] raidconf=[/etc/mdadm/mdadm.conf] crm_feature_set=[3.0.5] OCF_CHECK_LEVEL=[1] raiddev=[/dev/md8] CRM_meta_name=[monitor] CRM_meta_timeout=[6] CRM_meta_interval=[6] stayed in operation list for 24010 ms (longer than 1 ms) Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_local_callback: Expanded fail-count-prm_cbw_ci_mnt_lvm=value++ to 9 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-prm_cbw_ci_mnt_lvm (9) Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_perform_update: Sent update 416: fail-count-prm_cbw_ci_mnt_lvm=9 Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] ocf::LVM monitor needs excessive time to complete
Hi, On Fri, Aug 05, 2011 at 01:55:25PM +0200, Ulrich Windl wrote: Hi, we run a cluster that has about 30 LVM VGs that are monitored every minute with a timeout interval of 90s. Surprisingly even if the system is in nominal state, the LVM monitor times out. I suspect this has to do with multiple LVM commands being run in parallel like this: # ps ax |grep vg 2014 pts/0D+ 0:00 vgs 2580 ?D 0:00 vgdisplay -v NFS_C11_IO 2638 ?D 0:00 vgck CBW_DB_BTD 2992 ?D 0:00 vgdisplay -v C11_DB_Exe 3002 ?D 0:00 vgdisplay -v C11_DB_15k 4564 pts/2S+ 0:00 grep vg # ps ax |grep vg 8095 ?D 0:00 vgck CBW_DB_Exe 8119 ?D 0:00 vgdisplay -v C11_DB_FATA 8194 ?D 0:00 vgdisplay -v NFS_SAP_Exe When I tried a vgs manually, it could not be suspended or killed, and it took more than 30 seconds to complete. Thus the LVM monitoring is quite useless as it is now (SLES 11 SP1 x86_64 on a machine with lots of disks, RAM and CPUs). I guess that this is somehow related to the storage. Best to report directly to SUSE. As I had changed all the timeouts via crm configure edit, I suspect the LRM starts all these monitors at the same time, creating massive parallelism. Maybe a random star delay would be more useful than having the user specify a variable start delay for the monitor. Possibly those stuck monitor operations also affect monitors that would finish in time. lrmd starts at most max-children operations in parallel. That's 4 by default. Thanks, Dejan Here's a part of the mess on one node: Aug 5 13:50:55 h03 lrmd: [14526]: WARN: operation monitor[360] on ocf::LVM::prm_cbw_ci_mnt_lvm for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[3] CRM_meta_interval=[1] volgrpname=[CBW_CI] : pid [29910] timed out Aug 5 13:50:55 h03 crmd: [14529]: ERROR: process_lrm_event: LRM operation prm_cbw_ci_mnt_lvm_monitor_1 (360) Timed Out (timeout=3ms) Aug 5 13:50:55 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[154] on ocf::IPaddr2::prm_a20_ip_1 for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[2] CRM_meta_interval=[1] iflabel=[a20] ip=[172.20.17.54] stayed in operation list for 24020 ms (longer than 1 ms) Aug 5 13:50:56 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[179] on ocf::Raid1::prm_nfs_cbw_trans_raid1 for client 14529, its parameters: CRM_meta_record_pending=[true] raidconf=[/etc/mdadm/mdadm.conf] crm_feature_set=[3.0.5] OCF_CHECK_LEVEL=[1] raiddev=[/dev/md8] CRM_meta_name=[monitor] CRM_meta_timeout=[6] CRM_meta_interval=[6] stayed in operation list for 24010 ms (longer than 1 ms) Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_local_callback: Expanded fail-count-prm_cbw_ci_mnt_lvm=value++ to 9 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-prm_cbw_ci_mnt_lvm (9) Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_perform_update: Sent update 416: fail-count-prm_cbw_ci_mnt_lvm=9 Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] ocf::LVM monitor needs excessive time to complete
On 8/5/2011 7:18 AM, Dejan Muhamedagic wrote: Hi, On Fri, Aug 05, 2011 at 01:55:25PM +0200, Ulrich Windl wrote: ... When I tried a vgs manually, it could not be suspended or killed, and it took more than 30 seconds to complete. Thus the LVM monitoring is quite useless as it is now (SLES 11 SP1 x86_64 on a machine with lots of disks, RAM and CPUs). I guess that this is somehow related to the storage. Best to report directly to SUSE. What sort of disks and how many? -- last time we ran out of room, I had to add a different-sized ide disk (smaller, because you couldn't buy a big one anymore) so I had to use lvm. I/O performance went down the drain right away. (That was centos5 a couple of years ago.) Dima (thank Cthulhu for sata and mdadm) ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] ocf::LVM monitor needs excessive time to complete
Hi, processes in state D looks like locked in a kernel call/device request. Do you have a problem with your storage? This is not cluster related . Kind regards Fabian On 08/05/2011 01:55 PM, Ulrich Windl wrote: Hi, we run a cluster that has about 30 LVM VGs that are monitored every minute with a timeout interval of 90s. Surprisingly even if the system is in nominal state, the LVM monitor times out. I suspect this has to do with multiple LVM commands being run in parallel like this: # ps ax |grep vg 2014 pts/0D+ 0:00 vgs 2580 ?D 0:00 vgdisplay -v NFS_C11_IO 2638 ?D 0:00 vgck CBW_DB_BTD 2992 ?D 0:00 vgdisplay -v C11_DB_Exe 3002 ?D 0:00 vgdisplay -v C11_DB_15k 4564 pts/2S+ 0:00 grep vg # ps ax |grep vg 8095 ?D 0:00 vgck CBW_DB_Exe 8119 ?D 0:00 vgdisplay -v C11_DB_FATA 8194 ?D 0:00 vgdisplay -v NFS_SAP_Exe When I tried a vgs manually, it could not be suspended or killed, and it took more than 30 seconds to complete. Thus the LVM monitoring is quite useless as it is now (SLES 11 SP1 x86_64 on a machine with lots of disks, RAM and CPUs). As I had changed all the timeouts via crm configure edit, I suspect the LRM starts all these monitors at the same time, creating massive parallelism. Maybe a random star delay would be more useful than having the user specify a variable start delay for the monitor. Possibly those stuck monitor operations also affect monitors that would finish in time. Here's a part of the mess on one node: Aug 5 13:50:55 h03 lrmd: [14526]: WARN: operation monitor[360] on ocf::LVM::prm_cbw_ci_mnt_lvm for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[3] CRM_meta_interval=[1] volgrpname=[CBW_CI] : pid [29910] timed out Aug 5 13:50:55 h03 crmd: [14529]: ERROR: process_lrm_event: LRM operation prm_cbw_ci_mnt_lvm_monitor_1 (360) Timed Out (timeout=3ms) Aug 5 13:50:55 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[154] on ocf::IPaddr2::prm_a20_ip_1 for client 14529, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.5] CRM_meta_record_pending=[true] CRM_meta_timeout=[2] CRM_meta_interval=[1] iflabel=[a20] ip=[172.20.17.54] stayed in operation list for 24020 ms (longer than 1 ms) Aug 5 13:50:56 h03 lrmd: [14526]: WARN: perform_ra_op: the operation operation monitor[179] on ocf::Raid1::prm_nfs_cbw_trans_raid1 for client 14529, its parameters: CRM_meta_record_pending=[true] raidconf=[/etc/mdadm/mdadm.conf] crm_feature_set=[3.0.5] OCF_CHECK_LEVEL=[1] raiddev=[/dev/md8] CRM_meta_name=[monitor] CRM_meta_timeout=[6] CRM_meta_interval=[6] stayed in operation list for 24010 ms (longer than 1 ms) Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_local_callback: Expanded fail-count-prm_cbw_ci_mnt_lvm=value++ to 9 Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-prm_cbw_ci_mnt_lvm (9) Aug 5 13:50:56 h03 attrd: [14527]: info: attrd_perform_update: Sent update 416: fail-count-prm_cbw_ci_mnt_lvm=9 Aug 5 13:50:56 h03 attrd: [14527]: notice: attrd_ais_dispatch: Update relayed from h04 Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
