Hi, On Tue, Jan 08, 2008 at 05:17:49PM +0100, Franck Ganachaud wrote: > 2 nodes cluster configuration with a VIP managed by heartbeat and a set of > ressources. > Heartbeat version 2.0.8 running on RHEL 4 update 4 > > I have the following problem, at first, everything runs ok and after some > time of working fine (can be a few days), I got the OCF tests getting > zombified waiting for lrmd that doesn't react anymore. > I can't stop heartbeat anymore, I can't get the control back and I have to > kill -9 all the heartbeat processes to restart and get a sane state. > > You can find attached the outout of the ps command at the time and the logs > from heartbeat. > You can also find the output of the syslog (message) when those defunct > processes are detected. > > Do you guys have an idea how to avoid this problem
Please upgrade to 2.1.3. It'd be really hard to say what's going on with the version you're running. The lrmd in the meantime got a thorough overhaul. Thanks, Dejan > > -- > GANACHAUD Franck > Consultant > Tel. : +33 (0)2 98 05 43 21 > http://www.altran.com > -- > Technop?le Brest Iroise > Site du Vernis - CS 23866 > 29238 Brest Cedex 3 - France > Tel. : +33 (0)2 98 05 43 21 > Fax. : +33 (0)2 98 05 20 34 > e-mail: [EMAIL PROTECTED] > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0 4744 556 ? S Nov15 0:01 init [3] > > root 2 0.0 0.0 0 0 ? S Nov15 0:03 [migration/0] > root 3 0.0 0.0 0 0 ? SN Nov15 0:05 [ksoftirqd/0] > root 4 0.0 0.0 0 0 ? S Nov15 0:03 [migration/1] > root 5 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/1] > root 6 0.0 0.0 0 0 ? S Nov15 0:03 [migration/2] > root 7 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/2] > root 8 0.0 0.0 0 0 ? S Nov15 0:03 [migration/3] > root 9 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/3] > root 10 0.0 0.0 0 0 ? S< Nov15 0:00 [events/0] > root 11 0.0 0.0 0 0 ? S< Nov15 0:00 [events/1] > root 12 0.0 0.0 0 0 ? S< Nov15 0:00 [events/2] > root 13 0.0 0.0 0 0 ? S< Nov15 0:00 [events/3] > root 14 0.0 0.0 0 0 ? S< Nov15 0:00 [khelper] > root 15 0.0 0.0 0 0 ? S< Nov15 0:00 [kacpid] > root 41 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/0] > root 42 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/1] > root 43 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/2] > root 44 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/3] > root 45 0.0 0.0 0 0 ? S Nov15 0:00 [khubd] > root 62 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/0] > root 61 0.0 0.0 0 0 ? S Nov15 1:12 [kswapd0] > root 63 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/1] > root 64 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/2] > root 65 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/3] > root 138 0.0 0.0 0 0 ? S Nov15 0:00 [kseriod] > root 199 0.0 0.0 0 0 ? S Nov15 0:00 [scsi_eh_0] > root 214 0.0 0.0 0 0 ? S Nov15 1:20 [kjournald] > root 1015 0.0 0.0 3596 444 ? S<s Nov15 0:00 udevd > root 1230 0.0 0.0 0 0 ? S Nov15 0:00 [kjournald] > root 1231 0.0 0.0 0 0 ? S Nov15 3:44 [kjournald] > root 1232 0.0 0.0 0 0 ? S Nov15 0:24 [kjournald] > root 1233 0.0 0.0 0 0 ? S Nov15 1:22 [kjournald] > root 1762 0.0 0.0 3620 664 ? Ss Nov15 1:07 syslogd -m 0 > root 1766 0.0 0.0 2528 492 ? Ss Nov15 0:00 klogd -x > root 1777 0.0 0.0 2524 500 ? Ss Nov15 0:04 irqbalance > rpc 1795 0.0 0.0 4724 608 ? Ss Nov15 0:00 portmap > root 1951 0.0 0.0 2532 552 ? Ss Nov15 0:00 /usr/sbin/acpid > ntp 2076 0.0 0.1 17492 5248 ? SLs Nov15 0:06 ntpd -u > ntp:ntp -p /var/run/ntpd.pid > root 2186 0.0 0.0 4168 536 ? Ss Nov15 0:00 gpm -m > /dev/input/mice -t imps2 > root 2909 0.0 0.0 46460 816 ? Ss Nov15 0:00 crond > xfs 2944 0.0 0.0 10172 1648 ? Ss Nov15 0:00 xfs -droppriv > -daemon > daemon 2973 0.0 0.0 4680 660 ? Ss Nov15 0:00 /usr/sbin/atd > dbus 2997 0.0 0.0 8548 1204 ? Ss Nov15 0:00 dbus-daemon-1 > --system > root 3007 0.0 0.1 18700 6236 ? Ss Nov15 0:11 hald > root 3016 0.0 0.0 25396 1220 ? Ss Nov15 0:00 login -- root > > root 3017 0.0 0.0 2512 416 tty2 Ss+ Nov15 0:00 /sbin/mingetty > tty2 > root 3018 0.0 0.0 2512 416 tty3 Ss+ Nov15 0:00 /sbin/mingetty > tty3 > root 3019 0.0 0.0 2512 416 tty4 Ss+ Nov15 0:00 /sbin/mingetty > tty4 > root 3020 0.0 0.0 2512 416 tty5 Ss+ Nov15 0:00 /sbin/mingetty > tty5 > root 3021 0.0 0.0 2512 416 tty6 Ss+ Nov15 0:00 /sbin/mingetty > tty6 > root 3689 0.0 0.0 45216 1616 tty1 Ss+ Nov15 0:00 -bash > root 20967 0.0 0.0 48100 1228 ? S Nov19 0:00 vi exportStats > root 4964 0.0 0.0 0 0 ? S Nov21 0:04 [pdflush] > root 26042 0.0 0.0 44004 1060 ? S Dec17 0:03 /bin/sh > ./scout4zombies.sh > root 18308 0.0 0.0 11884 664 ? Sl Dec19 0:00 > /usr/software/systemsupervision/wrapper-linux-x86-32 > /usr/software/systemsupervision/conf/systemsupervisiond.conf > wrapper.syslog.ident=systemsupervision > wrapper.pidfile=/etc/rc.d/init.d/./systemsupervision.pid > wrapper.daemonize=TRUE wrapper.lockfile=/var/lock/subsys/systemsupervision > root 18310 0.0 1.2 294960 49916 ? Sl Dec19 0:00 > /usr/local/jdk/jre/bin/java -Xmx64m -Djava.library.path=./lib -classpath > ./lib/wrapper.jar:./systemsupervision-full.jar -Dwrapper.key=QmLFu9kBb76Ti6vr > -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 > -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=18308 -Dwrapper.version=3.2.3 > -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE > -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 > org.tanukisoftware.wrapper.WrapperSimpleApp > com.software.proserv.businesscontact.supervision.redhat.main.Main > root 28197 0.0 0.0 20816 2000 ? Ss Dec19 0:01 /usr/sbin/sshd > root 29667 0.0 0.0 0 0 ? S Dec19 0:02 [pdflush] > userm 2801 0.0 1.7 1311708 71012 ? Ssl Dec20 0:00 > /usr/local/jdk/jre/bin/java -cp > /usr/local/software/supervisionMysql/etc:/usr/local/software/supervisionMysql/lib/software-supervision-mysql-full.jar > com.software.proserv.software.supervisionmysql.SupervisionMain > root 8182 0.0 0.0 45076 1308 ? S Dec20 0:00 /bin/sh > /usr/local/mysql/bin/mysqld_safe --datadir=/opt/mysql > --pid-file=/opt/mysql/ptrimd01.pid > mysql 8214 0.0 7.7 658300 313168 ? Sl Dec20 0:01 > /usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql > --datadir=/opt/mysql --user=mysql --pid-file=/opt/mysql/ptrimd01.pid > --skip-external-locking --socket=/tmp/mysql.sock > root 8206 0.0 0.0 19696 1380 ? S Dec21 0:01 ha_logd: read > process > root 8207 0.0 0.0 19696 1328 ? S Dec21 0:00 ha_logd: write > process > root 8249 0.0 0.3 34496 12580 ? SLs Dec21 0:02 heartbeat: > master control process > nobody 8252 0.0 0.1 27260 5828 ? SL Dec21 0:00 heartbeat: > FIFO reader > nobody 8253 0.0 0.1 27388 5956 ? SL Dec21 0:03 heartbeat: > write: ping 192.168.87.254 > nobody 8254 0.0 0.1 27388 5956 ? SL Dec21 0:02 heartbeat: > read: ping 192.168.87.254 > nobody 8255 0.0 0.1 27388 5956 ? SL Dec21 0:00 heartbeat: > write: ucast bond0 > nobody 8256 0.0 0.1 27388 5956 ? SL Dec21 0:01 heartbeat: > read: ucast bond0 > root 8260 0.0 0.0 36844 3024 ? S Dec21 0:00 > /usr/lib/heartbeat/mgmtd -t > 90 8261 0.0 0.0 19184 1416 ? S Dec21 0:00 > /usr/lib64/heartbeat/ccm > 90 8262 0.0 0.1 34548 4212 ? S Dec21 0:00 > /usr/lib64/heartbeat/cib > nobody 8263 0.0 0.0 28488 1992 ? S Dec21 0:05 > /usr/lib64/heartbeat/lrmd -r > nobody 8264 0.0 0.1 22008 4652 ? SL Dec21 0:00 > /usr/lib64/heartbeat/stonithd > 90 8265 0.0 0.0 20920 1468 ? S Dec21 0:00 > /usr/lib64/heartbeat/attrd > 90 8266 0.0 0.0 28368 2432 ? S Dec21 0:00 > /usr/lib64/heartbeat/crmd > root 8587 0.0 0.0 18948 2320 ? Ss Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 8590 0.0 0.0 19084 2504 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 8591 0.0 0.0 19084 2508 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 8592 0.0 0.0 19084 2508 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 8594 0.0 0.0 19084 2504 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 8595 0.0 0.0 19084 2508 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 17975 0.0 0.0 19084 2516 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 17984 0.0 0.0 19084 2520 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 17985 0.0 0.0 19084 2520 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 17989 0.0 0.0 19084 2452 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > daemon 17990 0.0 0.0 19084 2452 ? S Dec21 0:00 > /usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf > root 7971 0.0 0.2 128360 10252 ? Ssl Dec21 0:00 > /usr/local/OB/bin/balancer -ORBconfig /etc/OB/balancer.conf > root 24140 0.0 0.0 4440 840 ? S Dec21 0:00 > /usr/local/nmon/nmon_x86_64_rhel4 -f -s 60 -c 720 > root 6727 0.0 0.0 0 0 ? Z 00:25 0:00 [IPaddr] > <defunct> > root 6735 0.0 0.0 0 0 ? Z 00:25 0:00 [apache] > <defunct> > root 6826 0.0 0.0 0 0 ? Z 00:25 0:00 [crontab] > <defunct> > root 6835 0.0 0.0 0 0 ? Z 00:25 0:00 [IPaddr] > <defunct> > root 6861 0.0 0.0 5436 788 ? R 00:25 0:00 ps aux > Dec 22 00:25:07 ptrimd01 apache[6735]: [6825]: INFO: 00:25:07 > URL:http://192.168.87.100:80/server-status [1,748/1,748] -> "-" [1] > Dec 22 00:25:12 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT: > operation monitor[10] on ocf::IPaddr::IPaddr_Cluster for client 8266, its > parameters: CRM_meta_interval=[5000] ip=[192.168.87.100] > CRM_meta_op_target_rc=[7] CRM_meta_id=[IPaddr_Cluster_mon] > CRM_meta_timeout=[5000] crm_feature_set=[1.0.7] CRM_meta_name=[m. > Dec 22 00:25:12 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM > operation IPaddr_Cluster_monitor_5000 (10) Timed Out (timeout=5000ms) > Dec 22 00:25:13 ptrimd01 crmd: [8266]: info: do_lrm_rsc_op: Performing > op=crontab_4_stop_0 key=24:6:6a979367-5437-4a5a-b02e-33aa065dce7f) > Dec 22 00:25:13 ptrimd01 crmd: [8266]: WARN: process_lrm_event: LRM operation > crontab_4_monitor_120000 (call=16, rc=-2) Cancelled > Dec 22 00:25:13 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 8266, call:30): 0.330.4447 -> 0.330.4448 (ok) > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) Reloading cron daemon configuration: > Dec 22 00:25:13 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 16369, call:3): 0.330.4448 -> 0.330.4449 (ok) > Dec 22 00:25:13 ptrimd01 crond: crond -HUP succeeded > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) [ > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) OK > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) ] > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) > Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: > (crontab_4:stop:stdout) > Dec 22 00:25:13 ptrimd01 cib: [6833]: info: write_cib_contents: Wrote version > 0.330.4449 of the CIB to disk (digest: 5b05cd7acc2eec2fdf563b8501f4333d) > Dec 22 00:25:18 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT: > operation stop[18] on heartbeat::crontab::crontab_4 for client 8266, its > parameters: CRM_meta_op_target_rc=[7] CRM_meta_timeout=[5000] > crm_feature_set=[1.0.7] . > Dec 22 00:25:18 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM > operation crontab_4_stop_0 (18) Timed Out (timeout=5000ms) > Dec 22 00:25:19 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 8266, call:31): 0.330.4449 -> 0.330.4450 (ok) > Dec 22 00:25:19 ptrimd01 cib: [6843]: info: write_cib_contents: Wrote version > 0.330.4450 of the CIB to disk (digest: 045218370b1d381ea48d0aadd2fa7951) > Dec 22 00:25:22 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT: > operation monitor[10] on ocf::IPaddr::IPaddr_Cluster for client 8266, its > parameters: CRM_meta_interval=[5000] ip=[192.168.87.100] > CRM_meta_op_target_rc=[7] CRM_meta_id=[IPaddr_Cluster_mon] > CRM_meta_timeout=[5000] crm_feature_set=[1.0.7] CRM_meta_name=[m. > Dec 22 00:25:22 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM > operation IPaddr_Cluster_monitor_5000 (10) Timed Out (timeout=5000ms) > Dec 22 00:25:23 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 8266, call:32): 0.330.4450 -> 0.330.4451 (ok) > Dec 22 00:25:23 ptrimd01 cib: [6852]: info: write_cib_contents: Wrote version > 0.330.4451 of the CIB to disk (digest: 711f833687063eec16e1e0c8cb5b5904) > Dec 22 00:25:37 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT: > operation monitor[12] on ocf::apache::apache_2 for client 8266, its > parameters: CRM_meta_interval=[30000] CRM_meta_op_target_rc=[7] > CRM_meta_id=[apache_2_mon] CRM_meta_timeout=[30000] crm_feature_set=[1.0.7] > configfile=[/usr/local/apache/conf/httpd.conf]. > Dec 22 00:25:37 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM > operation apache_2_monitor_30000 (12) Timed Out (timeout=30000ms) > Dec 22 00:25:38 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 8266, call:33): 0.330.4451 -> 0.330.4452 (ok) > Dec 22 00:25:38 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client: > 16369, call:5): 0.330.4452 -> 0.330.4453 (ok) > Dec 22 00:25:38 ptrimd01 cib: [6854]: info: write_cib_contents: Wrote version > 0.330.4453 of the CIB to disk (digest: e08ad97399ed32a4c36637ae653a0153) > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
