2 nodes cluster configuration with a VIP managed by heartbeat and a set
of ressources.
Heartbeat version 2.0.8 running on RHEL 4 update 4
I have the following problem, at first, everything runs ok and after
some time of working fine (can be a few days), I got the OCF tests
getting zombified waiting for lrmd that doesn't react anymore.
I can't stop heartbeat anymore, I can't get the control back and I have
to kill -9 all the heartbeat processes to restart and get a sane state.
You can find attached the outout of the ps command at the time and the
logs from heartbeat.
You can also find the output of the syslog (message) when those defunct
processes are detected.
Do you guys have an idea how to avoid this problem
--
GANACHAUD Franck
Consultant
Tel. : +33 (0)2 98 05 43 21
http://www.altran.com
--
Technopôle Brest Iroise
Site du Vernis - CS 23866
29238 Brest Cedex 3 - France
Tel. : +33 (0)2 98 05 43 21
Fax. : +33 (0)2 98 05 20 34
e-mail: [EMAIL PROTECTED]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4744 556 ? S Nov15 0:01 init [3]
root 2 0.0 0.0 0 0 ? S Nov15 0:03 [migration/0]
root 3 0.0 0.0 0 0 ? SN Nov15 0:05 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Nov15 0:03 [migration/1]
root 5 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/1]
root 6 0.0 0.0 0 0 ? S Nov15 0:03 [migration/2]
root 7 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/2]
root 8 0.0 0.0 0 0 ? S Nov15 0:03 [migration/3]
root 9 0.0 0.0 0 0 ? SN Nov15 0:00 [ksoftirqd/3]
root 10 0.0 0.0 0 0 ? S< Nov15 0:00 [events/0]
root 11 0.0 0.0 0 0 ? S< Nov15 0:00 [events/1]
root 12 0.0 0.0 0 0 ? S< Nov15 0:00 [events/2]
root 13 0.0 0.0 0 0 ? S< Nov15 0:00 [events/3]
root 14 0.0 0.0 0 0 ? S< Nov15 0:00 [khelper]
root 15 0.0 0.0 0 0 ? S< Nov15 0:00 [kacpid]
root 41 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/0]
root 42 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/1]
root 43 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/2]
root 44 0.0 0.0 0 0 ? S< Nov15 0:00 [kblockd/3]
root 45 0.0 0.0 0 0 ? S Nov15 0:00 [khubd]
root 62 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/0]
root 61 0.0 0.0 0 0 ? S Nov15 1:12 [kswapd0]
root 63 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/1]
root 64 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/2]
root 65 0.0 0.0 0 0 ? S< Nov15 0:00 [aio/3]
root 138 0.0 0.0 0 0 ? S Nov15 0:00 [kseriod]
root 199 0.0 0.0 0 0 ? S Nov15 0:00 [scsi_eh_0]
root 214 0.0 0.0 0 0 ? S Nov15 1:20 [kjournald]
root 1015 0.0 0.0 3596 444 ? S<s Nov15 0:00 udevd
root 1230 0.0 0.0 0 0 ? S Nov15 0:00 [kjournald]
root 1231 0.0 0.0 0 0 ? S Nov15 3:44 [kjournald]
root 1232 0.0 0.0 0 0 ? S Nov15 0:24 [kjournald]
root 1233 0.0 0.0 0 0 ? S Nov15 1:22 [kjournald]
root 1762 0.0 0.0 3620 664 ? Ss Nov15 1:07 syslogd -m 0
root 1766 0.0 0.0 2528 492 ? Ss Nov15 0:00 klogd -x
root 1777 0.0 0.0 2524 500 ? Ss Nov15 0:04 irqbalance
rpc 1795 0.0 0.0 4724 608 ? Ss Nov15 0:00 portmap
root 1951 0.0 0.0 2532 552 ? Ss Nov15 0:00 /usr/sbin/acpid
ntp 2076 0.0 0.1 17492 5248 ? SLs Nov15 0:06 ntpd -u ntp:ntp
-p /var/run/ntpd.pid
root 2186 0.0 0.0 4168 536 ? Ss Nov15 0:00 gpm -m
/dev/input/mice -t imps2
root 2909 0.0 0.0 46460 816 ? Ss Nov15 0:00 crond
xfs 2944 0.0 0.0 10172 1648 ? Ss Nov15 0:00 xfs -droppriv
-daemon
daemon 2973 0.0 0.0 4680 660 ? Ss Nov15 0:00 /usr/sbin/atd
dbus 2997 0.0 0.0 8548 1204 ? Ss Nov15 0:00 dbus-daemon-1
--system
root 3007 0.0 0.1 18700 6236 ? Ss Nov15 0:11 hald
root 3016 0.0 0.0 25396 1220 ? Ss Nov15 0:00 login -- root
root 3017 0.0 0.0 2512 416 tty2 Ss+ Nov15 0:00 /sbin/mingetty
tty2
root 3018 0.0 0.0 2512 416 tty3 Ss+ Nov15 0:00 /sbin/mingetty
tty3
root 3019 0.0 0.0 2512 416 tty4 Ss+ Nov15 0:00 /sbin/mingetty
tty4
root 3020 0.0 0.0 2512 416 tty5 Ss+ Nov15 0:00 /sbin/mingetty
tty5
root 3021 0.0 0.0 2512 416 tty6 Ss+ Nov15 0:00 /sbin/mingetty
tty6
root 3689 0.0 0.0 45216 1616 tty1 Ss+ Nov15 0:00 -bash
root 20967 0.0 0.0 48100 1228 ? S Nov19 0:00 vi exportStats
root 4964 0.0 0.0 0 0 ? S Nov21 0:04 [pdflush]
root 26042 0.0 0.0 44004 1060 ? S Dec17 0:03 /bin/sh
./scout4zombies.sh
root 18308 0.0 0.0 11884 664 ? Sl Dec19 0:00
/usr/software/systemsupervision/wrapper-linux-x86-32
/usr/software/systemsupervision/conf/systemsupervisiond.conf
wrapper.syslog.ident=systemsupervision
wrapper.pidfile=/etc/rc.d/init.d/./systemsupervision.pid wrapper.daemonize=TRUE
wrapper.lockfile=/var/lock/subsys/systemsupervision
root 18310 0.0 1.2 294960 49916 ? Sl Dec19 0:00
/usr/local/jdk/jre/bin/java -Xmx64m -Djava.library.path=./lib -classpath
./lib/wrapper.jar:./systemsupervision-full.jar -Dwrapper.key=QmLFu9kBb76Ti6vr
-Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999
-Dwrapper.pid=18308 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper
-Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
org.tanukisoftware.wrapper.WrapperSimpleApp
com.software.proserv.businesscontact.supervision.redhat.main.Main
root 28197 0.0 0.0 20816 2000 ? Ss Dec19 0:01 /usr/sbin/sshd
root 29667 0.0 0.0 0 0 ? S Dec19 0:02 [pdflush]
userm 2801 0.0 1.7 1311708 71012 ? Ssl Dec20 0:00
/usr/local/jdk/jre/bin/java -cp
/usr/local/software/supervisionMysql/etc:/usr/local/software/supervisionMysql/lib/software-supervision-mysql-full.jar
com.software.proserv.software.supervisionmysql.SupervisionMain
root 8182 0.0 0.0 45076 1308 ? S Dec20 0:00 /bin/sh
/usr/local/mysql/bin/mysqld_safe --datadir=/opt/mysql
--pid-file=/opt/mysql/ptrimd01.pid
mysql 8214 0.0 7.7 658300 313168 ? Sl Dec20 0:01
/usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/opt/mysql
--user=mysql --pid-file=/opt/mysql/ptrimd01.pid --skip-external-locking
--socket=/tmp/mysql.sock
root 8206 0.0 0.0 19696 1380 ? S Dec21 0:01 ha_logd: read
process
root 8207 0.0 0.0 19696 1328 ? S Dec21 0:00 ha_logd: write
process
root 8249 0.0 0.3 34496 12580 ? SLs Dec21 0:02 heartbeat:
master control process
nobody 8252 0.0 0.1 27260 5828 ? SL Dec21 0:00 heartbeat: FIFO
reader
nobody 8253 0.0 0.1 27388 5956 ? SL Dec21 0:03 heartbeat:
write: ping 192.168.87.254
nobody 8254 0.0 0.1 27388 5956 ? SL Dec21 0:02 heartbeat: read:
ping 192.168.87.254
nobody 8255 0.0 0.1 27388 5956 ? SL Dec21 0:00 heartbeat:
write: ucast bond0
nobody 8256 0.0 0.1 27388 5956 ? SL Dec21 0:01 heartbeat: read:
ucast bond0
root 8260 0.0 0.0 36844 3024 ? S Dec21 0:00
/usr/lib/heartbeat/mgmtd -t
90 8261 0.0 0.0 19184 1416 ? S Dec21 0:00
/usr/lib64/heartbeat/ccm
90 8262 0.0 0.1 34548 4212 ? S Dec21 0:00
/usr/lib64/heartbeat/cib
nobody 8263 0.0 0.0 28488 1992 ? S Dec21 0:05
/usr/lib64/heartbeat/lrmd -r
nobody 8264 0.0 0.1 22008 4652 ? SL Dec21 0:00
/usr/lib64/heartbeat/stonithd
90 8265 0.0 0.0 20920 1468 ? S Dec21 0:00
/usr/lib64/heartbeat/attrd
90 8266 0.0 0.0 28368 2432 ? S Dec21 0:00
/usr/lib64/heartbeat/crmd
root 8587 0.0 0.0 18948 2320 ? Ss Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 8590 0.0 0.0 19084 2504 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 8591 0.0 0.0 19084 2508 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 8592 0.0 0.0 19084 2508 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 8594 0.0 0.0 19084 2504 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 8595 0.0 0.0 19084 2508 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 17975 0.0 0.0 19084 2516 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 17984 0.0 0.0 19084 2520 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 17985 0.0 0.0 19084 2520 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 17989 0.0 0.0 19084 2452 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
daemon 17990 0.0 0.0 19084 2452 ? S Dec21 0:00
/usr/local/apache/bin/httpd -DSTATUS -f /usr/local/apache/conf/httpd.conf
root 7971 0.0 0.2 128360 10252 ? Ssl Dec21 0:00
/usr/local/OB/bin/balancer -ORBconfig /etc/OB/balancer.conf
root 24140 0.0 0.0 4440 840 ? S Dec21 0:00
/usr/local/nmon/nmon_x86_64_rhel4 -f -s 60 -c 720
root 6727 0.0 0.0 0 0 ? Z 00:25 0:00 [IPaddr]
<defunct>
root 6735 0.0 0.0 0 0 ? Z 00:25 0:00 [apache]
<defunct>
root 6826 0.0 0.0 0 0 ? Z 00:25 0:00 [crontab]
<defunct>
root 6835 0.0 0.0 0 0 ? Z 00:25 0:00 [IPaddr]
<defunct>
root 6861 0.0 0.0 5436 788 ? R 00:25 0:00 ps auxDec 22 00:25:07 ptrimd01 apache[6735]: [6825]: INFO: 00:25:07
URL:http://192.168.87.100:80/server-status [1,748/1,748] -> "-" [1]
Dec 22 00:25:12 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT:
operation monitor[10] on ocf::IPaddr::IPaddr_Cluster for client 8266, its
parameters: CRM_meta_interval=[5000] ip=[192.168.87.100]
CRM_meta_op_target_rc=[7] CRM_meta_id=[IPaddr_Cluster_mon]
CRM_meta_timeout=[5000] crm_feature_set=[1.0.7] CRM_meta_name=[m.
Dec 22 00:25:12 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM operation
IPaddr_Cluster_monitor_5000 (10) Timed Out (timeout=5000ms)
Dec 22 00:25:13 ptrimd01 crmd: [8266]: info: do_lrm_rsc_op: Performing
op=crontab_4_stop_0 key=24:6:6a979367-5437-4a5a-b02e-33aa065dce7f)
Dec 22 00:25:13 ptrimd01 crmd: [8266]: WARN: process_lrm_event: LRM operation
crontab_4_monitor_120000 (call=16, rc=-2) Cancelled
Dec 22 00:25:13 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
8266, call:30): 0.330.4447 -> 0.330.4448 (ok)
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
Reloading cron daemon configuration:
Dec 22 00:25:13 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
16369, call:3): 0.330.4448 -> 0.330.4449 (ok)
Dec 22 00:25:13 ptrimd01 crond: crond -HUP succeeded
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
[
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
OK
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
]
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
Dec 22 00:25:13 ptrimd01 lrmd: [8263]: info: RA output: (crontab_4:stop:stdout)
Dec 22 00:25:13 ptrimd01 cib: [6833]: info: write_cib_contents: Wrote version
0.330.4449 of the CIB to disk (digest: 5b05cd7acc2eec2fdf563b8501f4333d)
Dec 22 00:25:18 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT:
operation stop[18] on heartbeat::crontab::crontab_4 for client 8266, its
parameters: CRM_meta_op_target_rc=[7] CRM_meta_timeout=[5000]
crm_feature_set=[1.0.7] .
Dec 22 00:25:18 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM operation
crontab_4_stop_0 (18) Timed Out (timeout=5000ms)
Dec 22 00:25:19 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
8266, call:31): 0.330.4449 -> 0.330.4450 (ok)
Dec 22 00:25:19 ptrimd01 cib: [6843]: info: write_cib_contents: Wrote version
0.330.4450 of the CIB to disk (digest: 045218370b1d381ea48d0aadd2fa7951)
Dec 22 00:25:22 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT:
operation monitor[10] on ocf::IPaddr::IPaddr_Cluster for client 8266, its
parameters: CRM_meta_interval=[5000] ip=[192.168.87.100]
CRM_meta_op_target_rc=[7] CRM_meta_id=[IPaddr_Cluster_mon]
CRM_meta_timeout=[5000] crm_feature_set=[1.0.7] CRM_meta_name=[m.
Dec 22 00:25:22 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM operation
IPaddr_Cluster_monitor_5000 (10) Timed Out (timeout=5000ms)
Dec 22 00:25:23 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
8266, call:32): 0.330.4450 -> 0.330.4451 (ok)
Dec 22 00:25:23 ptrimd01 cib: [6852]: info: write_cib_contents: Wrote version
0.330.4451 of the CIB to disk (digest: 711f833687063eec16e1e0c8cb5b5904)
Dec 22 00:25:37 ptrimd01 lrmd: [8263]: WARN: on_op_timeout_expired: TIMEOUT:
operation monitor[12] on ocf::apache::apache_2 for client 8266, its parameters:
CRM_meta_interval=[30000] CRM_meta_op_target_rc=[7] CRM_meta_id=[apache_2_mon]
CRM_meta_timeout=[30000] crm_feature_set=[1.0.7]
configfile=[/usr/local/apache/conf/httpd.conf].
Dec 22 00:25:37 ptrimd01 crmd: [8266]: ERROR: process_lrm_event: LRM operation
apache_2_monitor_30000 (12) Timed Out (timeout=30000ms)
Dec 22 00:25:38 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
8266, call:33): 0.330.4451 -> 0.330.4452 (ok)
Dec 22 00:25:38 ptrimd01 cib: [8262]: info: cib_diff_notify: Update (client:
16369, call:5): 0.330.4452 -> 0.330.4453 (ok)
Dec 22 00:25:38 ptrimd01 cib: [6854]: info: write_cib_contents: Wrote version
0.330.4453 of the CIB to disk (digest: e08ad97399ed32a4c36637ae653a0153)
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
