I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster services (IP addresses, Postgresql, applications) work fine.
The mysql config file and data files are located on shared, cluster-wide storage (GPFS). On each node, I can successfully start mysql via: service mysqld start and via: rg_test test /etc/cluster/cluster.conf start service mysql (in each case, the corresponding command with the 'stop' option will also successfully shut down mysql). However, attempting to start the mysql service with clusvcadm results in the service failing over from one node to the next, and being marked as "stopped" after the last node. Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long enough for mysql to start before checking if it is running and I have added startup delays in cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this time -- no startup or failure messages. The only log entries (/var/log/messages, /var/log/cluster/*, etc) reference rgmanager, not the mysql process itself. Any suggestions? RHCS components: cman-3.0.12.1-78.el6.x86_64 luci-0.26.0-78.el6.centos.x86_64 rgmanager-3.0.12.1-26.el6_8.3.x86_64 ricci-0.16.2-86.el6.x86_64 --------------------- /etc/cluster/cluster.conf (edited) ----------------- <cluster config_version="63" name="example-rhcs"> <rm> <resources> <postgres-8 config_file="/var/lib/pgsql/data/postgresql.conf" name="PostgreSQL8" postmaster_user="postgres" startup_wait="25"/> <ip address="192.168.169.173" sleeptime="10"/> <mysql config_file="/cluster_shared/mysql_centos6/etc/my.cnf" listen_address="192.168.169.173" name="mysql" shutdown_wait="10" startup_wait="30"/> </resources> <service max_restarts="3" name="mysql" recovery="restart" restart_expire_time="180"> <ip ref="192.168.169.173"> <mysql ref="mysql"/> </ip> </service> </rm> </cluster> -------------------------------------------------------------------------- --------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm ----------------------- Aug 08 11:58:16 rgmanager Recovering failed service service:mysql Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2 Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2 Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql Aug 08 11:58:21 rgmanager 1 events processed Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified) Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1 Aug 08 11:58:21 rgmanager Stopping service service:mysql Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed -------------------------------------------------------------------------------- ------------------------ rg_test output ----------------------------------------- Running in test mode. <debug> Link for eth2: Detected [ip] Link for eth2: Detected <info> Adding IPv4 address 192.168.169.173/24 to eth2 [ip] Adding IPv4 address 192.168.169.173/24 to eth2 <debug> Pinging addr 192.168.169.173 from dev eth2 [ip] Pinging addr 192.168.169.173 from dev eth2 <debug> Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff <debug> Verifying Configuration Of mysql:mysql [mysql] Verifying Configuration Of mysql:mysql <debug> Verifying Configuration Of mysql:mysql > Succeed [mysql] Verifying Configuration Of mysql:mysql > Succeed <debug> Monitoring Service mysql:mysql [mysql] Monitoring Service mysql:mysql <error> Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed <error> Monitoring Service mysql:mysql > Service Is Not Running [mysql] Monitoring Service mysql:mysql > Service Is Not Running <info> Starting Service mysql:mysql [mysql] Starting Service mysql:mysql <debug> Looking For IP Address > Succeed - IP Address Found [mysql] Looking For IP Address > Succeed - IP Address Found <debug> Starting Service mysql:mysql > Succeed [mysql] Starting Service mysql:mysql > Succeed <debug> Monitoring Service mysql:mysql [mysql] Monitoring Service mysql:mysql <debug> Monitoring Service mysql:mysql > Service Is Running [mysql] Monitoring Service mysql:mysql > Service Is Running Loading resource rule from /usr/share/cluster/checkquorum Loading resource rule from /usr/share/cluster/mysql.sh Loading resource rule from /usr/share/cluster/ip.sh Loading resource rule from /usr/share/cluster/SAPDatabase Loading resource rule from /usr/share/cluster/fs.sh Loading resource rule from /usr/share/cluster/fence_scsi_check.pl Loading resource rule from /usr/share/cluster/script.sh Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl Loading resource rule from /usr/share/cluster/samba.sh Loading resource rule from /usr/share/cluster/oradg.sh Loading resource rule from /usr/share/cluster/nfsexport.sh Loading resource rule from /usr/share/cluster/orainstance.sh Loading resource rule from /usr/share/cluster/service.sh Loading resource rule from /usr/share/cluster/postgres-8.sh Loading resource rule from /usr/share/cluster/named.sh Loading resource rule from /usr/share/cluster/bind-mount.sh Loading resource rule from /usr/share/cluster/svclib_nfslock Loading resource rule from /usr/share/cluster/lvm_by_lv.sh Loading resource rule from /usr/share/cluster/ASEHAagent.sh Loading resource rule from /usr/share/cluster/netfs.sh Loading resource rule from /usr/share/cluster/tomcat-6.sh Loading resource rule from /usr/share/cluster/clusterfs.sh Loading resource rule from /usr/share/cluster/lvm.sh Loading resource rule from /usr/share/cluster/openldap.sh Loading resource rule from /usr/share/cluster/vm.sh Loading resource rule from /usr/share/cluster/ocf-shellfuncs Loading resource rule from /usr/share/cluster/lvm_by_vg.sh Loading resource rule from /usr/share/cluster/SAPInstance Loading resource rule from /usr/share/cluster/nfsserver.sh Loading resource rule from /usr/share/cluster/nfsclient.sh Loading resource rule from /usr/share/cluster/apache.sh Loading resource rule from /usr/share/cluster/oracledb.sh Loading resource rule from /usr/share/cluster/oralistener.sh Starting mysql... Start of mysql complete --------------------------------------------------------------------------------- -- Mark Bergman -- Linux-cluster mailing list Linux-cluster@redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster