Hello,
We have a problem with slurmdbd. Everytime we start slurmdbd it dies:
# /etc/init.d/slurmdbd restart ; ps -ef |grep slurm
stopping slurmdbd: [FALLÓ]
Se está ejecutando slurmdbd (pid 2942)...
Se está ejecutando slurmdbd (pid 2942)...
Se está ejecutando slurmdbd (pid 2942)...
Se está ejecutando slurmdbd (pid 2942)...
starting slurmdbd: [ OK ]
root 6607 1 0 Mar26 pts/0
00:00:00 /bin/sh /usr/bin/mysqld_safe
--datadir=/usr/local/slurm-2.4.3/mysql/
--socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid
--basedir=/usr --user=mysql
mysql 6793 6607 41 Mar26 pts/0 07:42:21 /usr/libexec/mysqld
--basedir=/usr --datadir=/usr/local/slurm-2.4.3/mysql/ --user=mysql
--log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/lib/mysql/mysql.sock
slurm 9652 1 0 08:46 ?
00:00:00 /usr/local/slurm-2.4.3/sbin/slurmdbd
root 9657 29357 0 08:46 pts/0 00:00:00 grep slurm
slurm 29567 1 1 Mar26 ?
00:19:52 /usr/local/slurm-2.4.3/sbin/slurmctld
After two seconds slurmdbd does not exist:
# ps -ef |grep slurm
root 6607 1 0 Mar26 pts/0
00:00:00 /bin/sh /usr/bin/mysqld_safe
--datadir=/usr/local/slurm-2.4.3/mysql/
--socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid
--basedir=/usr --user=mysql
mysql 6793 6607 41 Mar26 pts/0 07:42:30 /usr/libexec/mysqld
--basedir=/usr --datadir=/usr/local/slurm-2.4.3/mysql/ --user=mysql
--log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/lib/mysql/mysql.sock
root 16920 29357 0 08:46 pts/0 00:00:00 grep slurm
slurm 29567 1 1 Mar26 ?
00:19:52 /usr/local/slurm-2.4.3/sbin/slurmctld
Last week our system was working without any problems, but at this time
we are not able to start slurmdbd.
In the log file we have:
[2013-03-27T08:46:03] slurmdbd version 2.4.3 started
[2013-03-27T08:46:10] error: couldn't suspend job 51029
[2013-03-27T08:46:10] error: couldn't suspend job 50361
[2013-03-27T08:46:10] error: couldn't suspend job 50763
[2013-03-27T08:46:10] error: couldn't suspend job 50764
(... --> 19 jobs)
and nothing else.
The queue system is working as usual:
$ squeue |wc -l
5037
But we are not able to work with sacctmgr, ...
Do you have any idea about what is happening?
Thanks in advance,
Pablo Sanz
--
Pablo Sanz Mercado
Director Técnico
Centro de Computación Científica
Universidad Autónoma de Madrid
