Hi Pablo, Try to raise the DebugLevel for the slurmdbd daemon at slurmdbd.conf
It should give you more info about what's going on. On 03/27/2013 09:13 AM, Pablo Sanz Mercado wrote: > > Hello, > > We have a problem with slurmdbd. Everytime we start slurmdbd it dies: > > # /etc/init.d/slurmdbd restart ; ps -ef |grep slurm > stopping slurmdbd: [FALLÓ] > Se está ejecutando slurmdbd (pid 2942)... > Se está ejecutando slurmdbd (pid 2942)... > Se está ejecutando slurmdbd (pid 2942)... > Se está ejecutando slurmdbd (pid 2942)... > starting slurmdbd: [ OK ] > > > root 6607 1 0 Mar26 pts/0 > 00:00:00 /bin/sh /usr/bin/mysqld_safe > --datadir=/usr/local/slurm-2.4.3/mysql/ > --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid > --basedir=/usr --user=mysql > mysql 6793 6607 41 Mar26 pts/0 07:42:21 /usr/libexec/mysqld > --basedir=/usr --datadir=/usr/local/slurm-2.4.3/mysql/ --user=mysql > --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid > --socket=/var/lib/mysql/mysql.sock > slurm 9652 1 0 08:46 ? > 00:00:00 /usr/local/slurm-2.4.3/sbin/slurmdbd > root 9657 29357 0 08:46 pts/0 00:00:00 grep slurm > slurm 29567 1 1 Mar26 ? > 00:19:52 /usr/local/slurm-2.4.3/sbin/slurmctld > > After two seconds slurmdbd does not exist: > > # ps -ef |grep slurm > root 6607 1 0 Mar26 pts/0 > 00:00:00 /bin/sh /usr/bin/mysqld_safe > --datadir=/usr/local/slurm-2.4.3/mysql/ > --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid > --basedir=/usr --user=mysql > mysql 6793 6607 41 Mar26 pts/0 07:42:30 /usr/libexec/mysqld > --basedir=/usr --datadir=/usr/local/slurm-2.4.3/mysql/ --user=mysql > --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid > --socket=/var/lib/mysql/mysql.sock > root 16920 29357 0 08:46 pts/0 00:00:00 grep slurm > slurm 29567 1 1 Mar26 ? > 00:19:52 /usr/local/slurm-2.4.3/sbin/slurmctld > > Last week our system was working without any problems, but at this time > we are not able to start slurmdbd. > > In the log file we have: > > [2013-03-27T08:46:03] slurmdbd version 2.4.3 started > [2013-03-27T08:46:10] error: couldn't suspend job 51029 > [2013-03-27T08:46:10] error: couldn't suspend job 50361 > [2013-03-27T08:46:10] error: couldn't suspend job 50763 > [2013-03-27T08:46:10] error: couldn't suspend job 50764 > (... --> 19 jobs) > > and nothing else. > > The queue system is working as usual: > > $ squeue |wc -l > 5037 > > But we are not able to work with sacctmgr, ... > > Do you have any idea about what is happening? > > Thanks in advance, > > Pablo Sanz > > WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer
