Hello Danny,
Our cluster has been working six months at least before this problem
started.
When we start slumdbd it works for a while, but after some seconds dies
as you can see:
# /etc/init.d/slurmdbd-2.4.3 start; sacct -vvvv -u
user1
starting slurmdbd: daemon /usr/local/slurm-2.4.3/sbin/slurmdbd
sacct: Jobs eligible from Tue Apr 9 00:00:00 2013 - Now
sacct: debug: Options selected:
opt_completion=0
opt_dump=0
opt_dup=0
opt_fdump=0
opt_field_list=(null)
opt_help=0
opt_allocs=0
sacct: debug3: Trying to load
plugin /usr/local/slurm-2.4.3/lib/slurm/accounting_storage_slurmdbd.so
sacct: Accounting storage SLURMDBD plugin loaded with AuthInfo=(null)
sacct: debug3: Success.
sacct: debug3: Trying to load
plugin /usr/local/slurm-2.4.3/lib/slurm/auth_munge.so
sacct: auth plugin for Munge (http://code.google.com/p/munge/) loaded
sacct: debug3: Success.
sacct: debug: slurmdbd: Sent DbdInit msg
sacct: debug2: Clusters requested: cccuam
sacct: debug2: Userids requested:
sacct: debug2: : 888
JobID JobName Partition Account AllocCPUS State
ExitCode
------------ ---------- ---------- ---------- ---------- ----------
--------
alida+ 1 RUNNING 0:0
147281 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147282 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147283 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147284 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147285 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147288 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147289 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147290 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147293 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147294 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147295 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147298 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147301 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147304 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147305 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147308 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147309 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147310 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147311 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147315 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147316 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147320 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147327 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147328 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147332 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147336 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147337 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147338 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147342 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147343 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147344 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147348 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147349 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147350 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147351 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147352 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147356 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147360 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147361 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147362 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147366 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147367 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147371 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147374 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147378 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147382 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147383 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147384 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147388 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147392 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147396 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147400 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147401 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147405 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147406 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147408 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147412 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
147413 launchnaf+ ccc quiralida+ 1 RUNNING
0:0
sacct: debug: slurmdbd: Sent fini msg
Take into account that cccuam is the cluster name, and it is
configured.
Our slurm.conf file is:
ControlMachine=mastermachine
ControlAddr=mastermachine
BackupController=slavemachine
BackupAddr=slavemachine
AuthType=auth/munge
CacheGroups=1
CryptoType=crypto/munge
EnforcePartLimits=yes
JobCredentialPrivateKey=/usr/local/slurm-2.4.3/credential/private.key
JobCredentialPublicCertificate=/usr/local/slurm-2.4.3/credential/public.key
MailProg=/bin/mail
MpiDefault=none
MpiParams=ports=12000-12999
ProctrackType=proctrack/linuxproc
ReturnToService=1
SlurmctldPidFile=/usr/local/slurm-2.4.3/pid/slurmctld.pid
SlurmctldPort=6820
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6821
SlurmdSpoolDir=/var/log/slurm/slurmd.spool
SlurmUser=slurm
StateSaveLocation=/usr/local/slurm-2.4.3/savelocation
SwitchType=switch/none
TaskPlugin=task/affinity
TaskPluginParam=sched
TopologyPlugin=topology/none
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=36000
Waittime=0
FastSchedule=1
SchedulerType=sched/backfill
SchedulerPort=7322
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory
PreemptMode=SUSPEND,GANG
PreemptType=preempt/partition_prio
PriorityType=priority/multifactor
PriorityWeightAge=10000
PriorityWeightJobSize=1000
PriorityWeightQOS=10000
AccountingStorageEnforce=limits,qos
AccountingStorageHost=mastermachine
AccountingStorageBackupHost=slavemachine
AccountingStorageLoc=slurm_acct_db
AccountingStoragePort=6819
AccountingStorageType=accounting_storage/slurmdbd
ClusterName=CCCUAM
JobCompLoc=/var/log/slurm/job_completions
JobCompType=jobcomp/mysql
JobAcctGatherFrequency=240
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=7
SlurmctldLogFile=/usr/local/slurm-2.4.3/log/slurmctld.log
SlurmdDebug=7
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmSchedLogFile=/var/log/slurm/sched.log
SlurmSchedLogLevel=1
Thanks in advance,
Pablo Sanz
--
Pablo Sanz Mercado
Director Técnico
Centro de Computación Científica
Universidad Autónoma de Madrid
