Hi Alejandro,
Sorry, the messages we obtain about the "couldn't suspend job" are:
[2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:50260 STATE:RUNNING
[2013-03-27T09:19:28] debug2: as_mysql_slurmdb_job_start() called
[2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:299) query
select job_db_inx from "(null)_job_table" where id_job=50260 and
time_submit=1361967308 and time_eligible=0 and time_start=0;
[2013-03-27T09:19:28] error: couldn't suspend job 50260
The first lines in slurmdbd.log are:
[2013-03-27T09:19:27] debug3: Trying to load
plugin /usr/local/slurm-2.4.3/lib/slurm/auth_munge.so
[2013-03-27T09:19:27] auth plugin for Munge
(http://code.google.com/p/munge/) loaded
[2013-03-27T09:19:27] debug3: Success.
[2013-03-27T09:19:27] debug3: Trying to load
plugin /usr/local/slurm-2.4.3/lib/slurm/accounting_storage_mysql.so
[2013-03-27T09:19:27] debug2: mysql_connect() called for db
slurm_acct_db
[2013-03-27T09:19:27] Accounting storage MYSQL plugin loaded
[2013-03-27T09:19:27] debug3: Success.
[2013-03-27T09:19:27] pidfile not locked, assuming no running daemon
[2013-03-27T09:19:27] debug2: ArchiveDir = /tmp
[2013-03-27T09:19:27] debug2: ArchiveScript = (null)
[2013-03-27T09:19:27] debug2: AuthInfo = (null)
[2013-03-27T09:19:27] debug2: AuthType = auth/munge
[2013-03-27T09:19:27] debug2: DbdAddr = localhost
[2013-03-27T09:19:27] debug2: DbdBackupHost = (null)
[2013-03-27T09:19:27] debug2: DbdHost = localhost
[2013-03-27T09:19:27] debug2: DbdPort = 6819
[2013-03-27T09:19:27] debug2: DebugLevel = 7
[2013-03-27T09:19:27] debug2: DefaultQOS = (null)
[2013-03-27T09:19:27] debug2: LogFile
= /usr/local/slurm-2.4.3/log/slurmdbd.log
[2013-03-27T09:19:27] debug2: MessageTimeout = 10
[2013-03-27T09:19:27] debug2: PidFile = /var/run/slurmdbd.pid
[2013-03-27T09:19:27] debug2: PluginDir
= /usr/local/slurm-2.4.3/lib/slurm
[2013-03-27T09:19:27] debug2: PrivateData = none
[2013-03-27T09:19:27] debug2: PurgeEventAfter = 2 months*
[2013-03-27T09:19:27] debug2: PurgeJobAfter = 12 months*
[2013-03-27T09:19:27] debug2: PurgeStepAfter = 2 days*
[2013-03-27T09:19:27] debug2: PurgeSuspendAfter = 2 hours*
[2013-03-27T09:19:27] debug2: SlurmUser = slurm(106)
[2013-03-27T09:19:27] debug2: StorageBackupHost = (null)
[2013-03-27T09:19:27] debug2: StorageHost = localhost
[2013-03-27T09:19:27] debug2: StorageLoc = slurm_acct_db
[2013-03-27T09:19:27] debug2: StoragePass = (null)
[2013-03-27T09:19:27] debug2: StoragePort = 3306
[2013-03-27T09:19:27] debug2: StorageType =
accounting_storage/mysql
[2013-03-27T09:19:27] debug2: StorageUser = slurm
[2013-03-27T09:19:27] debug2: TrackWCKey = 1
[2013-03-27T09:19:27] debug2: TrackSlurmctldDown= 0
[2013-03-27T09:19:27] debug2: acct_storage_p_get_connection: request new
connection 0
[2013-03-27T09:19:27] debug3: 0(as_mysql_qos.c:1110) query
select name, description, id, flags, grace_time, grp_cpu_mins,
grp_cpu_run_mins, grp_cpus, grp_jobs, grp_mem, grp_nodes,
grp_submit_jobs, grp_wall, max_cpu_mins_per_job, max_cpu_run_mins_per
_user, max_cpus_per_job, max_cpus_per_user, max_jobs_per_user,
max_nodes_per_job, max_nodes_per_user, max_submit_jobs_per_user,
max_wall_duration_per_job, preempt, preempt_mode, priority, us
age_factor, usage_thres from qos_table where deleted=0
[2013-03-27T09:19:27] debug3: 0(as_mysql_user.c:1206) query
select name, admin_level from user_table where deleted=0
[2013-03-27T09:19:27] debug3: 0(as_mysql_wckey.c:435) query
select distinct t1.id_wckey, t1.is_def, t1.wckey_name, t1.user from
"cccuam_wckey_table" as t1 where t1.deleted=0 order by wckey_name, user;
Then we have some lines like:
[2013-03-27T09:19:27] debug2: user usuario1 default wckey is *
And we have:
2013-03-27T09:19:27] debug3: 0(as_mysql_cluster.c:667) query
select name, classification, control_host, control_port, rpc_version,
dimensions, flags, plugin_id_select from cluster_table where deleted=0
[2013-03-27T09:19:27] debug3: cccuam curr hour is now
1362582000-1362585600
[2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:353) query
select node_name, cpu_count, time_start, time_end, state from
"cccuam_event_table" where !(state & 32768) && (time_start < 1362585600
&& (time_end >= 1362582000 || time_end = 0)) order by no
de_name, time_start
[2013-03-27T09:19:27] debug3: 0(as_mysql_assoc.c:2027) query
select distinct t1.id_assoc, t1.lft, t1.rgt, t1.user, t1.acct,
t1.partition, t1.shares, t1.grp_cpu_mins, t1.grp_cpu_run_mins,
t1.grp_cpus, t1.grp_jobs, t1.grp_mem, t1.grp_nodes, t1.grp_submi
t_jobs, t1.grp_wall, t1.max_cpu_mins_pj, t1.max_cpu_run_mins,
t1.max_cpus_pj, t1.max_jobs, t1.max_nodes_pj, t1.max_submit_jobs,
t1.max_wall_pj, t1.parent_acct, t1.def_qos_id, t1.qos, t1.delt
a_qos, t1.is_def from "cccuam_assoc_table" as t1 where t1.deleted=0 &&
(t1.acct='root') && (t1.user='') order by lft;
[2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:586) query
select id_resv, assoclist, cpus, flags, time_start, time_end from
"cccuam_resv_table" where (time_start < 1362585600 && time_end >=
1362582000) order by time_start
[2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:684) query
select job_db_inx, id_job, id_assoc, id_wckey, time_eligible,
time_start, time_end, time_suspended, cpus_alloc, cpus_req, id_resv from
"cccuam_job_table" where (time_eligible < 1362585600 &&
(time_end >= 1362582000 || time_end = 0)) order by id_assoc,
time_eligible
[2013-03-27T09:19:27] debug2: Opened connection 7 from 192.168.2.19
[2013-03-27T09:19:27] debug: DBD_INIT: CLUSTER:cccuam VERSION:10
UID:106 IP:192.168.2.19 CONN:7
[2013-03-27T09:19:27] debug2: acct_storage_p_get_connection: request new
connection 0
[2013-03-27T09:19:27] debug2: DBD_REGISTER_CTLD: called for cccuam(6820)
[2013-03-27T09:19:27] debug2: slurmctld at ip:192.168.2.19, port:6820
[2013-03-27T09:19:27] debug3: 7(as_mysql_cluster.c:407) query
select name, control_port from cluster_table where deleted=0 &&
(name='cccuam');
[2013-03-27T09:19:27] debug3: 7(accounting_storage_mysql.c:1698) query
update cluster_table set mod_time=1364372367,
control_host='192.168.2.19', control_port=6820, last_port=6820,
rpc_version=10, dimensions=1, plugin_id_select=101, flags=0 where
deleted=0 &&
(name='cccuam');insert into txn_table (timestamp, action, name, actor,
info) values (1364372367, 1430, '(name=\'cccuam\')', 'slurm',
'control_host=\'192.168.2.9\', control_port=6820, last_
port=6820, rpc_version=10, dimensions=1, plugin_id_select=101,
flags=0');
[2013-03-27T09:19:28] debug2: Opened connection 9 from 192.168.2.18
[2013-03-27T09:19:28] debug: DBD_INIT: CLUSTER:(null) VERSION:10
UID:106 IP:192.168.2.18 CONN:9
[2013-03-27T09:19:28] debug2: acct_storage_p_get_connection: request new
connection 0
[2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:78125 STATE:RUNNIN
After this we have messages about jobs. Some logs does not report any
error:
2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:78125
STATE:RUNNING
[2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:1146) query
update "(null)_job_table" set time_suspended=1362581650-time_suspended,
state=1 where job_db_inx=85113;update "(null)_suspend_table" set
time_end=1362581650 where job_db_inx=85113 && time_en
d=0;
but some of them:
[2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:50415 STATE:SUSPENDED
[2013-03-27T09:19:28] debug2: as_mysql_slurmdb_job_start() called
[2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:299) query
select job_db_inx from "(null)_job_table" where id_job=50415 and
time_submit=1361983365 and time_eligible=0 and time_start=0;
[2013-03-27T09:19:28] error: couldn't suspend job 50415
and finally:
[2013-03-27T09:19:28] debug2: DBD_REGISTER_CTLD: called for
(null)(6820)
[2013-03-27T09:19:28] debug2: slurmctld at ip:192.168.2.18,
port:6820
[2013-03-27T09:26:29] slurmdbd version 2.4.3 started
but slurmdbd finally dies.
Thanks in advance,
Pablo Sanz
--
Pablo Sanz Mercado
Director Técnico
Centro de Computación Científica
Universidad Autónoma de Madrid
