Try to start it so it does not damonize I think the option is -D but better check the man page, see if it core dumps.
sent from galaxy nexus On Mar 27, 2013 9:31 AM, "Pablo Sanz Mercado" <[email protected]> wrote: > > > Hi Alejandro, > > Sorry, the messages we obtain about the "couldn't suspend job" are: > [2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:50260 STATE:RUNNING > [2013-03-27T09:19:28] debug2: as_mysql_slurmdb_job_start() called > [2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:299) query > select job_db_inx from "(null)_job_table" where id_job=50260 and > time_submit=1361967308 and time_eligible=0 and time_start=0; > [2013-03-27T09:19:28] error: couldn't suspend job 50260 > > > The first lines in slurmdbd.log are: > > > [2013-03-27T09:19:27] debug3: Trying to load > plugin /usr/local/slurm-2.4.3/lib/slurm/auth_munge.so > [2013-03-27T09:19:27] auth plugin for Munge > (http://code.google.com/p/munge/) loaded > [2013-03-27T09:19:27] debug3: Success. > [2013-03-27T09:19:27] debug3: Trying to load > plugin /usr/local/slurm-2.4.3/lib/slurm/accounting_storage_mysql.so > [2013-03-27T09:19:27] debug2: mysql_connect() called for db > slurm_acct_db > [2013-03-27T09:19:27] Accounting storage MYSQL plugin loaded > [2013-03-27T09:19:27] debug3: Success. > [2013-03-27T09:19:27] pidfile not locked, assuming no running daemon > [2013-03-27T09:19:27] debug2: ArchiveDir = /tmp > [2013-03-27T09:19:27] debug2: ArchiveScript = (null) > [2013-03-27T09:19:27] debug2: AuthInfo = (null) > [2013-03-27T09:19:27] debug2: AuthType = auth/munge > [2013-03-27T09:19:27] debug2: DbdAddr = localhost > [2013-03-27T09:19:27] debug2: DbdBackupHost = (null) > [2013-03-27T09:19:27] debug2: DbdHost = localhost > [2013-03-27T09:19:27] debug2: DbdPort = 6819 > [2013-03-27T09:19:27] debug2: DebugLevel = 7 > [2013-03-27T09:19:27] debug2: DefaultQOS = (null) > [2013-03-27T09:19:27] debug2: LogFile > = /usr/local/slurm-2.4.3/log/slurmdbd.log > [2013-03-27T09:19:27] debug2: MessageTimeout = 10 > [2013-03-27T09:19:27] debug2: PidFile = /var/run/slurmdbd.pid > [2013-03-27T09:19:27] debug2: PluginDir > = /usr/local/slurm-2.4.3/lib/slurm > [2013-03-27T09:19:27] debug2: PrivateData = none > [2013-03-27T09:19:27] debug2: PurgeEventAfter = 2 months* > [2013-03-27T09:19:27] debug2: PurgeJobAfter = 12 months* > [2013-03-27T09:19:27] debug2: PurgeStepAfter = 2 days* > [2013-03-27T09:19:27] debug2: PurgeSuspendAfter = 2 hours* > [2013-03-27T09:19:27] debug2: SlurmUser = slurm(106) > [2013-03-27T09:19:27] debug2: StorageBackupHost = (null) > [2013-03-27T09:19:27] debug2: StorageHost = localhost > [2013-03-27T09:19:27] debug2: StorageLoc = slurm_acct_db > [2013-03-27T09:19:27] debug2: StoragePass = (null) > [2013-03-27T09:19:27] debug2: StoragePort = 3306 > [2013-03-27T09:19:27] debug2: StorageType = > accounting_storage/mysql > [2013-03-27T09:19:27] debug2: StorageUser = slurm > [2013-03-27T09:19:27] debug2: TrackWCKey = 1 > [2013-03-27T09:19:27] debug2: TrackSlurmctldDown= 0 > [2013-03-27T09:19:27] debug2: acct_storage_p_get_connection: request new > connection 0 > [2013-03-27T09:19:27] debug3: 0(as_mysql_qos.c:1110) query > select name, description, id, flags, grace_time, grp_cpu_mins, > grp_cpu_run_mins, grp_cpus, grp_jobs, grp_mem, grp_nodes, > grp_submit_jobs, grp_wall, max_cpu_mins_per_job, max_cpu_run_mins_per > _user, max_cpus_per_job, max_cpus_per_user, max_jobs_per_user, > max_nodes_per_job, max_nodes_per_user, max_submit_jobs_per_user, > max_wall_duration_per_job, preempt, preempt_mode, priority, us > age_factor, usage_thres from qos_table where deleted=0 > [2013-03-27T09:19:27] debug3: 0(as_mysql_user.c:1206) query > select name, admin_level from user_table where deleted=0 > [2013-03-27T09:19:27] debug3: 0(as_mysql_wckey.c:435) query > select distinct t1.id_wckey, t1.is_def, t1.wckey_name, t1.user from > "cccuam_wckey_table" as t1 where t1.deleted=0 order by wckey_name, user; > > Then we have some lines like: > > [2013-03-27T09:19:27] debug2: user usuario1 default wckey is * > > And we have: > > > 2013-03-27T09:19:27] debug3: 0(as_mysql_cluster.c:667) query > select name, classification, control_host, control_port, rpc_version, > dimensions, flags, plugin_id_select from cluster_table where deleted=0 > [2013-03-27T09:19:27] debug3: cccuam curr hour is now > 1362582000-1362585600 > [2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:353) query > select node_name, cpu_count, time_start, time_end, state from > "cccuam_event_table" where !(state & 32768) && (time_start < 1362585600 > && (time_end >= 1362582000 || time_end = 0)) order by no > de_name, time_start > [2013-03-27T09:19:27] debug3: 0(as_mysql_assoc.c:2027) query > select distinct t1.id_assoc, t1.lft, t1.rgt, t1.user, t1.acct, > t1.partition, t1.shares, t1.grp_cpu_mins, t1.grp_cpu_run_mins, > t1.grp_cpus, t1.grp_jobs, t1.grp_mem, t1.grp_nodes, t1.grp_submi > t_jobs, t1.grp_wall, t1.max_cpu_mins_pj, t1.max_cpu_run_mins, > t1.max_cpus_pj, t1.max_jobs, t1.max_nodes_pj, t1.max_submit_jobs, > t1.max_wall_pj, t1.parent_acct, t1.def_qos_id, t1.qos, t1.delt > a_qos, t1.is_def from "cccuam_assoc_table" as t1 where t1.deleted=0 && > (t1.acct='root') && (t1.user='') order by lft; > [2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:586) query > select id_resv, assoclist, cpus, flags, time_start, time_end from > "cccuam_resv_table" where (time_start < 1362585600 && time_end >= > 1362582000) order by time_start > [2013-03-27T09:19:27] debug3: 0(as_mysql_rollup.c:684) query > select job_db_inx, id_job, id_assoc, id_wckey, time_eligible, > time_start, time_end, time_suspended, cpus_alloc, cpus_req, id_resv from > "cccuam_job_table" where (time_eligible < 1362585600 && > (time_end >= 1362582000 || time_end = 0)) order by id_assoc, > time_eligible > [2013-03-27T09:19:27] debug2: Opened connection 7 from 192.168.2.19 > [2013-03-27T09:19:27] debug: DBD_INIT: CLUSTER:cccuam VERSION:10 > UID:106 IP:192.168.2.19 CONN:7 > [2013-03-27T09:19:27] debug2: acct_storage_p_get_connection: request new > connection 0 > [2013-03-27T09:19:27] debug2: DBD_REGISTER_CTLD: called for cccuam(6820) > [2013-03-27T09:19:27] debug2: slurmctld at ip:192.168.2.19, port:6820 > [2013-03-27T09:19:27] debug3: 7(as_mysql_cluster.c:407) query > select name, control_port from cluster_table where deleted=0 && > (name='cccuam'); > [2013-03-27T09:19:27] debug3: 7(accounting_storage_mysql.c:1698) query > update cluster_table set mod_time=1364372367, > control_host='192.168.2.19', control_port=6820, last_port=6820, > rpc_version=10, dimensions=1, plugin_id_select=101, flags=0 where > deleted=0 && > (name='cccuam');insert into txn_table (timestamp, action, name, actor, > info) values (1364372367, 1430, '(name=\'cccuam\')', 'slurm', > 'control_host=\'192.168.2.9\', control_port=6820, last_ > port=6820, rpc_version=10, dimensions=1, plugin_id_select=101, > flags=0'); > [2013-03-27T09:19:28] debug2: Opened connection 9 from 192.168.2.18 > [2013-03-27T09:19:28] debug: DBD_INIT: CLUSTER:(null) VERSION:10 > UID:106 IP:192.168.2.18 CONN:9 > [2013-03-27T09:19:28] debug2: acct_storage_p_get_connection: request new > connection 0 > [2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:78125 STATE:RUNNIN > > After this we have messages about jobs. Some logs does not report > any > error: > > > 2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:78125 > STATE:RUNNING > [2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:1146) query > update "(null)_job_table" set time_suspended=1362581650-time_suspended, > state=1 where job_db_inx=85113;update "(null)_suspend_table" set > time_end=1362581650 where job_db_inx=85113 && time_en > d=0; > > but some of them: > > [2013-03-27T09:19:28] debug2: DBD_JOB_SUSPEND: ID:50415 STATE:SUSPENDED > [2013-03-27T09:19:28] debug2: as_mysql_slurmdb_job_start() called > [2013-03-27T09:19:28] debug3: 9(as_mysql_job.c:299) query > select job_db_inx from "(null)_job_table" where id_job=50415 and > time_submit=1361983365 and time_eligible=0 and time_start=0; > [2013-03-27T09:19:28] error: couldn't suspend job 50415 > > and finally: > > [2013-03-27T09:19:28] debug2: DBD_REGISTER_CTLD: called for > (null)(6820) > [2013-03-27T09:19:28] debug2: slurmctld at ip:192.168.2.18, > port:6820 > [2013-03-27T09:26:29] slurmdbd version 2.4.3 started > > but slurmdbd finally dies. > > Thanks in advance, > > Pablo Sanz > > -- > Pablo Sanz Mercado > Director Técnico > Centro de Computación Científica > Universidad Autónoma de Madrid >
