Greetings !
We are running Maui 3.3.1 combined with Torque Version
3.0.3-snap.201107121616
on a small cluster with mixed resources 7 GPU nodes (i7) and 14 Nodes
with Opteron Magny-Cours-CPUs. The Maui and Torque service is running on
our portal node called gpu
We are able to submit simple jobs and OpenMPI jobs. They are being
processed correctly until the resources get exhausted and the jobs get
queued.
After processing the current running jobs, the nodes get free again
and stay that way, althogh the queue is still full.
One should expect, that the jobs queued should then get executed, one by
one, until the queue is empty again (as no further jobs are submitted).
But, although pbsnodes states every single node as free (twisting
thumbs..), the queue does not get processed.
We have defined several queues, for each type of resouces one short
(24h), medium (72h), one long (infinite), called gpushort. gpumedium,
gpulong (for the GPU nodes) and respective optshort,optmedium, optlong
(for the Opteron nodes). I have tried to set reservations, so that the
Opterons get assigned to the Opteron queues (exclusively) and the GPU
nodes to the GPU queues, so that one does not need to set the pbs requests
(opteron or gpunode) in the submit script. I have failed in that .
1) How do I get the nodes assigned to their queues selected by their
properties "opteron" and "gpunode" properly ?
2)More important: How can I fix this bad behaviour, that queued jobs are
never been processed?
I'd be grateful for any help, since I'm fairly new to this matter and I
did not find my answers in the documentation.
Greetings from Salzburg/Austria/Europe
Vlad Popa
University of Salzburg
Computer Science/HPC Computing
Jakob-Harringer-Str. 2
5020 Salzburg
Austria
PS: Below our configuration...
pbsnodes:
gpu01
state = free
np = 8
properties = i7,i7-new,gpunode,16G
ntype = cluster
status =
rectime=1317905387,varattr=,jobs=,state=free,netload=12779454,gres
=,loadave=0.03,ncpus=8,physmem=16315316kb,availmem=48458408kb,totmem=49083308kb,
idletime=16275,nusers=0,nsessions=0,uname=Linux gpu01
2.6.32-131.6.1.el6.x86_64
#1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 2
gpu_status =
gpu[1]=gpu_id=0000:06:00.0;,gpu[0]=gpu_id=0000:05:00.0;,driver
_ver=280.13,timestamp=Thu Oct 6 14:53:52 201
.... and so on until gpu07:
gpu07
state = free
np = 8
properties = fermi,16G,gpunode,i7
ntype = cluster
status =
rectime=1317905499,varattr=,jobs=,state=free,netload=5695667,gres=,loadave=0.01,ncpus=8,physmem=16310908kb,availmem=48600784kb,totmem=49078900kb,idletime=14281,nusers=0,nsessions=0,uname=Linux
gpu07 2.6.32-131.6.1.el6.x86_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 1
gpu_status =
gpu[0]=gpu_id=0000:01:00.0;,driver_ver=280.13,timestamp=Thu Oct 6
10:57:46 2011
... followed by our Oteron Nodes called hex01-hex14, all staying free
..hex07
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g
res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230
0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378
10844 1
0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux
hex07 2.6
.32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
hex06
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr
es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068
kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06
2.6.32-131.6.1.el6.x86
_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0hex07
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g
res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230
0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378
10844 1
0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux
hex07 2.6
.32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002hex07
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g
res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230
0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378
10844 1
0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux
hex07 2.6
.32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
hex06
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr
es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068
kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06
2.6.32-131.6.1.el6.x86
_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
mom_manager_port = 15003
gpus = 0
hex06
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr
es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068
kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06
2.6.32-131.6.1.el6.x86
_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
[vlad@gpu ~]$ qmgr -c 'p s'
#
# Create queues and set their attributes.
#
#
# Create and define queue gpushort
#
create queue gpushort
set queue gpushort queue_type = Execution
set queue gpushort resources_min.nodes = 1
set queue gpushort resources_default.neednodes = gpunode
set queue gpushort resources_default.nodes = 1
set queue gpushort resources_default.walltime = 24:00:00
set queue gpushort enabled = True
set queue gpushort started = True
#
# Create and define queue optlong
#
create queue optlong
set queue optlong queue_type = Execution
set queue optlong resources_default.neednodes = opteron
set queue optlong resources_default.nodes = 1
set queue optlong enabled = True
set queue optlong started = True
#
# Create and define queue gpumedium
#
create queue gpumedium
set queue gpumedium queue_type = Execution
set queue gpumedium resources_default.neednodes = gpunode
set queue gpumedium resources_default.nodes = 1
set queue gpumedium resources_default.walltime = 72:00:00
set queue gpumedium enabled = True
set queue gpumedium started = True
#
# Create and define queue gpulong
#
create queue gpulong
set queue gpulong queue_type = Execution
set queue gpulong resources_default.neednodes = gpunode
set queue gpulong resources_default.nodes = 1
set queue gpulong enabled = True
set queue gpulong started = True
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.nodes = 1
set queue batch resources_default.walltime = 01:00:00
set queue batch enabled = True
set queue batch started = True
#
# Create and define queue optshort
#
create queue optshort
set queue optshort queue_type = Execution
set queue optshort resources_default.neednodes = opteron
set queue optshort resources_default.nodes = 1
set queue optshort resources_default.walltime = 24:00:00
set queue optshort enabled = True
set queue optshort started = True
#
# Create and define queue optmedium
#
create queue optmedium
set queue optmedium queue_type = Execution
set queue optmedium resources_default.neednodes = opteron
set queue optmedium resources_default.nodes = 1
set queue optmedium resources_default.walltime = 72:00:00
set queue optmedium enabled = True
set queue optmedium started = True
#
# Create and define queue short
#
create queue short
set queue short queue_type = Execution
set queue short resources_default.walltime = 24:00:00
set queue short enabled = True
set queue short started = True
#
# Set server attributes.[vlad@gpu ~]$ qmgr -c 'p s'
#
# Create queues and set their attributes.
#
#
# Create and define queue gpushort
#
create queue gpushort
set queue gpushort queue_type = Execution
set queue gpushort resources_min.nodes = 1
set queue gpushort resources_default.neednodes = gpunode
set queue gpushort resources_default.nodes = 1hex07
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g
res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230
0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378
10844 1
0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux
hex07 2.6
.32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
hex06
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr
es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068
kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06
2.6.32-131.6.1.el6.x86
_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
set queue gpushort resources_default.walltime = 24:00:00
set queue gpushort enabled = True
set queue gpushort started = True
#
# Create and define queue optlong
#
create queue optlong
set queue optlong queue_type = Execution
set queue optlong resources_default.neednodes = opteron
set queue optlong resources_default.nodes = 1
set queue optlong enabled = True
set queue optlong started = True
#
# Create and define queue gpumedium
#
create queue gpumedium
set queue gpumedium queue_type = Execution
set queue gpumedium resources_default.neednodes = gpunode
set queue gpumedium resources_default.nodes = 1hex07
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g
res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230
0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378
10844 1
0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux
hex07 2.6
.32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
hex06
state = free
np = 14
properties = opteron
ntype = cluster
status =
rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr
es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068
kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06
2.6.32-131.6.1.el6.x86
_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
set queue gpumedium resources_default.walltime = 72:00:00
set queue gpumedium enabled = True
set queue gpumedium started = True
#
# Create and define queue gpulong
#
create queue gpulong
set queue gpulong queue_type = Execution
set queue gpulong resources_default.neednodes = gpunode
set queue gpulong resources_default.nodes = 1
set queue gpulong enabled = True
set queue gpulong started = True
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.nodes = 1
set queue batch resources_default.walltime = 01:00:00
set queue batch enabled = True
set queue batch started = True
#
# Create and define queue optshort
#
create queue optshort
set queue optshort queue_type = Execution
set queue optshort resources_default.neednodes = opteron
set queue optshort resources_default.nodes = 1
set queue optshort resources_default.walltime = 24:00:00
set queue optshort enabled = True
set queue optshort started = True
#
# Create and define queue optmedium
#
create queue optmedium
set queue optmedium queue_type = Execution
set queue optmedium resources_default.neednodes = opteron
set queue optmedium resources_default.nodes = 1
set queue optmedium resources_default.walltime = 72:00:00
set queue optmedium enabled = True
set queue optmedium started = True
#
# Create and define queue short
#
create queue short
set queue short queue_type = Execution
set queue short resources_default.walltime = 24:00:00
set queue short enabled = True
set queue short started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_hosts = gpu
set server managers = forsthof@gpu
set server managers += peter@gpu
set server managers += root@gpu
set server managers += vlad@gpu
set server operators = forsthof@gpu
set server operators += peter@gpu
set server operators += root@gpu
set server operators += vlad@gpu
set server default_queue = batch
set server log_events = 511
set server mail_from = adm
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 6
set server log_level = 7
set server mom_job_sync = True
set server keep_completed = 300
set server next_job_number = 293
#
set server scheduling = True
set server acl_hosts = gpu
set server managers = forsthof@gpu
set server managers += peter@gpu
set server managers += root@gpu
set server managers += vlad@gpu
set server operators = forsthof@gpu
set server operators += peter@gpu
set server operators += root@gpu
set server operators += vlad@gpu
set server default_queue = batch
set server log_events = 511
set server mail_from = adm
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 6
set server log_level = 7
set server mom_job_sync = True
set server keep_completed = 300
set server next_job_number = 293
[vlad@gpu ~]$ showconfig
# Maui version 3.3.1 (PID: 22120)
# global policies
REJECTNEGPRIOJOBS[0] FALSE
ENABLENEGJOBPRIORITY[0] FALSE
ENABLEMULTINODEJOBS[0] TRUE
ENABLEMULTIREQJOBS[0] FALSE
BFPRIORITYPOLICY[0] [NONE]
JOBPRIOACCRUALPOLICY QUEUEPOLICY
NODELOADPOLICY ADJUSTSTATE
USEMACHINESPEEDFORFS FALSE
USEMACHINESPEED FALSE
USESYSTEMQUEUETIME TRUE
USELOCALMACHINEPRIORITY FALSE
NODEUNTRACKEDLOADFACTOR 1.2
JOBNODEMATCHPOLICY[0]
JOBMAXSTARTTIME[0] INFINITY
METAMAXTASKS[0] 0
NODESETPOLICY[0] [NONE]
NODESETATTRIBUTE[0] [NONE]
NODESETLIST[0]
NODESETDELAY[0] 00:00:00
NODESETPRIORITYTYPE[0] MINLOSS
NODESETTOLERANCE[0] 0.00
BACKFILLPOLICY[0] FIRSTFIT
BACKFILLDEPTH[0] 0
BACKFILLPROCFACTOR[0] 0
BACKFILLMAXSCHEDULES[0] 10000
BACKFILLMETRIC[0] PROCS
BFCHUNKDURATION[0] 00:00:00
BFCHUNKSIZE[0] 0
PREEMPTPOLICY[0] REQUEUE
MINADMINSTIME[0] 00:00:00
RESOURCELIMITPOLICY[0]
NODEAVAILABILITYPOLICY[0] COMBINED:[DEFAULT]
NODEALLOCATIONPOLICY[0] CPULOAD
TASKDISTRIBUTIONPOLICY[0] DEFAULT
RESERVATIONPOLICY[0] CURRENTHIGHEST
RESERVATIONRETRYTIME[0] 00:00:00
RESERVATIONTHRESHOLDTYPE[0] NONE
RESERVATIONTHRESHOLDVALUE[0] 0
FSPOLICY [NONE]
FSPOLICY [NONE]
FSINTERVAL 12:00:00
FSDEPTH 8
FSDECAY 1.00
# Priority Weights
SERVICEWEIGHT[0] 1
TARGETWEIGHT[0] 1
CREDWEIGHT[0] 1
ATTRWEIGHT[0] 1
FSWEIGHT[0] 1
RESWEIGHT[0] 1
USAGEWEIGHT[0] 1
QUEUETIMEWEIGHT[0] 1
XFACTORWEIGHT[0] 0
SPVIOLATIONWEIGHT[0] 0
BYPASSWEIGHT[0] 0
TARGETQUEUETIMEWEIGHT[0] 0
TARGETXFACTORWEIGHT[0] 0
USERWEIGHT[0] 0
GROUPWEIGHT[0] 0
ACCOUNTWEIGHT[0] 0
QOSWEIGHT[0] 0
CLASSWEIGHT[0] 0
FSUSERWEIGHT[0] 0
FSGROUPWEIGHT[0] 0
FSACCOUNTWEIGHT[0] 0
FSQOSWEIGHT[0] 0
FSCLASSWEIGHT[0] 0
ATTRATTRWEIGHT[0] 0
ATTRSTATEWEIGHT[0] 0
NODEWEIGHT[0] 0
PROCWEIGHT[0] 0
MEMWEIGHT[0] 0
SWAPWEIGHT[0] 0
DISKWEIGHT[0] 0
PSWEIGHT[0] 0
PEWEIGHT[0] 0
WALLTIMEWEIGHT[0] 0
UPROCWEIGHT[0] 0
UJOBWEIGHT[0] 0
CONSUMEDWEIGHT[0] 0
USAGEEXECUTIONTIMEWEIGHT[0] 0
REMAININGWEIGHT[0] 0
PERCENTWEIGHT[0] 0
XFMINWCLIMIT[0] 00:02:00
# partition DEFAULT policies
REJECTNEGPRIOJOBS[1] FALSE
ENABLENEGJOBPRIORITY[1] FALSE
ENABLEMULTINODEJOBS[1] TRUE
ENABLEMULTIREQJOBS[1] FALSE
BFPRIORITYPOLICY[1] [NONE]
JOBPRIOACCRUALPOLICY QUEUEPOLICY
NODELOADPOLICY ADJUSTSTATE
JOBNODEMATCHPOLICY[1]
JOBMAXSTARTTIME[1] INFINITY
METAMAXTASKS[1] 0
NODESETPOLICY[1] [NONE]
NODESETATTRIBUTE[1] [NONE]
NODESETLIST[1]
NODESETDELAY[1] 00:00:00
NODESETPRIORITYTYPE[1] MINLOSS
NODESETTOLERANCE[1] 0.00
# Priority Weights
XFMINWCLIMIT[1] 00:00:00
RMAUTHTYPE[0] CHECKSUM
CLASSCFG[[NONE]] DEFAULT.FEATURES=[NONE]
CLASSCFG[[ALL]] DEFAULT.FEATURES=[NONE]
CLASSCFG[gpushort] DEFAULT.FEATURES=[gpunode]
CLASSCFG[optlong] DEFAULT.FEATURES=[opteron]
CLASSCFG[gpumedium] DEFAULT.FEATURES=[gpunode]
CLASSCFG[gpulong] DEFAULT.FEATURES=[gpunode]
CLASSCFG[batch] DEFAULT.FEATURES=[NONE]
CLASSCFG[optshort] DEFAULT.FEATURES=[opteron]
CLASSCFG[optmedium] DEFAULT.FEATURES=[opteron]
CLASSCFG[short] DEFAULT.FEATURES=[NONE]
QOSPRIORITY[0] 0
QOSQTWEIGHT[0] 0
QOSXFWEIGHT[0] 0
QOSTARGETXF[0] 0.00
QOSTARGETQT[0] 00:00:00
QOSFLAGS[0]
QOSPRIORITY[1] 0
QOSQTWEIGHT[1] 0
QOSXFWEIGHT[1] 0
QOSTARGETXF[1] 0.00
QOSTARGETQT[1] 00:00:00
QOSFLAGS[1]
# SERVER MODULES: MX
SERVERMODE NORMAL
SERVERNAME
SERVERHOST gpu
SERVERPORT 42559
LOGFILE maui.log
LOGFILEMAXSIZE 10000000
LOGFILEROLLDEPTH 1
LOGLEVEL 9
LOGFACILITY fALL
SERVERHOMEDIR /var/spool/maui/
TOOLSDIR /var/spool/maui/tools/
LOGDIR /var/spool/maui/log/
STATDIR /var/spool/maui/stats/
LOCKFILE /var/spool/maui/maui.pid
SERVERCONFIGFILE /var/spool/maui/maui.cfg
CHECKPOINTFILE /var/spool/maui/maui.ck
CHECKPOINTINTERVAL 00:05:00
CHECKPOINTEXPIRATIONTIME 3:11:20:00
TRAPJOB
TRAPNODE
TRAPFUNCTION
RESDEPTH 24
RMPOLLINTERVAL 00:00:30
NODEACCESSPOLICY SHARED
ALLOCLOCALITYPOLICY [NONE]
SIMTIMEPOLICY [NONE]
ADMIN1 root vlad peter forsthof
ADMINHOSTS ALL
NODEPOLLFREQUENCY 0
DISPLAYFLAGS
DEFAULTDOMAIN
DEFAULTCLASSLIST [DEFAULT:1]
FEATURENODETYPEHEADER
FEATUREPROCSPEEDHEADER
FEATUREPARTITIONHEADER
DEFERTIME 1:00:00
DEFERCOUNT 24
DEFERSTARTCOUNT 1
JOBPURGETIME 0
NODEPURGETIME 2140000000
APIFAILURETHRESHHOLD 6
NODESYNCTIME 600
JOBSYNCTIME 600
JOBMAXOVERRUN 00:10:00
NODEMAXLOAD 0.0
PLOTMINTIME 120
PLOTMAXTIME 245760
PLOTTIMESCALE 11
PLOTMINPROC 1
PLOTMAXPROC 512
PLOTPROCSCALE 9
SCHEDCFG[] MODE=NORMAL SERVER=gpu:42559
# RM MODULES: PBS SSS WIKI NATIVE
RMCFG[GPU] AUTHTYPE=CHECKSUM EPORT=15004 TIMEOUT=00:00:09 TYPE=PBS
SIMWORKLOADTRACEFILE workload
SIMRESOURCETRACEFILE resource
SIMAUTOSHUTDOWN OFF
SIMSTARTTIME 0
SIMSCALEJOBRUNTIME FALSE
SIMFLAGS
SIMJOBSUBMISSIONPOLICY CONSTANTJOBDEPTH
SIMINITIALQUEUEDEPTH 16
SIMWCACCURACY 0.00
SIMWCACCURACYCHANGE 0.00
SIMNODECOUNT 0
SIMNODECONFIGURATION NORMAL
SIMWCSCALINGPERCENT 100
SIMCOMRATE 0.10
SIMCOMTYPE ROUNDROBIN
COMINTRAFRAMECOST 0.30
COMINTERFRAMECOST 0.30
SIMSTOPITERATION -1
SIMEXITITERATION -1
_______________________________________________
mauiusers mailing list
[email protected]
http://www.supercluster.org/mailman/listinfo/mauiusers
