Re: [gridengine users] Strange SGE PE issue (threaded PE with 999 slots but scheduler thinks the value is 0)

[Feng Zhang]

Is "threads" added into all.q?

Also can check "qconf -srqs" is there's any limit

On Thu, Jun 11, 2020 at 2:33 PM Chris Dagdigian  wrote:
>
> Hi folks,
>
> Got a bewildering situation I've never seen before with simple SMP/threaded 
> PE techniques
>
> I made a brand new PE called threaded:
>
> $ qconf -sp threaded
> pe_namethreaded
> slots  999
> user_lists NONE
> xuser_listsNONE
> start_proc_argsNONE
> stop_proc_args NONE
> allocation_rule$pe_slots
> control_slaves FALSE
> job_is_first_task  TRUE
> urgency_slots  min
> accounting_summary FALSE
> qsort_args NONE
>
>
> And I attached that to all.q on an IDLE grid and submitted a job with '-pe 
> threaded 1' argument
>
> However all "qstat -j" data is showing this scheduler decision line:
>
> cannot run in PE "threaded" because it only offers 0 slots
>
>
> I'm sort of lost on how to debug this because I can't figure out how to probe 
> where SGE is keeping track of PE specific slots.  With other stuff I can look 
> at complex_values reported by execution hosts or I can use an "-F" argument 
> to qstat to dump the live state and status of a requestable resource but I 
> don't really have any debug or troubleshooting ideas for "how to figure out 
> why SGE thinks there are 0 slots when the static PE on an idle cluster has. 
> been set to contain 999 slots"
>
> Anyone seen something like this before?  I don't think I've ever seen this 
> particular issue with an SGE parallel environment before ...
>
>
> Chris
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] scripting help with per user job submit restrictions

[Feng Zhang]

You can try to write the script to first scan all the files to get their full 
path names and then run the Array jobs.


> On Jun 13, 2019, at 1:20 PM, VG  wrote:
> 
> HI Joshua,
> I like the array job option because essentially it will still be 1 job and it 
> will run them in parallel.
> 
> I have one issue though. I can create an array script, but here I presented a 
> simple problem. Actually my individual tar.gz files are under respective 
> directories
> For example 
> dir1 has file1.tar.gz
> dir2 has file2.tar.gz
> dir3 has file3.tar.gz
> 
> The way I was then submitting them was
> 
> for i in *
> do 
>   if [ -d "$i" ] 
>   then cd "$i"
>   qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xf 
> *.tar"
>   cd .. 
>   fi
> done
> 
> One way is I can pull out all the tar.gz in one folder and run array script 
> as you told, other wise is there a work around where everything runs and also 
> remains in the respective directories.
> 
> Thanks for your help.
> 
> Regards
> Varun
> 
> 
> 
> 
> 
>> On Thu, Jun 13, 2019 at 1:11 PM Joshua Baker-LePain  wrote:
>> On Thu, 13 Jun 2019 at 9:32am, VG wrote
>> 
>> > I have a scripting question regarding submitting jobs to the cluster.
>> > There is a limitation per user of 1000 jobs only.
>> > Let's say I have 1200 tar.gz files
>> >
>> > I tried to submit all the jobs together but after 1000 jobs it gave me an
>> > error message saying per user limit is 1000 and after that it did not
>> > queued the remaining jobs.
>> > I want to write a script where if the submitted jobs goes below
>> > 1000(because they finished running), then next jobs are submitted in the
>> > queue. How can I do that?
>> > I have written something like this:
>> >
>> > for i in *tar.gz
>> > do
>> >   qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf $i"
>> > done
>> 
>> The right answer to this problem (not the scripting issue, but how to 
>> untar all 1200 files without running afoul of the 1000 job limit) is an 
>> array job.  You can submit 1 job with 1200 tasks to untar all the files. 
>> The relevant bits of the job script would include (assuming bash):
>> 
>> targzs=(0 file1.tar.gz file2.tar.gz ... file1200.tar.gz)
>> tar xzf ${targzs[$SGE_TASK_ID]}
>> 
>> To submit it:
>> 
>> qsub -t 1-1200 job.sh
>> 
>> -- 
>> Joshua Baker-LePain
>> QB3 Shared Cluster Sysadmin
>> UCSF
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] jobs randomly die

[Feng Zhang]

looks like your job used a lot of ram:

mem  7.463TBs
io   70.435GB
iow  0.000s
maxvmem  532.004MB

Do you have CGROUP to limit resource of jobs?

Best,

Feng

On Tue, May 14, 2019 at 9:53 AM hiller  wrote:
>
> ~> qconf -srqs
> No resource quota set found
>
> 'dmesg -T' does not give an oom or other weird messages.
>
> 'free -h' looks good and also looked good at 'kill time':
>
> ~> free -h
>   totalusedfree  shared  buff/cache   
> available
> Mem:   188G1.0G185G2.6M2.0G
> 186G
> Swap:   49G  0B 49G
>
> Full output of qacct:
> ~>  qacct -j 635659
> ==
> qnameall.q
> hostname karun10
> groupusers
> ownercalj
> project  NONE
> department   defaultdepartment
> jobname  dsc_gdr2
> jobnumber635659
> taskid   undefined
> account  sge
> priority 0
> qsub_timeMon May 13 13:06:58 2019
> start_time   Mon May 13 13:06:56 2019
> end_time Mon May 13 18:31:42 2019
> granted_pe   make
> slots1
> failed   100 : assumedly after job
> exit_status  137  (Killed)
> ru_wallclock 19486s
> ru_utime 0.048s
> ru_stime 0.006s
> ru_maxrss11.566KB
> ru_ixrss 0.000B
> ru_ismrss0.000B
> ru_idrss 0.000B
> ru_isrss 0.000B
> ru_minflt7885
> ru_majflt0
> ru_nswap 0
> ru_inblock   0
> ru_oublock   8
> ru_msgsnd0
> ru_msgrcv0
> ru_nsignals  0
> ru_nvcsw 142
> ru_nivcsw3
> cpu  19305.760s
> mem  7.463TBs
> io   70.435GB
> iow  0.000s
> maxvmem  532.004MB
> arid undefined
> ar_sub_time  undefined
> category -l hostname=karun10 -pe make 1
>
>
> Thanks, ulrich
>
>
> On 5/14/19 3:28 PM, MacMullan IV, Hugh wrote:
> > It's a limit being reached, of some sort. Do you have a RQS of any kind 
> > (qconf -srqs)? We see this for job-requested, or system set RAM exhaustion 
> > (OOM killer, as mentioned 'dmesg -T' on compute nodes often useful), as 
> > well as time limits reached. What is the whole output from 'qacct -j JOBID'?
> >
> > Cheers,
> > -Hugh
> >
> > -Original Message-
> > From: users-boun...@gridengine.org  On Behalf 
> > Of hiller
> > Sent: Tuesday, May 14, 2019 9:02 AM
> > To: users@gridengine.org
> > Subject: Re: [gridengine users] jobs randomly die
> >
> > Hi,
> > nope, there are no oom messages in the journal.
> > Regards, ulrich
> >
> >
> > On 5/14/19 12:49 PM, Arnau wrote:
> >> Hi,
> >>
> >> _maybe_ the OOM killer killed the job ? a look to messages will give you 
> >> an answer (I've seen this in my cluster).
> >>
> >> HTH,
> >> Arnau
> >>
> >> El mar., 14 may. 2019 a las 12:37, hiller ( >> >) escribió:
> >>
> >> Dear all,
> >> i have a problem that jobs sent to gridengine randomly die.
> >> The gridengine version is 8.1.9
> >> The OS is opensuse 15.0
> >> The gridengine messages file says:
> >> 05/13/2019 18:31:45|worker|karun|E|master task of job 635659.1 failed 
> >> - killing job
> >> 05/13/2019 18:31:46|worker|karun|W|job 635659.1 failed on host karun10 
> >> assumedly after job because: job 635659.1 died through signal KILL (9)
> >>
> >> qacct -j 635659 says:
> >> failed   100 : assumedly after job
> >> exit_status  137  (Killed)
> >>
> >>
> >> The was no kill triggered by the user. Also there are no other 
> >> limitations, neither ulimit nor in the gridengine queue
> >> The 'qconf -sq all.q' command gives:
> >> s_rt  INFINITY
> >> h_rt  INFINITY
> >> s_cpu INFINITY
> >> h_cpu INFINITY
> >> s_fsize   INFINITY
> >> h_fsize   INFINITY
> >> s_dataINFINITY
> >> h_dataINFINITY
> >> s_stack   INFINITY
> >> h_stack   INFINITY
> >> s_coreINFINITY
> >> h_coreINFINITY
> >> s_rss INFINITY
> >> h_rss INFINITY
> >> s_vmemINFINITY
> >> h_vmemINFINITY
> >>
> >> Years ago there were some threads about the same issue, but i did not 
> >> find a solution.
> >>
> >> Does somebody have a hint what i can do or check/debug?
> >>
> >> With kind regards and many thanks for any help, ulrich
> >> ___
> >> users mailing list
> >> users@gridengine.org 
> >> https://gridengine.org/mailman/listinfo/users
> >>
> > ___
> > users mailing list
> > users@gridengine.org
> > https://gridengine.org/mailman/listinfo/users
> >
> ___
> users mailing list
> users@gridengine.org
> 

Re: [gridengine users] Processes not exiting

[Feng Zhang]

probably it is the Maker which does not have proper handling of signals?

Maybe you can try to use a script to run the job, rather than run
binary directly, to see if it can work. Also you can add some signal
handling commands in your script to check...

Best,

Feng

On Tue, Nov 13, 2018 at 7:07 PM  wrote:
>
> We have a cluster with gridengine 6.5u2 and noticing a strange behavior
> when running MPI jobs.  Our application will finish, yet the processes
> continue to run and use up the CPU.  We did configure a parallel
> environment for MPI as follows:
>
> pe_namempi
> slots  500
> user_lists NONE
> xuser_listsNONE
> start_proc_argsNONE
> stop_proc_args NONE
> allocation_rule$round_robin
> control_slaves TRUE
> job_is_first_task  FALSE
> urgency_slots  min
> accounting_summary FALSE
>
> Then we have run our application "Maker" like this,
> qsub -cwd -N  -b y -V -pe mpi 
> /opt/mpich-install/bin/mpiexec  maker 
>
> It seems to run fine and qstat will show it running.  Once it has
> completed, qstat is empty again and we have the desired output.
> However, the "maker" process have continued to run on the compute nodes
> until I login to each node and "kill -9" the processes.  We did not have
> this problem when running mpiexec directly with Maker, or running Maker
> in stand-alone mode (without MPI), so I guess it is a problem with our
> qsub command or parallel environment?  Any Ideas?
>
> Thanks,
> --
> Chandler / Systems Administrator
> Arizona Genomics Institute
> www.genome.arizona.edu
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Effects of login_shells settings

[Feng Zhang]

add an extra command line in job script:

unset module

may be helpful.

Best,

Feng
Best,

Feng


On Wed, Oct 31, 2018 at 2:48 PM Ilya M <4ilya.m+g...@gmail.com> wrote:
>
> Hello,
>
> I had an unexpected effect after adding bash to the login_shells list in SGE 
> 6.2.u5. Some array tasks started to get scheduled twice on the same node 
> (accounting info corroborates that) and 1-2 tasks in a job would not produce 
> any output at all, as if they did not run. This quickly became a wide-spread 
> problem in the cluster and I had to revert the change by removing bash from 
> the list.
>
> I am trying to understand the cause of this behavior. I did not find anything 
> in the logs. The qmaster logs and accounting only showed that some tasks were 
> scheduled twice, running for the same number of seconds and both times 
> completing successfully. Was this an uncovered bug?
>
> The rationale for adding bash to the list, were the errors some users 
> reported when trying to launch their jobs in bash:
>
>> /bin/bash: SPEcho: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPEcho'
>> /bin/bash: SPCheckFile: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCheckFile'
>> /bin/bash: SPNotify: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPNotify'
>> /bin/bash: SPCheckFileNonEmpty: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for 
>> `BASH_FUNC_SPCheckFileNonEmpty'
>> /bin/bash: SPCheckExec: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCheckExec'
>> /bin/bash: SPLogError: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPLogError'
>> /bin/bash: SPCkptDone: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCkptDone'
>> /bin/bash: SPCheckFunc: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCheckFunc'
>> /bin/bash: SPCkptShouldRun: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for 
>> `BASH_FUNC_SPCkptShouldRun'
>> /bin/bash: SPCheckVar: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCheckVar'
>> /bin/bash: SPCheckDir: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCheckDir'
>> /bin/bash: SPLogInfo: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPLogInfo'
>> /bin/bash: SPCleanDir: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPCleanDir'
>> /bin/bash: SPLogWarning: line 1: syntax error: unexpected end of file
>> /bin/bash: error importing function definition for `BASH_FUNC_SPLogWarning'
>>
>>
>
> And the discussion here: 
> https://sourceforge.net/p/modules/mailman/message/28650889/, suggesting that 
> the cause might be absence of bash in the login_shells list.
>
> Looking forward to some ideas and explanations.
>
> Thank you,
> Ilya.
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Instalation issue on version 8.1.10-1

[Feng Zhang]

Maybe you can check the install script, like inst_sge, to see if there's
any typo.

Or some script files may have special characters, from like Widows, Linux,
etc.?


Best,

Feng

Best,

Feng


On Wed, Oct 17, 2018 at 12:43 PM Jerome  wrote:

> Dear all
>
> I've trying to install a fresh version of SGE on a Debian server v 9.5
> (on virtualbox), using the git repository of Dave Love:
>
> https://gitlab.com/loveshack/sge.git
>
> The issue occurs on the commit "0271520806868d6be018a1a5c019fc64d5faddf3" .
>
> I do the test installing the deb package file and on compiling the
> source of sge directly on the machine.
> That's seems to be some easy bug to resolve althought i can't catch
> where the problem is.
>
> During the install_qmaster i've get this error during the check of the
> directory /opt/sge:
>
> Can't create a temporary file in the SGE_ROOT directory
>
>/opt/sgen
>
> This may be a permission problem (e.g. no read/write permission
> on a NFS mounted filesystem).
> Please check your permissions. You may cancel the installation now
> and restart it or continue and try again.
>
> Hit  to continue >>
>
>
> That'a a "n" letter at the end of the directory name.. I change fo some
> /opt/foo and get back "/opt/foon". The process stop here.
>
> Hope that someone could detect where is the problem?
>
> Regards
>
>
> --
> -- Jérôme
> Non, j'aime Cléopatre : ais-ja l'air d'un César ?
> J'adore Bérénice : ais-ja l'aspect d'un Tite ?
> (Edmond Rostand, Cyrano de Bergerac)
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] User job fails silently

[Feng Zhang]

I also did some test. If provide extra interrupting signal processing in
all these scripts(to catch any TERM signal from OS), it can kind of solve
the issue.

On Wed, Aug 8, 2018 at 11:04 PM, Feng Zhang  wrote:

>
> I am guessing it may be very similar to what I have met before. My issue
> was: one user used a bash script as a batch job script, and in it, it calls
> another script(python), and this script then calls a third script(and may
> be so on...).  For these kind of jobs, if there's anything wrong, it can
> cause some "defunct" processes( especialy for parallel jobs).  Theses jobs
> then actually corrupted, but since the defunct and other processes are not
> properly terminated(sometimes, even after the shephard process get
> terminated already), SGE still thinks it is running, and on the compute
> nodes, there are still some processes of the failed job running(hanging).
>
> What I did to resolve the issue is: write a script to check those jobs, if
> found any defunct processes, then kill the job related process(es), which
> will trigger SGE to kill the job automatically.
>
>
>
> On Wed, Aug 8, 2018 at 7:26 PM, Derrick Lin  wrote:
>
>> >  What state of the job you see in this line? Is it just hanging there
>> and doing nothing? They do not appear in `top`? And it never vanishes
>> automatically but you have to kill the job by hand?
>>
>> Sorry for the confusion. The job state is "r" according to SGE, but as
>> you mentioned qstat output is not related to any process.
>>
>> The line I coped is what it shown in top/htop. So basically, all his jobs
>> became:
>>
>> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671
>> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187677
>> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187690
>>
>> Each of this scripts does copy & untar a file to the local XFS file
>> system, then a python script is called to operate on these untared files.
>>
>> The job log shows that untaring is done, but the python script has never
>> started and the job process stuck as shown above.
>>
>> We don't see any storage related contention.
>>
>> I am more interested in knowing where this process
>> bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671 come
>> from?
>>
>> Cheers,
>>
>>
>> On Wed, Aug 8, 2018 at 6:53 PM, Reuti  wrote:
>>
>>>
>>> > Am 08.08.2018 um 08:15 schrieb Derrick Lin :
>>> >
>>> > Hi guys,
>>> >
>>> > I have a user reported his jobs stuck running for much longer than
>>> usual.
>>> >
>>> > So I go to the exec host, check the process and all processes owned by
>>> that user look like:
>>> >
>>> > `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671
>>>
>>> What state of the job you see in this line? Is it just hanging there and
>>> doing nothing? They do not appear in `top`? And it never vanishes
>>> automatically but you have to kill the job by hand?
>>>
>>>
>>> > In qstat, it still shows job is in running state.
>>>
>>> The `qstat`output is not really related to any running process. It's
>>> just what SGE granted and think it is running or granted to run. Especially
>>> with parallel jobs across nodes, the might or might not be any process on
>>> one of the granted slave nodes.
>>>
>>>
>>> > The user resubmitted the jobs and they ran and completed without an
>>> problem.
>>>
>>> Could it be a race condition with the shared file system?
>>>
>>> -- Reuti
>>>
>>>
>>> > I am wondering what may has caused this situation in general?
>>> >
>>> > Cheers,
>>> > Derrick
>>> > ___
>>> > users mailing list
>>> > users@gridengine.org
>>> > https://gridengine.org/mailman/listinfo/users
>>>
>>>
>>
>> ___
>> users mailing list
>> users@gridengine.org
>> https://gridengine.org/mailman/listinfo/users
>>
>>
>
>
> --
> Best,
>
> Feng
>



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] User job fails silently

[Feng Zhang]

I am guessing it may be very similar to what I have met before. My issue
was: one user used a bash script as a batch job script, and in it, it calls
another script(python), and this script then calls a third script(and may
be so on...).  For these kind of jobs, if there's anything wrong, it can
cause some "defunct" processes( especialy for parallel jobs).  Theses jobs
then actually corrupted, but since the defunct and other processes are not
properly terminated(sometimes, even after the shephard process get
terminated already), SGE still thinks it is running, and on the compute
nodes, there are still some processes of the failed job running(hanging).

What I did to resolve the issue is: write a script to check those jobs, if
found any defunct processes, then kill the job related process(es), which
will trigger SGE to kill the job automatically.



On Wed, Aug 8, 2018 at 7:26 PM, Derrick Lin  wrote:

> >  What state of the job you see in this line? Is it just hanging there
> and doing nothing? They do not appear in `top`? And it never vanishes
> automatically but you have to kill the job by hand?
>
> Sorry for the confusion. The job state is "r" according to SGE, but as you
> mentioned qstat output is not related to any process.
>
> The line I coped is what it shown in top/htop. So basically, all his jobs
> became:
>
> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671
> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187677
> `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187690
>
> Each of this scripts does copy & untar a file to the local XFS file
> system, then a python script is called to operate on these untared files.
>
> The job log shows that untaring is done, but the python script has never
> started and the job process stuck as shown above.
>
> We don't see any storage related contention.
>
> I am more interested in knowing where this process
> bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671 come
> from?
>
> Cheers,
>
>
> On Wed, Aug 8, 2018 at 6:53 PM, Reuti  wrote:
>
>>
>> > Am 08.08.2018 um 08:15 schrieb Derrick Lin :
>> >
>> > Hi guys,
>> >
>> > I have a user reported his jobs stuck running for much longer than
>> usual.
>> >
>> > So I go to the exec host, check the process and all processes owned by
>> that user look like:
>> >
>> > `- -bash /opt/gridengine/default/spool/omega-6-20/job_scripts/1187671
>>
>> What state of the job you see in this line? Is it just hanging there and
>> doing nothing? They do not appear in `top`? And it never vanishes
>> automatically but you have to kill the job by hand?
>>
>>
>> > In qstat, it still shows job is in running state.
>>
>> The `qstat`output is not really related to any running process. It's just
>> what SGE granted and think it is running or granted to run. Especially with
>> parallel jobs across nodes, the might or might not be any process on one of
>> the granted slave nodes.
>>
>>
>> > The user resubmitted the jobs and they ran and completed without an
>> problem.
>>
>> Could it be a race condition with the shared file system?
>>
>> -- Reuti
>>
>>
>> > I am wondering what may has caused this situation in general?
>> >
>> > Cheers,
>> > Derrick
>> > ___
>> > users mailing list
>> > users@gridengine.org
>> > https://gridengine.org/mailman/listinfo/users
>>
>>
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
>


-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Debugging crash when running program through GridEngine

[Feng Zhang]

It maybe caused by the environment variables.

One simple way may be adding something like:

>source myscriptforenv.sh (bash)

in your job script.

On Fri, May 4, 2018 at 9:45 AM, Simon Andrews 
wrote:

> I’ve got a strange problem on our cluster where some python programs are
> segfaulting when run through qsub, but work fine on the command line, or
> even if run remotely through SSH.
>
>
>
> Really simple (hello world) programs work OK, but anything which does a
> significant amount of imports seems to fail.  So for example;
>
>
>
> htseq-count
>
>
>
> works locally, but
>
>
>
> qsub -o test.log -cwd -V -j y -b y htseq-count
>
>
>
> Produces a segfault in the executed program.
>
>
>
> ssh compute-0-0 htseq-count
>
>
>
> ..works fine (we’re using ssh to launch jobs on our cluster)
>
>
>
> Any suggestions for how to go about trying to track this down?
>
>
>
> Thanks
>
>
>
> Simon.
>
>
>
> The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT 
> *Registered
> Charity No. 1053902.*
>
> The information transmitted in this email is directed only to the
> addressee. If you received this in error, please contact the sender and
> delete this email from your system. The contents of this e-mail are the
> views of the sender and do not necessarily represent the views of the
> Babraham Institute. Full conditions at: www.babraham.ac.uk
> 
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
>


-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] can't delete an exec host

[Feng Zhang]

Is there any running jobs on queue instance of compute-2-4@basic.q?

On Wed, Sep 6, 2017 at 11:33 AM, Michael Stauffer <mgsta...@gmail.com> wrote:
> On Wed, Sep 6, 2017 at 11:16 AM, Feng Zhang <prod.f...@gmail.com> wrote:
>>
>> It seems SGE master did not get refreshed with new hostgroup. Maybe you
>> can try:
>>
>> 1. restart SGE master
>
>
> Is it safe to do this with jobs queued and running? I think it's not
> reliable, i.e. jobs can get killed and de-queued?
>
>>
>> or
>>
>> 2. change basic.q, "hostlist" to any node, like "compute-1-0.local",
>>
>> wait till it gets refreshed; then change it back to "@basichosts".
>
>
> I've done this, but it's not refreshing (been about 10 minutes now). I'm
> still getting the error when I try to delete exec host compute-2-4, and
> qhost is still showing basic.q on the nodes in @basichosts.
>
> Interestingly, host compute-2-4 was removed from another queue
> (qlogin.basic.q) that also uses @basichosts, so it's something about basic.q
> that's stuck.
>
> Is there some way to refresh things other than restarting qmaster?
>
> -M
>
>
>>
>>
>>
>>
>> On Wed, Sep 6, 2017 at 10:29 AM, Michael Stauffer <mgsta...@gmail.com>
>> wrote:
>> > SoGE 8.1.8
>> >
>> > Hi,
>> >
>> > I'm having trouble deleting an execution host. I've removed it from the
>> > host group, but when I try to delete with qconf, it says it's still part
>> > of
>> > 'basic.q'. Here's the relevant output? Anyone have any suggestions?
>> >
>> > [root@chead ~]# qconf -de compute-2-4.local
>> > Host object "compute-2-4.local" is still referenced in cluster queue
>> > "basic.q".
>> >
>> > [root@chead ~]# qconf -sq basic.q
>> > qname basic.q
>> > hostlist  @basichosts
>> > seq_no0
>> > load_thresholds   np_load_avg=1.74
>> > suspend_thresholdsNONE
>> > nsuspend  1
>> > suspend_interval  00:05:00
>> > priority  0
>> > min_cpu_interval  00:05:00
>> > processorsUNDEFINED
>> > qtype BATCH
>> > ckpt_list NONE
>> > pe_list   make mpich mpi orte unihost serial
>> > rerun FALSE
>> > slots 8,[compute-1-2.local=3],[compute-1-0.local=7], \
>> >   [compute-1-1.local=7],[compute-1-3.local=7], \
>> >   [compute-1-5.local=8],[compute-1-6.local=8], \
>> >   [compute-1-7.local=8],[compute-1-8.local=8], \
>> >   [compute-1-9.local=8],[compute-1-10.local=8], \
>> >   [compute-1-11.local=8],[compute-1-12.local=8], \
>> >   [compute-1-13.local=8],[compute-1-14.local=8], \
>> >   [compute-1-15.local=8]
>> > tmpdir/tmp
>> > shell /bin/bash
>> > prologNONE
>> > epilogNONE
>> > shell_start_mode  posix_compliant
>> > starter_methodNONE
>> > suspend_methodNONE
>> > resume_method NONE
>> > terminate_method  NONE
>> > notify00:00:60
>> > owner_listNONE
>> > user_listsNONE
>> > xuser_lists   NONE
>> > subordinate_list  NONE
>> > complex_valuesNONE
>> > projects  NONE
>> > xprojects NONE
>> > calendar  NONE
>> > initial_state default
>> > s_rt  INFINITY
>> > h_rt  INFINITY
>> > s_cpu INFINITY
>> > h_cpu INFINITY
>> > s_fsize   INFINITY
>> > h_fsize   INFINITY
>> > s_dataINFINITY
>> > h_dataINFINITY
>> > s_stack   INFINITY
>> > h_stack   INFINITY
>> > s_coreINFINITY
>> > h_coreINFINITY
>> > s_rss INFINITY
>> > h_rss INFINITY
>> > s_vmem19G
>> > h_vmem19G
>> >
>> > [root@chead ~]# qconf -shgrp @basichosts
>> > group_name @basichosts
>> > hostlist compute-1-0.local compute-1-2.local compute-1-3.local \
>> >  compute-1-5.local compute-1-6.local compute-1-7.local \
>> >  compute-1-8.local compute-1-9.local compute-1-10.local \
>> >  compute-1-11.local compute-1-12.local compute-1-13.local \
>> >  compute-1-14.local compute-1-15.local compute-2-0.local \
>> >  compute-2-2.local compute-2-5.local compute-2-7.local \
>> >  compute-2-8.local compute-2-9.local compute-2-11.local \
>> >  compute-2-12.local compute-2-13.local compute-2-15.local \
>> >  compute-2-6.local
>> >
>> > Thanks
>> >
>> > -M
>> >
>> > ___
>> > users mailing list
>> > users@gridengine.org
>> > https://gridengine.org/mailman/listinfo/users
>> >
>>
>>
>>
>> --
>> Best,
>>
>> Feng
>
>



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] can't delete an exec host

[Feng Zhang]

It seems SGE master did not get refreshed with new hostgroup. Maybe you can try:

1. restart SGE master

or

2. change basic.q, "hostlist" to any node, like "compute-1-0.local",
wait till it gets refreshed; then change it back to "@basichosts".



On Wed, Sep 6, 2017 at 10:29 AM, Michael Stauffer  wrote:
> SoGE 8.1.8
>
> Hi,
>
> I'm having trouble deleting an execution host. I've removed it from the
> host group, but when I try to delete with qconf, it says it's still part of
> 'basic.q'. Here's the relevant output? Anyone have any suggestions?
>
> [root@chead ~]# qconf -de compute-2-4.local
> Host object "compute-2-4.local" is still referenced in cluster queue
> "basic.q".
>
> [root@chead ~]# qconf -sq basic.q
> qname basic.q
> hostlist  @basichosts
> seq_no0
> load_thresholds   np_load_avg=1.74
> suspend_thresholdsNONE
> nsuspend  1
> suspend_interval  00:05:00
> priority  0
> min_cpu_interval  00:05:00
> processorsUNDEFINED
> qtype BATCH
> ckpt_list NONE
> pe_list   make mpich mpi orte unihost serial
> rerun FALSE
> slots 8,[compute-1-2.local=3],[compute-1-0.local=7], \
>   [compute-1-1.local=7],[compute-1-3.local=7], \
>   [compute-1-5.local=8],[compute-1-6.local=8], \
>   [compute-1-7.local=8],[compute-1-8.local=8], \
>   [compute-1-9.local=8],[compute-1-10.local=8], \
>   [compute-1-11.local=8],[compute-1-12.local=8], \
>   [compute-1-13.local=8],[compute-1-14.local=8], \
>   [compute-1-15.local=8]
> tmpdir/tmp
> shell /bin/bash
> prologNONE
> epilogNONE
> shell_start_mode  posix_compliant
> starter_methodNONE
> suspend_methodNONE
> resume_method NONE
> terminate_method  NONE
> notify00:00:60
> owner_listNONE
> user_listsNONE
> xuser_lists   NONE
> subordinate_list  NONE
> complex_valuesNONE
> projects  NONE
> xprojects NONE
> calendar  NONE
> initial_state default
> s_rt  INFINITY
> h_rt  INFINITY
> s_cpu INFINITY
> h_cpu INFINITY
> s_fsize   INFINITY
> h_fsize   INFINITY
> s_dataINFINITY
> h_dataINFINITY
> s_stack   INFINITY
> h_stack   INFINITY
> s_coreINFINITY
> h_coreINFINITY
> s_rss INFINITY
> h_rss INFINITY
> s_vmem19G
> h_vmem19G
>
> [root@chead ~]# qconf -shgrp @basichosts
> group_name @basichosts
> hostlist compute-1-0.local compute-1-2.local compute-1-3.local \
>  compute-1-5.local compute-1-6.local compute-1-7.local \
>  compute-1-8.local compute-1-9.local compute-1-10.local \
>  compute-1-11.local compute-1-12.local compute-1-13.local \
>  compute-1-14.local compute-1-15.local compute-2-0.local \
>  compute-2-2.local compute-2-5.local compute-2-7.local \
>  compute-2-8.local compute-2-9.local compute-2-11.local \
>  compute-2-12.local compute-2-13.local compute-2-15.local \
>  compute-2-6.local
>
> Thanks
>
> -M
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] users Digest, Vol 72, Issue 13

[Feng Zhang]

Have you checked the status of the queue instances? Sometime if a queue
instance goes into error status, it can not run jobs like this.

qstat -F

can list the status. And qmod -c (queue instatnce) can clear it.

On Mon, Dec 12, 2016 at 12:35 AM, Coleman, Marcus [JRDUS Non-J] <
mcole...@its.jnj.com> wrote:

> Hi
>
> I am sure this is your problemYou are submitting a job that requires 2
> cores, to a queue that has only 1 slot available.
> If your host all have the same amount of cores...it is no reason to
> separate them via commons. This is only needed if the host have different
> amount of slots or you want to manipulate  the slots...
>
> slots 1,[ibm021=8],[ibm037=8],[ibm038=8]
> slots 8
>
>
> I would only list the pe I am using I am requesting...unless you plan to
> use each of those PE's
> pe_list   make mpi smp cores
> pe_list   cores
>
>
> Also you mentioned parallel env, I WOULD change allocation to $fill_up
> unless your software (not sge) control jobs distribution..
>
> qconf -sp core
>   allocation_rule$pe_slots <---( ONLY USE ONE MACHINE)
>  control_slaves FALSE <--- (I think you want tight integration)
>  job_is_first_task  TRUE  <( this is true if the  first job submitted
> only kicks off other jobs)
>
> allocation_rule $fill_up <---work better for parallel jobs
> control_slaves TRUE <  you want tight integration with SGE
> job_is_first_task   software will control job distro...
>
>
> Also what does qmaster message and the associated node sge message say...
>
>
>
>
>
>
>
>
> -Original Message-
> From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org]
> On Behalf Of users-requ...@gridengine.org
> Sent: Sunday, December 11, 2016 9:05 PM
> To: users@gridengine.org
> Subject: [EXTERNAL] users Digest, Vol 72, Issue 13
>
> Send users mailing list submissions to
> users@gridengine.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://gridengine.org/mailman/listinfo/users
> or, via email, send a message with subject or body 'help' to
> users-requ...@gridengine.org
>
> You can reach the person managing the list at
> users-ow...@gridengine.org
>
> When replying, please edit your Subject line so it is more specific than
> "Re: Contents of users digest..."
>
>
> Today's Topics:
>
>1. Re: CPU complex (John_Tai)
>
>
> --
>
> Message: 1
> Date: Mon, 12 Dec 2016 05:04:33 +
> From: John_Tai 
> To: Christopher Heiny 
> Cc: "users@gridengine.org" 
> Subject: Re: [gridengine users] CPU complex
> Message-ID: 
> Content-Type: text/plain; charset="utf-8"
>
> # qconf -sq all.q
> qname all.q
> hostlist  @allhosts
> seq_no0
> load_thresholds   np_load_avg=1.75
> suspend_thresholdsNONE
> nsuspend  1
> suspend_interval  00:05:00
> priority  0
> min_cpu_interval  00:05:00
> processorsUNDEFINED
> qtype BATCH INTERACTIVE
> ckpt_list NONE
> pe_list   make mpi smp cores
> rerun FALSE
> slots 1,[ibm021=8],[ibm037=8],[ibm038=8]
> tmpdir/tmp
> shell /bin/sh
> prologNONE
> epilogNONE
> shell_start_mode  posix_compliant
> starter_methodNONE
> suspend_methodNONE
> resume_method NONE
> terminate_method  NONE
> notify00:00:60
> owner_listNONE
> user_listsNONE
> xuser_lists   NONE
> subordinate_list  NONE
> complex_valuesNONE
> projects  NONE
> xprojects NONE
> calendar  NONE
> initial_state default
> s_rt  INFINITY
> h_rt  INFINITY
> s_cpu INFINITY
> h_cpu INFINITY
> s_fsize   INFINITY
> h_fsize   INFINITY
> s_dataINFINITY
> h_dataINFINITY
> s_stack   INFINITY
> h_stack   INFINITY
> s_coreINFINITY
> h_coreINFINITY
> s_rss INFINITY
> h_rss INFINITY
> s_vmemINFINITY
> h_vmemINFINITY
>
>
>
> From: Christopher Heiny [mailto:christopherhe...@gmail.com]
> Sent: Monday, December 12, 2016 12:22
> To: John_Tai
> Cc: users@gridengine.org; Reuti
> Subject: Re: [gridengine users] CPU complex
>
>
>
> On Dec 11, 2016 5:11 PM, "John_Tai"  n_...@smics.com>> wrote:
> I associated the queue with the PE:
>
> qconf -aattr queue pe_list cores all.q The only slots were defined
> in the all.q queue, and just the total slots in 

Re: [gridengine users] How to removed jobs stuck in DR state

[Feng Zhang]

qdel -f jobid

 with systemadmin account may work.

On Tue, Jun 28, 2016 at 4:22 PM, Sean Smith  wrote:
> Hi All,
>
> I have some jobs stuck in our Queue that I don't know how to remove.
> Originally I was not worried about these but I discovered later that they
> are blocking a slot so I want to clean them up.  Exec Hosts and Qmaster have
> been rebooted but these stuck jobs remain.  Any idea how to remove them and
> reclaim these slots...
>
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:27:57
> dvh...@dvgrid94.smi.local  1 192
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:00
> dvh...@dvgrid95.smi.local  1 196
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:03
> dvh...@dvgrid100.smi.local 1 200
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:15
> dvh...@dvgrid100.smi.local 1 222
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:18
> dvh...@dvgrid95.smi.local  1 227
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:21
> dvh...@dvgrid99.smi.local  1 232
> 1976739 0.47681 Nmaster_r4 rtlbuild dr05/28/2016 21:28:21
> dvh...@dvgrid97.smi.local  1 234
>
> Any advise would be greatly appreciated.
>
> Regards,
> Sean Smith
>
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Defining slots in grid queue

[Feng Zhang]

I think that set "slots" on the node to be 2 X number-of-cpu-cores will work.

On Fri, May 6, 2016 at 8:56 AM,   wrote:
> Hi,
>
>
>
> I have only one host defined in a queue and want to allot 2 slots per core
> instead of one slot per core.
>
>
>
> How do we need to define the slots to allocate more than 1 slot per core in
> the queue.
>
>
>
> Regards,
>
> Sudha
>
> The information contained in this electronic message and any attachments to
> this message are intended for the exclusive use of the addressee(s) and may
> contain proprietary, confidential or privileged information. If you are not
> the intended recipient, you should not disseminate, distribute or copy this
> e-mail. Please notify the sender immediately and destroy all copies of this
> message and any attachments. WARNING: Computer viruses can be transmitted
> via email. The recipient should check this email and any attachments for the
> presence of viruses. The company accepts no liability for any damage caused
> by any virus transmitted by this email. www.wipro.com
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Applying RQS limits across a set of hosts

[Feng Zhang]

Are the 2 groups of nodes defined as queues? If so, limit on queues
may work for you:

limitqueues XXX to slots=16

On Wed, Sep 23, 2015 at 10:27 AM, Simon Andrews
 wrote:
>
>
> On 23/09/2015 15:20, "Jesse Becker"  wrote:
>
>>On Wed, Sep 23, 2015 at 10:56:21AM +, Simon Andrews wrote:
>>>I?m trying to use RQS to limit the total number of slots which users can
>>>take up on our cluster.  This is slightly complicated by the hosts being
>>>split into two groups, one for batch jobs (@compute) and one for
>>>interactive jobs (@interactive).  The idea would be that I?d limit the
>>>total slots taken up by a user on the @compute set, but not restrict
>>>@interactive.
>>>
>>>The RQS rule I?m using is:
>>>
>>>{
>>>   name per_user_slot_limit
>>>   description  "limit the number of slots per user"
>>>   enabled  TRUE
>>>   limitusers {*} hosts {@compute} to slots=16
>>>}
>>>
>>>However I don?t think this is generating the right effect.  I think what
>>>this is doing is saying that the limit of 16 slots applies on each of
>>>the individual members of @compute, and not on the sum of the slots
>>>across all of @compute.
>>
>>I'm guessing that using:
>>
>>  limit  users {*} hosts @compute to slots=16
>>
>>doesn't work?
>
> I think that¹s functionally the same as my previous version, the brackets
> aren¹t strictly required in this case.
>
> I think I have a work round which is:
>
> {
>  name per_user_slot_limit
>  description  "limit the number of slots per user"
>  enabled  TRUE
>  limit users {*} hosts {@interactive} to slots=16
>  limitusers {*} to slots=16
> }
>
> ..where the @interactive limit is a no-op since that¹s as many cores as
> any of the interactive hosts have.  It works because the interactive jobs
> hit the first rule, match and then don¹t proceed to the more general rule.
>  I think it¹s working, but it seems somewhat inelegant and I suspect
> there¹s a better way to do this.
>
>
> Simon.
>
>
>
> The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT 
> Registered Charity No. 1053902.
> The information transmitted in this email is directed only to the addressee. 
> If you received this in error, please contact the sender and delete this 
> email from your system. The contents of this e-mail are the views of the 
> sender and do not necessarily represent the views of the Babraham Institute. 
> Full conditions at: www.babraham.ac.uk
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users



-- 
Best,

Feng

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Applying RQS limits across a set of hosts

[Feng Zhang]

On the wiki ge page:http://wiki.gridengine.info/wiki/index.php/RQS_Common_Uses

Looks like the brackets does matter the rules applied. Without
brackets, it means on all hosts, while with brackets, it means per
host. Not sure it works.

On Wed, Sep 23, 2015 at 10:51 AM, Simon Andrews
<simon.andr...@babraham.ac.uk> wrote:
>
>
> On 23/09/2015 15:48, "Feng Zhang" <prod.f...@gmail.com> wrote:
>
>>Are the 2 groups of nodes defined as queues? If so, limit on queues
>>may work for you:
>>
>>limitqueues XXX to slots=16
>
>
> They¹re not separate queues, they¹re simply split by the job type they can
> process.  I guess moving them to different queues would work, but I like
> the simplicity we have now where the users just have to use qsub or qlogin
> and the job is distributed to the appropriate set of nodes.
>
> Simon.
>
> The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT 
> Registered Charity No. 1053902.
> The information transmitted in this email is directed only to the addressee. 
> If you received this in error, please contact the sender and delete this 
> email from your system. The contents of this e-mail are the views of the 
> sender and do not necessarily represent the views of the Babraham Institute. 
> Full conditions at: www.babraham.ac.uk<http://www.babraham.ac.uk/terms>
>
> ___
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users



-- 
Best,

Feng

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Monitoring slot usage

[Feng Zhang]

I have similar issue too. Especially when users run MPI+Multithreads
jobs. Some Multithreading programs by default use all of the cores on
a node they find.

Now I have a script to scan the usage of CPU and RAM on all nodes, and
it will warn me if it find any overloaded nodes.

Not sure SGE has built-in ability to track the CPU cores each job
uses. But it may not be difficult to prepare a script to do that
routinely out of SGE.



On Thu, Jul 30, 2015 at 11:00 AM, Simon Andrews
simon.andr...@babraham.ac.uk wrote:
 What is the recommended way of identifying jobs which are consuming more CPU
 than they’ve requested?  I have an environment set up where people mostly
 submit SMP jobs through a parallel environment and we can use this
 information to schedule them appropriately.  We’ve had several cases though
 where the jobs have used significantly more cores on the machine they’re
 assigned to than they requested, so the nodes become overloaded and go into
 an alarm state.

 What options do I have for monitoring the number of cores simultaneously
 used by a job and comparing this to the number which were requested so I can
 find cases where the actual usage is way above the request and kill them?

 Thanks

 Simon.

 The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT
 Registered Charity No. 1053902.

 The information transmitted in this email is directed only to the addressee.
 If you received this in error, please contact the sender and delete this
 email from your system. The contents of this e-mail are the views of the
 sender and do not necessarily represent the views of the Babraham Institute.
 Full conditions at: www.babraham.ac.uk


 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users




-- 
Best,

Feng

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Negative complex values

[Feng Zhang]

Hi Simon,

As you defined the h_vmem as JOB, according to the manual:

 A consumable defined by 'y' is a per slot consumables  which
 means  the  limit is multiplied by the number of slots being
 used by the job before being applied.  In case  of  'j'  the
 consumable is a per job consumable. This resource is debited
 as requested (without  multiplication)  from  the  allocated
 master  queue.  The  resource needs not be available for the
 slave task queues.

I am wondering if you can set it to be YES, other than JOB, and to
see  if it can work for parallel jobs?


On Mon, Jun 8, 2015 at 11:10 AM, Simon Andrews
simon.andr...@babraham.ac.uk wrote:
 Having done a bit of investigation it seems that the problem we're hitting is 
 that our h_vmem limits aren't being respected if the jobs are being submitted 
 as parallel jobs.

 If I put two jobs in:

 $ qsub -o test.log -l h_vmem=1000G hostname
 Your job 343719 (hostname) has been submitted

 $ qsub -o test.log -l h_vmem=1000G -pe cores 2 hostname
 Your job 343720 (hostname) has been submitted

 The first job won't be scheduled:
 scheduling info:cannot run in queue instance 
 all.q@compute-1-6.local because it is not of type batch
 cannot run in queue instance 
 all.q@compute-1-1.local because it is not of type batch
 cannot run in queue instance 
 all.q@compute-1-5.local because it is not of type batch
 cannot run in queue instance 
 all.q@compute-1-3.local because it is not of type batch
 cannot run in queue instance 
 all.q@compute-1-2.local because it is not of type batch
 (-l h_vmem=1000G) cannot run at host 
 compute-0-2.local because it offers only hc:h_vmem=4.000G
 cannot run in queue instance 
 all.q@compute-1-4.local because it is not of type batch
 cannot run in queue instance 
 all.q@compute-1-0.local because it is not of type batch
 (-l h_vmem=1000G) cannot run at host 
 compute-0-4.local because it offers only hc:h_vmem=16.000G
 cannot run in queue instance 
 all.q@compute-1-7.local because it is not of type batch
 (-l h_vmem=1000G) cannot run at host 
 compute-0-3.local because it offers only hc:h_vmem=25.000G
 (-l h_vmem=1000G) cannot run at host 
 compute-0-6.local because it offers only hc:h_vmem=-968.000G
 (-l h_vmem=1000G) cannot run at host 
 compute-0-5.local because it offers only hc:h_vmem=32.000G
 (-l h_vmem=1000G) cannot run at host 
 compute-0-0.local because it offers only hc:h_vmem=32.000G
 (-l h_vmem=1000G) cannot run at host 
 compute-0-1.local because it offers only hc:h_vmem=12.000G


 But the second is immediately scheduled and overcommits the node it's on (and 
 the overcommit is reflected by qstat -F h_vmem).

 The memory usage is recorded and will prevent other jobs from running on that 
 node, but I need to figure out how to make the scheduler respect the resource 
 limit when the job is first submitted.

 Any suggestions would be very welcome

 Thanks.

 Simon.

 -Original Message-
 From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org] On 
 Behalf Of Simon Andrews
 Sent: 08 June 2015 13:53
 To: users@gridengine.org
 Subject: [gridengine users] Negative complex values

 Our cluster seems to have ended up in a strange state, and I don't understand 
 why.

 We have set up h_vmem to be a consumable resource so that users can't exhaust 
 the memory on any compute node.  This has been working OK and in our tests it 
 all seemed to be right, but we've now found that somehow we've ended up with 
 nodes with negative amounts of memory remaining.

 We only have one queue on the system, all.q.

 $ qstat -F h_vmem -q all.q@compute-0-3
 queuename  qtype resv/used/tot. load_avg arch  
 states
 -
 all.q@compute-0-3.localBP0/44/6413.13lx26-amd64
 hc:h_vmem=-172.000G

 ..so the node is somehow at -172G memory.

 The setup for the resource is as follows:

 $ qconf -sc | grep h_vmem
 h_vmem  h_vmem MEMORY  =YES JOB0 
0

 We use a jsv to add a default memory allocation to all jobs, and the jobs 
 listed all provide an h_vmem condition (see later).

 ..the initialisation of the complex value for the node looks OK:

 $ qconf -se compute-0-3 | grep complex
 complex_valuesh_vmem=128G

 The problem seems to stem from an individual job which has managed to commit 
 a 200G job on a node with only 128G. These are the jobs which are running on 
 that node.

 qstat -j 341706 | grep hard resource_list
 hard 

Re: [gridengine users] frequent errors from the GRID messages

[Feng Zhang]

Is there any way to list all the files of the failed job in /tmp:

ls -l /tmp/

and

ls -l /tmp/8319760.1. rhel6.q/

On Fri, May 29, 2015 at 1:49 AM,  sudha.penme...@wipro.com wrote:
 Yes Hugh, Users have permissions for the directory



 drwxrwxrwt. 48 root root 163840 May 29 08:48 /tmp



 Regards,

 Sudha



 From: MacMullan, Hugh [mailto:hugh...@wharton.upenn.edu]
 Sent: Thursday, May 28, 2015 8:45 PM
 To: Sudha Padmini Penmetsa (WT01 - Global Media  Telecom);
 users@gridengine.org
 Subject: RE: frequent errors from the GRID messages



 Do users have permission to write to /tmp? 1777 is the 'normal' permissions
 for that directory:



 drwxrwxrwt.  12 root root 4096 May 28 11:10 tmp





 From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org] On
 Behalf Of sudha.penme...@wipro.com
 Sent: Thursday, May 28, 2015 11:02 AM
 To: users@gridengine.org
 Subject: [gridengine users] frequent errors from the GRID messages



 Hi,



 The below  errors are quiet often in the GRID messages file. Though there
 are enough permissions for the /tmp directory on host, the messages show the
 jobs failed because of this error.



 Can you please help in understanding the reason for these errors.



 05/27/2015 09:58:25|qmaster|gridmaster1|W|job 8318993.1 failed on host host1
 general before job because: 05/27/2015 09:58:25 [899:22217]: can't open file
 /tmp/8318993.1.rhel6.q/pid: Permission denied

 05/27/2015 10:55:37|qmaster|gridmaster1|W|job 8319636.1 failed on host host2
 general before job because: 05/27/2015 10:55:37 [899:11272]: can't open file
 /tmp/8319636.1. rhel6.q/pid: Permission denied

 05/27/2015 10:58:05|qmaster| gridmaster1|W|job 8319689.1 failed on host
 host3 general before job because: 05/27/2015 10:58:04 [899:48764]: can't
 open file /tmp/8319689.1. rhel6.q/pid: Permission denied

 05/27/2015 10:58:05|qmaster| gridmaster1|W|job 8319691.1 failed on host
 host6 general before job because: 05/27/2015 10:58:04 [899:46484]: can't
 open file /tmp/8319691.1. rhel6.q/pid: Permission denied

 05/27/2015 11:00:51|qmaster| gridmaster1|W|job 8319752.1 failed on host
 host4 general before job because: 05/27/2015 11:00:51 [899:14950]: can't
 open file /tmp/8319752.1. rhel6.q/pid: Permission denied

 05/27/2015 11:00:51|qmaster| gridmaster1|W|job 8319750.1 failed on host
 host7 general before job because: 05/27/2015 11:00:51 [899:50509]: can't
 open file /tmp/8319750.1. rhel6.q/pid: Permission denied

 05/27/2015 11:02:14|qmaster| gridmaster1|W|job 8319760.1 failed on host
 host5 general before job because: 05/27/2015 11:02:14 [899:17507]: can't
 open file /tmp/8319760.1. rhel6.q/pid: Permission denied



 Regards,

 Sudha

 The information contained in this electronic message and any attachments to
 this message are intended for the exclusive use of the addressee(s) and may
 contain proprietary, confidential or privileged information. If you are not
 the intended recipient, you should not disseminate, distribute or copy this
 e-mail. Please notify the sender immediately and destroy all copies of this
 message and any attachments. WARNING: Computer viruses can be transmitted
 via email. The recipient should check this email and any attachments for the
 presence of viruses. The company accepts no liability for any damage caused
 by any virus transmitted by this email. www.wipro.com

 The information contained in this electronic message and any attachments to
 this message are intended for the exclusive use of the addressee(s) and may
 contain proprietary, confidential or privileged information. If you are not
 the intended recipient, you should not disseminate, distribute or copy this
 e-mail. Please notify the sender immediately and destroy all copies of this
 message and any attachments. WARNING: Computer viruses can be transmitted
 via email. The recipient should check this email and any attachments for the
 presence of viruses. The company accepts no liability for any damage caused
 by any virus transmitted by this email. www.wipro.com

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users




-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Using multiple queues inherits s_rt h_rt

[Feng Zhang]

A question: for -q free64,bio, what GE does to choose an available
queue for a job? Will it sort and do alphabetical order?

On Fri, May 29, 2015 at 8:12 AM, William Hay w@ucl.ac.uk wrote:
 On Thu, 28 May 2015 19:27:07 +
 Joseph Farran jfar...@uci.edu wrote:

 Hi all.

 I am not sure if this is a bug or the way Grid Engine works.

 We have several queues our users submit jobs to.One of the queues
 free64 has a 3-day wall-clock limit:

 $ qconf -sq free64 | grep _rt
 s_rt  72:00:00
 h_rt  72:05:00

 While other queue bio does not:

 $ qconf -sq bio | grep _rt
 s_rt  INFINITY
 h_rt  INFINITY

 When a user submits a job to both queues  -q free64,bio, jobs that
 run longer than 3 days are killed whether they land on free64 or
 bio queue.Why are jobs that land on the bio queue being
 killed after 3 days?


 Are you sure the whole job is in the bio queue?  Might a slave task be
 in the free64 queue?

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users




-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] limit slots to core count no longer works

[Feng Zhang]

What is $num_proc? Did you try to set a real number? Like  limit
hosts {*} to slots=12?

On Tue, Apr 14, 2015 at 3:32 PM, John Young j.e.yo...@larc.nasa.gov wrote:
 Hello,

We (fairly) recently upgraded our cluster to Rocks 6.1.1
 and we now seem to be having problems with RQS.  On our old
 cluster, we had an RQS quota set as follows:

 {
name host-slots
description  restrict slots to core count
enabled  TRUE
limithosts {*} to slots=$num_proc
 }

 The reason for this was to try to prevent oversubscription
 of the processors on the clients.  Now, if I have this quota
 enabled, jobs that are submitted don't start and if I do a
 'qstat -j job-number' under scheduling info I see things like

 cannot run because it exceeds limit compute-0-7/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-7/ in rule host-slots/1
 (-l slots=1) cannot run in queue compute-0-39.local because it offers only 
 hc:slots=0.00
 cannot run because it exceeds limit compute-0-78/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-78/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-55/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-55/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-74/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-74/ in rule host-slots/1
 cannot run because it exceeds limit compute-2-7/ in rule host-slots/1
 cannot run because it exceeds limit compute-2-1/ in rule host-slots/1
 cannot run because it exceeds limit compute-2-2/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-22/ in rule host-slots/1
 cannot run because it exceeds limit compute-0-22/ in rule host-slots/1
 cannot run because it exceeds limit compute-1-2/ in rule host-slots/1
 cannot run in PE mpich because it only offers 0 slots

 But as soon as I run 'qconf -mrqs' and change TRUE to FALSE, the job runs.

 Has the process for preventing oversubscription changed?  Any ideas?

 JY

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] How to set up h_vmem as a consumable resource

[Feng Zhang]

On Wed, Feb 25, 2015 at 6:44 AM, Simon Andrews
simon.andr...@babraham.ac.uk wrote:


 From:  Mishkin Derakhshan mishkin...@gmail.com
 Date:  Tuesday, 24 February 2015 00:07
 To:  users@gridengine.org users@gridengine.org
 Subject:  [gridengine users] How to set up h_vmem as a consumable
resource

 We _think_ this has to do with setting a default h_vmem (on a queue
basis? host basis?) so jobs that
 don't explicitly request the resource will use something by default, but
we've been unable to figure
 out how to set this up.

 I don't think that not adding h_vmem should result in a zero allocation -
 quite the opposite, if you don't add a default allocation then you can use
 infinite memory.

 We tried a couple of ways to enforce this.  We started putting the -l
 h_vmem=1G (or whatever) in the $SGE_ROOT/default/common/sge_request file
 so it gets added to all jobs by default, however the users can remove this
 by putting -clear on their submissions.  We've now moved to using a server
 side job submission verifier (JSV).  I'd be happy to send you the script

Just came into my mind, that use JSV to disable -clear option, and
use sge_request to set defaults?

 we use, but its a little more complex than you might need in that we only
 add the h_vmem to batch jobs and we leave interactive jobs unrestricted.

 That seems to be working really well for us.

 Simon.

 The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT 
 Registered Charity No. 1053902.
 The information transmitted in this email is directed only to the addressee. 
 If you received this in error, please contact the sender and delete this 
 email from your system. The contents of this e-mail are the views of the 
 sender and do not necessarily represent the views of the Babraham Institute. 
 Full conditions at: www.babraham.ac.ukhttp://www.babraham.ac.uk/terms

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



-- 
Best,

Feng

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Huge amount of files generated in local disk

[Feng Zhang]

Thanks, Reuti!

Yes, the user intended to generate those millions files on the  local
scratch directory provided by SGE.

 Can you spot any oom-killer in the messages file of the node?

No, I did not find any useful info. from the log files.

The user happened to run 10 jobs on a node, and these jobs generated
10+million files. It took very long time to even ls these files. It
looks like it takes very long time for execd to remove these files,
since I noticed huge IO by execd after I had deleted these jobs.


On Mon, Jan 26, 2015 at 12:04 PM, Reuti re...@staff.uni-marburg.de wrote:
 Hi,

 Am 26.01.2015 um 17:15 schrieb Feng Zhang prod.f...@gmail.com:

 I just found a strange behavior of SGE 2011.

 One user's job generate 1+ million small files in local
 disk($TEMPDIR).

 Hence in the local scratch directory provided by SGE?


 It looks like it makes the execd very busy and from
 the side of qmaster, the node is lost and unavailable, while I can ssh
 to login. On the node, execd makes huge IOs( a few hundred KB/s to a
 few MB/s). Some nodes can survive and get back to normal, some nodes
 failed at the end(Since this kind of jobs also use a lot of memory, so
 it looks like these nodes failed while the RAM got used up).

 Can you spot any oom-killer in the messages file of the node?


 I am
 wondering that whether the execd handles the files that a job
 generates?

 Not that I'm aware of. It will just remove the generated directory after the 
 job.

 Is it intended by the user to generate this high number of files? It could be 
 limited with a set disk quota.

 -- Reuti


 Or execd does something else to communicate with qmaster
 while there are a lot of job generated files?


 --
 Best,

 Feng
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users




-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Cannot request resource if it is a load value of memory type: SGE reports it as unknown resource

[Feng Zhang]

Llya,

Can you please run:

qstat -j jobid

and past the output here? It may be useful for checking the problem

On Fri, Jan 23, 2015 at 12:08 PM, Ilya M 4ilya.m+g...@gmail.com wrote:
 Removed the quota limits. To no avail: same problems.


  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a load
 value of memory type: SGE reports it as unknown resource
 From: Reuti re...@staff.uni-marburg.de
 To: Ilya M 4ilya.m+g...@gmail.com
 Date: 1/23/15, 2:33 AM

 Can you remove them temporarily? I saw cases where suddenly the unknown
 resource popped up - and also suddenly vanished again, but it was somehow
 connected to RQS was my conclusion.

 -- Reuti


 Am 23.01.2015 um 00:16 schrieb Ilya M 4ilya.m+g...@gmail.com:

 There are two RQS, one is disabled:

 {
name limit_for_interns
description  limit to max 5 GPU jobs per intern.
enabled  TRUE
limitusers {int1,int2} hosts @gpu to slots=5
 }
 {
name limit_slots
description  NONE
enabled  FALSE
limithosts {@gpu} to slots=2
 }


  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a load
 value of memory type: SGE reports it as unknown resource
 From: Reuti re...@staff.uni-marburg.de
 To: Ilya 4ilya.m+g...@gmail.com
 Date: 1/21/15, 16:12

 Hi,

 Am 22.01.2015 um 00:52 schrieb Ilya:

 Something happened to the SGE (6.2u5) that had been running fine for
 many months, and users can no longer put resource requests for load values
 if they are of memory type, e.g.

 qsub -l mem_free=5G -w v  produces the following output:

 cannot run in queue gpu.q@gpu038 because job requests unknown
 resource (mem_free)

 The resource is available, though, when querying for it:
 qhost -F mem_free -h gpu038
 HOSTNAMEARCH NCPU  LOAD  MEMTOT  MEMUSE SWAPTO
 SWAPUS

 ---
 global  -   - -   - -   -
 -
 gpu038 lx24-amd64 16  2.11  126.1G 15.7G
 4.0G 0.0
 Host Resource(s):  hl:mem_free=110.416G


 This was first reported by a user when he tried to request custom hl
 resource. However, it now appears that all hl resources of type memory
 show this behavior. Integer hl are OK.

 Do you have any RQS in place?

 -- Reuti


 I bounced qmaster between master and shadow-master a couple of times,
 but it did not resolve the problem.

 Additionally, when I added MONITOR=1 to scheduler's configuration, the
 file $SGE_ROOT/$SGE_CELL/common/schedule contains only colons:
 
 
 

 Any ideas?

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users


 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



-- 
Best,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Cannot request resource if it is a load value of memory type: SGE reports it as unknown resource

[Feng Zhang]

It looks normal to me.

Can you run:

qhost -F mem_free  -l mem_free=80g

to see if it can list the nodes properly?

On Fri, Jan 23, 2015 at 4:35 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 So it is requestable, but not consumable, and there is no default set
 in the complex. Well, the default is set to zero, but I don't think
 that is treated as a default.

 Is that what was intended - requestable but not consumable?

 Ian

 On Fri, Jan 23, 2015 at 12:36 PM, Ilya M 4ilya.m+g...@gmail.com wrote:
 Natually, it does:

 qconf -sc | grep mem_free
 mem_freemfMEMORY  = YES NO
 00

 And it is reported on all nodes:

 qhost -F mem_free -h gpu001
 HOSTNAMEARCH NCPU  LOAD  MEMTOT  MEMUSE SWAPTO
 SWAPUS
 ---
 global  -   - -   - -   -   -
 gpu001 lx24-amd64 16  3.32  126.1G 37.2G4.0G
 0.0
 Host Resource(s):  hl:mem_free=88.885G

 And everything was working until a week ago.

 Ilya.

  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a load
 value of memory type: SGE reports it as unknown resource
 From: Ian Kaufman ikauf...@eng.ucsd.edu
 To: Ilya M 4ilya.m+g...@gmail.com
 Date: 1/23/15, 11:38 AM

 Is mem_free defined in the host complex_values? What does

 qconf -sc | grep mem_free

 show? Is there a default value defined?

 Ian

 On Fri, Jan 23, 2015 at 11:30 AM, Ilya M 4ilya.m+g...@gmail.com wrote:

 Because I am testing with qsub -w v, the jobs is not accepted for
 scheduling, job id is not generated, and qstat -j will not work. The
 output
 of qsub is as I showed in the original email:

 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu001 because
 job
 requests unknown resource (mem_free)
 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu002 because
 job
 requests unknown resource (mem_free)
 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu003 because
 job
 requests unknown resource (mem_free)
 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu004 because
 job
 requests unknown resource (mem_free)
 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu005 because
 job
 requests unknown resource (mem_free)
 Job 2210897 (mem_free=100G) cannot run in queue gpu.q@gpu006 because
 job
 requests unknown resource (mem_free)
 ...

 Ilya.


  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a load
 value of memory type: SGE reports it as unknown resource
 From: Feng Zhang prod.f...@gmail.com
 To: Ilya M 4ilya.m+g...@gmail.com
 Date: 1/23/15, 9:27 AM

 Llya,

 Can you please run:

 qstat -j jobid

 and past the output here? It may be useful for checking the problem

 On Fri, Jan 23, 2015 at 12:08 PM, Ilya M 4ilya.m+g...@gmail.com wrote:

 Removed the quota limits. To no avail: same problems.


  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a load
 value of memory type: SGE reports it as unknown resource
 From: Reuti re...@staff.uni-marburg.de
 To: Ilya M 4ilya.m+g...@gmail.com
 Date: 1/23/15, 2:33 AM

 Can you remove them temporarily? I saw cases where suddenly the
 unknown
 resource popped up - and also suddenly vanished again, but it was
 somehow
 connected to RQS was my conclusion.

 -- Reuti


 Am 23.01.2015 um 00:16 schrieb Ilya M 4ilya.m+g...@gmail.com:

 There are two RQS, one is disabled:

 {
  name limit_for_interns
  description  limit to max 5 GPU jobs per intern.
  enabled  TRUE
  limitusers {int1,int2} hosts @gpu to slots=5
 }
 {
  name limit_slots
  description  NONE
  enabled  FALSE
  limithosts {@gpu} to slots=2
 }


  Original Message 
 Subject: Re: [gridengine users] Cannot request resource if it is a
 load
 value of memory type: SGE reports it as unknown resource
 From: Reuti re...@staff.uni-marburg.de
 To: Ilya 4ilya.m+g...@gmail.com
 Date: 1/21/15, 16:12

 Hi,

 Am 22.01.2015 um 00:52 schrieb Ilya:

 Something happened to the SGE (6.2u5) that had been running fine
 for
 many months, and users can no longer put resource requests for load
 values
 if they are of memory type, e.g.

 qsub -l mem_free=5G -w v  produces the following output:

 cannot run in queue gpu.q@gpu038 because job requests unknown
 resource (mem_free)

 The resource is available, though, when querying for it:
 qhost -F mem_free -h gpu038
 HOSTNAMEARCH NCPU  LOAD  MEMTOT  MEMUSE
 SWAPTO
 SWAPUS



 ---
 global  -   - -   - -   -
 -
 gpu038 lx24-amd64 16  2.11  126.1G
 15.7G
 4.0G 0.0
   Host Resource(s):  hl:mem_free=110.416G


 This was first

Re: [gridengine users] Requesting a resource OR another resource

[Feng Zhang]

SGE has no information of GPU. Defining a consumable of ngpus is a
way to do that, but SGE still does not know which GPU is assigned to
which job(or process).

What I did is to set a script to assign available GPU id(s) to a
job(or MPI process) , like SGE load sensor, but put it in
/etc/prof.d(RedHat Linux), which can be run for each process(using ssh
as qrsh), which is very useful for parallel GPU jobs. And set GPU to
run in Exclusive  Thread mode.  If you run parallel job, your
program needs to know to choose a usable GPU(assigned by the script
and exported as $CUDA_VISIBLE_DEVICES) for each process, or do not set
the GPU id in the program for each process  and it can also work
through CUDA.

On Wed, Nov 19, 2014 at 11:41 AM, Kevin Taylor
groucho.64...@hotmail.com wrote:

 The catch for us is that we want to know which GPU we're using for the code
 we have.


 Date: Wed, 19 Nov 2014 16:58:32 +0100
 From: roberto.nunn...@supsi.ch
 To: groucho.64...@hotmail.com; users@gridengine.org
 Subject: Re: [gridengine users] Requesting a resource OR another resource


 Hi.

 You have two gpu on one host.. why not define a consumable resource
 gpu=2 and request it with -l gpu=1 ?

 the value of gpu will be decreased by one, and it would be possible for
 another job to ask for the remaining gpu.. or you could request two gpus
 for one job, with -l gpu=2

 Best regards.
 Robi


 Il 19.11.2014 14:44, Kevin Taylor ha scritto:
  I'm not sure if this is possible or not, but thought I'd ask it.
 
  We have a setup of consumable resources for our GPUs. If a system has
  two we have a complex called gpu1_free and gpu2_free. They'll be equal
  to 1 if they're free and zero if they're not. Typically we just request
  like this: qsub -l gpu1_free=1 job.sh
 
  Is there a way though qsub to say
 
  qsub -l gpu1_free=1 OR gpu2_free=1 job.sh
 
  I know putting multiple -l's will ask for both, but we just want one or
  the other, whichever.
 
  Univa has that nice RSMAP feature that would solve our issue, but we
  haven't worked out finances on that yet so we're seeing if we can just
  make it work a little easier with what we have.
 
  Thanks.
 
 
 
  ___
  users mailing list
  users@gridengine.org
  https://gridengine.org/mailman/listinfo/users


 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] SGE and NFS

[Feng Zhang]

Bright sets Spool to be local on each node, while the config and
excusables on NFS if you have a HA configuration on your head servers.
I think in theory, if the active head fails, you can bring it offline
and make the passive head active manually, and your jobs will not be
lost.

From the error message, looks like the NFS server is failed too, that
the node can not mount it. Is the NFS server installed on the failed
head server? I remember that Bright recommends to use a separate NFS
server.

On Wed, Nov 12, 2014 at 11:33 AM, Skylar Thompson
skyl...@u.washington.edu wrote:
 Hi Eric,

 We produce our own RPMs using FPM, just so we don't have to have the
 executables on NFS. When the NFS storage is busy, it can make GE unusable
 and sometimes unstable (if you hit protocol timeouts) if the executables
 and/or job spool are on NFS.

 On Wed, Nov 12, 2014 at 04:26:51PM +, Peskin, Eric wrote:
 All,

 Does SGE have to use NFS or can it work locally on each node?
 If parts of it have to be on NFS, what is the minimal subset?
 How much of this changes if you want redundant masters?

 We have a cluster running CentOS 6.3, Bright Cluster Manager 6.0, and SGE 
 2011.11.  Specifically, SGE is provided by a Bright package: 
 sge-2011.11-360_cm6.0.x86_64

 Twice, we have lost all the running SGE jobs when the cluster failed over 
 from one head node to the other.  =( Not supposed to happen.
 Since then, we have also had many individual jobs get lost.  The later 
 situation correlates with messages in the system logs saying

  abrt[9007]: File '/cm/shared/apps/sge/2011.11/bin/linux-x64/sge_execd' 
  seems to be deleted

 That file lives on an NFS mount on our Isilon storage.
 Surely, the executables don't have to be on NFS?
 Interesting, we are using local spooling, the spool directory on each node 
 is  /cm/local/apps/sge/var/spool , which is, indeed local.
 But the $SGE_ROOT ,  /cm/shared/apps/sge/2011.11 lives on NFS.
 Does any of it need to?
 Maybe just the var part would need to:  /cm/shared/apps/sge/var ?

 Thanks,
 Eric



 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

 --
 -- Skylar Thompson (skyl...@u.washington.edu)
 -- Genome Sciences Department, System Administrator
 -- Foege Building S046, (206)-685-7354
 -- University of Washington School of Medicine
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Enforce users to use specific amount of memory/slot

[Feng Zhang]

Guys,

Just curious, how does the h_vmem work on processes of MPI jobs(or
OPENMP, multi-threading)? I have some parallel jobs, the top command
shows VET of 40GB, while the RES is only 100MB.

On Mon, Jun 30, 2014 at 3:01 PM, Michael Stauffer mgsta...@gmail.com wrote:
 Message: 4
 Date: Mon, 30 Jun 2014 11:53:12 +0200
 From: Txema Heredia txema.llis...@gmail.com
 To: Derrick Lin klin...@gmail.com, SGE Mailing List
 users@gridengine.org
 Subject: Re: [gridengine users] Enforce users to use specific amount
 of  memory/slot
 Message-ID: 53b13388.5060...@gmail.com
 Content-Type: text/plain; charset=iso-8859-1; Format=flowed


 Hi Derrick,

 You could either set h_vmem as a consumable (consumable=yes) attribute
 and set a default value of 8GB for it. This way, whenever a job doesn't
 request any amount of h_vmem, it will automatically request 8GB per
 slot. This will affect all types of jobs.

 You could also define a JSV script that checks the username, and forces
 a -l h_vmem=8G for his/her jobs (
 jsv_sub_add_param('l_hard','h_vmem','8G') ). This will affect all jobs
 for that user, but could turn into a pain to manage.

 Or, you could set a different policy and allow all users to request the
 amount of memory they really need, trying to fit best the node. What is
 the point of forcing the user to reserve 63 additional cores when they
 only need 1 core and 500GB of memory? You could fit in that node one job
 like this, and, say, two 30-core-6GB-memory jobs.

 Txema



 El 30/06/14 08:55, Derrick Lin escribi?:

  Hi guys,
 
  A typical node on our cluster has 64 cores and 512GB memory. So it's
  about 8GB/core. Occasionally, we have some jobs that utilizes only 1
  core but 400-500GB of memory, that annoys lots of users. So I am
  seeking a way that can force jobs to run strictly below 8GB/core
  ration or it should be killed.
 
  For example, the above job should ask for 64 cores in order to use
  500GB of memory (we have user quota for slots).
 
  I have been trying to play around h_vmem, set it to consumable and
  configure RQS
 
  {
  namemax_user_vmem
  enabled true
  description Each user can utilize more than 8GB/slot
  limit   users {bad_user} to h_vmem=8g
  }
 
  but it seems to be setting a total vmem bad_user can use per job.
 
  I would love to set it on users instead of queue or hosts because we
  have applications that utilize the same set of nodes and app should be
  unlimited.
 
  Thanks
  Derrick


 I've been dealing with this too. I'm using h_vmem to kill processes that go
 above the limit, and s_vmem set slightly lower by default to give
 well-behaved processes a chance first to exit gracefully.

 The issue is that these use virtual memory, which is (always, more or less)
 great than resident memory, i.e. the actual ram usage. And with java apps
 like Matlab, the amount of virtual memory reserved/used is HUGE compared to
 resident, by 10x give or take. So it makes it really impracticle actually.
 However so far I've just set the default h_vmem and s_vmem values high
 enough to accomadate jvm apps, and increased the per-host consumable
 appropriately. We don't get fine-grained memory control, but it definitely
 controls out-of-control users/procs that otherwise might gobble up enough
 ram to slow dow the entire node.

 We may switch to UVE just for this reason, to get memory limits based on
 resident memory, if it seems worth it enough in the end.

 -M

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Enforce users to use specific amount of memory/slot

[Feng Zhang]

Sorry a typo. The VET should be VIRT.

On Mon, Jun 30, 2014 at 4:47 PM, Feng Zhang prod.f...@gmail.com wrote:
 Guys,

 Just curious, how does the h_vmem work on processes of MPI jobs(or
 OPENMP, multi-threading)? I have some parallel jobs, the top command
 shows VET of 40GB, while the RES is only 100MB.

 On Mon, Jun 30, 2014 at 3:01 PM, Michael Stauffer mgsta...@gmail.com wrote:
 Message: 4
 Date: Mon, 30 Jun 2014 11:53:12 +0200
 From: Txema Heredia txema.llis...@gmail.com
 To: Derrick Lin klin...@gmail.com, SGE Mailing List
 users@gridengine.org
 Subject: Re: [gridengine users] Enforce users to use specific amount
 of  memory/slot
 Message-ID: 53b13388.5060...@gmail.com
 Content-Type: text/plain; charset=iso-8859-1; Format=flowed


 Hi Derrick,

 You could either set h_vmem as a consumable (consumable=yes) attribute
 and set a default value of 8GB for it. This way, whenever a job doesn't
 request any amount of h_vmem, it will automatically request 8GB per
 slot. This will affect all types of jobs.

 You could also define a JSV script that checks the username, and forces
 a -l h_vmem=8G for his/her jobs (
 jsv_sub_add_param('l_hard','h_vmem','8G') ). This will affect all jobs
 for that user, but could turn into a pain to manage.

 Or, you could set a different policy and allow all users to request the
 amount of memory they really need, trying to fit best the node. What is
 the point of forcing the user to reserve 63 additional cores when they
 only need 1 core and 500GB of memory? You could fit in that node one job
 like this, and, say, two 30-core-6GB-memory jobs.

 Txema



 El 30/06/14 08:55, Derrick Lin escribi?:

  Hi guys,
 
  A typical node on our cluster has 64 cores and 512GB memory. So it's
  about 8GB/core. Occasionally, we have some jobs that utilizes only 1
  core but 400-500GB of memory, that annoys lots of users. So I am
  seeking a way that can force jobs to run strictly below 8GB/core
  ration or it should be killed.
 
  For example, the above job should ask for 64 cores in order to use
  500GB of memory (we have user quota for slots).
 
  I have been trying to play around h_vmem, set it to consumable and
  configure RQS
 
  {
  namemax_user_vmem
  enabled true
  description Each user can utilize more than 8GB/slot
  limit   users {bad_user} to h_vmem=8g
  }
 
  but it seems to be setting a total vmem bad_user can use per job.
 
  I would love to set it on users instead of queue or hosts because we
  have applications that utilize the same set of nodes and app should be
  unlimited.
 
  Thanks
  Derrick


 I've been dealing with this too. I'm using h_vmem to kill processes that go
 above the limit, and s_vmem set slightly lower by default to give
 well-behaved processes a chance first to exit gracefully.

 The issue is that these use virtual memory, which is (always, more or less)
 great than resident memory, i.e. the actual ram usage. And with java apps
 like Matlab, the amount of virtual memory reserved/used is HUGE compared to
 resident, by 10x give or take. So it makes it really impracticle actually.
 However so far I've just set the default h_vmem and s_vmem values high
 enough to accomadate jvm apps, and increased the per-host consumable
 appropriately. We don't get fine-grained memory control, but it definitely
 controls out-of-control users/procs that otherwise might gobble up enough
 ram to slow dow the entire node.

 We may switch to UVE just for this reason, to get memory limits based on
 resident memory, if it seems worth it enough in the end.

 -M

 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Enforce users to use specific amount of memory/slot

[Feng Zhang]

We are running GPU MPI jobs. There are huge amount of data transfering
between the host RAM and the GPU  RAM, which may be the reason.

On Mon, Jun 30, 2014 at 5:01 PM, Michael Stauffer mgsta...@gmail.com wrote:
 If I remember right, h_vmem amount applies to the job and is not scaled by #
 of slots like some other resources. Just did a simple test with an 8-slot
 job (pe_serial) and it only used one 'unit' of h_vmem, i.e. the default
 amount assigned as consumable.

 40GB VIRT vs 100MB RES is a huge difference! I thought I had it bad with
 matlab using 4GB VIRT for 100MB RES.

 -M


 On Mon, Jun 30, 2014 at 4:47 PM, Feng Zhang prod.f...@gmail.com wrote:

 Guys,

 Just curious, how does the h_vmem work on processes of MPI jobs(or
 OPENMP, multi-threading)? I have some parallel jobs, the top command
 shows VET of 40GB, while the RES is only 100MB.

 On Mon, Jun 30, 2014 at 3:01 PM, Michael Stauffer mgsta...@gmail.com
 wrote:
  Message: 4
  Date: Mon, 30 Jun 2014 11:53:12 +0200
  From: Txema Heredia txema.llis...@gmail.com
  To: Derrick Lin klin...@gmail.com, SGE Mailing List
  users@gridengine.org
  Subject: Re: [gridengine users] Enforce users to use specific amount
  of  memory/slot
  Message-ID: 53b13388.5060...@gmail.com
  Content-Type: text/plain; charset=iso-8859-1; Format=flowed
 
 
  Hi Derrick,
 
  You could either set h_vmem as a consumable (consumable=yes) attribute
  and set a default value of 8GB for it. This way, whenever a job doesn't
  request any amount of h_vmem, it will automatically request 8GB per
  slot. This will affect all types of jobs.
 
  You could also define a JSV script that checks the username, and forces
  a -l h_vmem=8G for his/her jobs (
  jsv_sub_add_param('l_hard','h_vmem','8G') ). This will affect all jobs
  for that user, but could turn into a pain to manage.
 
  Or, you could set a different policy and allow all users to request the
  amount of memory they really need, trying to fit best the node. What is
  the point of forcing the user to reserve 63 additional cores when they
  only need 1 core and 500GB of memory? You could fit in that node one
  job
  like this, and, say, two 30-core-6GB-memory jobs.
 
  Txema
 
 
 
  El 30/06/14 08:55, Derrick Lin escribi?:
 
   Hi guys,
  
   A typical node on our cluster has 64 cores and 512GB memory. So it's
   about 8GB/core. Occasionally, we have some jobs that utilizes only 1
   core but 400-500GB of memory, that annoys lots of users. So I am
   seeking a way that can force jobs to run strictly below 8GB/core
   ration or it should be killed.
  
   For example, the above job should ask for 64 cores in order to use
   500GB of memory (we have user quota for slots).
  
   I have been trying to play around h_vmem, set it to consumable and
   configure RQS
  
   {
   namemax_user_vmem
   enabled true
   description Each user can utilize more than 8GB/slot
   limit   users {bad_user} to h_vmem=8g
   }
  
   but it seems to be setting a total vmem bad_user can use per job.
  
   I would love to set it on users instead of queue or hosts because we
   have applications that utilize the same set of nodes and app should
   be
   unlimited.
  
   Thanks
   Derrick
 
 
  I've been dealing with this too. I'm using h_vmem to kill processes that
  go
  above the limit, and s_vmem set slightly lower by default to give
  well-behaved processes a chance first to exit gracefully.
 
  The issue is that these use virtual memory, which is (always, more or
  less)
  great than resident memory, i.e. the actual ram usage. And with java
  apps
  like Matlab, the amount of virtual memory reserved/used is HUGE compared
  to
  resident, by 10x give or take. So it makes it really impracticle
  actually.
  However so far I've just set the default h_vmem and s_vmem values high
  enough to accomadate jvm apps, and increased the per-host consumable
  appropriately. We don't get fine-grained memory control, but it
  definitely
  controls out-of-control users/procs that otherwise might gobble up
  enough
  ram to slow dow the entire node.
 
  We may switch to UVE just for this reason, to get memory limits based on
  resident memory, if it seems worth it enough in the end.
 
  -M
 
  ___
  users mailing list
  users@gridengine.org
  https://gridengine.org/mailman/listinfo/users
 


___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] sequential jobs on cluster

[Feng Zhang]

The method on Stephans' blog works fine.

load_formula  -slots
schedule_interval 00:00:10
load_adjustment_decay_time0:00:09

I am wondering that if it is possible to set the scheduling policy for
each queue? Right now the change will apply to all the queues.  For
example, I want queue FILLUP jobs to fill up a node as possible,
while for queue SERIAL job  fill nodes in a round robin way?

Thanks,

Feng

On Fri, May 23, 2014 at 5:47 PM, Reuti re...@staff.uni-marburg.de wrote:
 Please keep the list posted.

 Am 23.05.2014 um 23:37 schrieb Feng Zhang:

 Thanks, Reuti!

 Your method will work for sure. I am just wondering if SGE's scheduler
 policy can be set to do so. I know that for PE jobs, the PE env can be
 set to distribute processes in a round robin way.

 In principle yes. Either:

 - add an artificial load by setting 
 job_load_adjustments/load_adjustment_decay_time in a schedule to increase 
 np_load_avg

 - Use the least used host: 
 http://wiki.gridengine.info/wiki/index.php/StephansBlog

 But matching your original intention is only correctly mapped by the 
 consumable. The other two ways may fail.

 -- Reuti


 On Fri, May 23, 2014 at 5:31 PM, Reuti re...@staff.uni-marburg.de wrote:
 Hi,

 Am 23.05.2014 um 23:14 schrieb Feng Zhang:

 I am now running large disk IO jobs(sequential jobs NOT PE jobs) on my
 cluster. Is there any way I can submit my jobs evenly to all the
 nodes? The problem I have now is:

 I have 10 jobs, and I have 10 nodes, and each node has 10 CPU cores.
 When I submit my jobs, all the 10 jobs are distributed to one node,
 and the other nodes are idling(almost). How can I run 1 job on 1 node
 instead?

 By using a consumable complex like bigio and attach a complex_value of 
 one to each exechost for it. Then request this also during job submission.

 -- Reuti

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] sequential jobs on cluster

[Feng Zhang]

On Wed, May 28, 2014 at 11:13 AM, Feng Zhang prod.f...@gmail.com wrote:
 The method on Stephans' blog works fine.

Works fine:  I mean it can distribute jobs evenly into the cluster,
but not in round robin way.


 load_formula  -slots
 schedule_interval 00:00:10
 load_adjustment_decay_time0:00:09

 I am wondering that if it is possible to set the scheduling policy for
 each queue? Right now the change will apply to all the queues.  For
 example, I want queue FILLUP jobs to fill up a node as possible,
 while for queue SERIAL job  fill nodes in a round robin way?

 Thanks,

 Feng

 On Fri, May 23, 2014 at 5:47 PM, Reuti re...@staff.uni-marburg.de wrote:
 Please keep the list posted.

 Am 23.05.2014 um 23:37 schrieb Feng Zhang:

 Thanks, Reuti!

 Your method will work for sure. I am just wondering if SGE's scheduler
 policy can be set to do so. I know that for PE jobs, the PE env can be
 set to distribute processes in a round robin way.

 In principle yes. Either:

 - add an artificial load by setting 
 job_load_adjustments/load_adjustment_decay_time in a schedule to 
 increase np_load_avg

 - Use the least used host: 
 http://wiki.gridengine.info/wiki/index.php/StephansBlog

 But matching your original intention is only correctly mapped by the 
 consumable. The other two ways may fail.

 -- Reuti


 On Fri, May 23, 2014 at 5:31 PM, Reuti re...@staff.uni-marburg.de wrote:
 Hi,

 Am 23.05.2014 um 23:14 schrieb Feng Zhang:

 I am now running large disk IO jobs(sequential jobs NOT PE jobs) on my
 cluster. Is there any way I can submit my jobs evenly to all the
 nodes? The problem I have now is:

 I have 10 jobs, and I have 10 nodes, and each node has 10 CPU cores.
 When I submit my jobs, all the 10 jobs are distributed to one node,
 and the other nodes are idling(almost). How can I run 1 job on 1 node
 instead?

 By using a consumable complex like bigio and attach a complex_value of 
 one to each exechost for it. Then request this also during job submission.

 -- Reuti

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

[gridengine users] sequential jobs on cluster

[Feng Zhang]

Hi,

I am now running large disk IO jobs(sequential jobs NOT PE jobs) on my
cluster. Is there any way I can submit my jobs evenly to all the
nodes? The problem I have now is:

I have 10 jobs, and I have 10 nodes, and each node has 10 CPU cores.
When I submit my jobs, all the 10 jobs are distributed to one node,
and the other nodes are idling(almost). How can I run 1 job on 1 node
instead?


Thanks,

Feng
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Script in Parallel Environment

[Feng Zhang]

For red hat Linux, you may try to put the script into /etc/profile.d, for
bash env.


On Tuesday, April 29, 2014, Yago Fernández Pinilla yago...@gmail.com
wrote:

 Hi all,

 Is it possible to execute a script in Grid Engine as a startup script in
 every of the nodes with tight integration?

 The same as *starter_method * but instead of executing in the main node
 in all the nodes.

 Thanks in advance

 Yago

 --
 Yago Fernández Pinilla
 e-mail: yago...@gmail.comjavascript:_e(%7B%7D,'cvml','yago...@gmail.com');


___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

[gridengine users] SGE and GPU

[Feng Zhang]

Hi,

Is there's any plan to implement the GPU resource management in SGE in
the near future? Like Slurm or Torque? There are some ways to do this
using scripts/programs, but I wonder that if the SGE itself can
recognize and manage GPU(and Phi). Not need to be complicated and
powerful, just do basic work.

Thanks,
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] SGE and GPU

[Feng Zhang]

Thanks, Ian!

I haven't checked the GPU load sensor in detail, either. It sounds to
me it only handles the number of GPU allocated to a job, but the job
doesn't know which GPUs it actually get and set the
CUDA_VISIBLE_DEVICE(some programs need this env to be set). This can
be done by writing some scripts/programs, but to me, it is not an
accurate solution, since some jobs may still happen to collide to each
other on the same GPU on a multiple GPU node. If GE can have the
memory to record the GPUs allocated to a job, then this can be
perfect.


On Mon, Apr 14, 2014 at 1:46 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 I believe there already is support for GPUs - there is a GPU Load
 Sensor in Open Grid Engine. You may have to build it yourself, I
 haven't checked to see if it comes pre-packaged.

 Univa has Phi support, and I believe OGE/OGS has it as well, or at
 least has been working on it.

 Ian

 On Mon, Apr 14, 2014 at 10:35 AM, Feng Zhang prod.f...@gmail.com wrote:
 Hi,

 Is there's any plan to implement the GPU resource management in SGE in
 the near future? Like Slurm or Torque? There are some ways to do this
 using scripts/programs, but I wonder that if the SGE itself can
 recognize and manage GPU(and Phi). Not need to be complicated and
 powerful, just do basic work.

 Thanks,
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



 --
 Ian Kaufman
 Research Systems Administrator
 UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] SGE and GPU

[Feng Zhang]

Thanks, Reuti,

The socket solution looks like only work fine for serial jobs, not PE
jobs, right?

Our cluster has different nodes, some nodes each has 2 GPUs, some
others each has 4 GPUs. Most of the user jobs are PE jobs, some are
serial.

The socket solution can event work for PE jobs, but as my
understanding, it is not efficient? Since each node has, for example,
4 queues. If one user submit a PE job to a queue, he/she can not use
the other GPUs on the other queues?

On Mon, Apr 14, 2014 at 2:16 PM, Reuti re...@staff.uni-marburg.de wrote:
 Am 14.04.2014 um 20:06 schrieb Feng Zhang:

 Thanks, Ian!

 I haven't checked the GPU load sensor in detail, either. It sounds to
 me it only handles the number of GPU allocated to a job, but the job
 doesn't know which GPUs it actually get and set the
 CUDA_VISIBLE_DEVICE(some programs need this env to be set). This can
 be done by writing some scripts/programs, but to me, it is not an
 accurate solution, since some jobs may still happen to collide to each
 other on the same GPU on a multiple GPU node. If GE can have the
 memory to record the GPUs allocated to a job, then this can be
 perfect.

 Like the option to request sockets instead of cores which I posted in the 
 last couple of days, you can use a similar approach to get the number of the 
 granted GPU out of the queue name.

 -- Reuti


 On Mon, Apr 14, 2014 at 1:46 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 I believe there already is support for GPUs - there is a GPU Load
 Sensor in Open Grid Engine. You may have to build it yourself, I
 haven't checked to see if it comes pre-packaged.

 Univa has Phi support, and I believe OGE/OGS has it as well, or at
 least has been working on it.

 Ian

 On Mon, Apr 14, 2014 at 10:35 AM, Feng Zhang prod.f...@gmail.com wrote:
 Hi,

 Is there's any plan to implement the GPU resource management in SGE in
 the near future? Like Slurm or Torque? There are some ways to do this
 using scripts/programs, but I wonder that if the SGE itself can
 recognize and manage GPU(and Phi). Not need to be complicated and
 powerful, just do basic work.

 Thanks,
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



 --
 Ian Kaufman
 Research Systems Administrator
 UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users

___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] SGE and GPU

[Feng Zhang]

Thanks, Ian and Gowtham!


This is a very nice instruction.  One of my problem is, for example:

node1,  number of gpu=4
node2,  number of gpu=4
node3,  number of gpu=2

So in total I have 10 GPUs.

Right now, user A has a serial GPU job, which takes one GPU on
node1(Don't know which GPU though). So node1:3, node2:4 and node3:2
GPUs are still free for jobs.

I submit one job with PE=8. SGE allocate all the 3 nodes to me with 8
GPU slots. The problem is now: how my job knows what GPUs it can get
on node1?

Best




On Mon, Apr 14, 2014 at 4:13 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 Again, look into using it as a consumable resource as Gowtham posted above.

 Ian

 On Mon, Apr 14, 2014 at 11:57 AM, Feng Zhang prod.f...@gmail.com wrote:
 Thanks, Reuti,

 The socket solution looks like only work fine for serial jobs, not PE
 jobs, right?

 Our cluster has different nodes, some nodes each has 2 GPUs, some
 others each has 4 GPUs. Most of the user jobs are PE jobs, some are
 serial.

 The socket solution can event work for PE jobs, but as my
 understanding, it is not efficient? Since each node has, for example,
 4 queues. If one user submit a PE job to a queue, he/she can not use
 the other GPUs on the other queues?

 On Mon, Apr 14, 2014 at 2:16 PM, Reuti re...@staff.uni-marburg.de wrote:
 Am 14.04.2014 um 20:06 schrieb Feng Zhang:

 Thanks, Ian!

 I haven't checked the GPU load sensor in detail, either. It sounds to
 me it only handles the number of GPU allocated to a job, but the job
 doesn't know which GPUs it actually get and set the
 CUDA_VISIBLE_DEVICE(some programs need this env to be set). This can
 be done by writing some scripts/programs, but to me, it is not an
 accurate solution, since some jobs may still happen to collide to each
 other on the same GPU on a multiple GPU node. If GE can have the
 memory to record the GPUs allocated to a job, then this can be
 perfect.

 Like the option to request sockets instead of cores which I posted in the 
 last couple of days, you can use a similar approach to get the number of 
 the granted GPU out of the queue name.

 -- Reuti


 On Mon, Apr 14, 2014 at 1:46 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 I believe there already is support for GPUs - there is a GPU Load
 Sensor in Open Grid Engine. You may have to build it yourself, I
 haven't checked to see if it comes pre-packaged.

 Univa has Phi support, and I believe OGE/OGS has it as well, or at
 least has been working on it.

 Ian

 On Mon, Apr 14, 2014 at 10:35 AM, Feng Zhang prod.f...@gmail.com wrote:
 Hi,

 Is there's any plan to implement the GPU resource management in SGE in
 the near future? Like Slurm or Torque? There are some ways to do this
 using scripts/programs, but I wonder that if the SGE itself can
 recognize and manage GPU(and Phi). Not need to be complicated and
 powerful, just do basic work.

 Thanks,
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



 --
 Ian Kaufman
 Research Systems Administrator
 UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users




 --
 Ian Kaufman
 Research Systems Administrator
 UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] SGE and GPU

[Feng Zhang]

On Mon, Apr 14, 2014 at 5:36 PM, Reuti re...@staff.uni-marburg.de wrote:
 Am 14.04.2014 um 20:57 schrieb Feng Zhang:

 Thanks, Reuti,

 The socket solution looks like only work fine for serial jobs, not PE
 jobs, right?

 You mean using more than one GPU at a time, or using parallel processes as 
 usual in addition to the GPU?


Thanks, Reuti!
Right, using parallel processes running on GPUs(on the same node, and
also cross multiple nodes).

Best


 Our cluster has different nodes, some nodes each has 2 GPUs, some
 others each has 4 GPUs. Most of the user jobs are PE jobs, some are
 serial.

 The socket solution can event work for PE jobs, but as my
 understanding, it is not efficient? Since each node has, for example,
 4 queues. If one user submit a PE job to a queue, he/she can not use
 the other GPUs on the other queues?

 In SGE you don't submit to queues. You request resources. In case you want a 
 GPU job going to a set of queues the best way would be to attach a boolean 
 complex to these queues and submit the job with a request for this complex. 
 SGE is then free to elect any of the queues with this feature.

 Look at the link Gowtham posted for this.

 -- Reuti


 On Mon, Apr 14, 2014 at 2:16 PM, Reuti re...@staff.uni-marburg.de wrote:
 Am 14.04.2014 um 20:06 schrieb Feng Zhang:

 Thanks, Ian!

 I haven't checked the GPU load sensor in detail, either. It sounds to
 me it only handles the number of GPU allocated to a job, but the job
 doesn't know which GPUs it actually get and set the
 CUDA_VISIBLE_DEVICE(some programs need this env to be set). This can
 be done by writing some scripts/programs, but to me, it is not an
 accurate solution, since some jobs may still happen to collide to each
 other on the same GPU on a multiple GPU node. If GE can have the
 memory to record the GPUs allocated to a job, then this can be
 perfect.

 Like the option to request sockets instead of cores which I posted in the 
 last couple of days, you can use a similar approach to get the number of 
 the granted GPU out of the queue name.

 -- Reuti


 On Mon, Apr 14, 2014 at 1:46 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote:
 I believe there already is support for GPUs - there is a GPU Load
 Sensor in Open Grid Engine. You may have to build it yourself, I
 haven't checked to see if it comes pre-packaged.

 Univa has Phi support, and I believe OGE/OGS has it as well, or at
 least has been working on it.

 Ian

 On Mon, Apr 14, 2014 at 10:35 AM, Feng Zhang prod.f...@gmail.com wrote:
 Hi,

 Is there's any plan to implement the GPU resource management in SGE in
 the near future? Like Slurm or Torque? There are some ways to do this
 using scripts/programs, but I wonder that if the SGE itself can
 recognize and manage GPU(and Phi). Not need to be complicated and
 powerful, just do basic work.

 Thanks,
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



 --
 Ian Kaufman
 Research Systems Administrator
 UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
 ___
 users mailing list
 users@gridengine.org
 https://gridengine.org/mailman/listinfo/users



___
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users