Thank you all for the clarification. I have tried both as JOB and as YES, and
they both report the correct virtual_free usage in my testing.
# qstat -F mem
queuename qtype resv/used/tot. load_avg arch
states
---------------------------------------------------------------------------------
all.q@ibm037 BIP 0/4/8 0.00 lx-amd64
hc:virtual_free=10.000G
194 0.50500 xclock johnt r 12/21/2016 09:57:28 2
196 0.50500 xclock johnt r 12/21/2016 10:09:58 2
---------------------------------------------------------------------------------
all.q@ibm038 BIP 0/7/8 0.00 lx-amd64
hc:virtual_free=6.000G
195 0.60500 xclock johnt r 12/21/2016 09:59:43 7
I'm not super clear on what h_vmem is, is it better than using virtual_free? It
seems to work ok with virtual_free, I set it as the physical RAM so that it
doesn't use swap. I followed EXAMPLE 2 on this page:
https://docs.oracle.com/cd/E19957-01/820-0698/chp8-1501/index.html
-----Original Message-----
From: Reuti [mailto:re...@staff.uni-marburg.de]
Sent: Wednesday, December 21, 2016 7:05
To: Christopher Black
Cc: John_Tai; users@gridengine.org; Coleman, Marcus [JRDUS Non-J&J]
Subject: Re: [gridengine users] John's cores pe (Was: users Digest...)
Am 20.12.2016 um 23:42 schrieb Christopher Black:
> We have found that the behavior that multiples consumable memory resource
> requests by number of pe slots can be confusing (and requires extra math in
> automation scripts), so we've have the complex consumable value set to "JOB"
> rather than "YES". When this is done (at least on SoGE), the memory requested
> is NOT multiplied by the number of slots. We also use h_vmem rather than
> virtual_free.
Correct, it's not multiplied. But only the master exechost will get its memory
reduced in the bookeeping. The slave exechosts might still show a too high
value of the available memory I fear.
-- Reuti
> Best,
> Chris
>
> On 12/20/16, 5:11 AM, "users-boun...@gridengine.org on behalf of Reuti"
> <users-boun...@gridengine.org on behalf of re...@staff.uni-marburg.de> wrote:
>
>
>> Am 20.12.2016 um 02:45 schrieb John_Tai <john_...@smics.com>:
>>
>> I spoke too soon. I can request PE and virtual_free separately, but I cannot
>> request both:
>>
>>
>>
>> # qsub -V -b y -cwd -now n -pe cores 7 -l mem=10G -q all.q@ibm037
>> xclock
>
> Above you request "mem" (which is a snapshot of the actual usage and may
> vary over the runtime of other jobs [unless they request the total amount
> already at the beginning of the job and stay with it]).
>
>> Your job 180 ("xclock") has been submitted # qstat
>> job-ID prior name user state submit/start at queue
>> slots ja-task-ID
>> -----------------------------------------------------------------------------------------------------------------
>> 180 0.55500 xclock johnt qw 12/20/2016 09:43:41
>> 7
>> # qstat -j 180
>> ==============================================================
>> job_number: 180
>> exec_file: job_scripts/180
>> submission_time: Tue Dec 20 09:43:41 2016
>> owner: johnt
>> uid: 162
>> group: sa
>> gid: 4563
>> sge_o_home: /home/johnt
>> sge_o_log_name: johnt
>> sge_o_path:
>> /home/sge/sge8.1.9-1.el5/bin:/home/sge/sge8.1.9-1.el5/bin/lx-amd64:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/home/johnt/bin:.
>> sge_o_shell: /bin/tcsh
>> sge_o_workdir: /home/johnt/sge8
>> sge_o_host: ibm005
>> account: sge
>> cwd: /home/johnt/sge8
>> hard resource_list: virtual_free=10G
>
> 10G times 7 = 70 GB
>
> The node has this amount of memory installed and it is defined this way in
> `qconf -me ibm037`?
>
> -- Reuti
>
>
>> mail_list: johnt@ibm005
>> notify: FALSE
>> job_name: xclock
>> jobshare: 0
>> hard_queue_list: all.q@ibm037
>> env_list: TERM=xterm,DISPLAY=dsls11:3. [..]
>> script_file: xclock
>> parallel environment: cores range: 7
>> binding: NONE
>> job_type: binary
>> scheduling info: cannot run in queue "sim.q" because it is not
>> contained in its hard queue list (-q)
>> cannot run in queue "pc.q" because it is not
>> contained in its hard queue list (-q)
>> cannot run in PE "cores" because it only
>> offers 0 slots
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Reuti [mailto:re...@staff.uni-marburg.de]
>> Sent: Saturday, December 17, 2016 10:16
>> To: Reuti
>> Cc: John_Tai; users@gridengine.org; Coleman, Marcus [JRDUS Non-J&J]
>> Subject: Re: [gridengine users] John's cores pe (Was: users
>> Digest...)
>>
>>
>> Am 17.12.2016 um 11:34 schrieb Reuti:
>>
>>>
>>> Am 17.12.2016 um 02:01 schrieb John_Tai:
>>>
>>>> It is working!! Thank you to all that replied to me and helped me figure
>>>> this out.
>>>>
>>>> I meant to set the default to 2G so that was my mistake. I changed it to:
>>>>
>>>> virtual_free mem MEMORY <= YES YES 2G
>>>> 0
>>>
>>> That's strange. A plain "2" was for me always two bytes. A "h_vmem" of 2
>>> bytes would crash the job instantly when it got scheduled, but for
>>> "virtual_free" (which is only a guidance for SGE how to distribute jobs) it
>>> shouldn't hinder the scheduling at all.
>>>
>>> `man sge_types` also lists:
>>>
>>> If no multiplier is present, the value is just counted in bytes.
>>
>> We have set "-w e" in /usr/sge/default/common/sge_request, and then I even
>> face an "Unable to run job: error: no suitable queues." This happens whether
>> the low 2 byte value is specified in the complex definition `qconf -mc` or
>> on the command line as "-l virutal_free=2".
>>
>> It turns out, that the minimum value which is being accepted is: 33.
>>
>> -- Reuti
>>
>>
>>>
>>>> And it's working now. Although I'm not sure why it affected the PE.
>>>>
>>>> Also I didn't set a global one, what is the purpose of the global one?
>>>> Should I set it?
>>>
>>> No, it was only one place I would have checked too. The global complexes
>>> therein can for example be used for a limit in the number of licenses of an
>>> application you have and which can be used floating in the cluster (one
>>> could prefer to put such a limit in an RQS though).
>>>
>>> If you would have set it up there, it would have been the "overall limit of
>>> memory which can be used in the complete cluster at the same time".
>>>
>>> -- Reuti
>>>
>>>
>>>> # qconf -se global
>>>> hostname global
>>>> load_scaling NONE
>>>> complex_values NONE
>>>> load_values NONE
>>>> processors 0
>>>> user_lists NONE
>>>> xuser_lists NONE
>>>> projects NONE
>>>> xprojects NONE
>>>> usage_scaling NONE
>>>> report_variables NONE
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Reuti [mailto:re...@staff.uni-marburg.de]
>>>> Sent: Friday, December 16, 2016 7:36
>>>> To: John_Tai
>>>> Cc: Christopher Heiny; users@gridengine.org; Coleman, Marcus [JRDUS
>>>> Non-J&J]
>>>> Subject: Re: [gridengine users] John's cores pe (Was: users
>>>> Digest...)
>>>>
>>>>
>>>>> Am 16.12.2016 um 09:53 schrieb John_Tai <john_...@smics.com>:
>>>>>
>>>>> virtual_free mem MEMORY <= YES YES 2
>>>>> 0
>>>>
>>>> This would mean, that the default consumption is 2 bytes. I already feared
>>>> that a high values was programmed here. More suitable would be a default
>>>> of 1G or so.
>>>>
>>>> Is there any virtual_free complex defined on a global level: qconf
>>>> -se global
>>>>
>>>> -- Reuti
>>>> ________________________________
>>>>
>>>> This email (including its attachments, if any) may be confidential and
>>>> proprietary information of SMIC, and intended only for the use of the
>>>> named recipient(s) above. Any unauthorized use or disclosure of this email
>>>> is strictly prohibited. If you are not the intended recipient(s), please
>>>> notify the sender immediately and delete this email from your computer.
>>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users@gridengine.org
>>> https://gridengine.org/mailman/listinfo/users
>>
>> ________________________________
>>
>> This email (including its attachments, if any) may be confidential and
>> proprietary information of SMIC, and intended only for the use of the named
>> recipient(s) above. Any unauthorized use or disclosure of this email is
>> strictly prohibited. If you are not the intended recipient(s), please notify
>> the sender immediately and delete this email from your computer.
>>
>
>
> _______________________________________________
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
>
>
> This electronic message is intended for the use of the named recipient only,
> and may contain information that is confidential, privileged or protected
> from disclosure under applicable law. If you are not the intended recipient,
> or an employee or agent responsible for delivering this message to the
> intended recipient, you are hereby notified that any reading, disclosure,
> dissemination, distribution, copying or use of the contents of this message
> including any of its attachments is strictly prohibited. If you have received
> this message in error or are not the named recipient, please notify us
> immediately by contacting the sender at the electronic mail address noted
> above, and destroy all copies of this message. Please note, the recipient
> should check this email and any attachments for the presence of viruses. The
> organization accepts no liability for any damage caused by any virus
> transmitted by this email.
>
________________________________
This email (including its attachments, if any) may be confidential and
proprietary information of SMIC, and intended only for the use of the named
recipient(s) above. Any unauthorized use or disclosure of this email is
strictly prohibited. If you are not the intended recipient(s), please notify
the sender immediately and delete this email from your computer.
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
