Hi,
I'm not sure if this helps, but we have a working config with:
queue_sort_method seqno
load_formula slots
That puts single-slot jobs onto a single node if a bunch of nodes are
empty, rather than distributing them evenly across empty nodes.
Regards,
Alex
On 08/13/2012 09:14 AM, Joseph Farran wrote:
Hi Reuti / Rayson.
To make we are on the same page, are you saying that for PE jobs using
"$pe_slots" for the "allocation_rule", that Grid Engine does indeed
ignore the "load_formula" on the scheduler?
If Yes, a couple of questions please:
1) Was there a point in which GE did *not* ignore the
"load_formula" for PE jobs "$pe_slots"?
2) Will this be brought back to GE on a future release?
Joseph
On 08/13/2012 08:22 AM, Reuti wrote:
Am 12.08.2012 um 19:55 schrieb Joseph Farran:
Hi Rayson.
Here is one particular entry:
http://gridengine.org/pipermail/users/2012-May/003495.html
I am using Grid Engine 2011.11 binary
http://dl.dropbox.com/u/47200624/respin/ge2011.11.tar.gz
First of all sorry for using the wrong expression. If you used
"-cores_in_use", it should be the positive "slots". As a lower value
is taken first, a lower remaining number of slots should be taken
first. It's working as it should for serial jobs.
But for parallel ones, even with $pe_slots as allocation rule, it's
ignored already in 6.2u5.
-- Reuti
Thanks,
Joseph
On 8/12/2012 10:10 AM, Rayson Ho wrote:
On Sun, Aug 12, 2012 at 5:27 AM, Joseph Farran<[email protected]> wrote:
I saw some old postings that this used to be a bug with GE, that
parallel
jobs were not using the scheduler load_formula. Was this bug
corrected in
GE2011.11 ?
Hi Joseph,
Can you point me to the previous discussion? We did not receive bug
report related to this problem before...
So far, our main focus is to fix issues& bugs reported by our users
first, and may be we've missed the discussion on this bug.
Rayson
Anyone able to test this in GE2011.11 to see if it was fixed?
Joseph
On 8/11/2012 1:51 PM, Reuti wrote:
Am 11.08.2012 um 20:30 schrieb Joseph Farran:
Yes, all my queues have the same "0" for "seq_no".
Here is my scheduler load formula:
qconf -ssconf
algorithm default
schedule_interval 0:0:15
maxujobs 0
queue_sort_method load
job_load_adjustments NONE
load_adjustment_decay_time 0
load_formula -cores_in_use
Can you please try it with -slots? It should behave the same like
your own
complex. In one of your former post you mentioned a different
relation ==
for it.
-- Reuti
Here is a sample display of what is going on. My compute nodes
have 64
cores each:
I submit 4 1-core jobs to my bio queue. Note: I wait around 30
seconds
before submitting each 1-core job, long enough for my
"cores_in_use" to
report back correctly:
job-ID name user state queue slots
-----------------------------------------------------
2324 TEST me r bio@compute-2-3 1
2325 TEST me r bio@compute-2-3 1
2326 TEST me r bio@compute-2-3 1
2327 TEST me r bio@compute-2-3 1
Everything works great with single 1-core jobs. Jobs 2324
through 2327
packed unto one node ( compute-2-3 ) correctly. The
"cores_in_use" for
compute-2-3 reports "4".
Now I submit one 16-core "openmp" PE job:
job-ID name user state queue slots
-----------------------------------------------------
2324 TEST me r bio@compute-2-3 1
2325 TEST me r bio@compute-2-3 1
2326 TEST me r bio@compute-2-3 1
2327 TEST me r bio@compute-2-3 1
2328 TEST me r bio@compute-2-6 16
The scheduler should have picked compute-2-3 since it has 4
cores_in_use,
but instead, it picked compute-2-6 which had 0 cores_in_use.
So here the
scheduler is now behaving differently than with 1-core jobs.
As a further test I wait until my cores_in_use report back that
compute2-6 has "16" cores in use. I now submit another 16-core
"openmp"
job:
job-ID name user state queue slots
-----------------------------------------------------
2324 TEST me r bio@compute-2-3 1
2325 TEST me r bio@compute-2-3 1
2326 TEST me r bio@compute-2-3 1
2327 TEST me r bio@compute-2-3 1
2328 TEST me r bio@compute-2-6 16
2329 TEST me r bio@compute-2-7 16
The schedule now picks yet a different node compute-2-7 which had 0
cores_in_use. I have tried this several times with many config
changes to
the scheduler and it sure looks like that the scheduler is *not*
using the
"load_formula" for PE jobs. From what I can tell, the scheduler
chooses
nodes in random with PE jobs.
Here is my "openmp" PE:
# qconf -sp openmp
pe_name openmp
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args NONE
stop_proc_args NONE
allocation_rule $pe_slots
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE
Here is my "bio" Q showing relevant info:
# qconf -sq bio | egrep "qname|slots|pe_list"
qname bio
pe_list make mpi openmp
slots 64
Thanks for taking a look at this!
On 8/11/2012 4:32 AM, Reuti wrote:
Am 11.08.2012 um 02:57 schrieb Joseph Farran<[email protected]>:
Reuti,
Are you sure this works in GE2011.11?
I have defined my own complex called "cores_in_use" which
counts both
single cores and PE cores correctly.
It works great for single core jobs, but not for PE jobs using the
"$pe_slots" allocation rule.
# qconf -sp openmp
pe_name openmp
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args NONE
stop_proc_args NONE
allocation_rule $pe_slots
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE
# qconf -ssconf
algorithm default
schedule_interval 0:0:15
maxujobs 0
queue_sort_method seqno
The seq_no is the same for the queue instances in question?
-- Reuti
job_load_adjustments cores_in_use=1
load_adjustment_decay_time 0
load_formula -cores_in_use
schedd_job_info true
flush_submit_sec 5
flush_finish_sec 5
I wait until the node reports the correct "cores_in_use"
complex, I
then submit a PE openmp job and it totally ignores the
"load_formula" on the
scheduler.
Joseph
On 08/09/2012 12:50 PM, Reuti wrote:
Correct. It uses the "allocation_rule" specified in the PE
instead.
Only for "allocation_rule" set to $PE_SLOTS it will also use the
"load_formula". Unfortunately there is nothing what you can do
to change the
behavior.
-- Reuti
Am 09.08.2012 um 21:23 schrieb Joseph Farran<[email protected]>:
Howdy.
I am using GE2011.11.
I am successfully using GE "load_formula" to load jobs by
core count
using my own "load_sensor" script.
All works as expected with single core jobs, however, for PE
jobs, it
seems as if GE does not abide by the "load_formula".
Does the scheduler use a different "load" formula for single
core
jobs verses parallel jobs suing the PE environment setup?
Joseph
--
Alex Chekholko [email protected]
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
