Re: [gridengine users] Definitions of "complex" and "queue"?

Fritz Ferstl Mon, 09 May 2011 23:03:19 -0700

Let me provide some background regarding the goofy usage of the term "queue" in Grid Engine. No, it is not an artifact of German engineers being involved (trust me, I was the first and the lead ever since). The history of Grid Engine started late 1992 when we negotiated a license with Florida state university to productize and commercially distribute a public domain SW called DQS (Distributed Queuing System). The notion of a queue as a place where jobs execute is rooted in DQS. That's how it was in DQS and we kept it that way for compatibility reasons in the follow-on product called CODINE, which was renamed into Grid Engine then much much later. HOWEVER, this is no restriction - it even has some advantages. Classical queuing systems have one or more waiting queues. The problem with that is that jobs sort of get stuck in those queues once being in there. If conditions or configurations in the cluster change dynamically then thewaiting queues jobs are in may not be best suited anymore. You might be able to re-queue a job but that his its issues also. In Grid Engine all jobs wait in a single real queue - the pending jobs list. The resource requests and attributes you submit a job with associate it with one or several of the cluster queues which are defined. So while queues are really where jobs execute, that association, job characteristics to cluster queues, creates logical segmentation in the pending jobs list. Some job can go here, others can go there. The advantage is that this conserves full flexibility in the case of changes until a job really gets started. So if you change a queue configuration then new jobs may become eligible for it or others will not any longer. This will happen automatically. You don't need to re-queue jobs. I hope this helps. Cheers, Fritz PS: The term "complex" was also introduced not by us Germans. I'm not sure anymore whether it is another DQS artifact or something coming from the POSIX queuing system standard. Anyhow, the meaning is that it's a complex of attributes which is to be defined.
Am 10.05.11 05:37, schrieb Stuart Barkley:

On Mon, 9 May 2011 at 14:17 -0000, Reuti wrote:

I also have always had problems with the usage of "queue" in Grid
Engine.


To me a queue would be a set of jobs waiting to run, usually with
an ordering (first come first serve, priority, random).

Grid Engine seems to apply 'queue' to a group of resources with a
common set of configuration/attributes.

Yes, "queue" is something to do work in, not to wait. But it maybe
already be defined this way in POSIX.

Really?  To me a queue is where you wait until you get assigned to a
worker.  You have the email queue (mail waiting to go out or waiting
to be delivered).  In a network device the packet queue is where
packets sit until they are able to be processed.

As someone else pointed out, there might be a German/English language
issue in play here.  What is the German word for this concept
"something to do work in, not to wait"?

This next bit is quite off topic.  Feel free to ignore...

Using google translate starting from English "queue", most of the
German translations map back to about the same concept in English.

** A few double translations (en->de->en) go a little off the mark.

queue (noun) -> queue ->
   1. queue
   2. cue ** (pronounced the same, but very different concept)

(most of the other nouns are strange, but do translate back to queue
in some form)

queue (verb) -> anstehen ->
   1. queue
   2. hesitate (**)
   3. queue up
   4. befit (**)
   5. stand in line
   6. be due
   7. be on the agenda

queue (verb) -> Schlange stehen ->
   1. queue
   2. stand in line

queue (verb) -> sich anstellen ->
   1. queue
   2. behave (**)
   3. queue up
   4. act (**)
   5. line up
   6. stand in line
   7. make a fuss (**)
   8. act up (**)

queue (verb) -> eine Schlange bilden ->
   1. queue

As a simplified user concept the use of the word 'queue' can
convey a useful model and also be a method to apply differing
defaults to a job ("Submit to the short job queue").

No. Specify resource requests in time, memory,... and SGE will
select an appropriate queue for you. Submitting to a queue is
Torque/PBS style.

Actually, this is what I prefer (request what you really want), but
I've been unable to eradicate the concept of distinct queues from our
users.  The Internet is full of simplified user instructions to submit
jobs with "-q queuename" (unfortunately they usually don't show the
implementing SGE configurations).

I dislike the fact that someone created "long", "medium" and "short"
as SGE queues for our users and told them all they need to do is
select the correct queue for their job.  I can't change the parameters
of these without users getting unexpected behavior changes.

I do want to have resources with different wall clock limits so that
heavy compute users don't grab all of the available resources for
several weeks.  I think this is/should be another FAQ with a few
completely worked example configurations.

Do people actually have lots of queues?  Or would just 'interactive'
and 'batch' work suitably as queues?

I've picked up a lot of good information in the past couple of weeks.
Now I need to get some time to work with things and try some more
experiments.

Thanks,
Stuart

Univa Fritz Ferstl | CTO and Business Development, EMEA
Univa Corporation | The Data Center Optimization Company
E-Mail: [email protected] | Phone: +49.9471.200.195 | Mobile: +49.170.819.7390

_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users

Re: [gridengine users] Definitions of "complex" and "queue"?

Reply via email to