On Sun, Apr 7, 2013 at 7:21 PM, ik <[email protected]> wrote: > > > > On Sun, Apr 7, 2013 at 6:36 PM, Orna Agmon Ben-Yehuda > <[email protected]>wrote: > >> If you are looking for something reliable for many tasks scheduled at a >> time (e.g., one user may send a hundred tasks within one second) go for >> slurm. openPBS, torque, condor cannot take the heat. I heard that PBS (not >> open) can, too. >> > > One of the issues, is knowing what data to take, and there is a need for > a cycle. each cycle have finite number of data inside the queue, but each > new cycle recreate the queue to the actual content to be used. > > I still research this one. > >
With task schedulers, you can cancel jobs that are waiting in the queue easily, even with boinc (which is designed for loose connectivity, that is, it does not assume you can always contact the worker machines, so it fits volunteer computing, like SETI-at-home). However, if what you mean by "recreating the actual content to be used" is that you want to change the data on which the jobs operate, then it is even easier. Just have each job pull its data when it starts running from a known place. Then you can change this known place at no central cost until the job starts running. > >> >> On Sun, Apr 7, 2013 at 5:30 PM, ik <[email protected]> wrote: >> >>> On Sun, Apr 7, 2013 at 3:51 PM, Oleg Goldshmidt <[email protected]>wrote: >>> >>>> >>>> >>>> On Sun, Apr 7, 2013 at 2:27 PM, ik <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have a very complex queue management requirements, I need to >>>>> calculate task weight, time restrictions for execution, number of possible >>>>> execution at given time, changes of the data when it is inside the queue >>>>> and even disabling specific task. >>>>> some tasks have very short time to live, and required to be executed >>>>> faster, then others. but it's not a "simple" priority queue. >>>>> >>>> >>>> It is difficult to say from the description, but it sounds to me like >>>> your problem is that of "scheduling" rather than just "queue management". >>>> Note that a "scheduler" (e.g., that of the kernel) will have many queues >>>> holding tasks of different types and properties, and will need to manage >>>> all of them together. >>>> >>>> In particular, some of the things you mention sound to me like "real >>>> time" requirements. If you look at, say, the Linux scheduler it handles RT >>>> tasks separately from the "normal" tasks. You may need to do the same. >>>> >>>> Another thing that is missing is the target system. A single server? A >>>> cluster or multicomputer? Something else? >>>> >>>> I'd suggest googling for "scheduling" and "resource management" (of >>>> which "scheduling" is a component). Apart from things like SLURM and Condor >>>> that Orna mentioned, maybe Maui? There are all sorts of proprietary >>>> offerings as well. >>>> >>> >>> At first glance, condor looks like the thing that can help me most. >>> >>> Thanks >>> >>> >>>> >>>> -- >>>> Oleg Goldshmidt | [email protected] >>>> >>> >>> >>> _______________________________________________ >>> Linux-il mailing list >>> [email protected] >>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >>> >>> >> >> >> -- >> Orna Agmon Ben-Yehuda. >> http://ladypine.org >> > > -- Orna Agmon Ben-Yehuda. http://ladypine.org
_______________________________________________ Linux-il mailing list [email protected] http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
