Dave,
For what it's worth, here are my opinions.
I see here three levels of functionality in a scheduler.
Type 1: Single-threaded
Type 2: Multi-threaded, no dependencies
Type 3: Multi-threaded, with dependencies
My recommendations:
Type 1 functionality be fully implemented first and should go into the
first release of Turbine (release 2.0 ;-> ). This is a minimum
acceptable level of scheduler functionality.
Type 2 should be implemented thereafter.
Type 3 is a much more complex issue and should only be undertaken
after you have gathered a lot of requirements.
Stephen
details below...
At 10:22 PM 3/22/00 -0600, you wrote:
>I'm in the process of reworking the scheduler service. In doing so, I'm
>looking at
>the need to run each scheduled job in it's own thread.
This would be needed for Types 2 and 3.
>Under the current design, a job is rescheduled to run based upon the
>reoccuring time it's scheduled to run and the time it takes to complete.
>The run queue is sorted and prioritized by this factor. The possible
>problem
>with this is that a long running task can force other jobs to run late
>waiting on
>a job to complete.
This is Type 1.
>If I change this to run each job in it's own thread, then the queue will
>be independent
>of the time it takes an individual task to run. So, a task is scheduled
>to run again regardless
>of the time it takes to complete and the scheduler can (in theory) run
>each job on time.
This is Type 2.
>However by doing this, it makes the scheduler function more complex.
>Because now the
>service needs some type of a callback to update the queue and it would
>be possible for
>the scheduler to launch the task to run again even it's not finished
>from it's previous
>execution. For example, in the case that a run again time < time to
>complete.
This (Type 2) is how Unix "cron" works. It is multi-process (rather than
multi-threaded), and it launches new tasks without regard for whether
the previous tasks it launches have been completed yet (or whether they
succeeded or failed).
>This can
>be solved by setting some type of state check or priority on the job but
>this adds complexity
>to the service.
This is getting into Type 3 job scheduling. You are proposing here
a very limited set of dependency capability: a task from a single task
definition must complete before a second task from the same task
definition may be run. This is a business rule which is relevant to
some tasks and not to others. I would recommend that you implement
Type 2 scheduling, and people can put this logic into the task itself
if they really want it.
Once you start creating dependencies between the tasks, you will be
faced with many more issues. A full-fledged Type 3 scheduler could/should
support any number of the following features:
* arbitrary dependency definitions between tasks
(i.e. task C cannot run until task A is finished and task B is finished)
* limiting the number of tasks that can be run at a time
(i.e. don't run this task until the system load is < 80% CPU utilization)
(i.e. don't run more than 5 tasks at the same time)
(i.e. each task has a workload value, and the sum total workload values
for all tasks must not exceed a configured system-level workload capacity)
* dependencies on file existence
(i.e. task D cannot run until file "/path/to/file" exists)
* priorities of tasks
(i.e. if there is a limitation to the number of tasks that can be run,
the highest priority of the waiting tasks will in fact be run)
Once you start talking about dependencies, you have to think about
* what to do in the case of failures?
* how do I monitor tasks? (GUI?)
* how do I restart tasks?
* etc.
I think Type 3 scheduling can be valuable, but really requires a much
greater vision in order to do it right.
>So my question is: What's more important to you? Can we live with a job
>(possibly)
>running a little late; or is the importance of this service worth the
>complexity of seperate thread
>per job?
I think Type 1 is quite sufficient for most purposes and the easiest to
implement in order to get to Turbine release.
Type 2 is very valuable, and should be done by the first available
person who has got the itch.
Type 3 is more of a batch processing system... not really needed in a
servlet environment.
my $0.02 ...
Stephen
------------------------------------------------------------
To subscribe: [EMAIL PROTECTED]
To unsubscribe: [EMAIL PROTECTED]
Problems?: [EMAIL PROTECTED]
