On Wednesday 05 January 2011 15:36:13 Aaron Bentley wrote: > On 11-01-05 10:23 AM, Julian Edwards wrote: > > On Wednesday 05 January 2011 15:01:36 Aaron Bentley wrote: > >> You are focusing on a particular kind of solution to the "interruptible > >> processing of continuous data", but there are other kinds of solutions. > >> > >> Shouldn't we look at them, too? > >> > >>> Micro jobs are a nice idea, but are orthogonal to what I want to do > >>> here. They might even end up using this design. > >> > >> They provide a way of doing interruptible processing of continuous data. > >> > >> They only seem orthagonal because you're focusing on solutions that > >> > >> store restart points. > > > > I am open to your suggestions of a way of recovering from a data point > > (after a crash or interval between cron jobs) without storing where you > > left off. I cannot see how the loop tuner nor micro jobs do that. > > In my earlier example, each data record that needs to be processed would > have an associated micro-job. The BuildStatus-like enum would indicate > whether the job has completed. > > To recover, you just query for jobs that haven't been completed. (If > you need to pick up exactly where you left off, you can sort the jobs or > records.)
On what data would you sort? How do you work out what micro-jobs to create next by the thing that produces them? > Using a queue would be another way to implement micro-jobs :-) The queue would transparently handle the restart points. Without the queue, you still need a way of organising the job order and state. J _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
