Bug#918719: task-spooler option -w does not appear to work as intended
Greetings! Thus spoke Alexander Inyukhin: > On Tue, Jan 08, 2019 at 06:08:00PM +0100, Davide Monsorno > wrote: > > I have tried to use task spooler in order to launch 2 tasks > > (a1,a2) in parallel, followed by 3 tasks (b1,b2,b3) in > > parallel. To this end, I have increased the maximum number > > of slots to 3, with the option -S [number of slots]. > > Task spooler has a limited support for dependencies. Probably, > an external logic is required to handle this case. > > [...] > > I could forward request upstream to support multiple > dependencies. +1 for multiple dependency support. task-spooler 1.0-1 indeed sadly only supports single dependencies; it's baked into the "queue item" data structure it employs. I see two ways to implement the "external logic" you mention above, and both of them have drawbacks. The straight-forward way is to submit a task that first waits for all dependencies in turn and then runs the real task. This blocks at least one slot, and may block one of its own dependencies from actually running because there are no more free slots left, creating a livelock. The less straight-forward way is to wrap the task in a chain of task-spooler submissions, each with one dependency less than the previous link in the chain. For example, assume task D depends on C, B, and A. First, submit A as usual. Then create task B' which depends on A and whose action is to submit task C'. Task C' depends on B and its action is to submit task D'. Task D' depends on C and its action is to submit task D. Task D shall depend on no other tasks. Once A, B and C are done, then task D' will succeed in submitting task D, effectively making D depend on A, B and C. This approach does not unnecessarily block task-spooler slots since task-spooler is keeping track of when to actually run the respective tasks. But because it creates intermediate tasks it is difficult to use when A, B or C themselves have multiple dependencies and must be chained as well; the final task IDs of A, B and C are only known once the true A, B and C tasks are submitted. Having upstream support for multiple dependencies would move this logic into the task-spooler itself, where such dependency checks can be done without running a task too early or starting multiple intermediate tasks, and thus sidestepping the above-mentioned drawbacks. So yes, please do forward this request for support to upstream. Thank you, and kind regards, Marco signature.asc Description: PGP signature
Bug#918719: task-spooler option -w does not appear to work as intended
On Tue, Jan 08, 2019 at 06:08:00PM +0100, Davide Monsorno wrote: > I have tried to use task spooler in order to launch 2 tasks (a1,a2) in > parallel, followed by 3 tasks (b1,b2,b3) in parallel. > To this end, I have increased the maximum number of slots to 3, with > the option -S [number of slots]. Task spooler has a limited support for dependencies. Probably, an external logic is required to handle this case. > I would like the b1,b2,b3 tasks to wait until the task a1 (or a2, or > both) is completed. > > I have tried using the -w option as per the manual; however it does > not appear to work. For instance, if I specify the id > of the previous process, e.g. tsp -w [id] echo "hello world", the > command seems to hang until the process [id] is finished, > and then nothing happens. At a first glance things are working as expected. -w is an action to wait for task completion. Nothing else should happen here. -D is an option for declaring dependency when running a new task. In both cases completion of a single task is tracked. > On the other hand, if I use the option -D [id], which does the same > thing but also checks if the task [id] finished well, it > seems to work as intended. However, simply using option -D instead of > option -w is not ok in my application, since the tasks a1,a2 > and b1,b2,b3 are independent and the latter should start even if the > former return an error. Checking for exit status is a right thing to do, usually. A simple shell wrapper to hide error could help here. Another wrapper could collect results from tasks a* and provide a single dependency to tasks b*. > Are you able to replicate the bug and, if so, could you please provide a fix? I could forward request upstream to support multiple dependencies.
Bug#918719: task-spooler option -w does not appear to work as intended
Package: task-spooler Version: 1.0-1 Dear Mantainer, I have tried to use task spooler in order to launch 2 tasks (a1,a2) in parallel, followed by 3 tasks (b1,b2,b3) in parallel. To this end, I have increased the maximum number of slots to 3, with the option -S [number of slots]. I would like the b1,b2,b3 tasks to wait until the task a1 (or a2, or both) is completed. I have tried using the -w option as per the manual; however it does not appear to work. For instance, if I specify the id of the previous process, e.g. tsp -w [id] echo "hello world", the command seems to hang until the process [id] is finished, and then nothing happens. On the other hand, if I use the option -D [id], which does the same thing but also checks if the task [id] finished well, it seems to work as intended. However, simply using option -D instead of option -w is not ok in my application, since the tasks a1,a2 and b1,b2,b3 are independent and the latter should start even if the former return an error. Are you able to replicate the bug and, if so, could you please provide a fix? I am using Debian GNU/Linux 9.4, kernel Linux 4.9.0-6-amd64 (SMP w/2 CPU cores) and libc6 2.24-11+deb9u3 Thank you, DM
