On 2014-07-24, at 06:50, Tony Thigpen wrote:
> > May be faster to have the main build a queue of records and let the subtasks
> > feed themselves. There are interesting serialization concerns with multiple
> > eaters. The feeder can post all waiting tasks when food is available.
>
I've done something like this with POSIX pipes. Initially queue N
tokens into the pipe. The eater code is:
do forever
read one token from pipe /* blocks if empty. */
fork one child to process a unit of work.
end
the child code is:
process one unit of work
terminate one token into pipe
terminate.
The child workers are ephemeral. They could be made persistent
if the token were the worker's ID and "fork" replaced by POST
and "terminate" replaced by WAIT.
-- gil
