Hi again,
On May 29, 2012, at 4:40 PM, Alexander Burger wrote:
>
>
>> Another option to avoid the fork() would be to have a pool of
>> pre-forked instances reading from a jobs queue or something like that
>> (perhaps taking advantage of an additional database to implement the
>> queue?), but my skills are still lacking on how to implement that.
>
> That's also an interesting approach.
>
>
> An important question is: Does this parallel processing of database
> objects also involve modifications of these objects? If so, the
> necessary synchronization between the processes will produce additional
> costs.
Yes, I need to update the database with results from the executed commands.
So, you suggest limiting the updates to happen only in the main process, right?
In that case I would need something like the following to be able to invoke the
shell commands and update the database with the results.
I don't like it, too convoluted for my taste. Any suggestion on how to improve
the performance/style? Perhaps a different approach would be better?
Note that *maxProcesses should be set to something like the number of cores and
that increasing the length of *L can result in fork() errors in the simple
later/wait benchmark.
Thanks,
Jorge
(setq *maxProcesses 2)
(setq "*idle" '(NIL NIL nil))
(setq "*batches" (make (do *maxProcesses (link "*idle"))))
(de "findPred" (Pred L)
(seek '((X) (Pred (car X))) L) )
(de "completeJob" ("Pos")
(let (cdar "Pos")
(let RESULT (caar "Pos")
(eval (caddar "Pos")) ) )
(set "Pos" "*idle")
"Pos" )
(de "waitJob" ()
("completeJob" (wait NIL ("findPred" 'pair "*batches"))))
(de waitAll ()
(wait NIL (not ("findPred" 'atom "*batches")))
(map '"completeJob" "*batches"))
(de batch ("Cont" . "Job")
(let "Pos" ("waitJob")
(set "Pos" 'busy)
(later "Pos"
(list (run "Job") (env) "Cont") ) ) )
(mapcar '((X) (batch '(msg RESULT) (* X X))) (range 1 10))
(waitAll)
(de mycmd ()
(in '(cc "-c" "../picolisp/src/main.c" "-o" "/dev/null")) T)
(setq *L (range 1 50))
(bench
(mapcar '(() (batch '(nil RESULT) (mycmd))) *L)
(waitAll) )
(bench
(prog1
(mapcan '(() (later (cons) (mycmd))) *L)
(wait NIL (full @))
) )
(bench
(mapcan '(() (mycmd)) *L))
--
UNSUBSCRIBE: mailto:[email protected]?subject=Unsubscribe
