Hi again,

On May 29, 2012, at 4:40 PM, Alexander Burger wrote:

> 
> 
>> Another option to avoid the fork() would be to have a pool of
>> pre-forked instances reading from a jobs queue or something like that
>> (perhaps taking advantage of an additional database to implement the
>> queue?), but my skills are still lacking on how to implement that.
> 
> That's also an interesting approach.
> 
> 
> An important question is: Does this parallel processing of database
> objects also involve modifications of these objects? If so, the
> necessary synchronization between the processes will produce additional
> costs.

Yes, I need to update the database with results from the executed commands.

So, you suggest limiting the updates to happen only in the main process, right?

In that case I would need something like the following to be able to invoke the 
shell commands and update the database with the results. 
I don't like it, too convoluted for my taste. Any suggestion on how to improve 
the performance/style? Perhaps a different approach would be better?

Note that *maxProcesses should be set to something like the number of cores and 
that increasing the length of *L can result in fork() errors in the simple 
later/wait benchmark.

Thanks,
  Jorge

(setq *maxProcesses 2)
(setq "*idle" '(NIL NIL nil))
(setq "*batches" (make (do *maxProcesses (link "*idle"))))

(de "findPred" (Pred L)
   (seek '((X) (Pred (car X))) L) ) 

(de "completeJob" ("Pos")
   (let (cdar "Pos")
      (let RESULT (caar "Pos")
         (eval (caddar "Pos")) ) )
   (set "Pos" "*idle")
   "Pos" )

(de "waitJob" ()
   ("completeJob" (wait NIL ("findPred" 'pair "*batches"))))

(de waitAll ()
   (wait NIL (not ("findPred" 'atom "*batches")))
   (map '"completeJob" "*batches"))

(de batch ("Cont" . "Job")
   (let "Pos" ("waitJob")
      (set "Pos" 'busy)
      (later "Pos"
         (list (run "Job") (env) "Cont") ) ) )

(mapcar '((X) (batch '(msg RESULT) (* X X))) (range 1 10))
(waitAll)

(de mycmd ()
   (in '(cc "-c" "../picolisp/src/main.c" "-o" "/dev/null")) T)

(setq *L (range 1 50))

(bench 
   (mapcar '(() (batch '(nil RESULT) (mycmd))) *L)
   (waitAll) )

(bench
   (prog1
      (mapcan '(() (later (cons) (mycmd))) *L)
      (wait NIL (full @))
      ) )

(bench
   (mapcan '(() (mycmd)) *L))

--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Reply via email to