Hi Jorge, > > An important question is: Does this parallel processing of database > > objects also involve modifications of these objects? If so, the > > necessary synchronization between the processes will produce additional > > costs. > > Yes, I need to update the database with results from the executed commands.
Then I suspect that parallelizing the task may make things worse. Parallelizing something makes sense when CPUs load is the bottleneck. For database updates, however, the bottleneck is disk I/O and the system's disk buffer cache. What makes sense in such cases is to parallelize the work on separate databases, running on separate machines. > So, you suggest limiting the updates to happen only in the main process, > right? Yes. To be precise, the recommended way is to have all database operations run in child processes which are all children of a common parent process. If we call this parent process the "main" process, then the updates don't happen in that process but in one of the children. Having said this, I see that your test program doesn't operate on a database at all. Calling the C compiler in parallel tasks _does_ make sense. Then we talk about something completely different. > In that case I would need something like the following to be able to > invoke the shell commands and update the database with the results. This would indeed make sense, if the shell commands induce a heavy load. Then the CPU load is the bottleneck again, and the way to go is to fork several processes to do the work (i.e. call the shell commands), but still have a _single_ process operating on the database. This would be optimal. > I don't like it, too convoluted for my taste. Any suggestion on how to > improve the performance/style? Perhaps a different approach would be > better? Yes, I think so too. If I understand you right, you want to call a number of shell commands (in a batch list), and then store the results in a database. If so, you could use something like that: (de processJobs (CPUs Batch) (let Slots (need CPUs "free") (for Exe Batch (let Pos (wait NIL (seek '((Pos) (cond ((== "free" (car Pos)) # Found a free slot (set Pos "busy") ) ((n== "busy" (car Pos)) # Found a result (msg (car Pos)) # Instead of 'msg': Store result in DB (set Pos "busy") ) ) ) Slots ) ) (later Pos (eval Exe)) ) ) ) ) You pass the number of CPUs and a list of executable expressions which may do arbitrary work, including calls to a shell. The 'msg' call can be replaced with something more useful, e.g. a store the result into a database. Example call: (processJobs 4 (make (do 20 (link '(in '(sh "-c" "sleep 1; echo $RANDOM") (read))) ) ) ) This outpus 20 random numbers via 'msg', in maximally 4 parallel processes. I didn't completely analyze your code. Just a warning about an error: > (de "completeJob" ("Pos") > (let (cdar "Pos") > (let RESULT (caar "Pos") > (eval (caddar "Pos")) ) ) The second line redefines the function 'cdar'. Is that intended? Cheers, - Alex -- UNSUBSCRIBE: mailto:email@example.com?subject=Unsubscribe