Hi Matt! Thank you very much for the code! If I have any success I'll report here. Have a Happy New Year!
Cheers, Arthur Em 28 de dez de 2016 15:29, "Matt Welland" <[email protected]> escreveu: > Hi Arthur, > > You might find this bit of exploratory code useful: > http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54. > It is an example of spawning off dozens of sub-processes and using nanomsg > to communicate the data back. We needed to find changed files in gigs of > data where the originating process would not be able to directly see the > files (the program will be setuid). With this proof of concept code on a 32 > processor machine we saw a task that would take hours drop to minutes. No > surprise there but it was nice that it worked. It might be handy if this > idea was abstracted into an egg with special versions of map, for-each etc. > but I don't have time to attempt that now. > > Nanomsg is pretty neat but it has quirks. To work reliably all open and > closing of ports had to be protected with mutexes which seems very odd. > Note: the code was only a proof of concept trial and it probably won't work > for you out of the box. It is nice that with nanomsg you simply change the > connection URL to switch from in process communication to inter-process to > across hosts. We saw no difference between IPC and tcp but we were only > handing back and forth tiny amounts of data so that is to be expected. > > Matt > -=- > > On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel <[email protected]> > wrote: > >> Hi Kooda! >> >> Em 24 de dez de 2016 07:00, "Kooda" <[email protected]> escreveu: >> >> On Sat, 24 Dec 2016 02:11:37 -0200 >> Arthur Maciel <[email protected]> wrote: >> > Is there a way to implement map, for-each and other procedures in a >> > parallel way so >> > >> > (use srfi-1) >> > (map (lambda (x) (+ x 1)) (iota 1000000) >> > >> > would automatically split the list into smaller lists according to the >> > number of CPU cores and then gather the results back? >> >> I guess you could spawn a process pool and send these processes a thunk >> that calculates their part and send back the result. You could use s11n >> egg for that, I believe. >> >> I’m not sure it would be faster than the regular functions though. >> >> >> Do you recommend any specific way to create the pool and especially to >> communicate between the processes? >> >> About the speed, I'll test and report the results. >> >> Thanks! >> Arthur >> >> _______________________________________________ >> Chicken-users mailing list >> [email protected] >> https://lists.nongnu.org/mailman/listinfo/chicken-users >> >> >
_______________________________________________ Chicken-users mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/chicken-users
