Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-30 Thread Arthur Maciel 
Thanks for pointing it, Dan! Do you know if it is supported by CHICKEN on
Windows too?

Cheers,
Arthur

Em 30 de dez de 2016 03:43, "Dan Leslie" <d...@ironoxide.ca> escreveu:

> You could build something with the posix-shm egg, and the posix unit. If
> you need locks, there’s the posix-semaphore egg.
>
>
>
> -Dan
>
>
>
> *From:* Chicken-users [mailto:chicken-users-bounces+dan=
> ironoxide...@nongnu.org] *On Behalf Of *Arthur Maciel
> *Sent:* December 28, 2016 7:47 AM
> *To:* Kooda <ko...@upyum.com>
> *Cc:* chicken-users <chicken-users@nongnu.org>
> *Subject:* Re: [Chicken-users] Parallel procedures in CHICKEN
>
>
>
> Hi Kooda!
>
>
>
> Em 24 de dez de 2016 07:00, "Kooda" <ko...@upyum.com> escreveu:
>
> On Sat, 24 Dec 2016 02:11:37 -0200
> Arthur Maciel <arthurmac...@gmail.com> wrote:
> > Is there a way to implement map, for-each and other procedures in a
> > parallel way so
> >
> > (use srfi-1)
> > (map (lambda (x) (+ x 1)) (iota 100)
> >
> > would automatically split the list into smaller lists according to the
> > number of CPU cores and then gather the results back?
>
> I guess you could spawn a process pool and send these processes a thunk
> that calculates their part and send back the result. You could use s11n
> egg for that, I believe.
>
> I’m not sure it would be faster than the regular functions though.
>
>
>
> Do you recommend any specific way to create the pool and especially to
>  communicate between the processes?
>
>
>
> About the speed, I'll  test and report the results.
>
>
>
> Thanks!
>
> Arthur
>
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users

Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-29 Thread Dan Leslie 
You could build something with the posix-shm egg, and the posix unit. If you 
need locks, there’s the posix-semaphore egg.

 

-Dan

 

From: Chicken-users [mailto:chicken-users-bounces+dan=ironoxide...@nongnu.org] 
On Behalf Of Arthur Maciel
Sent: December 28, 2016 7:47 AM
To: Kooda <ko...@upyum.com>
Cc: chicken-users <chicken-users@nongnu.org>
Subject: Re: [Chicken-users] Parallel procedures in CHICKEN

 

Hi Kooda!

 

Em 24 de dez de 2016 07:00, "Kooda" <ko...@upyum.com <mailto:ko...@upyum.com> > 
escreveu:

On Sat, 24 Dec 2016 02:11:37 -0200
Arthur Maciel <arthurmac...@gmail.com <mailto:arthurmac...@gmail.com> > wrote:
> Is there a way to implement map, for-each and other procedures in a
> parallel way so
>
> (use srfi-1)
> (map (lambda (x) (+ x 1)) (iota 100)
>
> would automatically split the list into smaller lists according to the
> number of CPU cores and then gather the results back?

I guess you could spawn a process pool and send these processes a thunk
that calculates their part and send back the result. You could use s11n
egg for that, I believe.

I’m not sure it would be faster than the regular functions though.

 

Do you recommend any specific way to create the pool and especially to  
communicate between the processes? 

 

About the speed, I'll  test and report the results.

 

Thanks! 

Arthur 

___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users

Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-29 Thread Arthur Maciel 
Hi Matt! Thank you very much for the code! If I have any success I'll
report here. Have a Happy New Year!

Cheers,
Arthur

Em 28 de dez de 2016 15:29, "Matt Welland" 
escreveu:

> Hi Arthur,
>
> You might find this bit of exploratory code useful:
> http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54.
> It is an example of spawning off dozens of sub-processes and using nanomsg
> to communicate the data back. We needed to find changed files in gigs of
> data where the originating process would not be able to directly see the
> files (the program will be setuid). With this proof of concept code on a 32
> processor machine we saw a task that would take hours drop to minutes. No
> surprise there but it was nice that it worked. It might be handy if this
> idea was abstracted into an egg with special versions of map, for-each etc.
> but I don't have time to attempt that now.
>
> Nanomsg is pretty neat but it has quirks. To work reliably all open and
> closing of ports had to be protected with mutexes which seems very odd.
> Note: the code was only a proof of concept trial and it probably won't work
> for you out of the box. It is nice that with nanomsg you simply change the
> connection URL to switch from in process communication to inter-process to
> across hosts. We saw no difference between IPC and tcp but we were only
> handing back and forth tiny amounts of data so that is to be expected.
>
> Matt
> -=-
>
> On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel 
> wrote:
>
>> Hi Kooda!
>>
>> Em 24 de dez de 2016 07:00, "Kooda"  escreveu:
>>
>> On Sat, 24 Dec 2016 02:11:37 -0200
>> Arthur Maciel  wrote:
>> > Is there a way to implement map, for-each and other procedures in a
>> > parallel way so
>> >
>> > (use srfi-1)
>> > (map (lambda (x) (+ x 1)) (iota 100)
>> >
>> > would automatically split the list into smaller lists according to the
>> > number of CPU cores and then gather the results back?
>>
>> I guess you could spawn a process pool and send these processes a thunk
>> that calculates their part and send back the result. You could use s11n
>> egg for that, I believe.
>>
>> I’m not sure it would be faster than the regular functions though.
>>
>>
>> Do you recommend any specific way to create the pool and especially to
>>  communicate between the processes?
>>
>> About the speed, I'll  test and report the results.
>>
>> Thanks!
>> Arthur
>>
>> ___
>> Chicken-users mailing list
>> Chicken-users@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/chicken-users
>>
>>
>
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users

Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-28 Thread Matt Welland 
Hi Arthur,

You might find this bit of exploratory code useful:
http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54.
It is an example of spawning off dozens of sub-processes and using nanomsg
to communicate the data back. We needed to find changed files in gigs of
data where the originating process would not be able to directly see the
files (the program will be setuid). With this proof of concept code on a 32
processor machine we saw a task that would take hours drop to minutes. No
surprise there but it was nice that it worked. It might be handy if this
idea was abstracted into an egg with special versions of map, for-each etc.
but I don't have time to attempt that now.

Nanomsg is pretty neat but it has quirks. To work reliably all open and
closing of ports had to be protected with mutexes which seems very odd.
Note: the code was only a proof of concept trial and it probably won't work
for you out of the box. It is nice that with nanomsg you simply change the
connection URL to switch from in process communication to inter-process to
across hosts. We saw no difference between IPC and tcp but we were only
handing back and forth tiny amounts of data so that is to be expected.

Matt
-=-

On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel 
wrote:

> Hi Kooda!
>
> Em 24 de dez de 2016 07:00, "Kooda"  escreveu:
>
> On Sat, 24 Dec 2016 02:11:37 -0200
> Arthur Maciel  wrote:
> > Is there a way to implement map, for-each and other procedures in a
> > parallel way so
> >
> > (use srfi-1)
> > (map (lambda (x) (+ x 1)) (iota 100)
> >
> > would automatically split the list into smaller lists according to the
> > number of CPU cores and then gather the results back?
>
> I guess you could spawn a process pool and send these processes a thunk
> that calculates their part and send back the result. You could use s11n
> egg for that, I believe.
>
> I’m not sure it would be faster than the regular functions though.
>
>
> Do you recommend any specific way to create the pool and especially to
>  communicate between the processes?
>
> About the speed, I'll  test and report the results.
>
> Thanks!
> Arthur
>
> ___
> Chicken-users mailing list
> Chicken-users@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/chicken-users
>
>
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users

Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-28 Thread Arthur Maciel 
Hi Kooda!

Em 24 de dez de 2016 07:00, "Kooda"  escreveu:

On Sat, 24 Dec 2016 02:11:37 -0200
Arthur Maciel  wrote:
> Is there a way to implement map, for-each and other procedures in a
> parallel way so
>
> (use srfi-1)
> (map (lambda (x) (+ x 1)) (iota 100)
>
> would automatically split the list into smaller lists according to the
> number of CPU cores and then gather the results back?

I guess you could spawn a process pool and send these processes a thunk
that calculates their part and send back the result. You could use s11n
egg for that, I believe.

I’m not sure it would be faster than the regular functions though.


Do you recommend any specific way to create the pool and especially to
 communicate between the processes?

About the speed, I'll  test and report the results.

Thanks!
Arthur
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users

Re: [Chicken-users] Parallel procedures in CHICKEN

2016-12-24 Thread Kooda 
On Sat, 24 Dec 2016 02:11:37 -0200
Arthur Maciel  wrote:
> Is there a way to implement map, for-each and other procedures in a
> parallel way so
> 
> (use srfi-1)
> (map (lambda (x) (+ x 1)) (iota 100)
> 
> would automatically split the list into smaller lists according to the
> number of CPU cores and then gather the results back?

I guess you could spawn a process pool and send these processes a thunk
that calculates their part and send back the result. You could use s11n
egg for that, I believe.

I’m not sure it would be faster than the regular functions though.

___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users