Re: Simple parallel foreach and summation/reduction
Hi, Apologies for coming late to this thread. I started with: import std.random: uniform; import std.range: iota; import std.stdio: writeln; void main() { ulong sum; foreach(i; iota(1_000_000_000)) { if (uniform(0F,12F) > 6F) sum++; } writeln("The sum is ", sum); } and then transformed it to: import std.algorithm: map, reduce; import std.random: uniform; import std.range: iota; import std.stdio: writeln; void main() { ulong sum = iota(1_000_000_000).map!((_) => uniform(0F,12F) > 6F ? 1 : 0).reduce!"a +b"; writeln("The sum is ", sum); } and then made use of std.parallelism: import std.algorithm: map; import std.array:array; import std.parallelism: taskPool; import std.random: uniform; import std.range: iota; import std.stdio: writeln; void main() { ulong sum = taskPool().reduce!"a + b"(iota(1_000_000_000).map!((_) => uniform(0F,12F) > 6F ? 1 : 0)); writeln("The sum is ", sum); } I am not entirely sure how to capture the memory used but roughly (since this is a one off measure and not a statistically significant experiment): first takes 30s second takes 30s third takes 4s on an ancient twin Xeon workstation, so 8 cores but all ancient and slow. The issue here is that std.parallelism.reduce, std.parallelism.map, and std.parallelism.amap are all "top level" work scattering functions, they all assume total control of the resources. So the above is a parallel reduce using sequential map which works fine. Trying to mix parallel reduce and parallel map or amap ends up with two different attempts to make use of the resources to create tasks. std.parallelism isn't really a fork/join framework in the Java sense, if you want tree structure parallelism, you have to do things with futures. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Simple parallel foreach and summation/reduction
On Monday, 24 September 2018 at 07:13:24 UTC, Chris Katko wrote: On Monday, 24 September 2018 at 05:59:20 UTC, Chris Katko wrote: [...] Actually, I just realized/remembered that the error occurs inside parallelism itself, and MANY times at that: [...] This JUST occurred to me. When I use an outer taskPool.[a]map, am I NOT supposed to use the taskPool version of reduce?! But instead, the std.algorithm one? Because this is running with both/all cores, and only using 2.7MB of RAM: sum = taskPool.reduce!(test)( map!(monte)(range) //map, not taskPool.map ); If that's the correct case, the docs did NOT make that obvious! FYI, I went from ~5200 samples / mSec, to 7490 samples / mSec. 36% difference for second "real" core. Better than nothing, I guess. I'll have to try it on my main machine with a proper CPU.
Re: Simple parallel foreach and summation/reduction
On Monday, 24 September 2018 at 05:59:20 UTC, Chris Katko wrote: On Saturday, 22 September 2018 at 02:26:41 UTC, Chris Katko wrote: On Saturday, 22 September 2018 at 02:13:58 UTC, Chris Katko wrote: On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli wrote: On 09/21/2018 12:25 AM, Chris Katko wrote: [...] You can use a free-standing function as a workaround, which is included in the following chapter that explains most of std.parallelism: http://ddili.org/ders/d.en/parallelism.html That chapter is missing e.g. the newly-added fold(): https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold Ali Okay... so I've got it running. The problem is, it uses tons of RAM. In fact, proportional to the working set. T test(T)(T x, T y) { return x + y; } double monte(T)(T x) { double v = uniform(-1F, 1F); double u = uniform(-1F, 1F); if(sqrt(v*v + u*u) < 1.0) { return 1; }else{ return 0; } } auto taskpool = new TaskPool(); sum = taskpool.reduce!(test)( taskpool.amap!monte( iota(num) ) ); taskpool.finish(true); 100 becomes ~8MB 1000 becomes 80MB 1, I can't even run because it says "Exception: Memory Allocation failed" Also, when I don't call .finish(true) at the end, it just sits there forever (after running) like one of the threads won't terminate. Requiring a control-C. But the docs and examples don't seem to indicate I should need that... So I looked into it. It's amap that explodes in RAM. Per the docs, amap has "less overhead but more memory usage." While map has more overhead but less memory usage and "avoids the need to keep all results in memory." But, if I make a call to map... it doesn't compile! I get: Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map Simply changing amap to map here: sum = taskPool.reduce!(test) ( taskPool.map!(monte)(range) ); Actually, I just realized/remembered that the error occurs inside parallelism itself, and MANY times at that: /usr/include/dmd/phobos/std/parallelism.d(2590): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2596): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2616): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2626): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map /usr/include/dmd/phobos/std/parallelism.d(2634): Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map monte.d(64): Error: template instance std.parallelism.TaskPool.reduce!(test).reduce!(Map) error instantiating Though I tried looking up the git version of prallelism.d and the lines don't quite line up: https://github.com/dlang/phobos/blob/master/std/parallelism.d
Re: Simple parallel foreach and summation/reduction
On Saturday, 22 September 2018 at 02:26:41 UTC, Chris Katko wrote: On Saturday, 22 September 2018 at 02:13:58 UTC, Chris Katko wrote: On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli wrote: On 09/21/2018 12:25 AM, Chris Katko wrote: [...] You can use a free-standing function as a workaround, which is included in the following chapter that explains most of std.parallelism: http://ddili.org/ders/d.en/parallelism.html That chapter is missing e.g. the newly-added fold(): https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold Ali Okay... so I've got it running. The problem is, it uses tons of RAM. In fact, proportional to the working set. T test(T)(T x, T y) { return x + y; } double monte(T)(T x) { double v = uniform(-1F, 1F); double u = uniform(-1F, 1F); if(sqrt(v*v + u*u) < 1.0) { return 1; }else{ return 0; } } auto taskpool = new TaskPool(); sum = taskpool.reduce!(test)( taskpool.amap!monte( iota(num) ) ); taskpool.finish(true); 100 becomes ~8MB 1000 becomes 80MB 1, I can't even run because it says "Exception: Memory Allocation failed" Also, when I don't call .finish(true) at the end, it just sits there forever (after running) like one of the threads won't terminate. Requiring a control-C. But the docs and examples don't seem to indicate I should need that... So I looked into it. It's amap that explodes in RAM. Per the docs, amap has "less overhead but more memory usage." While map has more overhead but less memory usage and "avoids the need to keep all results in memory." But, if I make a call to map... it doesn't compile! I get: Error: no [] operator overload for type std.parallelism.TaskPool.map!(monte).map!(Result).map.Map Simply changing amap to map here: sum = taskPool.reduce!(test) ( taskPool.map!(monte)(range) );
Re: Simple parallel foreach and summation/reduction
On Saturday, 22 September 2018 at 02:13:58 UTC, Chris Katko wrote: On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli wrote: On 09/21/2018 12:25 AM, Chris Katko wrote: [...] You can use a free-standing function as a workaround, which is included in the following chapter that explains most of std.parallelism: http://ddili.org/ders/d.en/parallelism.html That chapter is missing e.g. the newly-added fold(): https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold Ali Okay... so I've got it running. The problem is, it uses tons of RAM. In fact, proportional to the working set. T test(T)(T x, T y) { return x + y; } double monte(T)(T x) { double v = uniform(-1F, 1F); double u = uniform(-1F, 1F); if(sqrt(v*v + u*u) < 1.0) { return 1; }else{ return 0; } } auto taskpool = new TaskPool(); sum = taskpool.reduce!(test)( taskpool.amap!monte( iota(num) ) ); taskpool.finish(true); 100 becomes ~8MB 1000 becomes 80MB 1, I can't even run because it says "Exception: Memory Allocation failed" Also, when I don't call .finish(true) at the end, it just sits there forever (after running) like one of the threads won't terminate. Requiring a control-C. But the docs and examples don't seem to indicate I should need that...
Re: Simple parallel foreach and summation/reduction
On Friday, 21 September 2018 at 12:15:59 UTC, Ali Çehreli wrote: On 09/21/2018 12:25 AM, Chris Katko wrote: On Thursday, 20 September 2018 at 05:51:17 UTC, Neia Neutuladh wrote: On Thursday, 20 September 2018 at 05:34:42 UTC, Chris Katko wrote: All I want to do is loop from 0 to [constant] with a for or foreach, and have it split up across however many cores I have. You're looking at std.parallelism.TaskPool, especially the amap and reduce functions. Should do pretty much exactly what you're asking. auto taskpool = new TaskPool(); taskpool.reduce!((a, b) => a + b)(iota(1_000_000_000_000L)); I get "Error: template instance `reduce!((a, b) => a + b)` cannot use local __lambda1 as parameter to non-global template reduce(functions...)" when trying to compile that using the online D editor with DMD and LDC. Any ideas? You can use a free-standing function as a workaround, which is included in the following chapter that explains most of std.parallelism: http://ddili.org/ders/d.en/parallelism.html That chapter is missing e.g. the newly-added fold(): https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold Ali Okay... so I've got it running. The problem is, it uses tons of RAM. In fact, proportional to the working set. T test(T)(T x, T y) { return x + y; } double monte(T)(T x) { double v = uniform(-1F, 1F); double u = uniform(-1F, 1F); if(sqrt(v*v + u*u) < 1.0) { return 1; }else{ return 0; } } auto taskpool = new TaskPool(); sum = taskpool.reduce!(test)( taskpool.amap!monte( iota(num) ) ); taskpool.finish(true); 100 becomes ~8MB 1000 becomes 80MB 1, I can't even run because it says "Exception: Memory Allocation failed"
Re: Simple parallel foreach and summation/reduction
On 09/21/2018 12:25 AM, Chris Katko wrote: On Thursday, 20 September 2018 at 05:51:17 UTC, Neia Neutuladh wrote: On Thursday, 20 September 2018 at 05:34:42 UTC, Chris Katko wrote: All I want to do is loop from 0 to [constant] with a for or foreach, and have it split up across however many cores I have. You're looking at std.parallelism.TaskPool, especially the amap and reduce functions. Should do pretty much exactly what you're asking. auto taskpool = new TaskPool(); taskpool.reduce!((a, b) => a + b)(iota(1_000_000_000_000L)); I get "Error: template instance `reduce!((a, b) => a + b)` cannot use local __lambda1 as parameter to non-global template reduce(functions...)" when trying to compile that using the online D editor with DMD and LDC. Any ideas? You can use a free-standing function as a workaround, which is included in the following chapter that explains most of std.parallelism: http://ddili.org/ders/d.en/parallelism.html That chapter is missing e.g. the newly-added fold(): https://dlang.org/phobos/std_parallelism.html#.TaskPool.fold Ali
Re: Simple parallel foreach and summation/reduction
On Friday, 21 September 2018 at 07:25:17 UTC, Chris Katko wrote: I get "Error: template instance `reduce!((a, b) => a + b)` cannot use local __lambda1 as parameter to non-global template reduce(functions...)" when trying to compile that using the online D editor with DMD and LDC. Any ideas? That's a long standing issue: https://issues.dlang.org/show_bug.cgi?id=5710 Using a string for the expression does work though: ``` import std.stdio, std.parallelism, std.range; void main() { taskPool.reduce!"a + b"(iota(1_000L)).writeln; } ```
Re: Simple parallel foreach and summation/reduction
On Thursday, 20 September 2018 at 05:51:17 UTC, Neia Neutuladh wrote: On Thursday, 20 September 2018 at 05:34:42 UTC, Chris Katko wrote: All I want to do is loop from 0 to [constant] with a for or foreach, and have it split up across however many cores I have. You're looking at std.parallelism.TaskPool, especially the amap and reduce functions. Should do pretty much exactly what you're asking. auto taskpool = new TaskPool(); taskpool.reduce!((a, b) => a + b)(iota(1_000_000_000_000L)); I get "Error: template instance `reduce!((a, b) => a + b)` cannot use local __lambda1 as parameter to non-global template reduce(functions...)" when trying to compile that using the online D editor with DMD and LDC. Any ideas?
Re: Simple parallel foreach and summation/reduction
On Thursday, 20 September 2018 at 05:34:42 UTC, Chris Katko wrote: All I want to do is loop from 0 to [constant] with a for or foreach, and have it split up across however many cores I have. You're looking at std.parallelism.TaskPool, especially the amap and reduce functions. Should do pretty much exactly what you're asking. auto taskpool = new TaskPool(); taskpool.reduce!((a, b) => a + b)(iota(1_000_000_000_000L));