On Tuesday, 19 December 2017 at 18:42:01 UTC, Ali Çehreli wrote:
On 12/19/2017 02:24 AM, Vino wrote:
> Hi All,
>
> Request your help in clarifying the below. As per the
document
>
> foreach (d; taskPool.parallel(xxx)) : The total number of
threads that
> will be created is total CPU -1 ( 2 processor with 6 core :
11 threads)
>
> foreach (d; taskPool.parallel(xxx,1)) : The total number of
threads that
> will be created is total CPU -1 ( 2 processor with 6 core :
12 threads)
That parameter is workUnitSize, meaning the number of elements
each thread will process per work unit. So, when you set it to
100, each thread will work on 100 elements before they go pick
more elements to work on. Experiment with different values to
find out which is faster for your work load. If each element
takes very short amount of time to work on, you need larger
values because you don't want to stop a happy thread that's
chugging along on elements. It really depends on each program,
so try different values.
> foreach (d; taskPool.parallel(xxx,20)) : As in Windows 2008
whatever
> value is set for the parallel the total number of threads
does not
> increase more than 12.
taskPool is just for convenience. You need to create your own
TaskPool if you want more threads:
import std.parallelism;
import core.thread;
import std.range;
void main() {
auto t = new TaskPool(20);
foreach (d; t.parallel(100.iota)) {
// ...
}
Thread.sleep(5.seconds);
t.finish();
}
Now there are 20 + 1 (main) threads.
Ali
Hi Ali,
Thank you very much, below are the observations, our program is
used to calculate the size of the folders, and we don't see any
improvements in the execution speed from the below test, are we
missing something. Basically we expected the total execution time
for the test 2 , as the time taken to calculate the size of the
biggest folder + few additional mins, the biggest folder size is
of 604 GB. Memory usage is just 12 MB, whereas the server has 65
GB and hardly 30% - 40% is used at any given point in time, so
there is no memory constrain.
Test 1:
foreach (d; taskPool.parallel(dFiles[],1))
auto SdFiles = Array!ulong(dirEntries(d, SpanMode.depth).map!(a
=> a.size).fold!((a,b) => a + b) (x))[].filter!(a => a > Size);
Execution Time is 26 mins with 11+1 (main) threads and 1 element
per thread
Test 2:
auto TL = dFiles.length;
auto TP = new TaskPool(TL);
foreach (d; TP.parallel(dFiles[],1))
auto SdFiles = Array!ulong(dirEntries(d, SpanMode.depth).map!(a
=> a.size).fold!((a,b) => a + b) (x))[].filter!(a => a > Size);
Thread.sleep(5.seconds); TP.finish();
Execution Time is 27 mins with 153+1 (main) threads and 1 element
per thread
From,
Vino.B
