Re: How do I limit the number of active threads (queuing spawn calls)

[Andrej Mitrovic]

Well I've worked around this by polling a variable which holds the
number of active threads. It's not a pretty solution, and I'd probably
be best with using std.parallelism or some upcoming module. My
solution for now is:

import std.stdio;
import std.file;
import std.path;
import std.process;
import std.concurrency;
import core.thread;

shared int threadsCount;

void compileObjfile(string name)
{
    system(r"dmd -od" ~ r".\cache\" ~ r" -c -version=Unicode
-version=WindowsNTonly -version=Windows2000 -version=WindowsXP -I..\ "
~ name ~ " ");
    atomicOp!"-="(threadsCount, 1);
}

int main()
{
    string libfileName = r".\cache\win32.lib ";
    string objFiles;
        foreach (string name; dirEntries(curdir, SpanMode.shallow))
    {
        if (name.isfile && name.basename.getName != "build" &&
(name.getExt == "d" || name.getExt == "di"))
        {
            string objfileName = r".\cache\" ~ name.basename.getName ~ ".obj";
            objFiles ~= objfileName ~ " ";

            atomicOp!"+="(threadsCount, 1);
            while (threadsCount > 3)
            {
                Thread.sleep(dur!("msecs")(1));
            }
            spawn(&compileObjfile, name);
        }
    }

    while (threadsCount)
    {
        Thread.sleep(dur!("msecs")(1));  // wait for threads to finish
before call to lib
    }
    system(r"lib -c -n -p64 " ~ objFiles);

    return 0;
}

The timing:

D:\dev\projects\win32\win32>timeit build
Digital Mars Librarian Version 8.02n
Copyright (C) Digital Mars 2000-2007 All Rights Reserved
http://www.digitalmars.com/ctg/lib.html
Digital Mars Librarian complete.

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:49 am, Sunday, March 27 2011
Elapsed Time:     0:00:06.437
Process Time:     0:00:00.062
System Calls:     627101
Context Switches: 123883
Page Faults:      734997
Bytes Read:       93800813
Bytes Written:    7138927
Bytes Other:      1043652

So about ~6.5 seconds. Now compare this to this build script which
simply invokes DMD with -lib and all the modules:

import std.stdio;
import std.process;
import std.path;
import std.file;

void main()
{
    string files;
        foreach (string name; dirEntries(curdir, SpanMode.shallow))
    {
        if (name.isfile && name.basename.getName != "build" &&
name.getExt == "d")
            files ~= name ~ " ";
    }

    system(r"dmd -lib -I..\ -version=Unicode -version=WindowsNTonly
-version=Windows2000 -version=WindowsXP " ~ files);
}

D:\dev\projects\win32\win32>timeit build.exe

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:54 am, Sunday, March 27 2011
Elapsed Time:     0:00:25.750
Process Time:     0:00:00.015
System Calls:     139172
Context Switches: 44648
Page Faults:      87440
Bytes Read:       7427284
Bytes Written:    7413372
Bytes Other:      45798

Compiling object by object is almost exactly 4 times faster with
threading than using -lib on all module files. And my multithreaded
script is probably wasting some time by calling thread.sleep(), but
I'm new to threading and I don't know how else to limit the number of
threads.