Re: Running Phobos unit tests in threads: I have data

2014-05-06 Thread Atila Neves via Digitalmars-d
https://issues.dlang.org/show_bug.cgi?id=12708 On Sunday, 4 May 2014 at 16:07:30 UTC, Andrei Alexandrescu wrote: On 5/4/14, 1:44 AM, Atila Neves wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results.

Re: Running Phobos unit tests in threads: I have data

2014-05-05 Thread Atila Neves via Digitalmars-d
On Sunday, 4 May 2014 at 17:01:23 UTC, safety0ff wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the

Re: Running Phobos unit tests in threads: I have data

2014-05-05 Thread Dicebot via Digitalmars-d
On Saturday, 3 May 2014 at 12:26:13 UTC, Rikki Cattermole wrote: On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not

Re: Running Phobos unit tests in threads: I have data

2014-05-05 Thread Orvid King via Digitalmars-d
Going to take a wild guess, but as core.atomic.casImpl will never be inlined anywhere with DMD, due to it's inline assembly, you have the cost of building and destroying a stack frame, the cost of passing the args in, moving them into registers, saving potentially trashed registers, etc. every

Re: Running Phobos unit tests in threads: I have data

2014-05-05 Thread Iain Buclaw via Digitalmars-d
On 5 May 2014 19:07, Orvid King via Digitalmars-d digitalmars-d@puremagic.com wrote: Going to take a wild guess, but as core.atomic.casImpl will never be inlined anywhere with DMD, due to it's inline assembly, you have the cost of building and destroying a stack frame, the cost of passing the

Re: Running Phobos unit tests in threads: I have data

2014-05-05 Thread Brad Anderson via Digitalmars-d
On Monday, 5 May 2014 at 17:56:11 UTC, Dicebot wrote: On Saturday, 3 May 2014 at 12:26:13 UTC, Rikki Cattermole wrote: On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Russel Winder via Digitalmars-d
On Sat, 2014-05-03 at 19:37 +, Atila Neves via Digitalmars-d wrote: […] I'm using parallel and taskPool from std.parallelism. I was under the impression it gave me a ready-to-use pool with as many threads as I have cores. There is a default, related to the number of cores the OS thinks

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Atila Neves via Digitalmars-d
On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Atila Neves via Digitalmars-d
Like I mentioned afterwards, I tried a different number of threads. On my machine, at least, std.parallelism.totalCPUs returns 8, the number of virtual cores. As it should. Atila On Sunday, 4 May 2014 at 07:49:51 UTC, Russel Winder via Digitalmars-d wrote: On Sat, 2014-05-03 at 19:37 +,

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Russel Winder via Digitalmars-d
On Sun, 2014-05-04 at 08:47 +, Atila Neves via Digitalmars-d wrote: Like I mentioned afterwards, I tried a different number of threads. On my machine, at least, std.parallelism.totalCPUs returns 8, the number of virtual cores. As it should. If you can create a small example of the

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Andrei Alexandrescu via Digitalmars-d
On 5/4/14, 3:06 AM, Russel Winder via Digitalmars-d wrote: On Sun, 2014-05-04 at 08:47 +, Atila Neves via Digitalmars-d wrote: Like I mentioned afterwards, I tried a different number of threads. On my machine, at least, std.parallelism.totalCPUs returns 8, the number of virtual cores. As it

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Andrei Alexandrescu via Digitalmars-d
On 5/4/14, 1:44 AM, Atila Neves wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread Joseph Rushton Wakeling via Digitalmars-d
On 04/05/14 09:49, Russel Winder via Digitalmars-d wrote: (*) Physical cores are not necessarily the number reported by the OS due to core hyperthreads. Quad core no hyperthreads, and dual core, two hyperthreads per core, both get reported as four processor systems. However if you benchmark them

Re: Running Phobos unit tests in threads: I have data

2014-05-04 Thread safety0ff via Digitalmars-d
On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded

Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
So I tried using unit-threaded to run Phobos unit tests again and had problems (which I'll look into later) with its compile-time reflection. Then I realised I was an idiot since I don't need to reflect on anything: all Phobos tests are in unittest blocks so all I need to do is include them in

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
I turned off all output to check. It was still slower with multiple threads. That was the only weird thing I was doing I could think of as the cause. Otherwise it's just a foreach(test; tests.parallel) { test(); }. Atila On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote: So I tried

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Rikki Cattermole via Digitalmars-d
On Saturday, 3 May 2014 at 12:08:56 UTC, Atila Neves wrote: I turned off all output to check. It was still slower with multiple threads. That was the only weird thing I was doing I could think of as the cause. Otherwise it's just a foreach(test; tests.parallel) { test(); }. Atila On

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Rikki Cattermole via Digitalmars-d
On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila I'm surprised. Threads should be cheap on

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
Ok, so I went and added __traits(getUnitTests) to unit-threaded. That way each unittest block is its own test case. I registered these modules in std to run: array, ascii, base64, bigint, bitmanip, concurrency, container, cstream. On the good news front, they all passed even though they

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. The Phobos modules are all called ustd because I couldn't/didn't know how to get this to work otherwise. So I copied the

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Andrei Alexandrescu via Digitalmars-d
On 5/3/14, 4:54 AM, Atila Neves wrote: So I tried using unit-threaded to run Phobos unit tests [snip] Thanks. Are you using thread pooling (a limited number of threads e.g. 1.5 * cores running all unittests)? -- Andrei

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Walter Bright via Digitalmars-d
On 5/3/2014 5:26 AM, Rikki Cattermole wrote: Something funky is definitely going on I bet. No doubt: http://www.youtube.com/watch?v=aZcbDESaxhY

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Walter Bright via Digitalmars-d
On 5/3/2014 10:22 AM, Atila Neves wrote: I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. I haven't investigated this, but my suspicions are: 1. thread creation/destruction is

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
On Saturday, 3 May 2014 at 18:26:37 UTC, Walter Bright wrote: On 5/3/2014 10:22 AM, Atila Neves wrote: I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. I haven't

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
On Saturday, 3 May 2014 at 18:16:52 UTC, Andrei Alexandrescu wrote: On 5/3/14, 4:54 AM, Atila Neves wrote: So I tried using unit-threaded to run Phobos unit tests [snip] Thanks. Are you using thread pooling (a limited number of threads e.g. 1.5 * cores running all unittests)? -- Andrei I'm

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Dmitry Olshansky via Digitalmars-d
03-May-2014 21:22, Atila Neves пишет: I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. The Phobos modules are all called ustd because I couldn't/didn't know how to get this to

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
if(single) { foreach(test; tests) { test(); } } else { foreach(test; tests.parallel) { Try different batch size: test.parallel(1), test.parallel(2) etc. So as to not have thread creation be disproportionately represented, I repeated the module list

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Atila Neves via Digitalmars-d
Same thing with unit_threaded on Phobos, 3x faster even without repeating the modules (0.1s vs 0.3s). Since the example is shorter than the other one, I'll post it here in case anyone else wants to try: import unit_threaded.runner; int main(string[] args) { return args.runTests!(

Re: Running Phobos unit tests in threads: I have data

2014-05-03 Thread Andrei Alexandrescu via Digitalmars-d
On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only