It seems performance are always the same whatever the number of threads. I ran all these tests on my laptop (which is not a "great supercomputer" !... As Matt noticed, and as I knew it !)
FH ~/Bandwidth>export OMP_NUM_THREADS=1 ~/Bandwidth>./stream_c.exe > stream_1thread.log ~/Bandwidth>export OMP_NUM_THREADS=2 ~/Bandwidth>./stream_c.exe > stream_2threads.log ~/Bandwidth>export OMP_NUM_THREADS=4 ~/Bandwidth>./stream_c.exe > stream_4threads.log ~/Bandwidth>grep Triad *log stream_1thread.log:Triad: 13057.6 0.018427 0.018380 0.018539 stream_2threads.log:Triad: 13044.9 0.018496 0.018398 0.018936 stream_4threads.log:Triad: 13049.1 0.018449 0.018392 0.018561 ________________________________________ De : Jed Brown [[email protected]] de la part de Jed Brown [[email protected]] Date d'envoi : vendredi 21 juin 2013 12:41 À : HOUSSEN Franck; Barry Smith Cc: [email protected] Objet : Re: [petsc-users] RE : How to get bandwidth peak out of PETSc log ? HOUSSEN Franck <[email protected]> writes: > MatMult 13216 1.0 5.4614e+01 1.0 5.52e+10 1.0 0.0e+00 0.0e+00 > 0.0e+00 21 56 0 0 0 21 56 0 0 0 1010 Much of your time is spent here, which is bandwidth limited. It needs 6 bytes per flop (plus a little, if the vector is perfectly reused) , so this number is about 6 GB/s. > MatMult 13550 1.0 3.8204e+01 1.0 2.85e+10 1.0 2.7e+04 3.5e+04 > 0.0e+00 23 55 79 93 0 23 55 79 93 0 1494 Here with two processes, you have about 9 GB/s. Was your STREAM test (getting 11 GB/s) using multiple threads/processes? Can you send STREAM results for one and for two threads? 50% of STREAM is not very good (though it's actually the best you can do on some funny architectures), 70-85% is what we expect. If you're getting a low fraction of peak in MatMult, try reordering your matrix to have lower bandwidth. You can use MatGetOrdering with RCM for this.
------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 10000000 (elements), Offset = 0 (elements) Memory per array = 76.3 MiB (= 0.1 GiB). Total memory required = 228.9 MiB (= 0.2 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 10722 microseconds. (= 10722 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 11508.2 0.013937 0.013903 0.013992 Scale: 11433.5 0.014024 0.013994 0.014049 Add: 13026.5 0.018449 0.018424 0.018479 Triad: 13057.6 0.018427 0.018380 0.018539 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays -------------------------------------------------------------
------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 10000000 (elements), Offset = 0 (elements) Memory per array = 76.3 MiB (= 0.1 GiB). Total memory required = 228.9 MiB (= 0.2 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 10965 microseconds. (= 10965 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 11490.1 0.013941 0.013925 0.013959 Scale: 11420.3 0.014031 0.014010 0.014063 Add: 13021.4 0.018453 0.018431 0.018503 Triad: 13044.9 0.018496 0.018398 0.018936 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays -------------------------------------------------------------
------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 10000000 (elements), Offset = 0 (elements) Memory per array = 76.3 MiB (= 0.1 GiB). Total memory required = 228.9 MiB (= 0.2 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 10723 microseconds. (= 10723 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 11487.5 0.013973 0.013928 0.014067 Scale: 11414.0 0.014044 0.014018 0.014082 Add: 13029.3 0.018469 0.018420 0.018577 Triad: 13049.1 0.018449 0.018392 0.018561 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays -------------------------------------------------------------
