Re: [OMPI devel] Fwd: [OMPI bugs] [Open MPI] #1250: Performance problem on SM

Wed, 23 Jul 2008 11:32:04 -0400

I added a tad more output to the debugging statement so you can see how many processors were found, how many children we have, and what the sched_yield will be set to... 

Besides, that way I got to be the one that hit r19000!

On Jul 23, 2008, at 9:21 AM, Jeff Squyres wrote:
It's PLPA that's at fault here; I'm running on an older Linux kernel that doesn't have the topology information available. So PLPA is saying "can't give you anything, sorry" (to include how many processors are available) -- but that might not be true. 

I need to think about this a bit to come up with the right solution...


On Jul 23, 2008, at 10:41 AM, Ralph Castain wrote:
Here is a real simple test that will tell us a bunch about what is going on: run this again with -mca odls_base_verbose 5. You'll get some output, but what we are looking for specifically is a message that includes "launch oversubscribed set to...". This will tell us what ORTE -thinks- the sched yield should be.
I'm wondering if maybe that get_processor_info call in paffinity is returning the wrong #processors. 

Ralph

On Jul 23, 2008, at 8:37 AM, Terry Dontje wrote:
This seems to work for me too. What is interesting is my experiments have shown that if you run on RH5.1 you don't need to set mpi_yield_when_idle to 0. 

--td

Jeff Squyres wrote:
Doh! I guess we still don't have that calculating right yet; I thought we had fixed that...
[7:12] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % mpirun --mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self --mca mpi_yield_when_idle 0 NPmpi 
0: svbu-mpi052
1: svbu-mpi052
Now starting the main loop
0:       1 bytes 131689 times -->     11.22 Mbps in       0.68 usec
1:       2 bytes 147026 times -->     22.54 Mbps in       0.68 usec
2:       3 bytes 147741 times -->     33.65 Mbps in       0.68 usec
...
[7:12] svbu-mpi052:~/svn/ompi-tests/osu % mpirun --mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self --mca mpi_yield_when_idle 0 osu_latency 
# OSU MPI Latency Test (Version 2.1)
# Size        Latency (us)
0        0.64
1        0.67
2        0.67
4        0.74
...

I'll check with Ralph.



On Jul 23, 2008, at 10:01 AM, George Bosilca wrote:


Can you try the HEAD with the mpi_yield_when_idle set to 0 please.

Thanks,
george.


On Jul 23, 2008, at 3:39 PM, Jeff Squyres wrote:
Short version: I'm seeing a large performance drop between r18850 and the SVN HEAD. 

Longer version:
FWIW, I ran the tests on 3 versions on a woodcrest-class x86_64 machine running RHEL4U4: 

* Trunk HEAD (r18997)
* r18973 --> had to patch the cpu64* thingy in openib btl to get it to compile 
* r18850
I ran both osu_latency and NetPIPE 3.7.1. In the r18997 and r18973, the latency for short sends over sm is *significantly* higher than that of r18850. Detailed results below. 

================================================================
r18997
[6:27] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % mpirun -- mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self NPmpi 
0: svbu-mpi052
1: svbu-mpi052
Now starting the main loop
0: 1 bytes 85423 times --> 8.23 Mbps in 0.93 usec 1: 2 bytes 107852 times --> 16.46 Mbps in 0.93 usec 2: 3 bytes 107874 times --> 24.65 Mbps in 0.93 usec 3: 4 bytes 71801 times --> 30.36 Mbps in 1.01 usec 4: 6 bytes 74610 times --> 45.27 Mbps in 1.01 usec 5: 8 bytes 49448 times --> 60.59 Mbps in 1.01 usec 6: 12 bytes 62044 times --> 90.72 Mbps in 1.01 usec 7: 13 bytes 41287 times --> 98.58 Mbps in 1.01 usec 8: 16 bytes 45872 times --> 120.81 Mbps in 1.01 usec 9: 19 bytes 55670 times --> 143.78 Mbps in 1.01 usec 10: 21 bytes 62644 times --> 156.63 Mbps in 1.02 usec 11: 24 bytes 65172 times --> 177.63 Mbps in 1.03 usec 12: 27 bytes 68714 times --> 187.21 Mbps in 1.10 usec 13: 29 bytes 40392 times --> 201.05 Mbps in 1.10 usec 14: 32 bytes 43868 times --> 220.92 Mbps in 1.11 usec 15: 35 bytes 48072 times --> 255.73 Mbps in 1.04 usec 16: 45 bytes 54725 times --> 308.90 Mbps in 1.11 usec 17: 48 bytes 59983 times --> 329.04 Mbps in 1.11 usec 18: 51 bytes 61772 times --> 348.53 Mbps in 1.12 usec 19: 61 bytes 35126 times --> 408.86 Mbps in 1.14 usec 20: 64 bytes 43206 times --> 453.67 Mbps in 1.08 usec 21: 67 bytes 47907 times --> 487.77 Mbps in 1.05 usec 22: 93 bytes 51271 times --> 561.32 Mbps in 1.26 usec 23: 96 bytes 52741 times --> 595.08 Mbps in 1.23 usec 24: 99 bytes 55012 times --> 617.64 Mbps in 1.22 usec 25: 125 bytes 29735 times --> 736.44 Mbps in 1.29 usec 26: 128 bytes 38301 times --> 779.33 Mbps in 1.25 usec 27: 131 bytes 40525 times --> 818.32 Mbps in 1.22 usec 28: 189 bytes 42501 times --> 1007.67 Mbps in 1.43 usec 29: 192 bytes 46588 times --> 1084.13 Mbps in 1.35 usec 30: 195 bytes 49725 times --> 1128.97 Mbps in 1.32 usec 31: 253 bytes 26462 times --> 1257.97 Mbps in 1.53 usec 32: 256 bytes 32457 times --> 1304.17 Mbps in 1.50 usec 33: 259 bytes 33647 times --> 1354.14 Mbps in 1.46 usec 34: 381 bytes 34925 times --> 1616.43 Mbps in 1.80 usec 35: 384 bytes 37072 times --> 1676.92 Mbps in 1.75 usec 36: 387 bytes 38308 times --> 1724.50 Mbps in 1.71 usec 37: 509 bytes 19921 times --> 1908.30 Mbps in 2.03 usec 38: 512 bytes 24521 times --> 2013.16 Mbps in 1.94 usec 39: 515 bytes 25869 times --> 2038.18 Mbps in 1.93 usec 40: 765 bytes 26188 times --> 2474.81 Mbps in 2.36 usec 41: 768 bytes 28268 times --> 2513.00 Mbps in 2.33 usec 42: 771 bytes 28648 times --> 2531.45 Mbps in 2.32 usec 43: 1021 bytes 14512 times --> 2831.70 Mbps in 2.75 usec 44: 1024 bytes 18158 times --> 2853.94 Mbps in 2.74 usec 45: 1027 bytes 18300 times --> 2872.58 Mbps in 2.73 usec 46: 1533 bytes 18420 times --> 3298.65 Mbps in 3.55 usec 47: 1536 bytes 18802 times --> 3320.86 Mbps in 3.53 usec 48: 1539 bytes 18910 times --> 3351.99 Mbps in 3.50 usec 49: 2045 bytes 9571 times --> 3599.21 Mbps in 4.33 usec 50: 2048 bytes 11528 times --> 3640.91 Mbps in 4.29 usec 51: 2051 bytes 11662 times --> 3638.62 Mbps in 4.30 usec 52: 3069 bytes 11654 times --> 3905.17 Mbps in 6.00 usec 53: 3072 bytes 11118 times --> 3917.67 Mbps in 5.98 usec 54: 3075 bytes 11149 times --> 3973.53 Mbps in 5.90 usec 55: 4093 bytes 5662 times --> 4450.80 Mbps in 7.02 usec 56: 4096 bytes 7124 times --> 4445.17 Mbps in 7.03 usec 57: 4099 bytes 7115 times --> 4412.88 Mbps in 7.09 usec 58: 6141 bytes 7064 times --> 4962.74 Mbps in 9.44 usec 59: 6144 bytes 7061 times --> 4941.94 Mbps in 9.49 usec 60: 6147 bytes 7030 times --> 4938.46 Mbps in 9.50 usec 61: 8189 bytes 3515 times --> 5263.65 Mbps in 11.87 usec 62: 8192 bytes 4211 times --> 5249.31 Mbps in 11.91 usec 63: 8195 bytes 4200 times --> 5202.08 Mbps in 12.02 usec 64: 12285 bytes 4162 times --> 6380.89 Mbps in 14.69 usec 65: 12288 bytes 4538 times --> 6385.27 Mbps in 14.68 usec 66: 12291 bytes 4541 times --> 6335.05 Mbps in 14.80 usec 67: 16381 bytes 2253 times --> 6535.76 Mbps in 19.12 usec 68: 16384 bytes 2614 times --> 6537.24 Mbps in 19.12 usec 69: 16387 bytes 2615 times --> 6514.52 Mbps in 19.19 usec 70: 24573 bytes 2606 times --> 6870.51 Mbps in 27.29 usec 71: 24576 bytes 2443 times --> 6866.57 Mbps in 27.31 usec 72: 24579 bytes 2441 times --> 6864.32 Mbps in 27.32 usec 73: 32765 bytes 1220 times --> 7124.85 Mbps in 35.09 usec 74: 32768 bytes 1425 times --> 7120.30 Mbps in 35.11 usec 75: 32771 bytes 1424 times --> 7127.15 Mbps in 35.08 usec 76: 49149 bytes 1425 times --> 8313.31 Mbps in 45.11 usec 77: 49152 bytes 1478 times --> 8312.58 Mbps in 45.11 usec 78: 49155 bytes 1477 times --> 8309.34 Mbps in 45.13 usec 79: 65533 bytes 738 times --> 8219.82 Mbps in 60.83 usec 80: 65536 bytes 822 times --> 8209.24 Mbps in 60.91 usec 81: 65539 bytes 820 times --> 8216.00 Mbps in 60.86 usec 82: 98301 bytes 821 times --> 8698.24 Mbps in 86.22 usec 83: 98304 bytes 773 times --> 8695.03 Mbps in 86.26 usec 84: 98307 bytes 772 times --> 8696.95 Mbps in 86.24 usec 85: 131069 bytes 386 times --> 8916.50 Mbps in 112.15 usec 86: 131072 bytes 445 times --> 8917.29 Mbps in 112.14 usec 87: 131075 bytes 445 times --> 8916.62 Mbps in 112.15 usec 88: 196605 bytes 445 times --> 9205.17 Mbps in 162.95 usec 89: 196608 bytes 409 times --> 9195.75 Mbps in 163.12 usec 90: 196611 bytes 408 times --> 9203.02 Mbps in 162.99 usec 91: 262141 bytes 204 times --> 9338.32 Mbps in 214.17 usec 92: 262144 bytes 233 times --> 9350.57 Mbps in 213.89 usec 93: 262147 bytes 233 times --> 9336.72 Mbps in 214.21 usec 94: 393213 bytes 233 times --> 9480.21 Mbps in 316.45 usec 95: 393216 bytes 210 times --> 9476.10 Mbps in 316.59 usec 96: 393219 bytes 210 times --> 9471.25 Mbps in 316.75 usec 97: 524285 bytes 105 times --> 9523.20 Mbps in 420.02 usec 98: 524288 bytes 119 times --> 9519.53 Mbps in 420.19 usec 99: 524291 bytes 118 times --> 9523.09 Mbps in 420.03 usec 100: 786429 bytes 119 times --> 9555.83 Mbps in 627.89 usec 101: 786432 bytes 106 times --> 9542.67 Mbps in 628.75 usec 102: 786435 bytes 106 times --> 9554.47 Mbps in 627.98 usec 103: 1048573 bytes 53 times --> 9527.96 Mbps in 839.63 usec 104: 1048576 bytes 59 times --> 9530.63 Mbps in 839.40 usec 105: 1048579 bytes 59 times --> 9500.65 Mbps in 842.05 usec 106: 1572861 bytes 59 times --> 9389.53 Mbps in 1278.02 usec 107: 1572864 bytes 52 times --> 9396.87 Mbps in 1277.02 usec 108: 1572867 bytes 52 times --> 9375.01 Mbps in 1280.00 usec 109: 2097149 bytes 26 times --> 9271.33 Mbps in 1725.75 usec 110: 2097152 bytes 28 times --> 9273.64 Mbps in 1725.32 usec 111: 2097155 bytes 28 times --> 9281.42 Mbps in 1723.88 usec 112: 3145725 bytes 29 times --> 9109.93 Mbps in 2634.48 usec 113: 3145728 bytes 25 times --> 9128.80 Mbps in 2629.04 usec 114: 3145731 bytes 25 times --> 9099.66 Mbps in 2637.46 usec 115: 4194301 bytes 12 times --> 8840.19 Mbps in 3619.83 usec 116: 4194304 bytes 13 times --> 8847.10 Mbps in 3617.00 usec 117: 4194307 bytes 13 times --> 8827.22 Mbps in 3625.15 usec 118: 6291453 bytes 13 times --> 8351.40 Mbps in 5747.54 usec 119: 6291456 bytes 11 times --> 8345.46 Mbps in 5751.63 usec 120: 6291459 bytes 11 times --> 8343.42 Mbps in 5753.04 usec 121: 8388605 bytes 5 times --> 8166.28 Mbps in 7837.10 usec 122: 8388608 bytes 6 times --> 8166.91 Mbps in 7836.50 usec 123: 8388611 bytes 6 times --> 8162.67 Mbps in 7840.57 usec 
[6:29] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % cd ../osu/
[6:29] svbu-mpi052:~/svn/ompi-tests/osu % mpirun --mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self osu_latency 
# OSU MPI Latency Test (Version 2.1)
# Size        Latency (us)
0        0.85
1        0.91
2        0.91
4        0.99
8        0.99
16        0.99
32        1.08
64        1.08
128        1.25
256        1.49
512        1.92
1024        2.71
2048        4.40
4096        6.85
8192        11.48
16384        19.25
32768        35.25
65536        61.03
131072        113.15
262144        215.54
524288        428.19
1048576        880.72
2097152        1839.12
4194304        3934.90
[6:29] svbu-mpi052:~/svn/ompi-tests/osu %

================================================================
r18973
[6:36] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % mpirun -- mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self NPmpi 
1: svbu-mpi052
0: svbu-mpi052
Now starting the main loop
0: 1 bytes 84392 times --> 8.29 Mbps in 0.92 usec 1: 2 bytes 108626 times --> 16.58 Mbps in 0.92 usec 2: 3 bytes 108657 times --> 24.91 Mbps in 0.92 usec 3: 4 bytes 72561 times --> 30.33 Mbps in 1.01 usec 4: 6 bytes 74529 times --> 45.51 Mbps in 1.01 usec 5: 8 bytes 49709 times --> 60.76 Mbps in 1.00 usec 6: 12 bytes 62222 times --> 90.84 Mbps in 1.01 usec 7: 13 bytes 41344 times --> 98.58 Mbps in 1.01 usec 8: 16 bytes 45875 times --> 121.19 Mbps in 1.01 usec 9: 19 bytes 55845 times --> 143.43 Mbps in 1.01 usec 10: 21 bytes 62491 times --> 156.66 Mbps in 1.02 usec 11: 24 bytes 65185 times --> 177.87 Mbps in 1.03 usec 12: 27 bytes 68806 times --> 187.63 Mbps in 1.10 usec 13: 29 bytes 40482 times --> 202.10 Mbps in 1.09 usec 14: 32 bytes 44096 times --> 222.11 Mbps in 1.10 usec 15: 35 bytes 48331 times --> 255.12 Mbps in 1.05 usec 16: 45 bytes 54593 times --> 308.42 Mbps in 1.11 usec 17: 48 bytes 59888 times --> 330.10 Mbps in 1.11 usec 18: 51 bytes 61970 times --> 348.31 Mbps in 1.12 usec 19: 61 bytes 35104 times --> 409.39 Mbps in 1.14 usec 20: 64 bytes 43261 times --> 451.69 Mbps in 1.08 usec 21: 67 bytes 47698 times --> 489.98 Mbps in 1.04 usec 22: 93 bytes 51504 times --> 565.69 Mbps in 1.25 usec 23: 96 bytes 53150 times --> 598.55 Mbps in 1.22 usec 24: 99 bytes 55333 times --> 623.24 Mbps in 1.21 usec 25: 125 bytes 30005 times --> 735.91 Mbps in 1.30 usec 26: 128 bytes 38274 times --> 781.32 Mbps in 1.25 usec 27: 131 bytes 40628 times --> 828.90 Mbps in 1.21 usec 28: 189 bytes 43050 times --> 1018.02 Mbps in 1.42 usec 29: 192 bytes 47066 times --> 1069.01 Mbps in 1.37 usec 30: 195 bytes 49032 times --> 1122.18 Mbps in 1.33 usec 31: 253 bytes 26303 times --> 1259.95 Mbps in 1.53 usec 32: 256 bytes 32508 times --> 1307.53 Mbps in 1.49 usec 33: 259 bytes 33734 times --> 1357.47 Mbps in 1.46 usec 34: 381 bytes 35011 times --> 1617.08 Mbps in 1.80 usec 35: 384 bytes 37087 times --> 1675.72 Mbps in 1.75 usec 36: 387 bytes 38280 times --> 1722.27 Mbps in 1.71 usec 37: 509 bytes 19895 times --> 1913.58 Mbps in 2.03 usec 38: 512 bytes 24589 times --> 1967.08 Mbps in 1.99 usec 39: 515 bytes 25276 times --> 2041.10 Mbps in 1.93 usec 40: 765 bytes 26226 times --> 2448.96 Mbps in 2.38 usec 41: 768 bytes 27973 times --> 2503.60 Mbps in 2.34 usec 42: 771 bytes 28541 times --> 2541.12 Mbps in 2.31 usec 43: 1021 bytes 14567 times --> 2845.46 Mbps in 2.74 usec 44: 1024 bytes 18246 times --> 2854.45 Mbps in 2.74 usec 45: 1027 bytes 18304 times --> 2939.64 Mbps in 2.67 usec 46: 1533 bytes 18850 times --> 3291.70 Mbps in 3.55 usec 47: 1536 bytes 18762 times --> 3310.45 Mbps in 3.54 usec 48: 1539 bytes 18851 times --> 3386.68 Mbps in 3.47 usec 49: 2045 bytes 9670 times --> 3635.22 Mbps in 4.29 usec 50: 2048 bytes 11644 times --> 3646.70 Mbps in 4.28 usec 51: 2051 bytes 11680 times --> 3640.09 Mbps in 4.30 usec 52: 3069 bytes 11659 times --> 3926.68 Mbps in 5.96 usec 53: 3072 bytes 11180 times --> 3962.33 Mbps in 5.92 usec 54: 3075 bytes 11276 times --> 3978.54 Mbps in 5.90 usec 55: 4093 bytes 5669 times --> 4398.66 Mbps in 7.10 usec 56: 4096 bytes 7041 times --> 4429.95 Mbps in 7.05 usec 57: 4099 bytes 7091 times --> 4378.99 Mbps in 7.14 usec 58: 6141 bytes 7009 times --> 5001.17 Mbps in 9.37 usec 59: 6144 bytes 7116 times --> 4984.01 Mbps in 9.41 usec 60: 6147 bytes 7090 times --> 5015.48 Mbps in 9.35 usec 61: 8189 bytes 3570 times --> 5286.90 Mbps in 11.82 usec 62: 8192 bytes 4230 times --> 5222.58 Mbps in 11.97 usec 63: 8195 bytes 4179 times --> 5261.91 Mbps in 11.88 usec 64: 12285 bytes 4210 times --> 6370.90 Mbps in 14.71 usec 65: 12288 bytes 4531 times --> 6376.57 Mbps in 14.70 usec 66: 12291 bytes 4535 times --> 6349.10 Mbps in 14.77 usec 67: 16381 bytes 2258 times --> 6521.57 Mbps in 19.16 usec 68: 16384 bytes 2608 times --> 6520.25 Mbps in 19.17 usec 69: 16387 bytes 2608 times --> 6504.81 Mbps in 19.22 usec 70: 24573 bytes 2602 times --> 6867.93 Mbps in 27.30 usec 71: 24576 bytes 2442 times --> 6869.27 Mbps in 27.30 usec 72: 24579 bytes 2442 times --> 6864.04 Mbps in 27.32 usec 73: 32765 bytes 1220 times --> 7118.03 Mbps in 35.12 usec 74: 32768 bytes 1423 times --> 7117.77 Mbps in 35.12 usec 75: 32771 bytes 1423 times --> 7120.85 Mbps in 35.11 usec 76: 49149 bytes 1424 times --> 8324.26 Mbps in 45.05 usec 77: 49152 bytes 1479 times --> 8328.77 Mbps in 45.02 usec 78: 49155 bytes 1480 times --> 8320.47 Mbps in 45.07 usec 79: 65533 bytes 739 times --> 8214.38 Mbps in 60.87 usec 80: 65536 bytes 821 times --> 8219.87 Mbps in 60.83 usec 81: 65539 bytes 822 times --> 8232.40 Mbps in 60.74 usec 82: 98301 bytes 823 times --> 8717.21 Mbps in 86.03 usec 83: 98304 bytes 774 times --> 8716.08 Mbps in 86.05 usec 84: 98307 bytes 774 times --> 8714.26 Mbps in 86.07 usec 85: 131069 bytes 387 times --> 8921.59 Mbps in 112.09 usec 86: 131072 bytes 446 times --> 8935.37 Mbps in 111.91 usec 87: 131075 bytes 446 times --> 8925.47 Mbps in 112.04 usec 88: 196605 bytes 446 times --> 9195.80 Mbps in 163.12 usec 89: 196608 bytes 408 times --> 9197.41 Mbps in 163.09 usec 90: 196611 bytes 408 times --> 9204.33 Mbps in 162.97 usec 91: 262141 bytes 204 times --> 9344.95 Mbps in 214.02 usec 92: 262144 bytes 233 times --> 9347.58 Mbps in 213.96 usec 93: 262147 bytes 233 times --> 9340.56 Mbps in 214.12 usec 94: 393213 bytes 233 times --> 9473.27 Mbps in 316.68 usec 95: 393216 bytes 210 times --> 9486.24 Mbps in 316.25 usec 96: 393219 bytes 210 times --> 9500.26 Mbps in 315.78 usec 97: 524285 bytes 105 times --> 9538.88 Mbps in 419.33 usec 98: 524288 bytes 119 times --> 9543.40 Mbps in 419.14 usec 99: 524291 bytes 119 times --> 9534.73 Mbps in 419.52 usec 100: 786429 bytes 119 times --> 9574.15 Mbps in 626.69 usec 101: 786432 bytes 106 times --> 9565.70 Mbps in 627.24 usec 102: 786435 bytes 106 times --> 9544.50 Mbps in 628.64 usec 103: 1048573 bytes 53 times --> 9530.85 Mbps in 839.38 usec 104: 1048576 bytes 59 times --> 9525.24 Mbps in 839.87 usec 105: 1048579 bytes 59 times --> 9511.86 Mbps in 841.06 usec 106: 1572861 bytes 59 times --> 9391.40 Mbps in 1277.76 usec 107: 1572864 bytes 52 times --> 9395.54 Mbps in 1277.20 usec 108: 1572867 bytes 52 times --> 9386.02 Mbps in 1278.50 usec 109: 2097149 bytes 26 times --> 9298.48 Mbps in 1720.71 usec 110: 2097152 bytes 29 times --> 9313.43 Mbps in 1717.95 usec 111: 2097155 bytes 29 times --> 9293.49 Mbps in 1721.64 usec 112: 3145725 bytes 29 times --> 9126.67 Mbps in 2629.65 usec 113: 3145728 bytes 25 times --> 9113.76 Mbps in 2633.38 usec 114: 3145731 bytes 25 times --> 9079.90 Mbps in 2643.20 usec 115: 4194301 bytes 12 times --> 8810.57 Mbps in 3632.00 usec 116: 4194304 bytes 13 times --> 8821.99 Mbps in 3627.30 usec 117: 4194307 bytes 13 times --> 8801.17 Mbps in 3635.88 usec 118: 6291453 bytes 13 times --> 8337.50 Mbps in 5757.12 usec 119: 6291456 bytes 11 times --> 8332.94 Mbps in 5760.27 usec 120: 6291459 bytes 11 times --> 8346.25 Mbps in 5751.09 usec 121: 8388605 bytes 5 times --> 8159.20 Mbps in 7843.90 usec 122: 8388608 bytes 6 times --> 8166.83 Mbps in 7836.58 usec 123: 8388611 bytes 6 times --> 8161.26 Mbps in 7841.92 usec 
[6:37] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % cd ../osu/
[6:37] svbu-mpi052:~/svn/ompi-tests/osu % mpirun --mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self osu_latency 
# OSU MPI Latency Test (Version 2.1)
# Size        Latency (us)
0        0.85
1        0.91
2        0.91
4        0.99
8        0.99
16        0.99
32        1.09
64        1.07
128        1.25
256        1.49
512        1.97
1024        2.69
2048        4.29
4096        6.83
8192        11.41
16384        19.69
32768        35.27
65536        61.06
131072        112.51
262144        215.47
524288        429.60
1048576        882.89
2097152        1836.45
4194304        3943.47
[6:37] svbu-mpi052:~/svn/ompi-tests/osu %

================================================================
r18850
[6:31] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % mpirun -- mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self NPmpi 
0: svbu-mpi052
1: svbu-mpi052
Now starting the main loop
0: 1 bytes 116185 times --> 11.32 Mbps in 0.67 usec 1: 2 bytes 148348 times --> 22.58 Mbps in 0.68 usec 2: 3 bytes 147969 times --> 33.88 Mbps in 0.68 usec 3: 4 bytes 98695 times --> 40.58 Mbps in 0.75 usec 4: 6 bytes 99737 times --> 60.85 Mbps in 0.75 usec 5: 8 bytes 66464 times --> 81.13 Mbps in 0.75 usec 6: 12 bytes 83076 times --> 121.58 Mbps in 0.75 usec 7: 13 bytes 55334 times --> 131.83 Mbps in 0.75 usec 8: 16 bytes 61344 times --> 161.81 Mbps in 0.75 usec 9: 19 bytes 74561 times --> 190.93 Mbps in 0.76 usec 10: 21 bytes 83186 times --> 207.97 Mbps in 0.77 usec 11: 24 bytes 86535 times --> 235.30 Mbps in 0.78 usec 12: 27 bytes 91024 times --> 241.36 Mbps in 0.85 usec 13: 29 bytes 52074 times --> 260.24 Mbps in 0.85 usec 14: 32 bytes 56782 times --> 286.57 Mbps in 0.85 usec 15: 35 bytes 62357 times --> 341.55 Mbps in 0.78 usec 16: 45 bytes 73090 times --> 400.53 Mbps in 0.86 usec 17: 48 bytes 77776 times --> 425.94 Mbps in 0.86 usec 18: 51 bytes 79963 times --> 449.27 Mbps in 0.87 usec 19: 61 bytes 45280 times --> 520.58 Mbps in 0.89 usec 20: 64 bytes 55011 times --> 589.77 Mbps in 0.83 usec 21: 67 bytes 62279 times --> 651.96 Mbps in 0.78 usec 22: 93 bytes 68530 times --> 706.75 Mbps in 1.00 usec 23: 96 bytes 66405 times --> 756.56 Mbps in 0.97 usec 24: 99 bytes 69940 times --> 786.11 Mbps in 0.96 usec 25: 125 bytes 37846 times --> 917.31 Mbps in 1.04 usec 26: 128 bytes 47708 times --> 991.21 Mbps in 0.99 usec 27: 131 bytes 51542 times --> 1030.40 Mbps in 0.97 usec 28: 189 bytes 53515 times --> 1228.14 Mbps in 1.17 usec 29: 192 bytes 56781 times --> 1317.94 Mbps in 1.11 usec 30: 195 bytes 60449 times --> 1372.28 Mbps in 1.08 usec 31: 253 bytes 32165 times --> 1506.60 Mbps in 1.28 usec 32: 256 bytes 38871 times --> 1590.08 Mbps in 1.23 usec 33: 259 bytes 41024 times --> 1657.90 Mbps in 1.19 usec 34: 381 bytes 42760 times --> 1894.98 Mbps in 1.53 usec 35: 384 bytes 43460 times --> 1958.92 Mbps in 1.50 usec 36: 387 bytes 44750 times --> 2029.44 Mbps in 1.45 usec 37: 509 bytes 23444 times --> 2176.96 Mbps in 1.78 usec 38: 512 bytes 27974 times --> 2268.97 Mbps in 1.72 usec 39: 515 bytes 29156 times --> 2340.62 Mbps in 1.68 usec 40: 765 bytes 30074 times --> 2698.17 Mbps in 2.16 usec 41: 768 bytes 30819 times --> 2778.48 Mbps in 2.11 usec 42: 771 bytes 31674 times --> 2847.11 Mbps in 2.07 usec 43: 1021 bytes 16322 times --> 3039.90 Mbps in 2.56 usec 44: 1024 bytes 19493 times --> 3161.06 Mbps in 2.47 usec 45: 1027 bytes 20270 times --> 3221.90 Mbps in 2.43 usec 46: 1533 bytes 20660 times --> 3455.95 Mbps in 3.38 usec 47: 1536 bytes 19698 times --> 3580.63 Mbps in 3.27 usec 48: 1539 bytes 20389 times --> 3623.40 Mbps in 3.24 usec 49: 2045 bytes 10346 times --> 3751.80 Mbps in 4.16 usec 50: 2048 bytes 12017 times --> 3833.40 Mbps in 4.08 usec 51: 2051 bytes 12278 times --> 3813.67 Mbps in 4.10 usec 52: 3069 bytes 12215 times --> 3997.25 Mbps in 5.86 usec 53: 3072 bytes 11381 times --> 4058.18 Mbps in 5.78 usec 54: 3075 bytes 11548 times --> 4102.09 Mbps in 5.72 usec 55: 4093 bytes 5845 times --> 4726.24 Mbps in 6.61 usec 56: 4096 bytes 7565 times --> 4679.74 Mbps in 6.68 usec 57: 4099 bytes 7491 times --> 4649.50 Mbps in 6.73 usec 58: 6141 bytes 7442 times --> 5072.39 Mbps in 9.24 usec 59: 6144 bytes 7217 times --> 5064.70 Mbps in 9.26 usec 60: 6147 bytes 7204 times --> 5067.07 Mbps in 9.26 usec 61: 8189 bytes 3606 times --> 5387.85 Mbps in 11.60 usec 62: 8192 bytes 4311 times --> 5393.87 Mbps in 11.59 usec 63: 8195 bytes 4316 times --> 5301.81 Mbps in 11.79 usec 64: 12285 bytes 4242 times --> 6568.81 Mbps in 14.27 usec 65: 12288 bytes 4672 times --> 6561.90 Mbps in 14.29 usec 66: 12291 bytes 4666 times --> 6548.01 Mbps in 14.32 usec 67: 16381 bytes 2329 times --> 6662.43 Mbps in 18.76 usec 68: 16384 bytes 2665 times --> 6655.18 Mbps in 18.78 usec 69: 16387 bytes 2662 times --> 6634.79 Mbps in 18.84 usec 70: 24573 bytes 2654 times --> 6937.26 Mbps in 27.02 usec 71: 24576 bytes 2466 times --> 6937.41 Mbps in 27.03 usec 72: 24579 bytes 2466 times --> 6931.40 Mbps in 27.05 usec 73: 32765 bytes 1232 times --> 7218.55 Mbps in 34.63 usec 74: 32768 bytes 1443 times --> 7213.85 Mbps in 34.66 usec 75: 32771 bytes 1442 times --> 7218.89 Mbps in 34.63 usec 76: 49149 bytes 1443 times --> 8387.79 Mbps in 44.71 usec 77: 49152 bytes 1491 times --> 8385.50 Mbps in 44.72 usec 78: 49155 bytes 1490 times --> 8390.79 Mbps in 44.69 usec 79: 65533 bytes 745 times --> 8261.32 Mbps in 60.52 usec 80: 65536 bytes 826 times --> 8260.34 Mbps in 60.53 usec 81: 65539 bytes 826 times --> 8265.33 Mbps in 60.50 usec 82: 98301 bytes 826 times --> 8747.13 Mbps in 85.74 usec 83: 98304 bytes 777 times --> 8746.72 Mbps in 85.75 usec 84: 98307 bytes 777 times --> 8733.81 Mbps in 85.88 usec 85: 131069 bytes 388 times --> 8956.71 Mbps in 111.65 usec 86: 131072 bytes 447 times --> 8967.16 Mbps in 111.52 usec 87: 131075 bytes 448 times --> 8960.56 Mbps in 111.60 usec 88: 196605 bytes 448 times --> 9247.58 Mbps in 162.20 usec 89: 196608 bytes 411 times --> 9234.30 Mbps in 162.44 usec 90: 196611 bytes 410 times --> 9231.32 Mbps in 162.49 usec 91: 262141 bytes 205 times --> 9365.98 Mbps in 213.54 usec 92: 262144 bytes 234 times --> 9368.25 Mbps in 213.49 usec 93: 262147 bytes 234 times --> 9363.09 Mbps in 213.61 usec 94: 393213 bytes 234 times --> 9512.63 Mbps in 315.37 usec 95: 393216 bytes 211 times --> 9497.01 Mbps in 315.89 usec 96: 393219 bytes 211 times --> 9510.80 Mbps in 315.43 usec 97: 524285 bytes 105 times --> 9553.55 Mbps in 418.69 usec 98: 524288 bytes 119 times --> 9561.59 Mbps in 418.34 usec 99: 524291 bytes 119 times --> 9551.86 Mbps in 418.77 usec 100: 786429 bytes 119 times --> 9582.63 Mbps in 626.13 usec 101: 786432 bytes 106 times --> 9576.72 Mbps in 626.52 usec 102: 786435 bytes 106 times --> 9584.78 Mbps in 625.99 usec 103: 1048573 bytes 53 times --> 9545.32 Mbps in 838.10 usec 104: 1048576 bytes 59 times --> 9532.37 Mbps in 839.25 usec 105: 1048579 bytes 59 times --> 9542.90 Mbps in 838.32 usec 106: 1572861 bytes 59 times --> 9434.44 Mbps in 1271.93 usec 107: 1572864 bytes 52 times --> 9400.64 Mbps in 1276.51 usec 108: 1572867 bytes 52 times --> 9409.24 Mbps in 1275.34 usec 109: 2097149 bytes 26 times --> 9305.75 Mbps in 1719.36 usec 110: 2097152 bytes 29 times --> 9314.56 Mbps in 1717.74 usec 111: 2097155 bytes 29 times --> 9278.43 Mbps in 1724.43 usec 112: 3145725 bytes 28 times --> 9065.15 Mbps in 2647.50 usec 113: 3145728 bytes 25 times --> 9095.10 Mbps in 2638.78 usec 114: 3145731 bytes 25 times --> 9073.88 Mbps in 2644.96 usec 115: 4194301 bytes 12 times --> 8772.63 Mbps in 3647.70 usec 116: 4194304 bytes 13 times --> 8768.32 Mbps in 3649.50 usec 117: 4194307 bytes 13 times --> 8771.37 Mbps in 3648.24 usec 118: 6291453 bytes 13 times --> 8321.22 Mbps in 5768.38 usec 119: 6291456 bytes 11 times --> 8320.00 Mbps in 5769.23 usec 120: 6291459 bytes 11 times --> 8335.25 Mbps in 5758.68 usec 121: 8388605 bytes 5 times --> 8167.02 Mbps in 7836.39 usec 122: 8388608 bytes 6 times --> 8165.44 Mbps in 7837.91 usec 123: 8388611 bytes 6 times --> 8162.24 Mbps in 7840.99 usec 
[6:32] svbu-mpi052:~/svn/ompi-tests/NetPIPE-3.7.1 % cd ../osu/
[6:32] svbu-mpi052:~/svn/ompi-tests/osu % mpirun --mca mpi_paffinity_alone 1 -np 2 --mca btl sm,self osu_latency 
# OSU MPI Latency Test (Version 2.1)
# Size        Latency (us)
0        0.65
1        0.69
2        0.69
4        0.76
8        0.76
16        0.76
32        0.85
64        0.83
128        1.03
256        1.25
512        1.73
1024        2.47
2048        4.18
4096        6.53
8192        11.23
16384        18.91
32768        34.97
65536        60.80
131072        112.09
262144        215.15
524288        427.97
1048576        880.90
2097152        1840.40
4194304        3945.23
[6:33] svbu-mpi052:~/svn/ompi-tests/osu %



On Jul 23, 2008, at 7:24 AM, Lenny Verkhovsky wrote:


Sorry Terry, :).

---------- Forwarded message ----------
From: Lenny Verkhovsky <lenny.verkhov...@gmail.com>
Date: Jul 23, 2008 2:22 PM
Subject: Re: [OMPI devel] [OMPI bugs] [Open MPI] #1250: Performance problem on SM 
To: Lenny Berkhovsky <lenny.verkhov...@gmail.com>
On 7/23/08, Terry Dontje <terry.don...@sun.com> wrote: I didn't see any attached results on the email. 

--td
Lenny Verkhovsky wrote:

I rechecked in on the same node, still no degradation,

see results attached.
On 7/22/08, *Open MPI* <b...@open-mpi.org <mailto:b...@open-mpi.org >> wrote: 

#1250: Performance problem on SM
-------------------- +------------------------------------------------------- 
Reporter:  bosilca  |        Owner:  bosilca
  Type:  defect   |       Status:  assigned
Priority:  blocker  |    Milestone:  Open MPI 1.3
Version:           |   Resolution:
Keywords:           |
-------------------- +------------------------------------------------------- 


Comment(by tdd):

Hmmm, Lennyve isn't your mpirun above going across nodes and not
on the
same node?  I am running netpipe on a single node.


--
Ticket URL:
<https://svn.open-mpi.org/trac/ompi/ticket/1250#comment:20>

Open MPI <http://www.open-mpi.org/>


_______________________________________________
bugs mailing list
b...@open-mpi.org <mailto:b...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/bugs


------------------------------------------------------------------------

_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




<NPmpi.log>_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel





_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Reply via email to