date:20160808

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-08-08 Thread Nathan Hjelm

No problem. Thanks for reporting this. Not all platforms see a slowdown so we 
missed it before the release. Let me know if that latest patch works for you.

-Nathan

> On Aug 8, 2016, at 8:50 PM, tmish...@jcity.maeda.co.jp wrote:
> 
> I understood. Thanks.
> 
> Tetsuya Mishima
> 
> 2016/08/09 11:33:15、"devel"さんは「Re: [OMPI devel] sm BTL performace of
> the openmpi-2.0.0」で書きました
>> I will add a control to have the new behavior or using all available RDMA
> btls or just the eager ones for the RDMA protocol. The flags will remain as
> they are. And, yes, for 2.0.0 you can set the btl
>> flags if you do not intend to use MPI RMA.
>> 
>> New patch:
>> 
>> 
> https://github.com/hjelmn/ompi/commit/43267012e58d78e3fc713b98c6fb9f782de977c7.patch
> 
>> 
>> -Nathan
>> 
>>> On Aug 8, 2016, at 8:16 PM, tmish...@jcity.maeda.co.jp wrote:
>>> 
>>> Then, my understanding is that you will restore the default value of
>>> btl_openib_flags to previous one( = 310) and add a new MCA parameter to
>>> control HCA inclusion for such a situation. The work arround so far for
>>> openmpi-2.0.0 is setting those flags manually. Right?
>>> 
>>> Tetsuya Mishima
>>> 
>>> 2016/08/09 9:56:29、"devel"さんは「Re: [OMPI devel] sm BTL performace
> of
>>> the openmpi-2.0.0」で書きました
 Hmm, not good. So we have a situation where it is sometimes better to
>>> include the HCA when it is the only rdma btl. Will have a new version
> up in
>>> a bit that adds an MCA parameter to control the
 behavior. The default will be the same as 1.10.x.
 
 -Nathan
 
> On Aug 8, 2016, at 4:51 PM, tmish...@jcity.maeda.co.jp wrote:
> 
> Hi, unfortunately it doesn't work well. The previous one was much
> better ...
> 
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2
> -report-bindings
> osu_bw
> [manage.cluster:25107] MCW rank 0 bound to socket 0[core 0[hwt 0]],
>>> socket
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
>>> 0]]:
> [B/B/B/B/B/B][./././././.]
> [manage.cluster:25107] MCW rank 1 bound to socket 0[core 0[hwt 0]],
>>> socket
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
>>> 0]]:
> [B/B/B/B/B/B][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 2.22
> 2 4.53
> 4 9.11
> 818.02
> 16   35.44
> 32   70.84
> 64  113.71
> 128 176.74
> 256 311.07
> 512 529.03
> 1024907.83
> 2048   1597.66
> 4096330.14
> 8192516.49
> 16384   780.31
> 32768  1038.43
> 65536  1186.36
> 131072 1268.87
> 262144 1222.24
> 524288 1232.30
> 10485761244.62
> 20971521260.25
> 41943041263.47
> 
> Tetsuya
> 
> 
> 2016/08/09 2:42:24、"devel"さんは「Re: [OMPI devel] sm BTL performace
>>> of
> the openmpi-2.0.0」で書きました
>> Ok, there was a problem with the selection logic when only one rdma
> capable btl is available. I changed the logic to always use the RDMA
>>> btl
> over pipelined send/recv. This works better for me on a
>> Intel Omnipath system. Let me know if this works for you.
>> 
>> 
> 
>>> 
> https://github.com/hjelmn/ompi/commit/dddb865b5337213fd73d0e226b02e2f049cfab47.patch
> 
>>> 
> 
>> 
>> -Nathan
>> 
>> On Aug 07, 2016, at 10:00 PM, tmish...@jcity.maeda.co.jp wrote:
>> 
>> Hi, here is the gdb output for additional information:
>> 
>> (It might be inexact, because I built openmpi-2.0.0 without debug
>>> option)
>> 
>> Core was generated by `osu_bw'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>> (gdb) where
>> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>> #1 0x0031d9008934 in _Unwind_Backtrace ()
>>> from /lib64/libgcc_s.so.1
>> #2 0x0037ab8e5ee8 in backtrace () from /lib64/libc.so.6
>> #3 0x2ad882bd4345 in opal_backtrace_print ()
>> at ./backtrace_execinfo.c:47
>> #4 0x2ad882bd1180 in show_stackframe () at ./stacktrace.c:331
>> #5 
>> #6 mca_pml_ob1_recv_request_schedule_once ()
>>> at ./pml_ob1_recvreq.c:983
>> #7 0x2aaab412f47a in mca_pml_ob1_recv_request_progress_rndv ()
>> 
>> 
> 
>>> 
> from /home/mishima/opt/mpi/openmpi-2.0.0-pgi16.5/lib/openmpi/mca_pml_ob1.so
>> #8 0x2aaab412c645 in mca_pml_ob1_recv_frag_match ()
>> at

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-08-08 Thread tmishima

I understood. Thanks.

Tetsuya Mishima

2016/08/09 11:33:15、"devel"さんは「Re: [OMPI devel] sm BTL performace of
the openmpi-2.0.0」で書きました
> I will add a control to have the new behavior or using all available RDMA
btls or just the eager ones for the RDMA protocol. The flags will remain as
they are. And, yes, for 2.0.0 you can set the btl
> flags if you do not intend to use MPI RMA.
>
> New patch:
>
>
https://github.com/hjelmn/ompi/commit/43267012e58d78e3fc713b98c6fb9f782de977c7.patch

>
> -Nathan
>
> > On Aug 8, 2016, at 8:16 PM, tmish...@jcity.maeda.co.jp wrote:
> >
> > Then, my understanding is that you will restore the default value of
> > btl_openib_flags to previous one( = 310) and add a new MCA parameter to
> > control HCA inclusion for such a situation. The work arround so far for
> > openmpi-2.0.0 is setting those flags manually. Right?
> >
> > Tetsuya Mishima
> >
> > 2016/08/09 9:56:29、"devel"さんは「Re: [OMPI devel] sm BTL performace
of
> > the openmpi-2.0.0」で書きました
> >> Hmm, not good. So we have a situation where it is sometimes better to
> > include the HCA when it is the only rdma btl. Will have a new version
up in
> > a bit that adds an MCA parameter to control the
> >> behavior. The default will be the same as 1.10.x.
> >>
> >> -Nathan
> >>
> >>> On Aug 8, 2016, at 4:51 PM, tmish...@jcity.maeda.co.jp wrote:
> >>>
> >>> Hi, unfortunately it doesn't work well. The previous one was much
> >>> better ...
> >>>
> >>> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2
-report-bindings
> >>> osu_bw
> >>> [manage.cluster:25107] MCW rank 0 bound to socket 0[core 0[hwt 0]],
> > socket
> >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
> > 0]]:
> >>> [B/B/B/B/B/B][./././././.]
> >>> [manage.cluster:25107] MCW rank 1 bound to socket 0[core 0[hwt 0]],
> > socket
> >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
> > 0]]:
> >>> [B/B/B/B/B/B][./././././.]
> >>> # OSU MPI Bandwidth Test v3.1.1
> >>> # SizeBandwidth (MB/s)
> >>> 1 2.22
> >>> 2 4.53
> >>> 4 9.11
> >>> 818.02
> >>> 16   35.44
> >>> 32   70.84
> >>> 64  113.71
> >>> 128 176.74
> >>> 256 311.07
> >>> 512 529.03
> >>> 1024907.83
> >>> 2048   1597.66
> >>> 4096330.14
> >>> 8192516.49
> >>> 16384   780.31
> >>> 32768  1038.43
> >>> 65536  1186.36
> >>> 131072 1268.87
> >>> 262144 1222.24
> >>> 524288 1232.30
> >>> 10485761244.62
> >>> 20971521260.25
> >>> 41943041263.47
> >>>
> >>> Tetsuya
> >>>
> >>>
> >>> 2016/08/09 2:42:24、"devel"さんは「Re: [OMPI devel] sm BTL performace
> > of
> >>> the openmpi-2.0.0」で書きました
>  Ok, there was a problem with the selection logic when only one rdma
> >>> capable btl is available. I changed the logic to always use the RDMA
> > btl
> >>> over pipelined send/recv. This works better for me on a
>  Intel Omnipath system. Let me know if this works for you.
> 
> 
> >>>
> >
https://github.com/hjelmn/ompi/commit/dddb865b5337213fd73d0e226b02e2f049cfab47.patch

> >
> >>>
> 
>  -Nathan
> 
>  On Aug 07, 2016, at 10:00 PM, tmish...@jcity.maeda.co.jp wrote:
> 
>  Hi, here is the gdb output for additional information:
> 
>  (It might be inexact, because I built openmpi-2.0.0 without debug
> > option)
> 
>  Core was generated by `osu_bw'.
>  Program terminated with signal 11, Segmentation fault.
>  #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>  (gdb) where
>  #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>  #1 0x0031d9008934 in _Unwind_Backtrace ()
> > from /lib64/libgcc_s.so.1
>  #2 0x0037ab8e5ee8 in backtrace () from /lib64/libc.so.6
>  #3 0x2ad882bd4345 in opal_backtrace_print ()
>  at ./backtrace_execinfo.c:47
>  #4 0x2ad882bd1180 in show_stackframe () at ./stacktrace.c:331
>  #5 
>  #6 mca_pml_ob1_recv_request_schedule_once ()
> > at ./pml_ob1_recvreq.c:983
>  #7 0x2aaab412f47a in mca_pml_ob1_recv_request_progress_rndv ()
> 
> 
> >>>
> >
from /home/mishima/opt/mpi/openmpi-2.0.0-pgi16.5/lib/openmpi/mca_pml_ob1.so
>  #8 0x2aaab412c645 in mca_pml_ob1_recv_frag_match ()
>  at ./pml_ob1_recvfrag.c:715
>  #9 0x2aaab412bba6 in mca_pml_ob1_recv_frag_callback_rndv ()
>  at ./pml_ob1_recvfrag.c:267
>  #10 0x2f2748d3 in mca_btl_vader_poll_handle_frag ()
>  at ./btl_vader_component.c:589
>  #11 0x2f274b9a in mca_btl_vader_component_progress ()
>  at

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-08-08 Thread tmishima

Then, my understanding is that you will restore the default value of
btl_openib_flags to previous one( = 310) and add a new MCA parameter to
control HCA inclusion for such a situation. The work arround so far for
openmpi-2.0.0 is setting those flags manually. Right?

Tetsuya Mishima

2016/08/09 9:56:29、"devel"さんは「Re: [OMPI devel] sm BTL performace of
the openmpi-2.0.0」で書きました
> Hmm, not good. So we have a situation where it is sometimes better to
include the HCA when it is the only rdma btl. Will have a new version up in
a bit that adds an MCA parameter to control the
> behavior. The default will be the same as 1.10.x.
>
> -Nathan
>
> > On Aug 8, 2016, at 4:51 PM, tmish...@jcity.maeda.co.jp wrote:
> >
> > Hi, unfortunately it doesn't work well. The previous one was much
> > better ...
> >
> > [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -report-bindings
> > osu_bw
> > [manage.cluster:25107] MCW rank 0 bound to socket 0[core 0[hwt 0]],
socket
> > 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
0]]:
> > [B/B/B/B/B/B][./././././.]
> > [manage.cluster:25107] MCW rank 1 bound to socket 0[core 0[hwt 0]],
socket
> > 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt
0]]:
> > [B/B/B/B/B/B][./././././.]
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 2.22
> > 2 4.53
> > 4 9.11
> > 818.02
> > 16   35.44
> > 32   70.84
> > 64  113.71
> > 128 176.74
> > 256 311.07
> > 512 529.03
> > 1024907.83
> > 2048   1597.66
> > 4096330.14
> > 8192516.49
> > 16384   780.31
> > 32768  1038.43
> > 65536  1186.36
> > 131072 1268.87
> > 262144 1222.24
> > 524288 1232.30
> > 10485761244.62
> > 20971521260.25
> > 41943041263.47
> >
> > Tetsuya
> >
> >
> > 2016/08/09 2:42:24、"devel"さんは「Re: [OMPI devel] sm BTL performace
of
> > the openmpi-2.0.0」で書きました
> >> Ok, there was a problem with the selection logic when only one rdma
> > capable btl is available. I changed the logic to always use the RDMA
btl
> > over pipelined send/recv. This works better for me on a
> >> Intel Omnipath system. Let me know if this works for you.
> >>
> >>
> >
https://github.com/hjelmn/ompi/commit/dddb865b5337213fd73d0e226b02e2f049cfab47.patch

> >
> >>
> >> -Nathan
> >>
> >> On Aug 07, 2016, at 10:00 PM, tmish...@jcity.maeda.co.jp wrote:
> >>
> >> Hi, here is the gdb output for additional information:
> >>
> >> (It might be inexact, because I built openmpi-2.0.0 without debug
option)
> >>
> >> Core was generated by `osu_bw'.
> >> Program terminated with signal 11, Segmentation fault.
> >> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
> >> (gdb) where
> >> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
> >> #1 0x0031d9008934 in _Unwind_Backtrace ()
from /lib64/libgcc_s.so.1
> >> #2 0x0037ab8e5ee8 in backtrace () from /lib64/libc.so.6
> >> #3 0x2ad882bd4345 in opal_backtrace_print ()
> >> at ./backtrace_execinfo.c:47
> >> #4 0x2ad882bd1180 in show_stackframe () at ./stacktrace.c:331
> >> #5 
> >> #6 mca_pml_ob1_recv_request_schedule_once ()
at ./pml_ob1_recvreq.c:983
> >> #7 0x2aaab412f47a in mca_pml_ob1_recv_request_progress_rndv ()
> >>
> >>
> >
from /home/mishima/opt/mpi/openmpi-2.0.0-pgi16.5/lib/openmpi/mca_pml_ob1.so
> >> #8 0x2aaab412c645 in mca_pml_ob1_recv_frag_match ()
> >> at ./pml_ob1_recvfrag.c:715
> >> #9 0x2aaab412bba6 in mca_pml_ob1_recv_frag_callback_rndv ()
> >> at ./pml_ob1_recvfrag.c:267
> >> #10 0x2f2748d3 in mca_btl_vader_poll_handle_frag ()
> >> at ./btl_vader_component.c:589
> >> #11 0x2f274b9a in mca_btl_vader_component_progress ()
> >> at ./btl_vader_component.c:231
> >> #12 0x2ad882b916fc in opal_progress () at
runtime/opal_progress.c:224
> >> #13 0x2ad8820a9aa5 in ompi_request_default_wait_all () at
> >> request/req_wait.c:77
> >> #14 0x2ad8820f10dd in PMPI_Waitall () at ./pwaitall.c:76
> >> #15 0x00401108 in main () at ./osu_bw.c:144
> >>
> >> Tetsuya
> >>
> >>
> >> 2016/08/08 12:34:57、"devel"さんは「Re: [OMPI devel] sm BTL performace
of
> >> the openmpi-2.0.0」で書きました
> >> Hi, it caused segfault as below:
> >> [manage.cluster:25436] MCW rank 0 bound to socket 0[core 0[hwt
0]],socket
> >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]],
> > socket 0[core 4[hwt 0]], socket 0[core 5[hwt
> > 0]]:[B/B/B/B/B/B][./././././.][manage.cluster:25436] MCW rank 1 bound
to
> > socket 0[core
> >> 0[hwt 0]],socket
> >> 0[core 1[hwt 0]], socket 0[core 2[hwt

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-08-08 Thread Nathan Hjelm

Hmm, not good. So we have a situation where it is sometimes better to include 
the HCA when it is the only rdma btl. Will have a new version up in a bit that 
adds an MCA parameter to control the behavior. The default will be the same as 
1.10.x.

-Nathan

> On Aug 8, 2016, at 4:51 PM, tmish...@jcity.maeda.co.jp wrote:
> 
> Hi, unfortunately it doesn't work well. The previous one was much
> better ...
> 
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -report-bindings
> osu_bw
> [manage.cluster:25107] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]:
> [B/B/B/B/B/B][./././././.]
> [manage.cluster:25107] MCW rank 1 bound to socket 0[core 0[hwt 0]], socket
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]:
> [B/B/B/B/B/B][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 2.22
> 2 4.53
> 4 9.11
> 818.02
> 16   35.44
> 32   70.84
> 64  113.71
> 128 176.74
> 256 311.07
> 512 529.03
> 1024907.83
> 2048   1597.66
> 4096330.14
> 8192516.49
> 16384   780.31
> 32768  1038.43
> 65536  1186.36
> 131072 1268.87
> 262144 1222.24
> 524288 1232.30
> 10485761244.62
> 20971521260.25
> 41943041263.47
> 
> Tetsuya
> 
> 
> 2016/08/09 2:42:24、"devel"さんは「Re: [OMPI devel] sm BTL performace of
> the openmpi-2.0.0」で書きました
>> Ok, there was a problem with the selection logic when only one rdma
> capable btl is available. I changed the logic to always use the RDMA btl
> over pipelined send/recv. This works better for me on a
>> Intel Omnipath system. Let me know if this works for you.
>> 
>> 
> https://github.com/hjelmn/ompi/commit/dddb865b5337213fd73d0e226b02e2f049cfab47.patch
> 
>> 
>> -Nathan
>> 
>> On Aug 07, 2016, at 10:00 PM, tmish...@jcity.maeda.co.jp wrote:
>> 
>> Hi, here is the gdb output for additional information:
>> 
>> (It might be inexact, because I built openmpi-2.0.0 without debug option)
>> 
>> Core was generated by `osu_bw'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>> (gdb) where
>> #0 0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
>> #1 0x0031d9008934 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
>> #2 0x0037ab8e5ee8 in backtrace () from /lib64/libc.so.6
>> #3 0x2ad882bd4345 in opal_backtrace_print ()
>> at ./backtrace_execinfo.c:47
>> #4 0x2ad882bd1180 in show_stackframe () at ./stacktrace.c:331
>> #5 
>> #6 mca_pml_ob1_recv_request_schedule_once () at ./pml_ob1_recvreq.c:983
>> #7 0x2aaab412f47a in mca_pml_ob1_recv_request_progress_rndv ()
>> 
>> 
> from /home/mishima/opt/mpi/openmpi-2.0.0-pgi16.5/lib/openmpi/mca_pml_ob1.so
>> #8 0x2aaab412c645 in mca_pml_ob1_recv_frag_match ()
>> at ./pml_ob1_recvfrag.c:715
>> #9 0x2aaab412bba6 in mca_pml_ob1_recv_frag_callback_rndv ()
>> at ./pml_ob1_recvfrag.c:267
>> #10 0x2f2748d3 in mca_btl_vader_poll_handle_frag ()
>> at ./btl_vader_component.c:589
>> #11 0x2f274b9a in mca_btl_vader_component_progress ()
>> at ./btl_vader_component.c:231
>> #12 0x2ad882b916fc in opal_progress () at runtime/opal_progress.c:224
>> #13 0x2ad8820a9aa5 in ompi_request_default_wait_all () at
>> request/req_wait.c:77
>> #14 0x2ad8820f10dd in PMPI_Waitall () at ./pwaitall.c:76
>> #15 0x00401108 in main () at ./osu_bw.c:144
>> 
>> Tetsuya
>> 
>> 
>> 2016/08/08 12:34:57、"devel"さんは「Re: [OMPI devel] sm BTL performace of
>> the openmpi-2.0.0」で書きました
>> Hi, it caused segfault as below:
>> [manage.cluster:25436] MCW rank 0 bound to socket 0[core 0[hwt 0]],socket
>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]],
> socket 0[core 4[hwt 0]], socket 0[core 5[hwt
> 0]]:[B/B/B/B/B/B][./././././.][manage.cluster:25436] MCW rank 1 bound to
> socket 0[core
>> 0[hwt 0]],socket
>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]],
> socket 0[core 4[hwt 0]], socket 0[core 5[hwt
> 0]]:[B/B/B/B/B/B][./././././.]# OSU MPI Bandwidth Test v3.1.1# Size
> Bandwidth (MB/s)1
>> 2.232 4.514 8.998 17.8316 35.1832 69.6664 109.84128 179.65256 303.52512
> 532.811024 911.742048 1605.294096 1598.738192 2135.9416384 2468.9832768
> 2818.3765536 3658.83131072 4200.50262144 4545.01524288
>> 4757.841048576 4831.75[manage:25442] *** Process received signal
> ***[manage:25442] Signal: Segmentation fault (11)[manage:25442] Signal
> code: Address not mapped (1)[manage:25442] Failing at address:

Re: [OMPI devel] PMIx Language Bindings

2016-08-08 Thread Pritchard Jr., Howard

HI Ralph,

If the java bindings are of use, I could see if my student how did a lot
of the recent work in the Open MPI java bindings would be interested.
He doesn¹t have a lot of extra cycles at the moment though.

Howard

-- 
Howard Pritchard

HPC-DES
Los Alamos National Laboratory





On 8/7/16, 3:21 PM, "devel on behalf of r...@open-mpi.org"
 wrote:

>Hi folks
>
>I¹m looking for someone(s) interested in writing some simple language
>bindings (e.g., Python, Java, Fortran) for the PMIx library (which is
>written in C). There aren¹t a lot of APIs, so I don¹t envision this as
>being a monstrous effort.
>
>Please let me know if you have any interest - and feel free to forward
>this to anyone you think might be!
>Ralph
>
>___
>devel mailing list
>devel@lists.open-mpi.org
>https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] openmpi2.0.0 - bad performance - ALIGNMENT

2016-08-08 Thread Paul Kapinos


Dear Open MPI developers,
there is already a thread about 'sm BTL performace of the openmpi-2.0.0'
https://www.open-mpi.org/community/lists/devel/2016/07/19288.php
and we also see 30% bandwidth loss, on communication *via InfiniBand*.

And we also have a clue: the IB buffers seem not to be aligned in 2.0.0 - in 
contrast to previous series (from at least 1.8.x).

That means,
- if we use a simple wrapper wrapping 'malloc' to the 32-bit-aligned-variant, we 
get the full bandwidth using the same compiled binary; and
- there is nothing to grep in 'ompi_info -all | grep memalign' in 2.0.0 while in 
1.10.3 there are 'btl_openib_memalign' and 'btl_openib_memalign_threshold' 
parameters.


=> seem the whole 'IB buffer alignment' part vanished in /2.0.0 ?

Could we get the aligned IB buffers in 2.x series back, please? It's about 30% 
of performance


Best

Paul


P.S. btl_openib_get_alignment and btl_openib_put_alignment are by default '0' - 
setting they high did not change the behaviour...



--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

Re: [OMPI devel] PMIx Language Bindings

[OMPI devel] openmpi2.0.0 - bad performance - ALIGNMENT

6 matches

Site Navigation

Mail list logo

Footer information