Re: [OMPI users] Rank specific argument to mpirun

2016-07-29 Thread Udayanga Wickramasinghe
Hi,
I think orte/ompi-mca foward number of environment variables (ie:- $OMPI_*)
to its ranks. So I believe you may use $OMPI_COMM_WORLD_LOCAL_RANK to
specifically filter out parameters within the script.

Regards
Udayanga Wickramasinghe
Research Assistant
School of Informatics and Computing | CREST
Indiana University, Bloomington





On Fri, Jul 29, 2016 at 2:50 PM, Saliya Ekanayake  wrote:

> Hi,
>
> I am invoking my Java MPI program as,
>
> mpirun opts script.sh
>
> Now, I want to pass some rank specific argument to this script. For
> example, I need to enable GC logs for one of the processes only, so I need
> to pass an argument to the JVM if that's rank0 to do GC logs.
>
> I've asked the same question a while back and Ralph suggested to use
>
> mpirun opts -np 1 java args-for-rank0 app : -np 1 java args-for-others ..
> style.
>
> It works for 1 node, but beyond that, there's something that goes wrong
> with how processes are assigned to communicators. In the code, the comm
> world is split into different communicators and this pattern breaks that
> logic.
>
> Is there an alternative approach to doing this?
>
> Thank you,
> Saliya
>
> --
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] OpenMPI non blocking I_Allreduce segfaults when using custom function..

2015-12-16 Thread Udayanga Wickramasinghe
Hi all,
I have a custom MPI_Op function which I use within a non blocking version
of all_reduce(). When executing the mpi program I am seeing a segfault
thrown from libNBC. It seems like this is a known issue in openMPI atleast
[1]. Is this somehow fixed in a later release version of openmpi ? I am
using 1.8.4.

Thanks
Udayanga

[1] http://www.open-mpi.org/community/lists/devel/2014/04/14588.php


Re: [OMPI users] OpenMPI non blocking I_Allreduce segfaults when using custom function..

2015-12-17 Thread Udayanga Wickramasinghe
I tried with 1.10.0, and is failing still. I will need to check whether it
works for later releases.

Thanks
Udayanga


On Wed, Dec 16, 2015 at 5:24 PM, Nathan Hjelm  wrote:

>
> I think this is fixed in the 1.10 series. We will not be making any more
> updates to the 1.8 series so you will need to update to 1.10 to get the
> fix.
>
> -Nathan
>
> On Wed, Dec 16, 2015 at 02:48:45PM -0500, Udayanga Wickramasinghe wrote:
> >Hi all,
> >I have a custom MPI_Op function which I use within a non blocking
> version
> >of all_reduce(). When executing the mpi program I am seeing a segfault
> >thrown from libNBC. It seems like this is a known issue in openMPI
> atleast
> >[1]. Is this somehow fixed in a later release version of openmpi ? I
> am
> >using 1.8.4.
> >Thanks
> >Udayanga
> >[1] http://www.open-mpi.org/community/lists/devel/2014/04/14588.php
>
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/12/28167.php
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/12/28170.php
>


[OMPI users] Using OpenMPI Thread Multiple mode

2016-01-19 Thread Udayanga Wickramasinghe
Hi devs,
I am using THREAD_MULTIPLE in openmpi version 1.8.4. However I occasionally
see following warning and my application gets hanged up intermittently.
Does this mean thread multiple mode is not supported in 1.8.4 ? Or does
openmpi has a version that fully supports this ?

opal_libevent2021_event_base_loop: reentrant invocation;  Only one
event_base_loop can run on each event_base at once.

Thanks and Regards
Udayanga


[OMPI users] A strange warning on Cray XC with Opemmpi-4.00

2019-01-07 Thread Udayanga Wickramasinghe
Hi,

I upgraded my open-mpi version to 4.00 on a Cray Aries cluster (GNI/uGNI).
Every time I run mpi, I get the following warning. Is there any way to
suppress this message? I am not seeing this in 3.1.3 version of open-mpi.
Any idea why openfabrics device is getting initialized when cray GNI
transport is actively used? (i.e. looks like it is related to UCX, but I
assume this could potentially be a configuration issue even though GNI
transport/btl seems to get detected correctly and works by default without
any explicit --mca parameters)



By default, for Open MPI 4.0 and later, infiniband ports on a device

are not used by default.  The intent is to use UCX for these devices.

You can override this policy by setting the btl_openib_allow_ib MCA
parameter

to true.


  Local host:  nid00301

  Local adapter:   ibgni

  Local port:  1


--

--

WARNING: There was an error initializing an OpenFabrics device.


  Local host:   nid00300

  Local device: ibgni


... [program output]

... [program output]

...


[login1:02032] 3 more processes have sent help message
help-mpi-btl-openib.txt / ib port not selected

[login1:02032] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
help / error messages

[login1:02032] 3 more processes have sent help message
help-mpi-btl-openib.txt / error in device init


Thanks,
Udayanga
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] A strange warning on Cray XC with Opemmpi-4.00

2019-01-07 Thread Udayanga Wickramasinghe
Hi,

I upgraded my open-mpi version to 4.00 on a Cray Aries cluster (GNI/uGNI).
Every time I run mpi, I get the following warning. Is there any way to
suppress this message? I am not seeing this in 3.1.3 version of open-mpi.
Any idea why openfabrics device is getting initialized when cray GNI
transport is actively used? (i.e. looks like it is related to UCX, but I
assume this could potentially be a configuration issue even though GNI
transport/btl seems to get detected correctly and works by default without
any explicit --mca parameters)



By default, for Open MPI 4.0 and later, infiniband ports on a device

are not used by default.  The intent is to use UCX for these devices.

You can override this policy by setting the btl_openib_allow_ib MCA
parameter

to true.


  Local host:  nid00301

  Local adapter:   ibgni

  Local port:  1


--

--

WARNING: There was an error initializing an OpenFabrics device.


  Local host:   nid00300

  Local device: ibgni


... [program output]

... [program output]

...


[login1:02032] 3 more processes have sent help message
help-mpi-btl-openib.txt / ib port not selected

[login1:02032] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
help / error messages

[login1:02032] 3 more processes have sent help message
help-mpi-btl-openib.txt / error in device init


Thanks,
Udayanga
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A strange warning on Cray XC with Opemmpi-4.00

2019-01-07 Thread Udayanga Wickramasinghe
Hi Salim,
Thank you. Yeah, I noticed warnings would vanish by turning on
btl_openib_allow_ib
-- but since this is quite annoying I am wondering if there is any other
way to suppress this by configuration.

Best,
Udayanga

On Mon, Jan 7, 2019 at 7:33 AM Salim Jamal-Eddine <
salim.jamal-edd...@lau.edu.lb> wrote:

> Hi Udayanga,
>
>
>
> I had the same issue, the default behavior of OpenMpi4.0.0 is to use UCX..
> add “–mca btl_openib_allow_ib 1” and everything should be fine.
>
>
>
> Regards,
>
>
>
>
>
> Salim Jamal-Eddine
>
> Lead Engineering Labs Supervisor
> Industrial & Mechanical Engineering Department, Byblos Campus
> School of Engineering
> <http://soe.lau.edu.lb/?utm_source=lau_email&utm_medium=email&utm_campaign=EmailSig2018>
>
> Office: +961 1 786456 ext. 2899
>
> f
> <https://www.facebook.com/pages/LAU-School-of-Engineering/271900902840495>
>
> SOE
>
>
>
> [image: Lebanese American University]
> <http://www.lau.edu.lb/?utm_source=lau_email&utm_medium=email&utm_campaign=EmailSig2018>
>
> Beirut
> <http://www.lau.edu.lb/about/contact/?utm_source=lau_email&utm_medium=email&utm_campaign=EmailSig2018>
>  | Byblos
> <http://www.lau.edu.lb/about/contact/?utm_source=lau_email&utm_medium=email&utm_campaign=EmailSig2018>
>  | New York
> <http://www.lau.edu.lb/locations/nyc/?utm_source=lau_email&utm_medium=email&utm_campaign=EmailSig2018>
>
> f <https://www.facebook.com/LebaneseAmericanUniversity>
>
> in <https://www.linkedin.com/edu/school?id=14810>
>
> t <https://twitter.com/lebamuniv>
>
> ig <https://instagram.com/lebamericanuni>
> --
>
>
>
> *From:* users  *On Behalf Of *Udayanga
> Wickramasinghe
> *Sent:* Monday, January 07, 2019 1:03 PM
> *To:* Open MPI Users 
> *Cc:* Open MPI Users 
> *Subject:* [OMPI users] A strange warning on Cray XC with Opemmpi-4.00
>
>
>
> Hi,
>
> I upgraded my open-mpi version to 4.00 on a Cray Aries cluster (GNI/uGNI).
> Every time I run mpi, I get the following warning. Is there any way to
> suppress this message? I am not seeing this in 3.1.3 version of open-mpi.
> Any idea why openfabrics device is getting initialized when cray GNI
> transport is actively used? (i.e. looks like it is related to UCX, but I
> assume this could potentially be a configuration issue even though GNI
> transport/btl seems to get detected correctly and works by default without
> any explicit --mca parameters)
>
>
>
>
>
> By default, for Open MPI 4.0 and later, infiniband ports on a device
>
> are not used by default.  The intent is to use UCX for these devices.
>
> You can override this policy by setting the btl_openib_allow_ib MCA
> parameter
>
> to true.
>
>
>
>   Local host:  nid00301
>
>   Local adapter:   ibgni
>
>   Local port:  1
>
>
>
> --
>
> --
>
> WARNING: There was an error initializing an OpenFabrics device.
>
>
>
>   Local host:   nid00300
>
>   Local device: ibgni
>
>
>
> ... [program output]
>
> ... [program output]
>
> ...
>
>
>
> [login1:02032] 3 more processes have sent help message
> help-mpi-btl-openib.txt / ib port not selected
>
> [login1:02032] Set MCA parameter "orte_base_help_aggregate" to 0 to see
> all help / error messages
>
> [login1:02032] 3 more processes have sent help message
> help-mpi-btl-openib.txt / error in device init
>
>
>
> Thanks,
> Udayanga
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A strange warning on Cray XC with Opemmpi-4.00

2019-01-07 Thread Udayanga Wickramasinghe
Thanks, that actually worked!

Best,
Udayanga


On Mon, Jan 7, 2019 at 8:23 AM Bennet Fauber  wrote:

> Used to be that you could put default MCA settings in
> OMPI_ROOT/etc/openmpi-mca-params.conf.
>
> btl_openib_allow_ib=1
>
> You could try that.
>
> -- bennet
>
>
>
> On Mon, Jan 7, 2019 at 8:16 AM Udayanga Wickramasinghe 
> wrote:
> >
> > Hi Salim,
> > Thank you. Yeah, I noticed warnings would vanish by turning on
> btl_openib_allow_ib -- but since this is quite annoying I am wondering if
> there is any other way to suppress this by configuration.
> >
> > Best,
> > Udayanga
> >
> > On Mon, Jan 7, 2019 at 7:33 AM Salim Jamal-Eddine <
> salim.jamal-edd...@lau.edu.lb> wrote:
> >>
> >> Hi Udayanga,
> >>
> >>
> >>
> >> I had the same issue, the default behavior of OpenMpi4.0.0 is to use
> UCX.. add “–mca btl_openib_allow_ib 1” and everything should be fine.
> >>
> >>
> >>
> >> Regards,
> >>
> >>
> >>
> >>
> >>
> >> Salim Jamal-Eddine
> >>
> >> Lead Engineering Labs Supervisor
> >> Industrial & Mechanical Engineering Department, Byblos Campus
> >> School of Engineering
> >>
> >> Office: +961 1 786456 ext. 2899
> >>
> >> f
> >>
> >> SOE
> >>
> >>
> >>
> >> Beirut | Byblos | New York
> >>
> >> f
> >>
> >> in
> >>
> >> t
> >>
> >> ig
> >>
> >> 
> >>
> >>
> >>
> >> From: users  On Behalf Of Udayanga
> Wickramasinghe
> >> Sent: Monday, January 07, 2019 1:03 PM
> >> To: Open MPI Users 
> >> Cc: Open MPI Users 
> >> Subject: [OMPI users] A strange warning on Cray XC with Opemmpi-4.00
> >>
> >>
> >>
> >> Hi,
> >>
> >> I upgraded my open-mpi version to 4.00 on a Cray Aries cluster
> (GNI/uGNI). Every time I run mpi, I get the following warning. Is there any
> way to suppress this message? I am not seeing this in 3.1.3 version of
> open-mpi. Any idea why openfabrics device is getting initialized when cray
> GNI transport is actively used? (i.e. looks like it is related to UCX, but
> I assume this could potentially be a configuration issue even though GNI
> transport/btl seems to get detected correctly and works by default without
> any explicit --mca parameters)
> >>
> >>
> >>
> >>
> >>
> >> By default, for Open MPI 4.0 and later, infiniband ports on a device
> >>
> >> are not used by default.  The intent is to use UCX for these devices.
> >>
> >> You can override this policy by setting the btl_openib_allow_ib MCA
> parameter
> >>
> >> to true.
> >>
> >>
> >>
> >>   Local host:  nid00301
> >>
> >>   Local adapter:   ibgni
> >>
> >>   Local port:  1
> >>
> >>
> >>
> >>
> --
> >>
> >>
> --
> >>
> >> WARNING: There was an error initializing an OpenFabrics device.
> >>
> >>
> >>
> >>   Local host:   nid00300
> >>
> >>   Local device: ibgni
> >>
> >>
> >>
> >> ... [program output]
> >>
> >> ... [program output]
> >>
> >> ...
> >>
> >>
> >>
> >> [login1:02032] 3 more processes have sent help message
> help-mpi-btl-openib.txt / ib port not selected
> >>
> >> [login1:02032] Set MCA parameter "orte_base_help_aggregate" to 0 to see
> all help / error messages
> >>
> >> [login1:02032] 3 more processes have sent help message
> help-mpi-btl-openib.txt / error in device init
> >>
> >>
> >>
> >> Thanks,
> >> Udayanga
> >>
> >>
> >>
> >> ___
> >> users mailing list
> >> users@lists.open-mpi.org
> >> https://lists.open-mpi.org/mailman/listinfo/users
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-08 Thread Udayanga Wickramasinghe
Hi,
I am running into an issue in open-mpi where it crashes abruptly
during MPI_WIN_ATTACH.

[nid00307:25463] *** An error occurred in MPI_Win_attach

[nid00307:25463] *** reported by process [140736284524545,140728898420736]

[nid00307:25463] *** on win rdma window 3

[nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment

[nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will now
abort,

[nid00307:25463] ***and potentially your MPI job)


Looking more into this issue, it seems like open-mpi has a restriction on
the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't
spec doesn't say a lot about this scenario --"The argument win must be a
window that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but
nonoverlapping) memory regions may be attached to the same window")

To workaround this, I have temporarily modified the variable
mca_osc_rdma_component.max_attach. Is there any way to configure this in
open-mpi?

Thanks
Udayanga
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-08 Thread Udayanga Wickramasinghe
Sorry should be corrected as MPI3.0 spec [1]

[1] https://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf ; page=443

Best Regards,
Udayanga

On Tue, Jan 8, 2019 at 11:36 PM Udayanga Wickramasinghe 
wrote:

> Hi,
> I am running into an issue in open-mpi where it crashes abruptly
> during MPI_WIN_ATTACH.
>
> [nid00307:25463] *** An error occurred in MPI_Win_attach
>
> [nid00307:25463] *** reported by process [140736284524545,140728898420736]
>
> [nid00307:25463] *** on win rdma window 3
>
> [nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment
>
> [nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will now
> abort,
>
> [nid00307:25463] ***and potentially your MPI job)
>
>
> Looking more into this issue, it seems like open-mpi has a restriction on
> the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't
> spec doesn't say a lot about this scenario --"The argument win must be a
> window that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but
> nonoverlapping) memory regions may be attached to the same window")
>
> To workaround this, I have temporarily modified the variable
> mca_osc_rdma_component.max_attach. Is there any way to configure this in
> open-mpi?
>
> Thanks
> Udayanga
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-09 Thread Udayanga Wickramasinghe
Thanks. Yes, I am aware of that however, I currently have a requirement to
increase the default.

Best,
Udayanga

On Wed, Jan 9, 2019 at 9:10 AM Nathan Hjelm via users <
users@lists.open-mpi.org> wrote:

> If you need to support more attachments you can set the value of that
> variable either by setting:
>
> Environment:
>
> OMPI_MCA_osc_rdma_max_attach
>
>
> mpirun command line:
>
> —mca osc_rdma_max_attach
>
>
> Keep in mind that each attachment may use an underlying hardware resource
> that may be easy to exhaust (hence the low default limit). It is
> recommended to keep the total number as small as possible.
>
> -Nathan
>
> > On Jan 8, 2019, at 9:36 PM, Udayanga Wickramasinghe 
> wrote:
> >
> > Hi,
> > I am running into an issue in open-mpi where it crashes abruptly during
> MPI_WIN_ATTACH.
> > [nid00307:25463] *** An error occurred in MPI_Win_attach
> > [nid00307:25463] *** reported by process
> [140736284524545,140728898420736]
> > [nid00307:25463] *** on win rdma window 3
> > [nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment
> > [nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will
> now abort,
> > [nid00307:25463] ***and potentially your MPI job)
> >
> > Looking more into this issue, it seems like open-mpi has a restriction
> on the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't
> spec doesn't say a lot about this scenario --"The argument win must be a
> window that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but
> nonoverlapping) memory regions may be attached to the same window")
> >
> > To workaround this, I have temporarily modified the variable
> mca_osc_rdma_component.max_attach. Is there any way to configure this in
> open-mpi?
> >
> > Thanks
> > Udayanga
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-09 Thread Udayanga Wickramasinghe
Thanks, I think that will be very useful.

Best,
Udayanga


On Wed, Jan 9, 2019 at 1:39 PM Jeff Squyres (jsquyres) via users <
users@lists.open-mpi.org> wrote:

> You can set this MCA var on a site-wide basis in a file:
>
> https://www.open-mpi.org/faq/?category=tuning#setting-mca-params
>
>
>
> > On Jan 9, 2019, at 1:18 PM, Udayanga Wickramasinghe 
> wrote:
> >
> > Thanks. Yes, I am aware of that however, I currently have a requirement
> to increase the default.
> >
> > Best,
> > Udayanga
> >
> > On Wed, Jan 9, 2019 at 9:10 AM Nathan Hjelm via users <
> users@lists.open-mpi.org> wrote:
> > If you need to support more attachments you can set the value of that
> variable either by setting:
> >
> > Environment:
> >
> > OMPI_MCA_osc_rdma_max_attach
> >
> >
> > mpirun command line:
> >
> > —mca osc_rdma_max_attach
> >
> >
> > Keep in mind that each attachment may use an underlying hardware
> resource that may be easy to exhaust (hence the low default limit). It is
> recommended to keep the total number as small as possible.
> >
> > -Nathan
> >
> > > On Jan 8, 2019, at 9:36 PM, Udayanga Wickramasinghe 
> wrote:
> > >
> > > Hi,
> > > I am running into an issue in open-mpi where it crashes abruptly
> during MPI_WIN_ATTACH.
> > > [nid00307:25463] *** An error occurred in MPI_Win_attach
> > > [nid00307:25463] *** reported by process
> [140736284524545,140728898420736]
> > > [nid00307:25463] *** on win rdma window 3
> > > [nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment
> > > [nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will
> now abort,
> > > [nid00307:25463] ***and potentially your MPI job)
> > >
> > > Looking more into this issue, it seems like open-mpi has a restriction
> on the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't
> spec doesn't say a lot about this scenario --"The argument win must be a
> window that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but
> nonoverlapping) memory regions may be attached to the same window")
> > >
> > > To workaround this, I have temporarily modified the variable
> mca_osc_rdma_component.max_attach. Is there any way to configure this in
> open-mpi?
> > >
> > > Thanks
> > > Udayanga
> > > ___
> > > users mailing list
> > > users@lists.open-mpi.org
> > > https://lists.open-mpi.org/mailman/listinfo/users
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-10 Thread Udayanga Wickramasinghe
I actually have a use case where my library will attach many
non-overlapping vm segments on demand to a single dynamic OMPI_Win_t
object. With the current static limit, I would either have to increase it
optimistically before startup or maintain a pool of dynamic win objects.
However, other MPI implementations (Cray-MPI2.2,  mvapich2) I tested do not
appear to have this constraint.

Regards,
Udayanga

On Thu, Jan 10, 2019 at 4:09 AM Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Jeff,
>
> At first glance, a comment in the code suggests the rationale is to
> minimize the number of allocations and hence the time spent registering the
> memory.
>
> Cheers,
>
> Gilles
>
> Jeff Hammond  wrote:
> Why is this allocated statically? I dont understand the difficulty of a
> dynamically allocates and thus unrestricted implementation. Is there some
> performance advantage to a bounded static allocation?  Or is it that you
> use O(n) lookups and need to keep n small to avoid exposing that to users?
>
> I have usage models with thousands of attached segments, hence need to
> understand how bad this will be with Open-MPI (yes I can amortize overhead
> but it’s a pain).
>
> Thanks,
>
> Jeff
>
> On Wed, Jan 9, 2019 at 8:12 AM Nathan Hjelm via users <
> users@lists.open-mpi.org> wrote:
>
>> If you need to support more attachments you can set the value of that
>> variable either by setting:
>>
>> Environment:
>>
>> OMPI_MCA_osc_rdma_max_attach
>>
>>
>> mpirun command line:
>>
>> —mca osc_rdma_max_attach
>>
>>
>> Keep in mind that each attachment may use an underlying hardware resource
>> that may be easy to exhaust (hence the low default limit). It is
>> recommended to keep the total number as small as possible.
>>
>> -Nathan
>>
>> > On Jan 8, 2019, at 9:36 PM, Udayanga Wickramasinghe 
>> wrote:
>> >
>> > Hi,
>> > I am running into an issue in open-mpi where it crashes abruptly during
>> MPI_WIN_ATTACH.
>> > [nid00307:25463] *** An error occurred in MPI_Win_attach
>> > [nid00307:25463] *** reported by process
>> [140736284524545,140728898420736]
>> > [nid00307:25463] *** on win rdma window 3
>> > [nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment
>> > [nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will
>> now abort,
>> > [nid00307:25463] ***and potentially your MPI job)
>> >
>> > Looking more into this issue, it seems like open-mpi has a restriction
>> on the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't
>> spec doesn't say a lot about this scenario --"The argument win must be a
>> window that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but
>> nonoverlapping) memory regions may be attached to the same window")
>> >
>> > To workaround this, I have temporarily modified the variable
>> mca_osc_rdma_component.max_attach. Is there any way to configure this in
>> open-mpi?
>> >
>> > Thanks
>> > Udayanga
>> > ___
>> > users mailing list
>> > users@lists.open-mpi.org
>> > https://lists.open-mpi.org/mailman/listinfo/users
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users