Re: [OMPI devel] OPAL_PMIX_NODEID is not set by orted

[r...@open-mpi.org]

I’m working on providing the info, guys - just sitting in a branch right now. 
Too many meetings...sigh.


> On Aug 11, 2016, at 10:09 AM, George Bosilca  wrote:
> 
> I just pushed a solution to this problem in 8d0baf140f. If we are unable to 
> extract the expected information from the RTE, we simply build a 
> non-reordered communicator and gracefully return.
> 
> That being said, not being able to correctly retrieve OPAL_PMIX_NODEID has 
> the potential to drastically decrease the performance as no specialized 
> hierarchies can be built without the RTE information.
> 
>   George.
> 
> 
> On Wed, Aug 10, 2016 at 3:57 AM, Gilles Gouaillardet  > wrote:
> Ralph,
> 
> 
> i noticed dist-graph/distgraph_test_4 from the ibm test suite fails when 
> using a hostfile and running no task on the host running mpirun.
> 
> n0$ mpirun --host n1:1,n2:1 -np 2 ./dist-graph/distgraph_test_4
> 
> 
> the root cause is OPAL_PMIX_NODEID is correctly set ( 0, 1, 2) by mpirun, but 
> for some reasons, orted sets it to -1 everywhere.
> 
> an indirect consequence is a crash of the test (it believes tasks run on zero 
> distinct nodes instead of 2)
> 
> 
> this occurs only master, and v2.x is fine.
> 
> 
> Could you please have a look ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] OPAL_PMIX_NODEID is not set by orted

[r...@open-mpi.org]

Fixed in https://github.com/open-mpi/ompi/pull/1959


> On Aug 11, 2016, at 6:23 PM, Gilles Gouaillardet  wrote:
> 
> Thanks George,
> 
> 
> fwiw, note the current behavior is a bit more "twisted" than that.
> 
> OPAL_MODEX_RECV_VALUE() returns successfully (e.g. err == OPAL_SUCCESS) but 
> the OPAL_PMIX_NODEID (e.g. val) value is -1.
> 
> that means orted did "push" OPAL_PMIX_NODEID, but with an unitialized value 
> of -1 (this is set in the constructor).
> 
> fortunatly, you used the same -1 special value if OPAL_MODEX_RECV_VALUE() had 
> failed (e.g. OPAL_ERR_NOT_FOUND),
> 
> so bottom line, your commit does fix the crash.
> 
> Cheers,
> 
> Gilles
> 
> On 8/12/2016 2:09 AM, George Bosilca wrote:
>> I just pushed a solution to this problem in 8d0baf140f. If we are unable to 
>> extract the expected information from the RTE, we simply build a 
>> non-reordered communicator and gracefully return.
>> 
>> That being said, not being able to correctly retrieve OPAL_PMIX_NODEID has 
>> the potential to drastically decrease the performance as no specialized 
>> hierarchies can be built without the RTE information.
>> 
>>   George.
>> 
>> 
>> On Wed, Aug 10, 2016 at 3:57 AM, Gilles Gouaillardet > > wrote:
>> Ralph,
>> 
>> 
>> i noticed dist-graph/distgraph_test_4 from the ibm test suite fails when 
>> using a hostfile and running no task on the host running mpirun.
>> 
>> n0$ mpirun --host n1:1,n2:1 -np 2 ./dist-graph/distgraph_test_4
>> 
>> 
>> the root cause is OPAL_PMIX_NODEID is correctly set ( 0, 1, 2) by mpirun, 
>> but for some reasons, orted sets it to -1 everywhere.
>> 
>> an indirect consequence is a crash of the test (it believes tasks run on 
>> zero distinct nodes instead of 2)
>> 
>> 
>> this occurs only master, and v2.x is fine.
>> 
>> 
>> Could you please have a look ?
>> 
>> 
>> Cheers,
>> 
>> 
>> Gilles
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>> 
>> 
>> 
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] OMPI v1.10.6rc1 ready for test

[r...@open-mpi.org]

Usual place: https://www.open-mpi.org/software/ompi/v1.10/ 


Scheduled release: Fri Feb 3rd

1.10.6
--
- Fix bug in timer code that caused problems at optimization settings
  greater than 2
- OSHMEM: make mmap allocator the default instead of sysv or verbs
- Support MPI_Dims_create with dimension zero
- Update USNIC support
- Prevent 64-bit overflow on timer counter
- Add support for forwarding signals
- Fix bug that caused truncated messages on large sends over TCP BTL
- Fix potential infinite loop when printing a stacktrace

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Reminder: assign as well as request review

[r...@open-mpi.org]

Thanks Paul - that does indeed help!

> On Jan 27, 2017, at 12:26 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
> 
> Ralph,
> 
> It looks like GitHub *might* have rolled out the solution to your problem 
> just this week:
>
> https://github.com/blog/2306-filter-pull-request-reviews-and-review-requests 
> <https://github.com/blog/2306-filter-pull-request-reviews-and-review-requests>
> 
> This appears to include an "Awaiting review from you" filter.
> Not quite a dashboard or notification, but at least a way to make the query.
> 
> -Paul
> 
> 
> On Fri, Jan 27, 2017 at 7:46 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Hey folks
> 
> Just a reminder. If you request a review from someone, GitHub doesn’t show 
> that person’s icon when looking at the list of PRs. It only shows their icon 
> and marks the PR with their ID if you actually “assign” it to that person. 
> Thus, just requesting a review without assigning the PR to someone makes it 
> impossible for them to see which PRs are awaiting their attention.
> 
> Speaking personally, I have no idea which PRs are awaiting my attention 
> unless you assign them to me. So please remember to do so.
> 
> Thanks
> Ralph
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
> 
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov 
> <mailto:phhargr...@lbl.gov>
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Reminder: assign as well as request review

[r...@open-mpi.org]

Hey folks

Just a reminder. If you request a review from someone, GitHub doesn’t show that 
person’s icon when looking at the list of PRs. It only shows their icon and 
marks the PR with their ID if you actually “assign” it to that person. Thus, 
just requesting a review without assigning the PR to someone makes it 
impossible for them to see which PRs are awaiting their attention.

Speaking personally, I have no idea which PRs are awaiting my attention unless 
you assign them to me. So please remember to do so.

Thanks
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Problem on master

[r...@open-mpi.org]

Hello all

There is a known issue on master that we are attempting to debug. Sadly, it is 
one that only shows on multi-node operations, and the signature varies based on 
your environment. We hope to have this resolved soon (and no, it doesn’t appear 
to be due to any one specific commit).

In the interim, setting the MCA param routed=direct appears to provide a 
workaround - at least, it is working for me, and hopefully will work for you too

Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] define a new ENV variable in etc/openmpi-mca-params.conf

[r...@open-mpi.org]

I think Jeff got lost in the weeds here. If you define a new MCA param in the 
default param file, we will automatically pick it up and it will be in the 
environment of your application. You don’t need to do anything. However, you 
checked for the wrong envar. Anything you provide is going to have an 
“OMPI_MCA_” attached to the front of it. So for your “apath” example, the envar 
will be

OMPI_MCA_apath

HTH
Ralph

> On Feb 24, 2017, at 7:25 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> On Feb 24, 2017, at 10:11 AM, Dahai Guo  wrote:
>> 
>> oops, I should use the word "MCA parameters". If I define a MCA parameter 
>> (apath, for example) in etc/openmpi-mca-params.conf, how can function 
>> setup_fork in ompi/orte/mca/schizo/ompi/schizo_ompi.c get its value?  I 
>> tried to call getenv("apath") there, which return null. 
> 
> Settings MCA parameters does not necessarily set environment variables.  More 
> below.
> 
>> However, If I defined it in the cmd line, mpirun -mca apath=sth .., then I 
>> could get it in setup_fork.
> 
> There are multiple mechanisms in which MCA params are passed to the back-end 
> MPI processes.
> 
> 1. If they are set via "--mca a b" on the mpirun command line, they are 
> passed back to the launching orteds as part of the launch schema for the job. 
>  I.e., the MCA param names and values are bundled up and sent in the network 
> message to each of the orteds that are involved in the launching of the 
> target processes.  IIRC, the launching orteds will fork, setenv a 
> corresponding environment variable for each of the MCA param values in the 
> launch schema, and then exec the MPI process.  The MPI process then picks up 
> those MCA param values from the environment variables.
> 
> 2. If they are set via the openmpi-mca-params.conf file, they are not sent 
> via network message in the launch schema.  Instead, each Open MPI process 
> (including MPI processes and orteds) reads that file upon startup to get the 
> values (more specifically: all Open MPI processes *always* read this file 
> before processing whatever MCA param values are sent in the launch schema).  
> A consequence: if openmpi-mca-params.conf is not on a network filesystem (and 
> visible to all Open MPI processes on all servers), you need to copy the 
> change you made to the one openmpi-mca-params.conf file to all copies of 
> openmpi-mca-params.conf.
> 
> -
> 
> What are you trying to do?
> 
> If you're working in an MPI application, you can use the MPI_T API to 
> retrieve MCA parameter names and their current values (regardless of whether 
> mechanism #1 or #2 -- or another mechanism -- is used).
> 
> If you're working in the Open MPI code base, you should be using the internal 
> opal_mca_var API to retrieve and/or set MCA param values.  See 
> https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_var.h for 
> all the functions available in this API.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Q: Using a hostfile in managed environment?

[r...@open-mpi.org]

> On Feb 24, 2017, at 11:57 AM, Thomas Naughton  wrote:
> 
> Hi,
> 
> We're trying to track down some curious behavior and decided to take a step
> back and check a base assumption.
> 
> When running within a managed environment (job allocation):
> 
>Q: Should you be able to use `--hostfile` or `--host` options to
>   operate on a subset of the resources in the allocation?
>   (Example: within 4 node SLURM allocation, run on just 2 nodes in
>allocation.)

Yes - those options are used to “filter” the allocation prior to launch

> 
>Q: Additionally, should this be the same when launching the DVM in
>   order to run on a subset of resources using subsequent
>   'mpirun --hnp ...' commands?
>   (Only 'orte-dvm' would need to have `--hostfile` or `--host` args.)

Yes - only the DVM needs to know the filter. When operating with a DVM, “mpirun 
--hnp...” only packages up the cmd line and sends it to the DVM. All the 
mapping occurs in orte-dvm.

> 
> There are a variety of interactions with ess/ras/rmaps and the resource
> manager, but the thought was that you "should" be able to use a hostfile to
> operate on a subset of the allocation. Is that a flawed assumption?
> 
> Thanks,
> --tjn
> 
> _
>  Thomas Naughton  naught...@ornl.gov
>  Research Associate   (865) 576-4184
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] OMPI v1.10.6

[r...@open-mpi.org]

Will someone be submitting that PR soon?

> On Jan 18, 2017, at 10:09 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
> 
> https://github.com/open-mpi/ompi/issues/2750 
> <https://github.com/open-mpi/ompi/issues/2750>
> 
>   George.
> 
> 
> 
> On Wed, Jan 18, 2017 at 12:57 PM, r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
> wrote:
> Last call for v1.10.6 changes - we still have a few pending for review, but 
> none marked as critical. If you want them included, please push for a review 
> _now_
> 
> Thanks
> Ralph
> 
> > On Jan 12, 2017, at 1:54 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> > wrote:
> >
> > Hi folks
> >
> > It looks like we may have motivation to release 1.10.6 in the near future. 
> > Please check to see if you have anything that should be included, or is 
> > pending review.
> >
> > Thanks
> > Ralph
> >
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] OMPI v1.10.6

[r...@open-mpi.org]

Last call for v1.10.6 changes - we still have a few pending for review, but 
none marked as critical. If you want them included, please push for a review 
_now_

Thanks
Ralph

> On Jan 12, 2017, at 1:54 PM, r...@open-mpi.org wrote:
> 
> Hi folks
> 
> It looks like we may have motivation to release 1.10.6 in the near future. 
> Please check to see if you have anything that should be included, or is 
> pending review.
> 
> Thanks
> Ralph
> 

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

I can not provide the user report as it is a proprietary problem. However, it 
consists of a large loop of calls to MPI_Bcast that crashes due to unexpected 
messages. We have been looking at instituting flow control, but that has way 
too widespread an impact. The coll/sync component would be a simple solution.

I honestly don’t believe the issue I was resolving was due to a bug - it was a 
simple problem of one proc running slow and creating an overload of unexpected 
messages that eventually consumed too much memory. Rather, I think you solved a 
different problem - by the time you arrived at LANL, the app I was working with 
had already modified their code to no longer create the problem (essentially 
refactoring the algorithm to avoid the massive loop over allreduce).

I have no issue supporting it as it takes near-zero effort to maintain, and 
this is a fairly common problem with legacy codes that don’t want to refactor 
their algorithms.


> On Aug 19, 2016, at 8:48 PM, Nathan Hjelm <hje...@me.com> wrote:
> 
>> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org wrote:
>> 
>> Hi folks
>> 
>> I had a question arise regarding a problem being seen by an OMPI user - has 
>> to do with the old bugaboo I originally dealt with back in my LANL days. The 
>> problem is with an app that repeatedly hammers on a collective, and gets 
>> overwhelmed by unexpected messages when one of the procs falls behind.
> 
> I did some investigation on roadrunner several years ago and determined that 
> the user code issue coll/sync was attempting to fix was due to a bug in 
> ob1/cksum (really can’t remember). coll/sync was simply masking a live-lock 
> problem. I committed a workaround for the bug in r26575 
> (https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35)
>  and tested it with the user code. After this change the user code ran fine 
> without coll/sync. Since lanl no longer had any users of coll/sync we stopped 
> supporting it.
> 
>> I solved this back then by introducing the “sync” component in 
>> ompi/mca/coll, which injected a barrier operation every N collectives. You 
>> could even “tune” it by doing the injection for only specific collectives.
>> 
>> However, I can no longer find that component in the code base - I find it in 
>> the 1.6 series, but someone removed it during the 1.7 series.
>> 
>> Can someone tell me why this was done??? Is there any reason not to bring it 
>> back? It solves a very real, not uncommon, problem.
>> Ralph
> 
> This was discussed during one (or several) tel-cons years ago. We agreed to 
> kill it and bring it back if there is 1) a use case, and 2) someone is 
> willing to support it. See 
> https://github.com/open-mpi/ompi/commit/5451ee46bd6fcdec002b333474dec919475d2d62
>  .
> 
> Can you link the user email?
> 
> -Nathan
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Warnings on master

[r...@open-mpi.org]

We seem to have gotten into a state again of generating a ton of warnings on 
master - can folks take a look at these and clean them up?

opal_datatype_pack.c: In function ‘pack_predefined_heterogeneous’:
opal_datatype_pack.c:421:24: warning: variable ‘_l_blength’ set but not used 
[-Wunused-but-set-variable]
 size_t _r_blength, _l_blength;
^~
opal_datatype_pack.c: In function ‘opal_pack_general_checksum’:
opal_datatype_pack.c:467:9: warning: variable ‘type’ set but not used 
[-Wunused-but-set-variable]
 int type;
 ^~~~
opal_datatype_pack.c: In function ‘pack_predefined_heterogeneous’:
opal_datatype_pack.c:421:24: warning: variable ‘_l_blength’ set but not used 
[-Wunused-but-set-variable]
 size_t _r_blength, _l_blength;
^~
opal_datatype_pack.c: In function ‘opal_pack_general’:
opal_datatype_pack.c:467:9: warning: variable ‘type’ set but not used 
[-Wunused-but-set-variable]
 int type;
 ^~~~


topology.c: In function ‘hwloc__check_children’:
topology.c:3126:9: warning: variable ‘prev_firstchild’ set but not used 
[-Wunused-but-set-variable]
 int prev_firstchild = -1; /* -1 works fine with first comparisons below */
 ^~~
At top level:
topology.c:637:31: warning: ‘obj_order_type’ defined but not used 
[-Wunused-const-variable=]
 static const hwloc_obj_type_t obj_order_type[] = {
   ^~
topology-xml.c: In function ‘opal_hwloc1113_hwloc_export_obj_userdata’:
topology-xml.c:1632:5: warning: ‘realname’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 hwloc__export_obj_userdata(state, encoded, realname, length, buffer, 
encoded_length);
 
^~~~
topology-xml.c:1632:5: warning: ‘encoded_length’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
topology-xml.c:1632:5: warning: ‘encoded’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]


class/opal_free_list.c: In function ‘opal_free_list_grow_st’:
class/opal_free_list.c:217:21: warning: ‘align’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
 payload_ptr = (unsigned char *) 
flist->fl_mpool->mpool_alloc(flist->fl_mpool, buffer_size, align, 0);
 
^~~~
class/opal_free_list.c:217:21: warning: ‘buffer_size’ may be used uninitialized 
in this function [-Wmaybe-uninitialized]


server/pmix_server.c: In function ‘PMIx_server_init’:
server/pmix_server.c:438:83: warning: ‘tl’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 pmix_show_help("help-pmix-server.txt", "listener-failed-start", 
true, tl->address.sun_path);

   ^~


rcache_grdma_module.c: In function ‘mca_rcache_grdma_check_cached’:
rcache_grdma_module.c:235:17: warning: unused variable ‘ref_cnt’ 
[-Wunused-variable]
 int32_t ref_cnt = opal_atomic_add_32 (_reg->ref_count, 1);
 ^~~
runtime/orte_init.c:129:2: warning: #ident is a GCC extension


coll_basic_reduce_scatter.c: In function ‘mca_coll_basic_reduce_scatter_inter’:
coll_basic_reduce_scatter.c:474:9: warning: ‘lbuf’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
 err = comm->c_local_comm->c_coll.coll_scatterv(lbuf, rcounts, disps, dtype,
 ^~~
rbuf, rcounts[rank], dtype, 0,
~~
comm->c_local_comm,
~~~
comm->c_local_comm->c_coll.coll_scatterv_module);

coll_basic_alltoallw.c: In function ‘mca_coll_basic_alltoallw_intra’:
coll_basic_alltoallw.c:71:16: warning: ‘gap’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 tmp_buffer -= gap;
^~
coll_basic_alltoallw.c:47:20: note: ‘gap’ was declared here
 ptrdiff_t ext, gap;
^~~


In file included from osc_pt2pt_module.c:23:0:
osc_pt2pt.h: In function ‘osc_pt2pt_gc_clean’:
osc_pt2pt.h:668:21: warning: unused variable ‘request’ [-Wunused-variable]
 ompi_request_t *request;
 ^~~
osc_pt2pt_module.c: In function ‘ompi_osc_pt2pt_free’:
osc_pt2pt_module.c:96:28: warning: comparison between signed and unsigned 
integer expressions [-Wsign-compare]
 for (int i = 0 ; i < module->recv_frag_count ; ++i) {
^
In file included from osc_pt2pt_frag.c:15:0:
osc_pt2pt.h: In function ‘osc_pt2pt_gc_clean’:
osc_pt2pt.h:668:21: warning: unused variable ‘request’ [-Wunused-variable]
 ompi_request_t *request;
 ^~~
In file included from osc_pt2pt_active_target.c:26:0:
osc_pt2pt.h: In function 

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

I don’t disagree with anything you said - however, this problem has been 
reported in our library for more than a decade (goes way back into the old Trac 
days), and has yet to be resolved. Meantime, we have a user that is “down” and 
needs a solution. Whether it is a “cheap shot” or not is irrelevant to them.

I’ll leave it to you deeper MPI wonks to solve the problem correctly :-) When 
you have done so, I will happily remove the coll/sync component and tell the 
user “all has been resolved”.


> On Aug 20, 2016, at 11:44 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
> 
> Ralph,
> 
> Bringing back the coll/sync is a cheap shot at hiding a real issue behind a 
> smoke curtain. As Nathan described in his email, Open MPI lacks of control 
> flow on eager messages is the real culprit here, and the loop around any 
> one-to-many collective (bcast and scatter*) was only helping to exacerbate 
> the issue. However, doing a loop around a small MPI_Send will also end on a 
> memory exhaustion issue, one that would not be easily circumvented by adding 
> synchronizations deep inside the library.
> 
>   George.
> 
> 
> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
> wrote:
> I can not provide the user report as it is a proprietary problem. However, it 
> consists of a large loop of calls to MPI_Bcast that crashes due to unexpected 
> messages. We have been looking at instituting flow control, but that has way 
> too widespread an impact. The coll/sync component would be a simple solution.
> 
> I honestly don’t believe the issue I was resolving was due to a bug - it was 
> a simple problem of one proc running slow and creating an overload of 
> unexpected messages that eventually consumed too much memory. Rather, I think 
> you solved a different problem - by the time you arrived at LANL, the app I 
> was working with had already modified their code to no longer create the 
> problem (essentially refactoring the algorithm to avoid the massive loop over 
> allreduce).
> 
> I have no issue supporting it as it takes near-zero effort to maintain, and 
> this is a fairly common problem with legacy codes that don’t want to refactor 
> their algorithms.
> 
> 
> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm <hje...@me.com 
> > <mailto:hje...@me.com>> wrote:
> >
> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> >> wrote:
> >>
> >> Hi folks
> >>
> >> I had a question arise regarding a problem being seen by an OMPI user - 
> >> has to do with the old bugaboo I originally dealt with back in my LANL 
> >> days. The problem is with an app that repeatedly hammers on a collective, 
> >> and gets overwhelmed by unexpected messages when one of the procs falls 
> >> behind.
> >
> > I did some investigation on roadrunner several years ago and determined 
> > that the user code issue coll/sync was attempting to fix was due to a bug 
> > in ob1/cksum (really can’t remember). coll/sync was simply masking a 
> > live-lock problem. I committed a workaround for the bug in r26575 
> > (https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35
> >  
> > <https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35>)
> >  and tested it with the user code. After this change the user code ran fine 
> > without coll/sync. Since lanl no longer had any users of coll/sync we 
> > stopped supporting it.
> >
> >> I solved this back then by introducing the “sync” component in 
> >> ompi/mca/coll, which injected a barrier operation every N collectives. You 
> >> could even “tune” it by doing the injection for only specific collectives.
> >>
> >> However, I can no longer find that component in the code base - I find it 
> >> in the 1.6 series, but someone removed it during the 1.7 series.
> >>
> >> Can someone tell me why this was done??? Is there any reason not to bring 
> >> it back? It solves a very real, not uncommon, problem.
> >> Ralph
> >
> > This was discussed during one (or several) tel-cons years ago. We agreed to 
> > kill it and bring it back if there is 1) a use case, and 2) someone is 
> > willing to support it. See 
> > https://github.com/open-mpi/ompi/commit/5451ee46bd6fcdec002b333474dec919475d2d62
> >  
> > <https://github.com/open-mpi/ompi/commit/5451ee46bd6fcdec002b333474dec919475d2d62>
> >  .
> >
> > Can you link the user email?
> >
> > -Nathan
> > ___

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

As I said earlier, modifying these legacy apps is not a desirable solution. The 
coll/sync component was developed specifically to alleviate these problems in 
an acceptable manner, albeit not optimal. Performance, in this case, is 
secondary to just getting the app to run.


> On Aug 20, 2016, at 7:38 PM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> in the meantime, and if not done already, your user can simply redefine 
> MPI_Bcast in the app.
> 
> 
> int MPI_Bcast(void *buffer, int count, MPI_Datatype type, int root, MPI_Comm 
> comm) {
> PMPI_Barrier(comm);
> return PMPI_Bcast(buffer, count, datatype, root, comm);
> }
> 
> the root causes are
> - no control flow in Open MPI for eager messages (as explained by George)
> and
> - some processes are much slower than others.
> 
> so even if Open MPI provides a fix or workaround, the end user will be left 
> with some important load imbalance, which is far from being optimal from 
> his/her performance point of view.
> 
> 
> Cheers,
> 
> Gilles
> 
> On Sunday, August 21, 2016, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> I don’t disagree with anything you said - however, this problem has been 
> reported in our library for more than a decade (goes way back into the old 
> Trac days), and has yet to be resolved. Meantime, we have a user that is 
> “down” and needs a solution. Whether it is a “cheap shot” or not is 
> irrelevant to them.
> 
> I’ll leave it to you deeper MPI wonks to solve the problem correctly :-) When 
> you have done so, I will happily remove the coll/sync component and tell the 
> user “all has been resolved”.
> 
> 
>> RROn Aug 20, 2016, at 11:44 AM, George Bosilca <bosi...@icl.utk.edu 
>> <javascript:_e(%7B%7D,'cvml','bosi...@icl.utk.edu');>> wrote:
>> 
>> Ralph,
>> 
>> Bringing back the coll/sync is a cheap shot at hiding a real issue behind a 
>> smoke curtain. As Nathan described in his email, Open MPI lacks of control 
>> flow on eager messages is the real culprit here, and the loop around any 
>> one-to-many collective (bcast and scatter*) was only helping to exacerbate 
>> the issue. However, doing a loop around a small MPI_Send will also end on a 
>> memory exhaustion issue, one that would not be easily circumvented by adding 
>> synchronizations deep inside the library.
>> 
>>   George.
>> 
>> 
>> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
>> <javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');> <r...@open-mpi.org 
>> <javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');>> wrote:
>> I can not provide the user report as it is a proprietary problem. However, 
>> it consists of a large loop of calls to MPI_Bcast that crashes due to 
>> unexpected messages. We have been looking at instituting flow control, but 
>> that has way too widespread an impact. The coll/sync component would be a 
>> simple solution.
>> 
>> I honestly don’t believe the issue I was resolving was due to a bug - it was 
>> a simple problem of one proc running slow and creating an overload of 
>> unexpected messages that eventually consumed too much memory. Rather, I 
>> think you solved a different problem - by the time you arrived at LANL, the 
>> app I was working with had already modified their code to no longer create 
>> the problem (essentially refactoring the algorithm to avoid the massive loop 
>> over allreduce).
>> 
>> I have no issue supporting it as it takes near-zero effort to maintain, and 
>> this is a fairly common problem with legacy codes that don’t want to 
>> refactor their algorithms.
>> 
>> 
>> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm <hje...@me.com 
>> > <javascript:_e(%7B%7D,'cvml','hje...@me.com');>> wrote:
>> >
>> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org 
>> >> <javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');> wrote:
>> >>
>> >> Hi folks
>> >>
>> >> I had a question arise regarding a problem being seen by an OMPI user - 
>> >> has to do with the old bugaboo I originally dealt with back in my LANL 
>> >> days. The problem is with an app that repeatedly hammers on a collective, 
>> >> and gets overwhelmed by unexpected messages when one of the procs falls 
>> >> behind.
>> >
>> > I did some investigation on roadrunner several years ago and determined 
>> > that the user code issue coll/sync was attempting to fix was due to a bug 
&

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

If you see ways to improve it, you are welcome to do so.

> On Aug 22, 2016, at 12:30 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Folks,
> 
> 
> i was reviewing the sources of the coll/sync module, and
> 
> 
> 1) i noticed the same pattern is used in *every* sources :
> 
> if (s->in_operation) {
> return s->c_coll.coll_xxx(...);
> } else {
> COLL_SYNC(s, s->c_coll.coll_xxx(...));
> 
>   }
> 
> is there any rationale for not moving the if(s->in_operation) test into the 
> COLL_SYNC macro ?
> 
> 
> 2) i could not find a rationale for using s->in_operation :
> - if a barrier must be performed, the barrier of the underlying module (e.g. 
> coll/tuned) is directly invoked, so coll/sync is not somehow reentrant
> - with MPI_THREAD_MULTIPLE, it is the enduser responsability that two threads 
> never invoke simultaneously a collective operation on the *same* communicator
>   (and s->in_operation is a per-communicator boolean), so i do not see how 
> s->in_operation can be true in a valid MPI program.
> 
> 
> Though the first point can be seen as a "matter of style", i am pretty 
> curious about the second one.
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> On 8/21/2016 3:44 AM, George Bosilca wrote:
>> Ralph,
>> 
>> Bringing back the coll/sync is a cheap shot at hiding a real issue behind a 
>> smoke curtain. As Nathan described in his email, Open MPI lacks of control 
>> flow on eager messages is the real culprit here, and the loop around any 
>> one-to-many collective (bcast and scatter*) was only helping to exacerbate 
>> the issue. However, doing a loop around a small MPI_Send will also end on a 
>> memory exhaustion issue, one that would not be easily circumvented by adding 
>> synchronizations deep inside the library.
>> 
>>   George.
>> 
>> 
>> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
>> wrote:
>> I can not provide the user report as it is a proprietary problem. However, 
>> it consists of a large loop of calls to MPI_Bcast that crashes due to 
>> unexpected messages. We have been looking at instituting flow control, but 
>> that has way too widespread an impact. The coll/sync component would be a 
>> simple solution.
>> 
>> I honestly don’t believe the issue I was resolving was due to a bug - it was 
>> a simple problem of one proc running slow and creating an overload of 
>> unexpected messages that eventually consumed too much memory. Rather, I 
>> think you solved a different problem - by the time you arrived at LANL, the 
>> app I was working with had already modified their code to no longer create 
>> the problem (essentially refactoring the algorithm to avoid the massive loop 
>> over allreduce).
>> 
>> I have no issue supporting it as it takes near-zero effort to maintain, and 
>> this is a fairly common problem with legacy codes that don’t want to 
>> refactor their algorithms.
>> 
>> 
>> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm <hje...@me.com 
>> > <mailto:hje...@me.com>> wrote:
>> >
>> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> >> wrote:
>> >>
>> >> Hi folks
>> >>
>> >> I had a question arise regarding a problem being seen by an OMPI user - 
>> >> has to do with the old bugaboo I originally dealt with back in my LANL 
>> >> days. The problem is with an app that repeatedly hammers on a collective, 
>> >> and gets overwhelmed by unexpected messages when one of the procs falls 
>> >> behind.
>> >
>> > I did some investigation on roadrunner several years ago and determined 
>> > that the user code issue coll/sync was attempting to fix was due to a bug 
>> > in ob1/cksum (really can’t remember). coll/sync was simply masking a 
>> > live-lock problem. I committed a workaround for the bug in r26575 
>> > (https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35
>> >  
>> > <https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35>)
>> >  and tested it with the user code. After this change the user code ran 
>> > fine without coll/sync. Since lanl no longer had any users of coll/sync we 
>> > stopped supporting it.
>> >
>> >> I solved this back then by introducing the “sync” component in 
>> >> ompi/mca/coll, which injecte

Re: [OMPI devel] v2.1.0rc1 has been released

[r...@open-mpi.org]

> On Feb 26, 2017, at 6:47 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> - Should be none.
>  ^^^ JMS Did we change --host or --hostfile behavior?
> 

Not that I am aware of - we talked about it, but I think we concluded that we 
couldn’t/shouldn’t as that would constitute a change to “3.0” in the release 
version


___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Binding with --oversubscribe in 2.0.0

[r...@open-mpi.org]

Yeah, that man page could use some work...will add it to the list.

> On Aug 24, 2016, at 4:35 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
> 
> Hi Ralph,
> 
> Thanks for that... that option's not on the man page for mpirun, but I can 
> see it in the --help message (as "overload-allowed", which also works).
> 
> Cheers,
> Ben
> 
> 
> -Original Message-
> From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of 
> r...@open-mpi.org
> Sent: Thursday, 25 August 2016 2:03 AM
> To: OpenMPI Devel <devel@lists.open-mpi.org>
> Subject: Re: [OMPI devel] Binding with --oversubscribe in 2.0.0
> 
> Actually, I stand corrected! Someone must have previously requested it, 
> because support already exists.
> 
> What you need to do is simply specify the desired binding. If you don’t 
> specify one, then we will disable it by default when oversubscribed. This was 
> done to protect performance for those who don’t have such kind scenarios, and 
> don’t realize we are otherwise binding by default.
> 
> So in your case, you’d want something like:
> 
> mpirun --map-by core:oversubscribe --bind-to core:overload
> 
> HTH
> Ralph
> 
>> On Aug 24, 2016, at 7:33 AM, r...@open-mpi.org wrote:
>> 
>> Well, that’s a new one! I imagine we could modify the logic to allow a 
>> combination of oversubscribe and overload flags. Won’t get out until 2.1, 
>> though you could pull the patch in advance if it is holding you up.
>> 
>> 
>>> On Aug 23, 2016, at 11:46 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
>>> 
>>> Hi,
>>> 
>>> One of our users has noticed that binding is disabled in 2.0.0 when 
>>> --oversubscribe is passed, which is hurting their performance, likely 
>>> through migrations between sockets. It looks to be because of 294793c 
>>> (PR#1228).
>>> 
>>> They need to use --oversubscribe as for some reason the developers 
>>> decided to run two processes for each MPI task for some reason (a 
>>> compute process and an I/O worker process, I think). Since the second 
>>> process in the pair is mostly idle, there's (almost) no harm in 
>>> launching two processes per core - and it's better than leaving half 
>>> the cores idle most of the time. In previous versions they were 
>>> binding each pair to a core and letting the hyper-threads argue over 
>>> which of the two processes to run, since this gave the best performance.
>>> 
>>> I tried creating a rankfile and binding each process to its own 
>>> hardware thread, but it refuses to launch more processes than the 
>>> number of cores (even if all these processes are on the first socket 
>>> because of the binding) unless --oversubscribe is passed, and thus 
>>> disabling the binding. Is there a way of bypassing the 
>>> disable-binding-if-oversubscribing check introduced by that commit? Or can 
>>> anyone think of a better way of running this program?
>>> 
>>> Alternatively, they could leave it with no binding at the mpirun 
>>> level and do the binding in a wrapper.
>>> 
>>> Thanks,
>>> Ben
>>> 
>>> 
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Binding with --oversubscribe in 2.0.0

[r...@open-mpi.org]

Hmmm...bet I know why. Let me poke a bit.

> On Aug 24, 2016, at 5:18 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
> 
> Actually, adding :oversubscribe to the --map-by option still disables 
> binding, even with :overload on the --bind-to option. While the :overload 
> option allows binding more than one process per CPU, it only has an effect if 
> binding actually happens - i.e. without :oversubscribe.
> 
> So, on one of our login nodes (2x8-core),
> 
>  mpirun --np 32 --bind-to core:overload --report-bindings true
> 
> works and does what you would expect (0 and 16 on core 0, 1 and 17 on core 1, 
> ...), while inside a PBS job on a compute node (same hardware) it fails with 
> "not enough slots available in the system". Adding --map-by 
> core:oversubscribe makes this to work, but then doesn't have binding.
> 
> Cheers,
> Ben
> 
> -Original Message-
> From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of Ben Menadue
> Sent: Thursday, 25 August 2016 9:36 AM
> To: 'Open MPI Developers' <devel@lists.open-mpi.org>
> Subject: Re: [OMPI devel] Binding with --oversubscribe in 2.0.0
> 
> Hi Ralph,
> 
> Thanks for that... that option's not on the man page for mpirun, but I can 
> see it in the --help message (as "overload-allowed", which also works).
> 
> Cheers,
> Ben
> 
> 
> -Original Message-
> From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of 
> r...@open-mpi.org
> Sent: Thursday, 25 August 2016 2:03 AM
> To: OpenMPI Devel <devel@lists.open-mpi.org>
> Subject: Re: [OMPI devel] Binding with --oversubscribe in 2.0.0
> 
> Actually, I stand corrected! Someone must have previously requested it, 
> because support already exists.
> 
> What you need to do is simply specify the desired binding. If you don’t 
> specify one, then we will disable it by default when oversubscribed. This was 
> done to protect performance for those who don’t have such kind scenarios, and 
> don’t realize we are otherwise binding by default.
> 
> So in your case, you’d want something like:
> 
> mpirun --map-by core:oversubscribe --bind-to core:overload
> 
> HTH
> Ralph
> 
>> On Aug 24, 2016, at 7:33 AM, r...@open-mpi.org wrote:
>> 
>> Well, that’s a new one! I imagine we could modify the logic to allow a 
>> combination of oversubscribe and overload flags. Won’t get out until 2.1, 
>> though you could pull the patch in advance if it is holding you up.
>> 
>> 
>>> On Aug 23, 2016, at 11:46 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
>>> 
>>> Hi,
>>> 
>>> One of our users has noticed that binding is disabled in 2.0.0 when 
>>> --oversubscribe is passed, which is hurting their performance, likely 
>>> through migrations between sockets. It looks to be because of 294793c 
>>> (PR#1228).
>>> 
>>> They need to use --oversubscribe as for some reason the developers 
>>> decided to run two processes for each MPI task for some reason (a 
>>> compute process and an I/O worker process, I think). Since the second 
>>> process in the pair is mostly idle, there's (almost) no harm in 
>>> launching two processes per core - and it's better than leaving half 
>>> the cores idle most of the time. In previous versions they were 
>>> binding each pair to a core and letting the hyper-threads argue over 
>>> which of the two processes to run, since this gave the best performance.
>>> 
>>> I tried creating a rankfile and binding each process to its own 
>>> hardware thread, but it refuses to launch more processes than the 
>>> number of cores (even if all these processes are on the first socket 
>>> because of the binding) unless --oversubscribe is passed, and thus 
>>> disabling the binding. Is there a way of bypassing the 
>>> disable-binding-if-oversubscribing check introduced by that commit? Or can 
>>> anyone think of a better way of running this program?
>>> 
>>> Alternatively, they could leave it with no binding at the mpirun 
>>> level and do the binding in a wrapper.
>>> 
>>> Thanks,
>>> Ben
>>> 
>>> 
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Binding with --oversubscribe in 2.0.0

[r...@open-mpi.org]

Okay, I found the issue and fixed it:

https://github.com/open-mpi/ompi-release/pull/1340

We are very close to v2.0.1 release, so it may not get into that one. Still, 
you are welcome to pull down the patch and locally apply it if it would help.

Ralph

> On Aug 24, 2016, at 5:29 PM, r...@open-mpi.org wrote:
> 
> Hmmm...bet I know why. Let me poke a bit.
> 
>> On Aug 24, 2016, at 5:18 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
>> 
>> Actually, adding :oversubscribe to the --map-by option still disables 
>> binding, even with :overload on the --bind-to option. While the :overload 
>> option allows binding more than one process per CPU, it only has an effect 
>> if binding actually happens - i.e. without :oversubscribe.
>> 
>> So, on one of our login nodes (2x8-core),
>> 
>> mpirun --np 32 --bind-to core:overload --report-bindings true
>> 
>> works and does what you would expect (0 and 16 on core 0, 1 and 17 on core 
>> 1, ...), while inside a PBS job on a compute node (same hardware) it fails 
>> with "not enough slots available in the system". Adding --map-by 
>> core:oversubscribe makes this to work, but then doesn't have binding.
>> 
>> Cheers,
>> Ben
>> 
>> -Original Message-
>> From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of Ben 
>> Menadue
>> Sent: Thursday, 25 August 2016 9:36 AM
>> To: 'Open MPI Developers' <devel@lists.open-mpi.org>
>> Subject: Re: [OMPI devel] Binding with --oversubscribe in 2.0.0
>> 
>> Hi Ralph,
>> 
>> Thanks for that... that option's not on the man page for mpirun, but I can 
>> see it in the --help message (as "overload-allowed", which also works).
>> 
>> Cheers,
>> Ben
>> 
>> 
>> -Original Message-
>> From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of 
>> r...@open-mpi.org
>> Sent: Thursday, 25 August 2016 2:03 AM
>> To: OpenMPI Devel <devel@lists.open-mpi.org>
>> Subject: Re: [OMPI devel] Binding with --oversubscribe in 2.0.0
>> 
>> Actually, I stand corrected! Someone must have previously requested it, 
>> because support already exists.
>> 
>> What you need to do is simply specify the desired binding. If you don’t 
>> specify one, then we will disable it by default when oversubscribed. This 
>> was done to protect performance for those who don’t have such kind 
>> scenarios, and don’t realize we are otherwise binding by default.
>> 
>> So in your case, you’d want something like:
>> 
>> mpirun --map-by core:oversubscribe --bind-to core:overload
>> 
>> HTH
>> Ralph
>> 
>>> On Aug 24, 2016, at 7:33 AM, r...@open-mpi.org wrote:
>>> 
>>> Well, that’s a new one! I imagine we could modify the logic to allow a 
>>> combination of oversubscribe and overload flags. Won’t get out until 2.1, 
>>> though you could pull the patch in advance if it is holding you up.
>>> 
>>> 
>>>> On Aug 23, 2016, at 11:46 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> One of our users has noticed that binding is disabled in 2.0.0 when 
>>>> --oversubscribe is passed, which is hurting their performance, likely 
>>>> through migrations between sockets. It looks to be because of 294793c 
>>>> (PR#1228).
>>>> 
>>>> They need to use --oversubscribe as for some reason the developers 
>>>> decided to run two processes for each MPI task for some reason (a 
>>>> compute process and an I/O worker process, I think). Since the second 
>>>> process in the pair is mostly idle, there's (almost) no harm in 
>>>> launching two processes per core - and it's better than leaving half 
>>>> the cores idle most of the time. In previous versions they were 
>>>> binding each pair to a core and letting the hyper-threads argue over 
>>>> which of the two processes to run, since this gave the best performance.
>>>> 
>>>> I tried creating a rankfile and binding each process to its own 
>>>> hardware thread, but it refuses to launch more processes than the 
>>>> number of cores (even if all these processes are on the first socket 
>>>> because of the binding) unless --oversubscribe is passed, and thus 
>>>> disabling the binding. Is there a way of bypassing the 
>>>> disable-binding-if-oversubscribing check introduced by that commit? Or can 
>>>> anyone think of a better way of running this progr

Re: [MTT devel] reporter error using pyclient

[r...@open-mpi.org]

I tried converting all the stdout/stderr input to single strings using a 
“join”, and now I’m getting a different error - looks like we are failing to 
get a submit id?

<<<<<<< Raw Output (Start) >>>>>>
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;>



500 Internal Server Error

#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}

#traceback {
color: red;
}



500 Internal Server Error
The server encountered an unexpected condition which prevented it 
from fulfilling the request.
Traceback (most recent call last):
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cprequest.py",
 line 670, in respond
response.body = self.handler()
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/encoding.py",
 line 217, in __call__
self.body = self.oldhandler(*args, **kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/jsontools.py",
 line 63, in json_handler
value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cpdispatch.py",
 line 60, in __call__
return self.callable(*self.args, **self.kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/dispatchers.py",
 line 318, in POST
submit_info = self._db.get_submit_id(data['metadata'])
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py",
 line 234, in get_submit_id
submit_id = self._select_insert("submit", "submit_id", fields, values)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py",
 line 183, in _select_insert
cursor.execute( select_stmt, values )
InternalError: current transaction is aborted, commands ignored until end of 
transaction block



  
Powered by http://www.cherrypy.org;>CherryPy 5.1.0
  




<<<<<<< Raw Output (End  ) >>>>>>


> On Aug 29, 2016, at 9:03 AM, Josh Hursey <jjhur...@open-mpi.org> wrote:
> 
> Looking at the cherrypy server log it looks like the error is related to this:
> "result_stderr": [
> "Option middleware is not supported"
> ],
> 
> This value was set to an array instead of string. 
> 
> result_stdout is coming through fine as a string, but it looks like you are 
> submitting the result_stderr as an array still. The server treats both of 
> those keys the same, so it must be on the python client side.
> 
> I don't think the server requires a MPI Install phase. It's listed as an 
> optional field for the test_build/test_run phases. If you don't submit an MPI 
> Install phase it might not show up in the MTT Reporter's default view, but 
> that's the same with the Perl client configured to use an already installed 
> build.
> 
> 
> On Fri, Aug 26, 2016 at 9:40 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Hmmm...Hey Josh - is it possible that your server is requiring an MPI_Install 
> phase? We don’t have one since we just build and then set the path 
> accordingly - is it complaining about missing data for an install phase?
> 
> 
>> On Aug 26, 2016, at 7:33 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> Okay, I cleaned things up some more and got a little bit further - now 
>> hitting an error in the server?
>> 
>> <<<<<<< Payload (End  ) -->>>>>>
>> INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: 
>> mtt.open-mpi.org <http://mtt.open-mpi.org/>
>> DEBUG:requests.packages.urllib3.connectionpool:"POST /submit/cpy/api/submit 
>> HTTP/1.1" 500 2535
>> <<<<<<< Response -->>>>>>
>> Result: 500: text/html;charset=utf-8
>> {'content-length': '2535', 'set-cookie': 
>> 'session_id=1b3f3c3df893e673e072430844afe27b4389be71; expires=Sat, 27 Aug 
>> 2016 03:31:28 GMT; Path=/', 'server': 'CherryPy/5.1.0', 'connection': 
>> 'close', 'allow': 'POST', 'date': 'Sat, 27 Aug 2016 02:31:28 GMT', 
>> 'access-

Re: [MTT devel] the boolean keyval error traceback with debug

[r...@open-mpi.org]

Should be fixed in https://github.com/open-mpi/mtt/pull/469

> On Aug 31, 2016, at 6:54 PM, r...@open-mpi.org wrote:
> 
> I’ll dig into this tomorrow
> 
>> On Aug 31, 2016, at 10:59 AM, Howard Pritchard <hpprit...@gmail.com 
>> <mailto:hpprit...@gmail.com>> wrote:
>> 
>> Hi Folks,
>> 
>> Here's what I'm seeing with the boolean keyval issue:
>> 
>> orking repo g...@github.com:open-mpi/ompi-tests
>> 
>> Working final repo g...@github.com:open-mpi/ompi-tests
>> 
>> LOGGING results for ASIS TestGet:IBM
>> 
>> ASIS TestBuild:IBMInstalled
>> 
>> DefaultTestBuild
>> 
>> processing kvkey merge_stdout_stderr
>> 
>> processing kvkey save_stdout_on_success
>> 
>> Autotools Execute
>> 
>> processing kvkey merge_stdout_stderr
>> 
>> Traceback (most recent call last):
>> 
>>   File "./pymtt.py", line 247, in 
>> 
>> testDef.executeTest()
>> 
>>   File "/Users/hpp/mtt/pylib/System/TestDef.py", line 602, in executeTest
>> 
>> executor.plugin_object.execute(self)
>> 
>>   File "/Users/hpp/mtt/pylib/Tools/Executor/sequential.py", line 223, in 
>> execute
>> 
>> plugin.execute(stageLog, keyvals, testDef)
>> 
>>   File "/Users/hpp/mtt/pylib/Stages/TestBuild/DefaultTestBuild.py", line 
>> 131, in execute
>> 
>> plugin.execute(log, cmds, testDef)
>> 
>>   File "/Users/hpp/mtt/pylib/Tools/Build/Autotools.py", line 63, in execute
>> 
>> testDef.parseOptions(log, self.options, keyvals, cmds)
>> 
>>   File "/Users/hpp/mtt/pylib/System/TestDef.py", line 125, in parseOptions
>> 
>> if keyvals[kvkey].upper() in ['TRUE', '1', 'T', 'Y', 'YES']:
>> 
>> 
>> 
>> ___
>> mtt-devel mailing list
>> mtt-devel@lists.open-mpi.org <mailto:mtt-devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/mtt-devel
> 
> ___
> mtt-devel mailing list
> mtt-devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/mtt-devel

___
mtt-devel mailing list
mtt-devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/mtt-devel

Re: [MTT devel] the boolean keyval error traceback with debug

[r...@open-mpi.org]

I’ll dig into this tomorrow

> On Aug 31, 2016, at 10:59 AM, Howard Pritchard  wrote:
> 
> Hi Folks,
> 
> Here's what I'm seeing with the boolean keyval issue:
> 
> orking repo g...@github.com:open-mpi/ompi-tests
> 
> Working final repo g...@github.com:open-mpi/ompi-tests
> 
> LOGGING results for ASIS TestGet:IBM
> 
> ASIS TestBuild:IBMInstalled
> 
> DefaultTestBuild
> 
> processing kvkey merge_stdout_stderr
> 
> processing kvkey save_stdout_on_success
> 
> Autotools Execute
> 
> processing kvkey merge_stdout_stderr
> 
> Traceback (most recent call last):
> 
>   File "./pymtt.py", line 247, in 
> 
> testDef.executeTest()
> 
>   File "/Users/hpp/mtt/pylib/System/TestDef.py", line 602, in executeTest
> 
> executor.plugin_object.execute(self)
> 
>   File "/Users/hpp/mtt/pylib/Tools/Executor/sequential.py", line 223, in 
> execute
> 
> plugin.execute(stageLog, keyvals, testDef)
> 
>   File "/Users/hpp/mtt/pylib/Stages/TestBuild/DefaultTestBuild.py", line 131, 
> in execute
> 
> plugin.execute(log, cmds, testDef)
> 
>   File "/Users/hpp/mtt/pylib/Tools/Build/Autotools.py", line 63, in execute
> 
> testDef.parseOptions(log, self.options, keyvals, cmds)
> 
>   File "/Users/hpp/mtt/pylib/System/TestDef.py", line 125, in parseOptions
> 
> if keyvals[kvkey].upper() in ['TRUE', '1', 'T', 'Y', 'YES']:
> 
> 
> 
> ___
> mtt-devel mailing list
> mtt-devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/mtt-devel

___
mtt-devel mailing list
mtt-devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/mtt-devel

Re: [OMPI devel] Question about Open MPI bindings

[r...@open-mpi.org]

I’ll dig more later, but just checking offhand, I can’t replicate this on my 
box, so it may be something in hwloc for that box (or maybe you have some MCA 
params set somewhere?):

$ mpirun -n 2 --bind-to core --report-bindings hostname
[rhc001:83938] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: 
[BB/../../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:83938] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: 
[../BB/../../../../../../../../../..][../../../../../../../../../../../..]

$ mpirun -n 2 --bind-to numa --report-bindings hostname
[rhc001:83927] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 
0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], 
socket 0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]], socket 0[core 10[hwt 0-1]], socket 0[core 11[hwt 0-1]]: 
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../../../..]
[rhc001:83927] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 
0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], 
socket 0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]], socket 0[core 10[hwt 0-1]], socket 0[core 11[hwt 0-1]]: 
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../../../..]


$ mpirun -n 2 --bind-to socket --report-bindings hostname
[rhc001:83965] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 
0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], 
socket 0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]], socket 0[core 10[hwt 0-1]], socket 0[core 11[hwt 0-1]]: 
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../../../..]
[rhc001:83965] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 
0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], 
socket 0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 
0-1]], socket 0[core 10[hwt 0-1]], socket 0[core 11[hwt 0-1]]: 
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../../../..]


I have seen the segfault when something fails early in the setup procedure - 
planned to fix that this weekend.


> On Sep 2, 2016, at 9:09 PM, George Bosilca  wrote:
> 
> While investigating the ongoing issue with OMPI messaging layer, I run into 
> some troubles with process binding. I read the documentation, but I still 
> find this puzzling.
> 
> Disclaimer: all experiments were done with current master (9c496f7) compiled 
> in optimized mode. The hardware: a single node 20 core Xeon E5-2650 v3 
> (hwloc-ls is at the end of this email).
> 
> First and foremost, trying to bind to NUMA nodes was a sure way to get a 
> segfault:
> 
> $ mpirun -np 2 --mca btl vader,self --bind-to numa --report-bindings true
> --
> No objects of the specified type were found on at least one node:
> 
>   Type: NUMANode
>   Node: arc00
> 
> The map cannot be done as specified.
> --
> [dancer:32162] *** Process received signal ***
> [dancer:32162] Signal: Segmentation fault (11)
> [dancer:32162] Signal code: Address not mapped (1)
> [dancer:32162] Failing at address: 0x3c
> [dancer:32162] [ 0] /lib64/libpthread.so.0[0x3126a0f7e0]
> [dancer:32162] [ 1] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(+0x560e0)[0x7f9075bc60e0]
> [dancer:32162] [ 2] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_grpcomm_API_xcast+0x84)[0x7f9075bc6f54]
> [dancer:32162] [ 3] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_plm_base_orted_exit+0x1a8)[0x7f9075bd9308]
> [dancer:32162] [ 4] 
> /home/bosilca/opt/trunk/fast/lib/openmpi/mca_plm_rsh.so(+0x384e)[0x7f907361284e]
> [dancer:32162] [ 5] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_state_base_check_all_complete+0x324)[0x7f9075bedca4]
> [dancer:32162] [ 6] 
> /home/bosilca/opt/trunk/fast/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0x53c)[0x7f90758eafec]
> [dancer:32162] [ 7] mpirun[0x401251]
> [dancer:32162] [ 8] mpirun[0x400e24]
> [dancer:32162] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x312621ed1d]
> [dancer:32162] [10] mpirun[0x400d49]
> [dancer:32162] *** End of error message ***
> Segmentation fault
> 
> As you can see on the hwloc output below, there are 2 NUMA nodes on the node 
> and HWLOC correctly identifies them, making OMPI error message confusing. 
> Anyway, we should not segfault but report a more meaning error message.
> 
> Binding to slot (I got this from the man page for 2.0) is apparently not 
> supported anymore. 

Re: [OMPI devel] Question about Open MPI bindings

[r...@open-mpi.org]

Okay, can you add --display-devel-map --mca rmaps_base_verbose 10 to your cmd 
line?

It sounds like there is something about that topo that is bothering the mapper

> On Sep 2, 2016, at 9:31 PM, George Bosilca  wrote:
> 
> Thanks Gilles, that's a very useful trick. The bindings reported by ORTE are 
> in sync with the one reported by the OS.
> 
> $ mpirun -np 2 --tag-output --bind-to core --report-bindings grep 
> Cpus_allowed_list /proc/self/status
> [1,0]:[arc00:90813] MCW rank 0 bound to socket 0[core 0[hwt 0]], 
> socket 0[core 4[hwt 0]]: 
> [B./../../../B./../../../../..][../../../../../../../../../..]
> [1,1]:[arc00:90813] MCW rank 1 bound to socket 1[core 10[hwt 0]], 
> socket 1[core 14[hwt 0]]: 
> [../../../../../../../../../..][B./../../../B./../../../../..]
> [1,0]:Cpus_allowed_list:0,8
> [1,1]:Cpus_allowed_list:1,9
> 
> George.
> 
> 
> 
> On Sat, Sep 3, 2016 at 12:27 AM, Gilles Gouaillardet 
> > wrote:
> George,
> 
> I cannot help much with this i am afraid
> 
> My best bet would be to rebuild OpenMPI with --enable-debug and an external 
> recent hwloc (iirc hwloc v2 cannot be used in Open MPI yet)
> 
> You might also want to try
> mpirun --tag-output --bind-to xxx --report-bindings grep Cpus_allowed_list 
> /proc/self/status
> 
> So you can confirm both openmpi and /proc/self/status report the same thing
> 
> Hope this helps a bit ...
> 
> Gilles
> 
> 
> George Bosilca > wrote:
> While investigating the ongoing issue with OMPI messaging layer, I run into 
> some troubles with process binding. I read the documentation, but I still 
> find this puzzling.
> 
> Disclaimer: all experiments were done with current master (9c496f7) compiled 
> in optimized mode. The hardware: a single node 20 core Xeon E5-2650 v3 
> (hwloc-ls is at the end of this email).
> 
> First and foremost, trying to bind to NUMA nodes was a sure way to get a 
> segfault:
> 
> $ mpirun -np 2 --mca btl vader,self --bind-to numa --report-bindings true
> --
> No objects of the specified type were found on at least one node:
> 
>   Type: NUMANode
>   Node: arc00
> 
> The map cannot be done as specified.
> --
> [dancer:32162] *** Process received signal ***
> [dancer:32162] Signal: Segmentation fault (11)
> [dancer:32162] Signal code: Address not mapped (1)
> [dancer:32162] Failing at address: 0x3c
> [dancer:32162] [ 0] /lib64/libpthread.so.0[0x3126a0f7e0]
> [dancer:32162] [ 1] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(+0x560e0)[0x7f9075bc60e0]
> [dancer:32162] [ 2] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_grpcomm_API_xcast+0x84)[0x7f9075bc6f54]
> [dancer:32162] [ 3] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_plm_base_orted_exit+0x1a8)[0x7f9075bd9308]
> [dancer:32162] [ 4] 
> /home/bosilca/opt/trunk/fast/lib/openmpi/mca_plm_rsh.so(+0x384e)[0x7f907361284e]
> [dancer:32162] [ 5] 
> /home/bosilca/opt/trunk/fast/lib/libopen-rte.so.0(orte_state_base_check_all_complete+0x324)[0x7f9075bedca4]
> [dancer:32162] [ 6] 
> /home/bosilca/opt/trunk/fast/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0x53c)[0x7f90758eafec]
> [dancer:32162] [ 7] mpirun[0x401251]
> [dancer:32162] [ 8] mpirun[0x400e24]
> [dancer:32162] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x312621ed1d]
> [dancer:32162] [10] mpirun[0x400d49]
> [dancer:32162] *** End of error message ***
> Segmentation fault
> 
> As you can see on the hwloc output below, there are 2 NUMA nodes on the node 
> and HWLOC correctly identifies them, making OMPI error message confusing. 
> Anyway, we should not segfault but report a more meaning error message.
> 
> Binding to slot (I got this from the man page for 2.0) is apparently not 
> supported anymore. Reminder: We should update the manpage accordingly.
> 
> Trying to bind to core looks better, the application at least starts. 
> Unfortunately the reported bindings (or at least my understanding of these 
> bindings) are troubling. Assuming that the way we report the bindings is 
> correct, why are my processes assigned to 2 cores far apart each ?
> 
> $ mpirun -np 2 --mca btl vader,self --bind-to core --report-bindings true
> [arc00:39350] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 
> 8[hwt 0]]: [B./../../../../../../../B./..][../../../../../../../../../..]
> [arc00:39350] MCW rank 1 bound to socket 0[core 1[hwt 0]], socket 0[core 
> 9[hwt 0]]: [../B./../../../../../../../B.][../../../../../../../../../..]
> 
> Maybe because I only used the binding option. Adding the mapping to the mix 
> (--map-by option) seem hopeless, the binding remains unchanged for 2 
> processes.
> 
> $ mpirun -np 2 --mca btl vader,self --bind-to core --report-bindings true
> [arc00:40401] MCW rank 0 bound to 

Re: [OMPI devel] Question about Open MPI bindings

[r...@open-mpi.org]

ancer.icl.utk.edu:17451/>] [[41198,0],0] 
> reset_usage: node arc00 has 3 procs on it
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> reset_usage: ignoring proc [[41198,1],0]
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> reset_usage: ignoring proc [[41198,1],1]
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> reset_usage: ignoring proc [[41198,1],2]
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> bind_depth: 5 map_depth 1
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps: bind 
> downward for job [41198,1] with bindings CORE
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> GOT 1 CPUS
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> PROC [[41198,1],0] BITMAP 0,8
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> BOUND PROC [[41198,1],0][arc00] TO socket 0[core 0[hwt 0-1]]: 
> [BB/../../..][../../../..
> ]
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> GOT 1 CPUS
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> PROC [[41198,1],1] BITMAP 4,12
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> BOUND PROC [[41198,1],1][arc00] TO socket 1[core 4[hwt 0-1]]: 
> [../../../..][BB/../../..
> ]
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> GOT 1 CPUS
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> PROC [[41198,1],2] BITMAP 1,9
> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0] 
> BOUND PROC [[41198,1],2][arc00] TO socket 0[core 1[hwt 0-1]]: 
> [../BB/../..][../../../..
> ]
> [1,0]:[arc00:07612] MCW rank 0 bound to socket 0[core 0[hwt 0]], 
> socket 0[core 8[hwt 0]]: [B./../../../../../../../B./..
> ][../../../../../../../../../..]
> [1,1]:[arc00:07612] MCW rank 1 bound to socket 0[core 4[hwt 0]], 
> socket 1[core 12[hwt 0]]: [../../../../B./../../../../.
> .][../../B./../../../../../../..]
> [1,2]:[arc00:07612] MCW rank 2 bound to socket 0[core 1[hwt 0]], 
> socket 0[core 9[hwt 0]]: [../B./../../../../../../../B.
> ][../../../../../../../../../..]
> 
> On Sat, Sep 3, 2016 at 9:44 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Okay, can you add --display-devel-map --mca rmaps_base_verbose 10 to your cmd 
> line?
> 
> It sounds like there is something about that topo that is bothering the mapper
> 
>> On Sep 2, 2016, at 9:31 PM, George Bosilca <bosi...@icl.utk.edu 
>> <mailto:bosi...@icl.utk.edu>> wrote:
>> 
>> Thanks Gilles, that's a very useful trick. The bindings reported by ORTE are 
>> in sync with the one reported by the OS.
>> 
>> $ mpirun -np 2 --tag-output --bind-to core --report-bindings grep 
>> Cpus_allowed_list /proc/self/status
>> [1,0]:[arc00:90813] MCW rank 0 bound to socket 0[core 0[hwt 0]], 
>> socket 0[core 4[hwt 0]]: 
>> [B./../../../B./../../../../..][../../../../../../../../../..]
>> [1,1]:[arc00:90813] MCW rank 1 bound to socket 1[core 10[hwt 0]], 
>> socket 1[core 14[hwt 0]]: 
>> [../../../../../../../../../..][B./../../../B./../../../../..]
>> [1,0]:Cpus_allowed_list:0,8
>> [1,1]:Cpus_allowed_list:1,9
>> 
>> George.
>> 
>> 
>> 
>> On Sat, Sep 3, 2016 at 12:27 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>> George,
>> 
>> I cannot help much with this i am afraid
>> 
>> My best bet would be to rebuild OpenMPI with --enable-debug and an external 
>> recent hwloc (iirc hwloc v2 cannot be used in Open MPI yet)
>> 
>> You might also want to try
>> mpirun --tag-output --bind-to xxx --report-bindings grep Cpus_allowed_list 
>> /proc/self/status
>> 
>> So you can confirm both openmpi and /proc/self/status report the same thing
>> 
>> Hope this helps a bit ...
>> 
>> Gilles
>> 
>> 
>> George Bosilca <bosi...@icl.utk.edu <mailto:bosi...@icl.utk.edu>> wrote:
>> While investigating the ongoing issue with OMPI messaging layer, I run into 
>> some troubles with process binding. I read the documentation, but I still 
>> find this puzzling.
>> 
>> Disclaimer: all experiments were done with current master (9c496f7) compiled 
>> in optimized mode. The hardware: a single node 20 core Xeon E5-

Re: [OMPI devel] OMPI devel] Question about Open MPI bindings

[r...@open-mpi.org]

Ah, indeed - if the node where mpirun is executing doesn’t match the compute 
nodes, then you must remove that --novm option. Otherwise, we have no way of 
knowing what the compute node topology looks like.


> On Sep 3, 2016, at 4:13 PM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> George,
> 
> If i understand correctly, you are running mpirun on dancer, which has
> 2 sockets, 4 cores per socket and 2 hwthreads per core,
> and orted are running on arc[00-08], though the tasks only run on arc00, 
> which has
> 2 sockets, 10 cores per socket and 2 hwthreads per core
> 
> to me, it looks like OpenMPI assumes all nodes are similar to dancer, which 
> is incorrect.
> 
> Can you try again with the --hetero-nodes option ?
> (iirc, that should not be needed because nodes should have different "hwloc 
> signatures", and OpenMPI is supposed to handle that automatically and 
> correctly)
> 
> That could be a side effect of your MCA params, can you try to remove them and
> mpirun --host arc00 --bind-to core -np 2 --report bindings grep 
> Cpus_alliwed_list /proc/self/status
> And one more test plus the --hetero-nodes option ?
> 
> Bottom line, you might have to set yet an other MCA param equivalent to the 
> --hetero-nodes option.
> 
> Cheers,
> 
> Gilles
> 
> r...@open-mpi.org wrote:
> Interesting - well, it looks like ORTE is working correctly. The map is what 
> you would expect, and so is planned binding.
> 
> What this tells us is that we are indeed binding (so far as ORTE is 
> concerned) to the correct places. Rank 0 is being bound to 0,8, and that is 
> what the OS reports. Rank 1 is bound to 4,12, and rank 2 is bound to 1,9. All 
> of this matches what the OS reported.
> 
> So it looks like it is report-bindings that is messed up for some reason.
> 
> 
>> On Sep 3, 2016, at 7:14 AM, George Bosilca <bosi...@icl.utk.edu 
>> <mailto:bosi...@icl.utk.edu>> wrote:
>> 
>> $mpirun -np 3 --tag-output --bind-to core --report-bindings 
>> --display-devel-map --mca rmaps_base_verbose 10 true
>> 
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0]: 
>> Final mapper priorities
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> ppr Priority: 90
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> seq Priority: 60
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> resilient Priority: 40
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> mindist Priority: 20
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> round_robin Priority: 10
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> staged Priority: 5
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> rank_file Priority: 0
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps: 
>> mapping job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps: 
>> setting mapping policies for job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps[153] 
>> mapping not set by user - using bysocket
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:ppr: 
>> job [41198,1] not using ppr mapper PPR NULL policy PPR NOTSET
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:seq: 
>> job [41198,1] not using seq mapper
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] 
>> mca:rmaps:resilient: cannot perform initial map of job [41198,1] - no fault 
>> groups
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] 
>> mca:rmaps:mindist: job [41198,1] not using mindist mapper
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:rr: 
>> mapping job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] AVAILABLE 
>> NODES FOR MAPPING:
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc00 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc01 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc02 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc03 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.e

Re: [OMPI devel] Question about Open MPI bindings

[r...@open-mpi.org]

I didn’t define the default behaviors - I just implemented what everyone said 
they wanted, as eventually captured in a Google spreadsheet Jeff posted (and 
was available and discussed for weeks before implemented). So the defaults are:

* if np <= 2, we map-by core bind-to core

* if np > 2, we map-by socket bind-to socket

In your case, you chose to specify a change in the binding pattern, but you 
left the mapping pattern to be the default. With 3 procs, that means you mapped 
by socket, and bound to core. ORTE did exactly what you told it to do 
(intentionally or not).

If you want the behavior you describe, then you simply tell ORTE to “--map-by 
core --bind-to core”

> On Sep 5, 2016, at 11:05 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
> 
> On Sat, Sep 3, 2016 at 10:34 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Interesting - well, it looks like ORTE is working correctly. The map is what 
> you would expect, and so is planned binding.
> 
> What this tells us is that we are indeed binding (so far as ORTE is 
> concerned) to the correct places. Rank 0 is being bound to 0,8, and that is 
> what the OS reports. Rank 1 is bound to 4,12, and rank 2 is bound to 1,9. All 
> of this matches what the OS reported.
> 
> So it looks like it is report-bindings that is messed up for some reason.
> 
> Ralph,
> 
> I have a hard time agreeing with you here. The binding you find correct is, 
> from a performance point of view, terrible. Why would anybody want a process 
> to be bound to 2 cores on different sockets ?
> 
> Please help me with the following exercise. How do I bind each process to a 
> single core, allocated in a round robin fashion (such as rank 0 on core 0, 
> rank 1 on core 1 and rank 2 on core 2) ?
> 
>   George.
> 
>  
> 
> 
>> On Sep 3, 2016, at 7:14 AM, George Bosilca <bosi...@icl.utk.edu 
>> <mailto:bosi...@icl.utk.edu>> wrote:
>> 
>> $mpirun -np 3 --tag-output --bind-to core --report-bindings 
>> --display-devel-map --mca rmaps_base_verbose 10 true
>> 
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] [[41198,0],0]: 
>> Final mapper priorities
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> ppr Priority: 90
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> seq Priority: 60
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> resilient Priority: 40
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> mindist Priority: 20
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> round_robin Priority: 10
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> staged Priority: 5
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>]  Mapper: 
>> rank_file Priority: 0
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps: 
>> mapping job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps: 
>> setting mapping policies for job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps[153] 
>> mapping not set by user - using bysocket
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:ppr: 
>> job [41198,1] not using ppr mapper PPR NULL policy PPR NOTSET
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:seq: 
>> job [41198,1] not using seq mapper
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] 
>> mca:rmaps:resilient: cannot perform initial map of job [41198,1] - no fault 
>> groups
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] 
>> mca:rmaps:mindist: job [41198,1] not using mindist mapper
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] mca:rmaps:rr: 
>> mapping job [41198,1]
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] AVAILABLE 
>> NODES FOR MAPPING:
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc00 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc01 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc02 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.utk.edu:17451/>] node: 
>> arc03 daemon: NULL
>> [dancer.icl.utk.edu:17451 <http://dancer.icl.u

[OMPI devel] Hanging tests

[r...@open-mpi.org]

Hey folks

All of the tests that involve either ISsend_ator, SSend_ator, ISsend_rtoa, or 
SSend_rtoa are hanging on master and v2.x. Does anyone know what these tests 
do, and why we never seem to pass them?

Do we care?
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Hanging tests

[r...@open-mpi.org]

I was just looking at the overnight MTT report, and these were present going 
back a long ways in both branches. They are in the Intel test suite.

If you have already addressed them, then thanks!

> On Sep 5, 2016, at 7:48 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> I fixed a hang earlier today in master, and the PR for v2.x is at 
> https://github.com/open-mpi/ompi-release/pull/1368
> 
> Can you please make sure you are running the latest master ?
> 
> Which testsuite do these tests come from ?
> I will have a look tomorrow if the hang is still there
> 
> Cheers,
> 
> Gilles
> 
> r...@open-mpi.org wrote:
>> Hey folks
>> 
>> All of the tests that involve either ISsend_ator, SSend_ator, ISsend_rtoa, 
>> or SSend_rtoa are hanging on master and v2.x. Does anyone know what these 
>> tests do, and why we never seem to pass them?
>> 
>> Do we care?
>> Ralph
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Hanging tests

[r...@open-mpi.org]

FWIW: those tests hang for me with TCP (I don’t have openib on my cluster). 
I’ll check it with your change as well


> On Sep 6, 2016, at 1:29 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Ralph,
> 
> 
> this looks like an other hang :-(
> 
> 
> i ran MPI_Issend_rtoa_c on 32 tasks (2 nodes, 2 sockets per node, 8 cores per 
> socket) with infiniband,
> 
> and i always observe the same hang at the same place.
> 
> 
> surprisingly, i do not get any hang if i blacklist the openib btl
> 
> 
> the patch below can be used to avoid the hang with infiniband or for 
> debugging purpose
> 
> the hang occurs in communicator 6, and if i skip tests on communicator 2, no 
> hang happens.
> 
> the hang occur on an intercomm :
> task 0 (from MPI_COMM_WORLD) has rank 0 in group A of the intercomm
> 
> task 1 (from MPI_COMM_WORLD) has rank 0 in group B of the intercomm
> 
> task 0 MPI_Issend to task 1, and task 1 MPI_Irecv from task 0, and then both 
> hang in MPI_Wait()
> 
> surprisingly, tasks 0 and 1 run on the same node, so it is very puzzling the 
> hang only occurs with the openib btl,
> 
> since vader should be used here.
> 
> 
> diff --git a/intel_tests/src/MPI_Issend_rtoa_c.c 
> b/intel_tests/src/MPI_Issend_rtoa_c.c
> index 8b26f84..b9a704b 100644
> --- a/intel_tests/src/MPI_Issend_rtoa_c.c
> +++ b/intel_tests/src/MPI_Issend_rtoa_c.c
> @@ -173,8 +177,9 @@ int main(int argc, char *argv[])
>  
>  for (comm_count = 0; comm_count < MPITEST_num_comm_sizes();
>   comm_count++) {
>  comm_index = MPITEST_get_comm_index(comm_count);
>  comm_type = MPITEST_get_comm_type(comm_count);
> +if (2 == comm_count) continue;
>  
>  /*
> @@ -312,6 +330,9 @@ int main(int argc, char *argv[])
>   * left sub-communicator
>   */
>  
> +if (6 == comm_count && 12 == length_count && 
> MPITEST_current_rank < 2) {
> +/* insert a breakpoint here */
> +}
>   * Reset a bunch of variables that will be set when we get our
> 
> 
> as a side note, which is very unlikely related to this issue, i noticed the 
> following programs works fine,
> 
> though it is reasonnable to expect a hang.
> the root cause is MPI_Send uses the eager protocol, and though communicators 
> used by MPI_Send and MPI_Recv
> 
> are different, they have the same (recycled) CID.
> 
> fwiw, the tests also completes with mpich.
> 
> 
> if not already done, should we provide an option not to recycle CIDs ?
> 
> or flush unexpected/unmatched messages when a communicator is free'd ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> #include 
> #include 
> 
> /* send a message (eager mode) in a communicator, and then
>  * receive it in an other communicator, but with the same CID
>  */
> int main(int argc, char *argv[]) {
> int rank, size;
> int b;
> MPI_Comm comm;
> 
> MPI_Init(, );
> MPI_Comm_rank(MPI_COMM_WORLD, );
> MPI_Comm_size(MPI_COMM_WORLD, );
> if (2 > size) MPI_Abort(MPI_COMM_WORLD, 1);
> 
> MPI_Comm_dup(MPI_COMM_WORLD, );
> if (0 == rank) {
> b = 0x;
> MPI_Send(, 1, MPI_INT, 1, 0, comm);
> }
> MPI_Comm_free();
> 
> MPI_Comm_dup(MPI_COMM_WORLD, );
> if (1 == rank) {
> b = 0x;
> MPI_Recv(, 1, MPI_INT, 0, 0, comm, MPI_STATUS_IGNORE);
> if (0x != b) MPI_Abort(MPI_COMM_WORLD, 2);
> }
> MPI_Comm_free();
> 
> MPI_Finalize();
> 
> return 0;
> }
> 
> 
> On 9/6/2016 12:03 AM, Gilles Gouaillardet wrote:
>> ok,  will double check tomorrow this was the very same hang i fixed earlier
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Monday, September 5, 2016, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
>> I was just looking at the overnight MTT report, and these were present going 
>> back a long ways in both branches. They are in the Intel test suite.
>> 
>> If you have already addressed them, then thanks!
>> 
>> > On Sep 5, 2016, at 7:48 AM, Gilles Gouaillardet 
>> > <gilles.gouaillar...@gmail.com <javascript:;>> wrote:
>> >
>> > Ralph,
>> >
>> > I fixed a hang earlier today in master, and the PR for v2.x is at 
>> > https://github.com/open-mpi/ompi-release/pull/1368 
>> > <https://github.com/open-mpi/ompi-release/pull/1368>
>> >
>> > Can you please make sure yo

[OMPI devel] PMIx shared memory dstore now off by default

[r...@open-mpi.org]

Hi folks

In order to let some folks continue working on dynamic operations on the 
master, I have turned the PMIx shared memory data store support “off” by 
default for the embedded code. You can turn it “on” using the 
--enable-pmix3-dstore option.

Once the dynamics support is functional, we will reverse the default setting of 
that flag. Sorry for the disruption.
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] C89 support

[r...@open-mpi.org]

Chris

At the risk of being annoying, it would really help me if you could answer my 
question: is Gilles correct in his feeling that we are looking at a scenario 
where you can support 90% of C99 (e.g., C99-style comments, named structure 
fields), and only the things modified in this PR are required?

I’m asking because these changes are minor and okay, but going back thru the 
code to revise all the comments and other C99isms would be a rather large task.


> On Aug 30, 2016, at 7:06 AM, C Bergström  wrote:
> 
> On Tue, Aug 30, 2016 at 9:20 PM, Jeff Squyres (jsquyres)
>  wrote:
>> On Aug 29, 2016, at 11:42 PM, C Bergström  wrote:
>>> 
>>> Paul - Is this your typical post? I can't tell if you're trying to be
>>> rude or it's accidental.
>> 
>> I believe that multiple people on this thread are reacting to the 
>> passive-aggressive tones and negative connotations in charged replies.
> 
> Total bullshit - If any of my replies were "charged", passive
> aggressive or otherwise that was not my intention. Anyone who I
> thought has replied rudely, I have called out directly and I don't
> mince words.
> 
> I'm not interested to spend 50 replies on 3 small patches. If you guys
> don't care about platform X, Foo compiler or older standards I respect
> that. My 1st email started with what I consider a humble tone. My
> patches are public and I've given all the details I have time for.
> 
> Last try
> 
>> 
>> I'd like to see:
>> 
>> 1. The specific link error that we're talking about.
> 
> As posted before - the error is *exactly* the same as in the public
> clang bug report.
> 
> (Thanks to Nathan)
> https://webcache.googleusercontent.com/search?q=cache:p2WZm7Vlt2gJ:https://llvm.org/bugs/show_bug.cgi%3Fid%3D5960+=1=en=clnk=us
> 
>> 
>> 2. All the information listed on https://www.open-mpi.org/community/help/ 
>> for compile/build problems.
> 
> I'm not going to shift threw this wall of text to try to figure out
> what you feel is missing. (Now my tone is "curt" if I have to be
> clear)
> 
>> 
>> 3. More complete patches for fixing the issues.  Specifically, the 3 
>> provided patches fix certain issues in some parts of the code base, but the 
>> same issues occur in other places in the code base.  As such, the provided 
>> patches are not complete.
> 
> The patches against 1.x are complete. If you want to test and fix 2.x
> branch or git master feel free to pull my patches when I push them to
> our github.
> 
> You can verify the patches with clang and SLES10. In the near future
> it's likely I'll even post prebuilt binaries of clang which could be
> used for easier validation. There's also of course the nightly EKOPath
> builds that are available.. etc etc
> 
> In parting - I will test LDFLAGS|CFLAGS=“-fgnu89-inline” and if it
> does indeed fix the issue without side effects I'll let you guys know.
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] C89 support

[r...@open-mpi.org]

Chris

For me, this is the critical point:


> On Aug 29, 2016, at 9:50 PM, Gilles Gouaillardet  wrote:
> 
> iirc, we use C99 struct initialisers, so stricly speaking, i do not think 
> Open MPI can be built with a pure C89 compiler when configure'd
> 
> with the --disable-c99 option. i'd rather imagine SLES10 clang and PathScale 
> compilers only implements 90% of the C99 standards,
> 
> and this PR avoids hitting the 10% that are not supported in this environment.


Can you please confirm Gilles’ “imagination”?
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

This has nothing to do with PMIx, Josh - the error is coming out of the usock 
OOB component.


> On Sep 14, 2016, at 7:17 AM, Joshua Ladd  wrote:
> 
> Eric,
> 
> We are looking into the PMIx code path that sets up the jobid. The session 
> directories are created based on the jobid. It might be the case that the 
> jobids (generated with rand) happen to be the same for different jobs 
> resulting in multiple jobs sharing the same session directory, but we need to 
> check. We will update.
> 
> Josh
> 
> On Wed, Sep 14, 2016 at 9:33 AM, Eric Chamberland 
> > 
> wrote:
> Lucky!
> 
> Since each runs have a specific TMP, I still have it on disc.
> 
> for the faulty run, the TMP variable was:
> 
> TMP=/tmp/tmp.wOv5dkNaSI
> 
> and into $TMP I have:
> 
> openmpi-sessions-40031@lorien_0
> 
> and into this subdirectory I have a bunch of empty dirs:
> 
> cmpbib@lorien:/tmp/tmp.wOv5dkNaSI/openmpi-sessions-40031@lorien_0> ls -la |wc 
> -l
> 1841
> 
> cmpbib@lorien:/tmp/tmp.wOv5dkNaSI/openmpi-sessions-40031@lorien_0> ls -la 
> |more
> total 68
> drwx-- 1840 cmpbib bib 45056 Sep 13 03:49 .
> drwx--3 cmpbib bib   231 Sep 13 03:50 ..
> drwx--2 cmpbib bib 6 Sep 13 02:10 10015
> drwx--2 cmpbib bib 6 Sep 13 03:05 10049
> drwx--2 cmpbib bib 6 Sep 13 03:15 10052
> drwx--2 cmpbib bib 6 Sep 13 02:22 10059
> drwx--2 cmpbib bib 6 Sep 13 02:22 10110
> drwx--2 cmpbib bib 6 Sep 13 02:41 10114
> ...
> 
> If I do:
> 
> lsof |grep "openmpi-sessions-40031"
> lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
>   Output information may be incomplete.
> lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
>   Output information may be incomplete.
> 
> nothing...
> 
> What else may I check?
> 
> Eric
> 
> 
> On 14/09/16 08:47 AM, Joshua Ladd wrote:
> Hi, Eric
> 
> I **think** this might be related to the following:
> 
> https://github.com/pmix/master/pull/145 
> 
> 
> I'm wondering if you can look into the /tmp directory and see if you
> have a bunch of stale usock files.
> 
> Best,
> 
> Josh
> 
> 
> On Wed, Sep 14, 2016 at 1:36 AM, Gilles Gouaillardet  
> >> wrote:
> 
> Eric,
> 
> 
> can you please provide more information on how your tests are launched ?
> 
> do you
> 
> mpirun -np 1 ./a.out
> 
> or do you simply
> 
> ./a.out
> 
> 
> do you use a batch manager ? if yes, which one ?
> 
> do you run one test per job ? or multiple tests per job ?
> 
> how are these tests launched ?
> 
> 
> do the test that crashes use MPI_Comm_spawn ?
> 
> i am surprised by the process name [[9325,5754],0], which suggests
> there MPI_Comm_spawn was called 5753 times (!)
> 
> 
> can you also run
> 
> hostname
> 
> on the 'lorien' host ?
> 
> if you configure'd Open MPI with --enable-debug, can you
> 
> export OMPI_MCA_plm_base_verbose 5
> 
> then run one test and post the logs ?
> 
> 
> from orte_plm_base_set_hnp_name(), "lorien" and pid 142766 should
> produce job family 5576 (but you get 9325)
> 
> the discrepancy could be explained by the use of a batch manager
> and/or a full hostname i am unaware of.
> 
> 
> orte_plm_base_set_hnp_name() generate a 16 bits job family from the
> (32 bits hash of the) hostname and the mpirun (32 bits ?) pid.
> 
> so strictly speaking, it is possible two jobs launched on the same
> node are assigned the same 16 bits job family.
> 
> 
> the easiest way to detect this could be to
> 
> - edit orte/mca/plm/base/plm_base_jobid.c
> 
> and replace
> 
> OPAL_OUTPUT_VERBOSE((5, orte_plm_base_framework.framework_output,
>  "plm:base:set_hnp_name: final jobfam %lu",
>  (unsigned long)jobfam));
> 
> with
> 
> OPAL_OUTPUT_VERBOSE((4, orte_plm_base_framework.framework_output,
>  "plm:base:set_hnp_name: final jobfam %lu",
>  (unsigned long)jobfam));
> 
> configure Open MPI with --enable-debug and rebuild
> 
> and then
> 
> export OMPI_MCA_plm_base_verbose=4
> 
> and run your tests.
> 
> 
> when the problem occurs, you will be able to check which pids
> produced the faulty jobfam, and that could hint to a conflict.
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> 
> On 9/14/2016 12:35 AM, Eric Chamberland wrote:
> 
> Hi,
> 
> It is the third time this happened into the last 10 days.
> 
> While running nighlty tests (~2200), we have one or two tests
> that fails at the very beginning with this strange error:
> 
> [lorien:142766] [[9325,5754],0] 

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

Many things are possible, given infinite time :-)

The issue with this notion lies in direct launch scenarios - i.e., when procs 
are launched directly by the RM and not via mpirun. In this case, there is 
nobody who can give us the session directory (well, until PMIx becomes 
universal), and so the apps must be able to generate a name that they all can 
know. Otherwise, we lose shared memory support because they can’t rendezvous.

However, that doesn’t seem to be the root problem here. I suspect there is a 
bug in the code that spawns the orted from the singleton, and subsequently 
parses the returned connection info. If you look at the error, you’ll see that 
both jobid’s have “zero” for their local jobid. This means that the two procs 
attempting to communicate both think they are daemons, which is impossible in 
this scenario.

So something garbled the string that the orted returns on startup to the 
singleton, and/or the singleton is parsing it incorrectly. IIRC, the singleton 
gets its name from that string, and so I expect it is getting the wrong name - 
and hence the error.

As you may recall, you made a change a little while back where we modified the 
code in ess/singleton to be a little less strict in its checking of that 
returned string. I wonder if that is biting us here? It wouldn’t fix the 
problem, but might generate a different error at a more obvious place.


> On Sep 14, 2016, at 8:00 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> is there any reason to use a session directory based on the jobid (or job 
> family) ?
> I mean, could we use mkstemp to generate a unique directory, and then 
> propagate the path via orted comm or the environment ?
> 
> Cheers,
> 
> Gilles
> 
> On Wednesday, September 14, 2016, r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
> wrote:
> This has nothing to do with PMIx, Josh - the error is coming out of the usock 
> OOB component.
> 
> 
>> On Sep 14, 2016, at 7:17 AM, Joshua Ladd <jladd.m...@gmail.com 
>> <javascript:_e(%7B%7D,'cvml','jladd.m...@gmail.com');>> wrote:
>> 
>> Eric,
>> 
>> We are looking into the PMIx code path that sets up the jobid. The session 
>> directories are created based on the jobid. It might be the case that the 
>> jobids (generated with rand) happen to be the same for different jobs 
>> resulting in multiple jobs sharing the same session directory, but we need 
>> to check. We will update.
>> 
>> Josh
>> 
>> On Wed, Sep 14, 2016 at 9:33 AM, Eric Chamberland 
>> <eric.chamberl...@giref.ulaval.ca 
>> <javascript:_e(%7B%7D,'cvml','eric.chamberl...@giref.ulaval.ca');>> wrote:
>> Lucky!
>> 
>> Since each runs have a specific TMP, I still have it on disc.
>> 
>> for the faulty run, the TMP variable was:
>> 
>> TMP=/tmp/tmp.wOv5dkNaSI
>> 
>> and into $TMP I have:
>> 
>> openmpi-sessions-40031@lorien_0
>> 
>> and into this subdirectory I have a bunch of empty dirs:
>> 
>> cmpbib@lorien:/tmp/tmp.wOv5dkNaSI/openmpi-sessions-40031@lorien_0> ls -la 
>> |wc -l
>> 1841
>> 
>> cmpbib@lorien:/tmp/tmp.wOv5dkNaSI/openmpi-sessions-40031@lorien_0> ls -la 
>> |more
>> total 68
>> drwx-- 1840 cmpbib bib 45056 Sep 13 03:49 .
>> drwx--3 cmpbib bib   231 Sep 13 03:50 ..
>> drwx--2 cmpbib bib 6 Sep 13 02:10 10015
>> drwx--2 cmpbib bib 6 Sep 13 03:05 10049
>> drwx--2 cmpbib bib 6 Sep 13 03:15 10052
>> drwx--2 cmpbib bib 6 Sep 13 02:22 10059
>> drwx--2 cmpbib bib 6 Sep 13 02:22 10110
>> drwx--2 cmpbib bib 6 Sep 13 02:41 10114
>> ...
>> 
>> If I do:
>> 
>> lsof |grep "openmpi-sessions-40031"
>> lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
>>   Output information may be incomplete.
>> lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
>>   Output information may be incomplete.
>> 
>> nothing...
>> 
>> What else may I check?
>> 
>> Eric
>> 
>> 
>> On 14/09/16 08:47 AM, Joshua Ladd wrote:
>> Hi, Eric
>> 
>> I **think** this might be related to the following:
>> 
>> https://github.com/pmix/master/pull/145 
>> <https://github.com/pmix/master/pull/145>
>> 
>> I'm wondering if you can look into the /tmp directory and see if you
>> have a bunch of stale usock files.
>> 
>> Best,
>> 
>> Josh
>> 
>> 
>> On Wed, Sep 14, 2016 at 1:36 AM, Gi

Re: [OMPI devel] Lots of new features rolled out on github.com today

[r...@open-mpi.org]

> On Sep 14, 2016, at 11:37 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> - Code reviews got better / more organized
> - Some project management tools now available
> - We can enforce the use of 2-factor authentication

Please don’t do that...

> 
> https://github.com/blog/2256-a-whole-new-github-universe-announcing-new-tools-forums-and-features
> 
> Sweet!
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Lots of new features rolled out on github.com today

[r...@open-mpi.org]

I’d want to _fully_ understand the implications before forcing something on 
everyone that might prove burdensome, especially when it “solves” a currently 
non-existent problem


> On Sep 14, 2016, at 11:43 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
> 
> On Sep 14, 2016, at 2:40 PM, r...@open-mpi.org wrote:
>> 
>>> - Code reviews got better / more organized
>>> - Some project management tools now available
>>> - We can enforce the use of 2-factor authentication
>> 
>> Please don’t do that...
> 
> Certainly wouldn't do the last one without talking it through with the 
> community first.
> 
> Is there a reason not to do it?  It's not 2-factor on every push -- it's 
> 2-factor for web logins only, gmail-style (periodically prompt for 2nd factor 
> on trusted machines, yadda yadda yadda)
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Lots of new features rolled out on github.com today

[r...@open-mpi.org]

The problem I hit, and the reason I’m pushing back, was that it required me to 
have a smart phone handy. Not everyone has a smart phone, nor do they always 
have it sitting next to them. In the case I hit, I was sitting somewhere that 
(a) had poor cell reception, and (b) didn’t have my cell phone next to me...and 
so Github refused to let me do something.

I’d rather not force that to be the only way we work until some evidence that 
we have a problem needing to be addressed.


> On Sep 14, 2016, at 11:59 AM, Pritchard Jr., Howard <howa...@lanl.gov> wrote:
> 
> Ralph,
> 
> I know with older versions of git you may have problems since you can’t use
> https.  I think with newer versions it will prompt not just for passed but
> also
> 2-factor.
> 
> That’s one problem I hit anyway when first enabling 2-factor.
> 
> Howard
> 
> -- 
> Howard Pritchard
> 
> HPC-DES
> Los Alamos National Laboratory
> 
> 
> 
> 
> 
> On 9/14/16, 12:53 PM, "devel on behalf of Jeff Squyres (jsquyres)"
> <devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com> wrote:
> 
>> Sure.  There's no rush at all; in fact, this is probably a decent topic
>> for our next face-to-face.
>> 
>> 
>>> On Sep 14, 2016, at 2:46 PM, r...@open-mpi.org wrote:
>>> 
>>> I’d want to _fully_ understand the implications before forcing
>>> something on everyone that might prove burdensome, especially when it
>>> “solves” a currently non-existent problem
>>> 
>>> 
>>>> On Sep 14, 2016, at 11:43 AM, Jeff Squyres (jsquyres)
>>>> <jsquy...@cisco.com> wrote:
>>>> 
>>>> On Sep 14, 2016, at 2:40 PM, r...@open-mpi.org wrote:
>>>>> 
>>>>>> - Code reviews got better / more organized
>>>>>> - Some project management tools now available
>>>>>> - We can enforce the use of 2-factor authentication
>>>>> 
>>>>> Please don’t do that...
>>>> 
>>>> Certainly wouldn't do the last one without talking it through with the
>>>> community first.
>>>> 
>>>> Is there a reason not to do it?  It's not 2-factor on every push --
>>>> it's 2-factor for web logins only, gmail-style (periodically prompt for
>>>> 2nd factor on trusted machines, yadda yadda yadda)
>>>> 
>>>> -- 
>>>> Jeff Squyres
>>>> jsquy...@cisco.com
>>>> For corporate legal information go to:
>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>> 
>>>> ___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> 
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] link issue on master with --disable-shared --enable-static --disable-dlopen

[r...@open-mpi.org]

I should think we could pass the disable-pdl-open option downward - can’t see 
any reason why not.

> On Sep 13, 2016, at 7:51 PM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> i configure'd Open MPI with
> 
> --disable-shared --enable-static --disable-dlopen
> 
> and i can no longer link a simple MPI or OpenSHMEM app
> 
> $ mpicc -g -O0 -o hello_c ../../src/ompi-master/examples/hello_c.c
> /usr/bin/ld: 
> /home/gilles/local/ompi-master-static/lib/libopen-pal.a(pdl_pdlopen_module.o):
>  undefined reference to symbol 'dlclose@@GLIBC_2.2.5'
> /usr/bin/ld: note: 'dlclose@@GLIBC_2.2.5' is defined in DSO /lib64/libdl.so.2 
> so try adding it to the linker command line
> /lib64/libdl.so.2: could not read symbols: Invalid operation
> collect2: error: ld returned 1 exit status
> 
> 
> a work around is to manually append -ldl to the mpicc command line
> 
> 
> dlopen (and other dl* symbols) are required by the pdlopen component of the 
> embedded pmix.
> 
> embedded pmix is configure'd with --disable-dlopen but not with 
> --disable-pdl-dlopen, so -ldl is required.
> 
> (fwiw, this is hidden on a system with infiniband libs since libibverbs.so do 
> depend on libdl.so)
> 
> 
> i am not sure of how that should be fixed :
> 
> 1) have Open MPI automatically add the --disable-pdl-open option to the 
> embedded pmix when Open MPI is configure'd with --disable-dlopen
> 
> 2) append -ldl to the wrappers unless both --disable-pdl-open was passed to 
> the embedded pmix (assuming there is a way to have Open MPI do that ...)
> 
> 
> Could someone please comment or fix that ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

I don’t understand this fascination with PMIx. PMIx didn’t calculate this jobid 
- OMPI did. Yes, it is in the opal/pmix layer, but it had -nothing- to do with 
PMIx.

So why do you want to continue to blame PMIx for this problem??


> On Sep 15, 2016, at 4:29 AM, Joshua Ladd  wrote:
> 
> Great catch, Gilles! Not much of a surprise though. 
> 
> Indeed, this issue has EVERYTHING to do with how PMIx is calculating the 
> jobid, which, in this case, results in hash collisions. ;-P
> 
> Josh
> 
> On Thursday, September 15, 2016, Gilles Gouaillardet  > wrote:
> Eric,
> 
> 
> a bug has been identified, and a patch is available at 
> https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1376.patch
>  
> 
> 
> the bug is specific to singleton mode (e.g. ./a.out vs mpirun -np 1 ./a.out), 
> so if applying a patch does not fit your test workflow,
> 
> it might be easier for you to update it and mpirun -np 1 ./a.out instead of 
> ./a.out
> 
> 
> basically, increasing verbosity runs some extra code, which include sprintf.
> so yes, it is possible to crash an app by increasing verbosity by running 
> into a bug that is hidden under normal operation.
> my intuition suggests this is quite unlikely ... if you can get a core file 
> and a backtrace, we will soon find out
> 
> 
> Cheers,
> 
> Gilles
> 
> 
> 
> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
> Ok,
> 
> one test segfaulted *but* I can't tell if it is the *same* bug because there 
> has been a segfault:
> 
> stderr:
> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>  
> 
>  
> 
> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
> NULL
> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
> 1366255883
> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
> [lorien:190552] [[53310,0],0] plm:base:receive start comm
> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
> ...
> ...
> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
> on the local node in file ess_singleton_module.c at line 573
> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
> on the local node in file ess_singleton_module.c at line 163
> *** An error occurred in MPI_Init_thread
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***and potentially your MPI job)
> [lorien:190306] Local abort before MPI_INIT completed completed successfully, 
> but am not able to aggregate error messages, and not able to guarantee that 
> all other processes were killed!
> 
> stdout:
> 
> -- 
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
> 
>   orte_ess_init failed
>   --> Returned value Unable to start a daemon on the local node (-127) 
> instead of ORTE_SUCCESS
> -- 
> -- 
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
> 
>   ompi_mpi_init: ompi_rte_init failed
>   --> Returned "Unable to start a daemon on the local node" (-127) instead of 
> "Success" (0)
> -- 
> 
> openmpi content of $TMP:
> 
> /tmp/tmp.GoQXICeyJl> ls -la
> total 1500
> drwx--3 cmpbib bib 250 Sep 14 13:34 .
> drwxrwxrwt  356 root   root  61440 Sep 14 13:45 ..
> ...
> drwx-- 1848 cmpbib bib   45056 Sep 14 13:34 
> openmpi-sessions-40031@lorien_0
> srw-rw-r--1 cmpbib bib   0 Sep 14 12:24 pmix-190552
> 
> cmpbib@lorien:/tmp/tmp.GoQXICeyJl/openmpi-sessions-40031@lorien_0> find . 
> -type f
> ./53310/contact.txt
> 
> cat 53310/contact.txt
> 3493724160.0;usock;tcp://132.203.7.36:54605 
> 190552
> 
> egrep 'jobfam|stop' */*/Cerr* ../BIBTV/*/*/*/Cerr*|grep 53310
> 

Re: [OMPI devel] OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

I don’t think a collision was the issue here. We were taking the 
mpirun-generated jobid and passing it thru the hash, thus creating an incorrect 
and invalid value. What I’m more surprised by is that it doesn’t -always- fail. 
Only thing I can figure is that, unlike with PMIx, the usock oob component 
doesn’t check the incoming identifier of the connecting proc to see if it is 
someone it knows. So unless you just happened to hash into a daemon jobid form, 
it would accept the connection (even though the name wasn’t correct).

I think this should fix the issue. Let’s wait and see


> On Sep 15, 2016, at 4:47 AM, Gilles Gouaillardet 
>  wrote:
> 
> I just realized i screwed up my test, and i was missing some relevant info...
> So on one hand, i fixed a bug in singleton,
> But on the other hand, i cannot tell whether a collision was involved in this 
> issue
> 
> Cheers,
> 
> Gilles
> 
> Joshua Ladd  wrote:
> Great catch, Gilles! Not much of a surprise though. 
> 
> Indeed, this issue has EVERYTHING to do with how PMIx is calculating the 
> jobid, which, in this case, results in hash collisions. ;-P
> 
> Josh
> 
> On Thursday, September 15, 2016, Gilles Gouaillardet  > wrote:
> Eric,
> 
> 
> a bug has been identified, and a patch is available at 
> https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1376.patch
>  
> 
> 
> the bug is specific to singleton mode (e.g. ./a.out vs mpirun -np 1 ./a.out), 
> so if applying a patch does not fit your test workflow,
> 
> it might be easier for you to update it and mpirun -np 1 ./a.out instead of 
> ./a.out
> 
> 
> basically, increasing verbosity runs some extra code, which include sprintf.
> so yes, it is possible to crash an app by increasing verbosity by running 
> into a bug that is hidden under normal operation.
> my intuition suggests this is quite unlikely ... if you can get a core file 
> and a backtrace, we will soon find out
> 
> 
> Cheers,
> 
> Gilles
> 
> 
> 
> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
> Ok,
> 
> one test segfaulted *but* I can't tell if it is the *same* bug because there 
> has been a segfault:
> 
> stderr:
> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>  
> 
>  
> 
> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
> NULL
> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
> 1366255883
> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
> [lorien:190552] [[53310,0],0] plm:base:receive start comm
> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
> ...
> ...
> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
> on the local node in file ess_singleton_module.c at line 573
> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
> on the local node in file ess_singleton_module.c at line 163
> *** An error occurred in MPI_Init_thread
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***and potentially your MPI job)
> [lorien:190306] Local abort before MPI_INIT completed completed successfully, 
> but am not able to aggregate error messages, and not able to guarantee that 
> all other processes were killed!
> 
> stdout:
> 
> -- 
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
> 
>   orte_ess_init failed
>   --> Returned value Unable to start a daemon on the local node (-127) 
> instead of ORTE_SUCCESS
> -- 
> -- 
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
> 
>   ompi_mpi_init: ompi_rte_init failed
>   --> Returned "Unable to start a daemon on the local node" (-127) instead of 
> "Success" (0)
> 

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

It’s okay - it was just confusing

This actually wound up having nothing to do with how the jobid is generated. 
The root cause of the problem was that we took an mpirun-generated jobid, and 
then mistakenly passed it back thru a hash function instead of just using it. 
So we hashed a perfectly good jobid.

What is puzzling is how it could ever have worked, yet the user said it only 
occasionally messed things up enough to cause breakage. You would think that 
hashing a valid jobid would create an unusable mess, but that doesn’t appear to 
be a definitive result.

 probably indicative of the weakness of the hash :-)


> On Sep 15, 2016, at 8:34 AM, Joshua Ladd <jladd.m...@gmail.com> wrote:
> 
> Ralph,
> 
> We love PMIx :). In this context, when I say PMIx, I am referring to the PMIx 
> framework in OMPI/OPAL, not the standalone PMIx library. Sorry that wasn't 
> clear.
> 
> Josh 
> 
> On Thu, Sep 15, 2016 at 10:07 AM, r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
> wrote:
> I don’t understand this fascination with PMIx. PMIx didn’t calculate this 
> jobid - OMPI did. Yes, it is in the opal/pmix layer, but it had -nothing- to 
> do with PMIx.
> 
> So why do you want to continue to blame PMIx for this problem??
> 
> 
>> On Sep 15, 2016, at 4:29 AM, Joshua Ladd <jladd.m...@gmail.com 
>> <mailto:jladd.m...@gmail.com>> wrote:
>> 
>> Great catch, Gilles! Not much of a surprise though. 
>> 
>> Indeed, this issue has EVERYTHING to do with how PMIx is calculating the 
>> jobid, which, in this case, results in hash collisions. ;-P
>> 
>> Josh
>> 
>> On Thursday, September 15, 2016, Gilles Gouaillardet <gil...@rist.or.jp 
>> <mailto:gil...@rist.or.jp>> wrote:
>> Eric,
>> 
>> 
>> a bug has been identified, and a patch is available at 
>> https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1376.patch
>>  
>> <https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1376.patch>
>> 
>> the bug is specific to singleton mode (e.g. ./a.out vs mpirun -np 1 
>> ./a.out), so if applying a patch does not fit your test workflow,
>> 
>> it might be easier for you to update it and mpirun -np 1 ./a.out instead of 
>> ./a.out
>> 
>> 
>> basically, increasing verbosity runs some extra code, which include sprintf.
>> so yes, it is possible to crash an app by increasing verbosity by running 
>> into a bug that is hidden under normal operation.
>> my intuition suggests this is quite unlikely ... if you can get a core file 
>> and a backtrace, we will soon find out
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> 
>> 
>> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
>> Ok,
>> 
>> one test segfaulted *but* I can't tell if it is the *same* bug because there 
>> has been a segfault:
>> 
>> stderr:
>> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>>  
>> <http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt>
>>  
>> 
>> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
>> NULL
>> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
>> 1366255883
>> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
>> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
>> [lorien:190552] [[53310,0],0] plm:base:receive start comm
>> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
>> ...
>> ...
>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
>> on the local node in file ess_singleton_module.c at line 573
>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
>> on the local node in file ess_singleton_module.c at line 163
>> *** An error occurred in MPI_Init_thread
>> *** on a NULL communicator
>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> ***and potentially your MPI job)
>> [lorien:190306] Local abort before MPI_INIT completed completed 
>> successfully, but am not able to aggregate error messages, and not able to 
>> guarantee that all other processes were killed!
>> 
>> stdout:
>> 
>> -- 
>> It looks like orte_init failed for some reason; your parallel process is
>> likely to abort.  There are many reasons that a parallel 

Re: [OMPI devel] toward a unique session directory

[r...@open-mpi.org]

Actually, you just use the envar that was previously cited on a different email 
thread: 

 if (NULL != getenv(OPAL_MCA_PREFIX"orte_launch")) {
/* you were launched by mpirun */
} else {
/* you were direct launched */
}

This is available from time of first instruction, so no worries as to when you 
look.


> On Sep 15, 2016, at 7:50 AM, Pritchard Jr., Howard <howa...@lanl.gov> wrote:
> 
> HI Gilles,
> 
> From what point in the job launch are you needed to determine whether
> or not the job was direct launched?
> 
> Howard
> 
> -- 
> Howard Pritchard
> 
> HPC-DES
> Los Alamos National Laboratory
> 
> 
> 
> 
> 
> On 9/15/16, 7:38 AM, "devel on behalf of Gilles Gouaillardet"
> <devel-boun...@lists.open-mpi.org on behalf of
> gilles.gouaillar...@gmail.com> wrote:
> 
>> Ralph,
>> 
>> that looks good to me.
>> 
>> can you please remind me how to test if an app was launched by
>> mpirun/orted or direct launched by the RM ?
>> 
>> right now, which direct launch method is supported ?
>> i am aware of srun (SLURM) and apron (CRAY), are there any other ?
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Thu, Sep 15, 2016 at 7:10 PM, r...@open-mpi.org <r...@open-mpi.org>
>> wrote:
>>> 
>>> On Sep 15, 2016, at 12:51 AM, Gilles Gouaillardet <gil...@rist.or.jp>
>>> wrote:
>>> 
>>> Ralph,
>>> 
>>> 
>>> my reply is in the text
>>> 
>>> 
>>> On 9/15/2016 11:11 AM, r...@open-mpi.org wrote:
>>> 
>>> If we are going to make a change, then let’s do it only once. Since we
>>> introduced PMIx and the concept of the string namespace, the plan has
>>> been
>>> to switch away from a numerical jobid and to the namespace. This
>>> eliminates
>>> the issue of the hash altogether. If we are going to make a disruptive
>>> change, then let’s do that one. Either way, this isn’t something that
>>> could
>>> go into the 2.x series. It is far too invasive, and would have to be
>>> delayed
>>> until a 3.x at the earliest.
>>> 
>>> got it !
>>> 
>>> Note that I am not yet convinced that is the issue here. We’ve had this
>>> hash
>>> for 12 years, and this is the first time someone has claimed to see a
>>> problem. That makes me very suspicious that the root cause isn’t what
>>> you
>>> are pursuing. This is only being reported for _singletons_, and that is
>>> a
>>> very unique code path. The only reason for launching the orted is to
>>> support
>>> PMIx operations such as notification and comm_spawn. If those aren’t
>>> being
>>> used, then we could use the “isolated” mode where the usock OOB isn’t
>>> even
>>> activated, thus eliminating the problem. This would be a much smaller
>>> “fix”
>>> and could potentially fit into 2.x
>>> 
>>> a bug has been identified and fixed, let's wait and see how things go
>>> 
>>> how can i use the isolated mode ?
>>> shall i simply
>>> export OMPI_MCA_pmix=isolated
>>> export OMPI_MCA_plm=isolated
>>> ?
>>> 
>>> out of curiosity, does "isolated" means we would not even need to fork
>>> the
>>> HNP ?
>>> 
>>> 
>>> Yes - that’s the idea. Simplify and make things faster. All you have to
>>> do
>>> is set OMPI_MCA_ess_singleton_isolated=1 on master, and I believe that
>>> code
>>> is in 2.x as well
>>> 
>>> 
>>> 
>>> FWIW: every organization I’ve worked with has an epilog script that
>>> blows
>>> away temp dirs. It isn’t the RM-based environment that is of concern -
>>> it’s
>>> the non-RM one where epilog scripts don’t exist that is the problem.
>>> 
>>> well, i was looking at this the other way around.
>>> if mpirun/orted creates the session directory with mkstemp(), then
>>> there is
>>> no more need to do any cleanup
>>> (as long as you do not run out of disk space)
>>> but with direct run, there is always a little risk that a previous
>>> session
>>> directory is used, hence the requirement for an epilogue.
>>> also, if the RM is configured to run one job at a time per a given node,
>>> epilog can be quite trivial.
>>> but if several jobs can run on a given node at the same time, epilog
>>> become
>>> less trivial
>>> 
>>> 
>>

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

Ah...I take that back. We changed this and now we _do_ indeed go down that code 
path. Not good.

So yes, we need that putenv so it gets the jobid from the HNP that was 
launched, like it used to do. You want to throw that in?

Thanks
Ralph

> On Sep 14, 2016, at 8:18 PM, r...@open-mpi.org wrote:
> 
> Nah, something isn’t right here. The singleton doesn’t go thru that code 
> line, or it isn’t supposed to do so. I think the problem lies in the way the 
> singleton in 2.x is starting up. Let me take a look at how singletons are 
> working over there.
> 
>> On Sep 14, 2016, at 8:10 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
>> 
>> Ralph,
>> 
>> 
>> i think i just found the root cause :-)
>> 
>> 
>> from pmix1_client_init() in opal/mca/pmix/pmix112/pmix1_client.c
>> 
>>  /* store our jobid and rank */
>>  if (NULL != getenv(OPAL_MCA_PREFIX"orte_launch")) {
>>   /* if we were launched by the OMPI RTE, then
>>* the jobid is in a special format - so get it */
>>   mca_pmix_pmix112_component.native_launch = true;
>>   opal_convert_string_to_jobid(, my_proc.nspace);
>>   } else {
>>   /* we were launched by someone else, so make the
>>* jobid just be the hash of the nspace */
>>   OPAL_HASH_STR(my_proc.nspace, pname.jobid);
>>   /* keep it from being negative */
>>   pname.jobid &= ~(0x8000);
>>   }
>> 
>> 
>> in this case, there is no OPAL_MCA_PREFIX"orte_launch" in the environment,
>> so we end up using a jobid that is a hash of the namespace
>> 
>> the initial error message was
>> [lorien:142766] [[9325,5754],0] usock_peer_recv_connect_ack: received 
>> unexpected process identifier [[9325,0],0] from [[5590,0],0]
>> 
>> and with little surprise
>> ((9325 << 16) + 1) and ((5590 << 16) + 1) have the same hash as calculated 
>> by OPAL_HASH_STR
>> 
>> 
>> i am now thinking the right fix is to simply
>> putenv(OPAL_MCA_PREFIX"orte_launch=1");
>> in ess/singleton
>> 
>> makes sense ?
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
>>> Ok,
>>> 
>>> one test segfaulted *but* I can't tell if it is the *same* bug because 
>>> there has been a segfault:
>>> 
>>> stderr:
>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>>>  
>>> 
>>> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
>>> NULL
>>> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
>>> 1366255883
>>> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
>>> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
>>> [lorien:190552] [[53310,0],0] plm:base:receive start comm
>>> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
>>> ...
>>> ...
>>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a 
>>> daemon on the local node in file ess_singleton_module.c at line 573
>>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a 
>>> daemon on the local node in file ess_singleton_module.c at line 163
>>> *** An error occurred in MPI_Init_thread
>>> *** on a NULL communicator
>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>>> ***and potentially your MPI job)
>>> [lorien:190306] Local abort before MPI_INIT completed completed 
>>> successfully, but am not able to aggregate error messages, and not able to 
>>> guarantee that all other processes were killed!
>>> 
>>> stdout:
>>> 
>>> -- 
>>> It looks like orte_init failed for some reason; your parallel process is
>>> likely to abort.  There are many reasons that a parallel process can
>>> fail during orte_init; some of which are due to configuration or
>>> environment problems.  This failure appears to be an internal failure;
>>> here's some additional information (which may only be relevant to an
>>> Open MPI developer):
>>> 
>>> orte_ess_init failed
>>> --> Returned value Unable to start a daemon on the local node (-127) 
>>> instead of ORTE_SUCCESS
>>> -- 
>>> -

Re: [OMPI devel] toward a unique session directory

[r...@open-mpi.org]

If we are going to make a change, then let’s do it only once. Since we 
introduced PMIx and the concept of the string namespace, the plan has been to 
switch away from a numerical jobid and to the namespace. This eliminates the 
issue of the hash altogether. If we are going to make a disruptive change, then 
let’s do that one. Either way, this isn’t something that could go into the 2.x 
series. It is far too invasive, and would have to be delayed until a 3.x at the 
earliest.

Note that I am not yet convinced that is the issue here. We’ve had this hash 
for 12 years, and this is the first time someone has claimed to see a problem. 
That makes me very suspicious that the root cause isn’t what you are pursuing. 
This is only being reported for _singletons_, and that is a very unique code 
path. The only reason for launching the orted is to support PMIx operations 
such as notification and comm_spawn. If those aren’t being used, then we could 
use the “isolated” mode where the usock OOB isn’t even activated, thus 
eliminating the problem. This would be a much smaller “fix” and could 
potentially fit into 2.x

FWIW: every organization I’ve worked with has an epilog script that blows away 
temp dirs. It isn’t the RM-based environment that is of concern - it’s the 
non-RM one where epilog scripts don’t exist that is the problem.


> On Sep 14, 2016, at 6:05 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Ralph,
> 
> On 9/15/2016 12:11 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> Many things are possible, given infinite time :-)
>> 
> i could not agree more :-D
>> The issue with this notion lies in direct launch scenarios - i.e., when 
>> procs are launched directly by the RM and not via mpirun. In this case, 
>> there is nobody who can give us the session directory (well, until PMIx 
>> becomes universal), and so the apps must be able to generate a name that 
>> they all can know. Otherwise, we lose shared memory support because they 
>> can’t rendezvous.
> thanks for the explanation,
> now let me rephrase that
> "a MPI task must be able to rebuild the path to the session directory, based 
> on the information it has when launched.
> if mpirun is used, we have several options to pass this option to the MPI 
> tasks.
> in case of direct run, this info is unlikely (PMIx is not universal (yet)) 
> passed by the batch manager, so we have to use what is available"
> 
> my concern is that, to keep things simple, session directory is based on the 
> Open MPI jobid, and since stepid is zero most of the time, jobid really means 
> job family
> which is stored on 16 bits.
> 
> in the case of mpirun, jobfam is a 16 bit hash of the hostname (reasonnable 
> sized string) and the mpirun pid (32 bits on Linux)
> if several mpirun are invoked on the same host at a given time, there is a 
> risk two distinct jobs are assigned the same jobfam (since we hash from 32 
> bits down to 16 bits).
> also, there is a risk the session directory already exists from a previous 
> job, with some/all files and unix sockets from a previous job, leading to 
> undefined behavior
> (an early crash if we are lucky, odd things otherwise).
> 
> in the case of direct run, i guess jobfam is a 16 bit hash of the jobid 
> passed by the RM, and once again, there is a risk of conflict and/or the 
> re-use of a previous session directory.
> 
> to me, the issue here is we are using the Open MPI jobfam in order to build 
> the session directory path
> instead, what if we
> 1) when mpirun, use a session directory created by mkstemp(), and pass it to 
> MPI tasks via the environment or retrieve it from orted/mpirun right after 
> the communication has been established.
> 2) for direct run, use a session directory based on the full jobid (which 
> might be a string or a number) as passed by the RM
> 
> in case of 1), there is no more risk of a hash conflict, or re-using a 
> previous session directory
> in case of 2), there is no more risk of a hash conflict, but there is still a 
> risk of re-using a session directory from a previous (e.g. terminated) job.
> that being said, once we document how the session directory is built from the 
> jobid, sysadmins will be able to write a RM epilog that do remove the session 
> directory.
> 
>  does that make sense ?
>> 
>> However, that doesn’t seem to be the root problem here. I suspect there is a 
>> bug in the code that spawns the orted from the singleton, and subsequently 
>> parses the returned connection info. If you look at the error, you’ll see 
>> that both jobid’s have “zero” for their local jobid. This means that the two 
>> procs attempting to communicate both think they are daemons, which is 
>> impossible in this s

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

Just in the FWIW category: the HNP used to send the singleton’s name down the 
pipe at startup, which eliminated the code line you identified. Now, we are 
pushing the name into the environment as a PMIx envar, and having the PMIx 
component pick it up. Roundabout way of getting it, and that’s what is causing 
the trouble.

> On Sep 14, 2016, at 8:26 PM, r...@open-mpi.org wrote:
> 
> Ah...I take that back. We changed this and now we _do_ indeed go down that 
> code path. Not good.
> 
> So yes, we need that putenv so it gets the jobid from the HNP that was 
> launched, like it used to do. You want to throw that in?
> 
> Thanks
> Ralph
> 
>> On Sep 14, 2016, at 8:18 PM, r...@open-mpi.org wrote:
>> 
>> Nah, something isn’t right here. The singleton doesn’t go thru that code 
>> line, or it isn’t supposed to do so. I think the problem lies in the way the 
>> singleton in 2.x is starting up. Let me take a look at how singletons are 
>> working over there.
>> 
>>> On Sep 14, 2016, at 8:10 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
>>> 
>>> Ralph,
>>> 
>>> 
>>> i think i just found the root cause :-)
>>> 
>>> 
>>> from pmix1_client_init() in opal/mca/pmix/pmix112/pmix1_client.c
>>> 
>>> /* store our jobid and rank */
>>> if (NULL != getenv(OPAL_MCA_PREFIX"orte_launch")) {
>>>  /* if we were launched by the OMPI RTE, then
>>>   * the jobid is in a special format - so get it */
>>>  mca_pmix_pmix112_component.native_launch = true;
>>>  opal_convert_string_to_jobid(, my_proc.nspace);
>>>  } else {
>>>  /* we were launched by someone else, so make the
>>>   * jobid just be the hash of the nspace */
>>>  OPAL_HASH_STR(my_proc.nspace, pname.jobid);
>>>  /* keep it from being negative */
>>>  pname.jobid &= ~(0x8000);
>>>  }
>>> 
>>> 
>>> in this case, there is no OPAL_MCA_PREFIX"orte_launch" in the environment,
>>> so we end up using a jobid that is a hash of the namespace
>>> 
>>> the initial error message was
>>> [lorien:142766] [[9325,5754],0] usock_peer_recv_connect_ack: received 
>>> unexpected process identifier [[9325,0],0] from [[5590,0],0]
>>> 
>>> and with little surprise
>>> ((9325 << 16) + 1) and ((5590 << 16) + 1) have the same hash as calculated 
>>> by OPAL_HASH_STR
>>> 
>>> 
>>> i am now thinking the right fix is to simply
>>> putenv(OPAL_MCA_PREFIX"orte_launch=1");
>>> in ess/singleton
>>> 
>>> makes sense ?
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
>>>> Ok,
>>>> 
>>>> one test segfaulted *but* I can't tell if it is the *same* bug because 
>>>> there has been a segfault:
>>>> 
>>>> stderr:
>>>> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>>>>  
>>>> 
>>>> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
>>>> NULL
>>>> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
>>>> 1366255883
>>>> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
>>>> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
>>>> [lorien:190552] [[53310,0],0] plm:base:receive start comm
>>>> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
>>>> ...
>>>> ...
>>>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a 
>>>> daemon on the local node in file ess_singleton_module.c at line 573
>>>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a 
>>>> daemon on the local node in file ess_singleton_module.c at line 163
>>>> *** An error occurred in MPI_Init_thread
>>>> *** on a NULL communicator
>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>>>> ***and potentially your MPI job)
>>>> [lorien:190306] Local abort before MPI_INIT completed completed 
>>>> successfully, but am not able to aggregate error messages, and not able to 
>>>> guarantee that all other processes were killed!
>>>> 
>>>> stdout:
>>>> 
>>>> -

Re: [OMPI devel] OpenMPI 2.x: bug: violent break at beginning with (sequential) runs...

[r...@open-mpi.org]

Nah, something isn’t right here. The singleton doesn’t go thru that code line, 
or it isn’t supposed to do so. I think the problem lies in the way the 
singleton in 2.x is starting up. Let me take a look at how singletons are 
working over there.

> On Sep 14, 2016, at 8:10 PM, Gilles Gouaillardet  wrote:
> 
> Ralph,
> 
> 
> i think i just found the root cause :-)
> 
> 
> from pmix1_client_init() in opal/mca/pmix/pmix112/pmix1_client.c
> 
>   /* store our jobid and rank */
>   if (NULL != getenv(OPAL_MCA_PREFIX"orte_launch")) {
>/* if we were launched by the OMPI RTE, then
> * the jobid is in a special format - so get it */
>mca_pmix_pmix112_component.native_launch = true;
>opal_convert_string_to_jobid(, my_proc.nspace);
>} else {
>/* we were launched by someone else, so make the
> * jobid just be the hash of the nspace */
>OPAL_HASH_STR(my_proc.nspace, pname.jobid);
>/* keep it from being negative */
>pname.jobid &= ~(0x8000);
>}
> 
> 
> in this case, there is no OPAL_MCA_PREFIX"orte_launch" in the environment,
> so we end up using a jobid that is a hash of the namespace
> 
> the initial error message was
> [lorien:142766] [[9325,5754],0] usock_peer_recv_connect_ack: received 
> unexpected process identifier [[9325,0],0] from [[5590,0],0]
> 
> and with little surprise
> ((9325 << 16) + 1) and ((5590 << 16) + 1) have the same hash as calculated by 
> OPAL_HASH_STR
> 
> 
> i am now thinking the right fix is to simply
> putenv(OPAL_MCA_PREFIX"orte_launch=1");
> in ess/singleton
> 
> makes sense ?
> 
> 
> Cheers,
> 
> Gilles
> 
> On 9/15/2016 2:58 AM, Eric Chamberland wrote:
>> Ok,
>> 
>> one test segfaulted *but* I can't tell if it is the *same* bug because there 
>> has been a segfault:
>> 
>> stderr:
>> http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.09.14.10h38m52s.faultyCerr.Triangle.h_cte_1.txt
>>  
>> 
>> [lorien:190552] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path 
>> NULL
>> [lorien:190552] plm:base:set_hnp_name: initial bias 190552 nodename hash 
>> 1366255883
>> [lorien:190552] plm:base:set_hnp_name: final jobfam 53310
>> [lorien:190552] [[53310,0],0] plm:rsh_setup on agent ssh : rsh path NULL
>> [lorien:190552] [[53310,0],0] plm:base:receive start comm
>> *** Error in `orted': realloc(): invalid next size: 0x01e58770 ***
>> ...
>> ...
>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
>> on the local node in file ess_singleton_module.c at line 573
>> [lorien:190306] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon 
>> on the local node in file ess_singleton_module.c at line 163
>> *** An error occurred in MPI_Init_thread
>> *** on a NULL communicator
>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> ***and potentially your MPI job)
>> [lorien:190306] Local abort before MPI_INIT completed completed 
>> successfully, but am not able to aggregate error messages, and not able to 
>> guarantee that all other processes were killed!
>> 
>> stdout:
>> 
>> -- 
>> It looks like orte_init failed for some reason; your parallel process is
>> likely to abort.  There are many reasons that a parallel process can
>> fail during orte_init; some of which are due to configuration or
>> environment problems.  This failure appears to be an internal failure;
>> here's some additional information (which may only be relevant to an
>> Open MPI developer):
>> 
>>  orte_ess_init failed
>>  --> Returned value Unable to start a daemon on the local node (-127) 
>> instead of ORTE_SUCCESS
>> -- 
>> -- 
>> It looks like MPI_INIT failed for some reason; your parallel process is
>> likely to abort.  There are many reasons that a parallel process can
>> fail during MPI_INIT; some of which are due to configuration or environment
>> problems.  This failure appears to be an internal failure; here's some
>> additional information (which may only be relevant to an Open MPI
>> developer):
>> 
>>  ompi_mpi_init: ompi_rte_init failed
>>  --> Returned "Unable to start a daemon on the local node" (-127) instead of 
>> "Success" (0)
>> -- 
>> 
>> openmpi content of $TMP:
>> 
>> /tmp/tmp.GoQXICeyJl> ls -la
>> total 1500
>> drwx--3 cmpbib bib 250 Sep 14 13:34 .
>> drwxrwxrwt  356 root   root  61440 Sep 14 13:45 ..
>> ...
>> drwx-- 1848 cmpbib bib   45056 Sep 14 13:34 
>> openmpi-sessions-40031@lorien_0
>> srw-rw-r--1 cmpbib bib   0 Sep 14 12:24 pmix-190552
>> 
>> cmpbib@lorien:/tmp/tmp.GoQXICeyJl/openmpi-sessions-40031@lorien_0> find . 
>> -type f
>> ./53310/contact.txt
>> 
>> cat 53310/contact.txt
>> 

Re: [OMPI devel] OMPI devel] RFC: Reenabling the TCP BTL over local interfaces (when specifically requested)

[r...@open-mpi.org]

FWIW: you know the location of every proc (to at least the host level) from 
time of orte_init, which should precede anything in the BTL

> On Sep 21, 2016, at 8:31 AM, Gilles Gouaillardet 
>  wrote:
> 
> George,
> 
> Is proc locality already set at that time ?
> 
> If yes, then we could keep a hard coded test so 127.x.y.z address (and IPv6 
> equivalent) are never used (even if included or not excluded) for inter node 
> communication
> 
> Cheers,
> 
> Gilles
> 
> "Jeff Squyres (jsquyres)"  wrote:
>> On Sep 21, 2016, at 10:56 AM, George Bosilca  wrote:
>>> 
>>> No, because 127.x.x.x is by default part of the exclude, so it will never 
>>> get into the modex. The problem today, is that even if you manually remove 
>>> it from the exclude and add it to the include, it will not work, because of 
>>> the hardcoded checks. Once we remove those checks, things will work the way 
>>> we expect, interfaces are removed because they don't match the provided 
>>> addresses.
>> 
>> Gotcha.
>> 
>>> I would have agreed with you if the current code was doing a better 
>>> decision of what is local and what not. But it is not, it simply remove all 
>>> 127.x.x.x interfaces (opal/util/net.c:222). Thus, the only thing the 
>>> current code does, is preventing a power-user from using the loopback 
>>> (despite being explicitly enabled via the corresponding MCA parameters).
>> 
>> Fair enough.
>> 
>> Should we have a keyword that can be used in the btl_tcp_if_include/exclude 
>> (e.g., "local") that removes all local-only interfaces?  I.E., all 
>> 127.x.x.x/8 interfaces *and* all local-only interfaces (e.g., bridging 
>> interfaces to local VMs and the like)?
>> 
>> We could then replace the default "127.0.0.0/8" value in btl_tcp_if_exclude 
>> with this token, and therefore actually exclude the VM-only interfaces 
>> (which have caused some users problems in the past).
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] toward a unique session directory

[r...@open-mpi.org]

> On Sep 15, 2016, at 12:51 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Ralph,
> 
> 
> 
> my reply is in the text
> 
> 
> On 9/15/2016 11:11 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> If we are going to make a change, then let’s do it only once. Since we 
>> introduced PMIx and the concept of the string namespace, the plan has been 
>> to switch away from a numerical jobid and to the namespace. This eliminates 
>> the issue of the hash altogether. If we are going to make a disruptive 
>> change, then let’s do that one. Either way, this isn’t something that could 
>> go into the 2.x series. It is far too invasive, and would have to be delayed 
>> until a 3.x at the earliest.
>> 
> got it !
>> Note that I am not yet convinced that is the issue here. We’ve had this hash 
>> for 12 years, and this is the first time someone has claimed to see a 
>> problem. That makes me very suspicious that the root cause isn’t what you 
>> are pursuing. This is only being reported for _singletons_, and that is a 
>> very unique code path. The only reason for launching the orted is to support 
>> PMIx operations such as notification and comm_spawn. If those aren’t being 
>> used, then we could use the “isolated” mode where the usock OOB isn’t even 
>> activated, thus eliminating the problem. This would be a much smaller “fix” 
>> and could potentially fit into 2.x
>> 
> a bug has been identified and fixed, let's wait and see how things go
> 
> how can i use the isolated mode ?
> shall i simply
> export OMPI_MCA_pmix=isolated
> export OMPI_MCA_plm=isolated
> ?
> 
> out of curiosity, does "isolated" means we would not even need to fork the 
> HNP ?

Yes - that’s the idea. Simplify and make things faster. All you have to do is 
set OMPI_MCA_ess_singleton_isolated=1 on master, and I believe that code is in 
2.x as well

> 
> 
>> FWIW: every organization I’ve worked with has an epilog script that blows 
>> away temp dirs. It isn’t the RM-based environment that is of concern - it’s 
>> the non-RM one where epilog scripts don’t exist that is the problem.
> well, i was looking at this the other way around.
> if mpirun/orted creates the session directory with mkstemp(), then there is 
> no more need to do any cleanup
> (as long as you do not run out of disk space)
> but with direct run, there is always a little risk that a previous session 
> directory is used, hence the requirement for an epilogue.
> also, if the RM is configured to run one job at a time per a given node, 
> epilog can be quite trivial.
> but if several jobs can run on a given node at the same time, epilog become 
> less trivial

Yeah, this session directory thing has always been problematic. We’ve had 
litter problems since day one, and tried multiple solutions over the years. 
Obviously, none of those has proven fully successful :-(

Artem came up with a good solution using PMIx that allows the RM to control the 
session directory location for both direct launch and mpirun launch, thus 
ensuring the RM can cleanup the correct place upon session termination. As we 
get better adoption of that method out there, then the RM-based solution (even 
for multiple jobs sharing a node) should be resolved.

This leaves the non-RM (i.e., ssh-based launch using mpirun) problem. Your 
proposal would resolve that one as (a) we always have orted’s in that scenario, 
and (b) the orted’s pass the session directory down to the apps. So maybe the 
right approach is to use mkstemp() in the scenario where we are launched via 
orted and the RM has not specified a session directory.

I’m not sure we can resolve the direct launch without PMIx problem - I think 
that’s best left as another incentive for RMs to get on-board the PMIx bus.

Make sense?


> 
> 
> Cheers,
> 
> Gilles
>> 
>>> On Sep 14, 2016, at 6:05 PM, Gilles Gouaillardet <gil...@rist.or.jp 
>>> <mailto:gil...@rist.or.jp>> wrote:
>>> 
>>> Ralph,
>>> 
>>> 
>>> On 9/15/2016 12:11 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>>>> Many things are possible, given infinite time :-)
>>>> 
>>> i could not agree more :-D
>>>> The issue with this notion lies in direct launch scenarios - i.e., when 
>>>> procs are launched directly by the RM and not via mpirun. In this case, 
>>>> there is nobody who can give us the session directory (well, until PMIx 
>>>> becomes universal), and so the apps must be able to generate a name that 
>>>> they all can know. Otherwise, we lose shared memory support because they 
>>>> can’t rendezvous.
>>> thanks for 

Re: [OMPI devel] Sample of merging ompi and ompi-release

[r...@open-mpi.org]

One question, to be discussed on the webex: now that github has a “reviewed” 
feature, so we still need/want the “thumbs-up” bot? If we retain it, then how 
do we deal with the non-sync’d, duplicative mechanisms?


> On Sep 19, 2016, at 4:23 PM, George Bosilca  wrote:
> 
> :+1:
> 
>   George.
> 
> 
> On Mon, Sep 19, 2016 at 6:56 PM, Jeff Squyres (jsquyres)  > wrote:
> (we can discuss all of this on the Webex tomorrow)
> 
> Here's a sample repo where I merged ompi and ompi-release:
> 
> https://github.com/open-mpi/ompi-all-the-branches 
> 
> 
> Please compare it to:
> 
> https://github.com/open-mpi/ompi 
> and https://github.com/open-mpi/ompi-release 
> 
> 
> It's current to as of within the last hour or so.
> 
> Feel free to make dummy commits / pull requests on this rep.  It's a sandbox 
> repo that will eventually be deleted; it's safe to make whatever changes you 
> want on this repo.
> 
> Notes:
> 
> - All current OMPI developers have been given the same push/merge access on 
> master
> - Force pushes are disabled on *all* branches
> - On release branches:
> - Pull requests cannot be merged without at least 1 review
>   *** ^^^ This is a new Github feature
> - Only the gatekeeper team can merge PRs
> 
> If no one sees any problem with this sandbox repo, I can merge all the 
> ompi-release branches to the ompi repo as soon as tomorrow, and config the 
> same settings.
> 
> --
> Jeff Squyres
> jsquy...@cisco.com 
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/ 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Error in hwloc configury

[r...@open-mpi.org]

Good job, Gilles! PR coming - this fixed the problem.

> On Sep 22, 2016, at 6:52 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> if you have libevent in your CPPFLAGS and you want to use the embedded
> one, then  please use the patch below
> pmix3x looks ok
> 
> Cheers,
> 
> Gilles
> 
> diff --git a/opal/mca/event/libevent2022/configure.m4
> b/opal/mca/event/libevent2022/configure.m4
> 
> index d299b5f..8124c11 100644
> 
> --- a/opal/mca/event/libevent2022/configure.m4
> 
> +++ b/opal/mca/event/libevent2022/configure.m4
> 
> @@ -75,9 +75,9 @@ EOF
> 
># Add some stuff to CPPFLAGS so that the rest of the source
> 
># tree can be built
> 
>libevent_file=$libevent_basedir/libevent
> 
> -   CPPFLAGS="$CPPFLAGS -I$OPAL_TOP_SRCDIR/$libevent_file
> -I$OPAL_TOP_SRCDIR/$libevent_file/include"
> 
>AS_IF([test "$OPAL_TOP_BUILDDIR" != "$OPAL_TOP_SRCDIR"],
> 
> - [CPPFLAGS="$CPPFLAGS
> -I$OPAL_TOP_BUILDDIR/$libevent_file/include"])
> 
> +
> [CPPFLAGS="-I$OPAL_TOP_BUILDDIR/$libevent_file/include $CPPFLAGS"])
> 
> +   CPPFLAGS="-I$OPAL_TOP_SRCDIR/$libevent_file
> -I$OPAL_TOP_SRCDIR/$libevent_file/include $CPPFLAGS"
> 
>unset libevent_file
> 
>   ])
> 
> ])
> 
> diff --git a/opal/mca/hwloc/hwloc1113/configure.m4
> b/opal/mca/hwloc/hwloc1113/configure.m4
> 
> index 7fe3527..c1646d4 100644
> 
> --- a/opal/mca/hwloc/hwloc1113/configure.m4
> 
> +++ b/opal/mca/hwloc/hwloc1113/configure.m4
> 
> @@ -52,9 +52,9 @@ AC_DEFUN([MCA_opal_hwloc_hwloc1113_POST_CONFIG],[
> 
># Add some stuff to CPPFLAGS so that the rest of the source
> 
># tree can be built
> 
>file=$opal_hwloc_hwloc1113_basedir/hwloc
> 
> -   CPPFLAGS="$CPPFLAGS -I$OPAL_TOP_SRCDIR/$file/include"
> 
>AS_IF([test "$OPAL_TOP_BUILDDIR" != "$OPAL_TOP_SRCDIR"],
> 
> - [CPPFLAGS="$CPPFLAGS -I$OPAL_TOP_BUILDDIR/$file/include"])
> 
> + [CPPFLAGS="-I$OPAL_TOP_BUILDDIR/$file/include $CPPFLAGS"])
> 
> +   CPPFLAGS="-I$OPAL_TOP_SRCDIR/$file/include $CPPFLAGS"
> 
>unset file
> 
>   ])
> 
> OPAL_VAR_SCOPE_POP
> 
> diff --git a/opal/mca/hwloc/hwloc1113/hwloc/config/hwloc.m4
> b/opal/mca/hwloc/hwloc1113/hwloc/config/hwloc.m4
> 
> index 6807624..3f6a6fe 100644
> 
> --- a/opal/mca/hwloc/hwloc1113/hwloc/config/hwloc.m4
> 
> +++ b/opal/mca/hwloc/hwloc1113/hwloc/config/hwloc.m4
> 
> @@ -988,7 +988,7 @@ EOF])
> 
> if test "x$enable_cpuid" != "xno"; then
> 
>AC_MSG_CHECKING([for x86 cpuid])
> 
>old_CPPFLAGS="$CPPFLAGS"
> 
> -   CPPFLAGS="$CPPFLAGS -I$HWLOC_top_srcdir/include"
> 
> +   CPPFLAGS="-I$HWLOC_top_srcdir/include $CPPFLAGS"
> 
># We need hwloc_uint64_t but we can't use
> hwloc/autogen/config.h before configure ends.
> 
># So pass #include/#define manually here for now.
> 
>CPUID_CHECK_HEADERS=
> 
> 
> On Thu, Sep 22, 2016 at 10:39 PM, Gilles Gouaillardet
> <gilles.gouaillar...@gmail.com> wrote:
>> Ralph,
>> 
>> do you use VPATH ?
>> 
>> can you give a try to the patch below ?
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> diff --git a/opal/mca/hwloc/hwloc1113/configure.m4
>> b/opal/mca/hwloc/hwloc1113/configure.m4
>> 
>> index 7fe3527..c1646d4 100644
>> 
>> --- a/opal/mca/hwloc/hwloc1113/configure.m4
>> 
>> +++ b/opal/mca/hwloc/hwloc1113/configure.m4
>> 
>> @@ -52,9 +52,9 @@ AC_DEFUN([MCA_opal_hwloc_hwloc1113_POST_CONFIG],[
>> 
>># Add some stuff to CPPFLAGS so that the rest of the source
>> 
>># tree can be built
>> 
>>file=$opal_hwloc_hwloc1113_basedir/hwloc
>> 
>> -   CPPFLAGS="$CPPFLAGS -I$OPAL_TOP_SRCDIR/$file/include"
>> 
>>AS_IF([test "$OPAL_TOP_BUILDDIR" != "$OPAL_TOP_SRCDIR"],
>> 
>> - [CPPFLAGS="$CPPFLAGS -I$OPAL_TOP_BUILDDIR/$file/include"])
>> 
>> + [CPPFLAGS="-I$OPAL_TOP_BUILDDIR/$file/include $CPPFLAGS"])
>> 
>> +   CPPFLAGS="-I$OPAL_TOP_SRCDIR/$file/include $CPPFLAGS"
>> 
>>unset file
>> 
>>   ])
>> 
>> OPAL_VAR_SCOPE_POP
>> 
>&

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. v2.x-dev-2911-gc7bf9a0

[r...@open-mpi.org]

Hey Gilles

This fix doesn’t look right to me. 

> +/* we read something - better get more */
> +num_chars_read += rc;
> +orted_uri = realloc((void*)orted_uri, buffer_length+chunk);
> +memset(_uri[buffer_length], 0, chunk);
> +buffer_length += chunk;
>}

Just because you read “something”, it doesn’t mean that you have to realloc the 
buffer. You might have read only a small piece, not an entire “chunk”.

What you need to do is to see if the buffer has been filled, and only read each 
time up to the remaining number of characters available in the buffer. If the 
buffer has been filled (or if you want to optimize, if the remaining space 
falls under some minimum limit), then you realloc another ORTE_URI_MSG_LNGTH 
block.


> On Sep 21, 2016, at 10:25 PM, git...@open-mpi.org wrote:
> 
> This is an automated email from the git hooks/post-receive script. It was
> generated because a ref change was pushed to the repository containing
> the project "open-mpi/ompi".
> 
> The branch, master has been updated
>  via  c7bf9a0ec9940d6fe0ff6da5212fc881de73cf21 (commit)
> from  e1d89a4dcf5feeaf978e6806b08d609ecfb9889a (commit)
> 
> Those revisions listed above that are new to this repository have
> not appeared on any other notification email; so we list those
> revisions in full, below.
> 
> - Log -
> https://github.com/open-mpi/ompi/commit/c7bf9a0ec9940d6fe0ff6da5212fc881de73cf21
> 
> commit c7bf9a0ec9940d6fe0ff6da5212fc881de73cf21
> Author: Gilles Gouaillardet 
> Date:   Thu Sep 22 13:18:54 2016 +0900
> 
>   ess/singleton: fix read on the pipe to spawn'ed orted
> 
>   and close the pipe on both ends when it is no more needed
> 
> diff --git a/orte/mca/ess/singleton/ess_singleton_module.c 
> b/orte/mca/ess/singleton/ess_singleton_module.c
> index 571d453..55c7985 100644
> --- a/orte/mca/ess/singleton/ess_singleton_module.c
> +++ b/orte/mca/ess/singleton/ess_singleton_module.c
> @@ -571,20 +571,20 @@ static int fork_hnp(void)
>orted_uri = (char*)malloc(buffer_length);
>memset(orted_uri, 0, buffer_length);
> 
> -while (chunk == (rc = read(p[0], _uri[num_chars_read], 
> chunk))) {
> +while (0 != (rc = read(p[0], _uri[num_chars_read], chunk))) {
>if (rc < 0 && (EAGAIN == errno || EINTR == errno)) {
>continue;
> -} else {
> -num_chars_read = 0;
> +} else if (rc < 0) {
> +num_chars_read = -1;
>break;
>}
> -/* we read an entire buffer - better get more */
> -num_chars_read += chunk;
> -orted_uri = realloc((void*)orted_uri, 
> buffer_length+ORTE_URI_MSG_LGTH);
> -memset(_uri[buffer_length], 0, ORTE_URI_MSG_LGTH);
> -buffer_length += ORTE_URI_MSG_LGTH;
> +/* we read something - better get more */
> +num_chars_read += rc;
> +orted_uri = realloc((void*)orted_uri, buffer_length+chunk);
> +memset(_uri[buffer_length], 0, chunk);
> +buffer_length += chunk;
>}
> -num_chars_read += rc;
> +close(p[0]);
> 
>if (num_chars_read <= 0) {
>/* we didn't get anything back - this is bad */
> diff --git a/orte/orted/orted_main.c b/orte/orted/orted_main.c
> index 3c7bc63..0e7c318 100644
> --- a/orte/orted/orted_main.c
> +++ b/orte/orted/orted_main.c
> @@ -628,6 +628,7 @@ int orte_daemon(int argc, char *argv[])
> 
>/* cleanup */
>free(tmp);
> +close(orted_globals.uri_pipe);
> 
>/* since a singleton spawned us, we need to harvest
> * any MCA params from the local environment so
> 
> 
> ---
> 
> Summary of changes:
> orte/mca/ess/singleton/ess_singleton_module.c |   18 +-
> orte/orted/orted_main.c   |1 +
> 2 files changed, 10 insertions(+), 9 deletions(-)
> 
> 
> hooks/post-receive
> -- 
> open-mpi/ompi
> ___
> ompi-commits mailing list
> ompi-comm...@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/ompi-commits

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Error in hwloc configury

[r...@open-mpi.org]

Hey folks

I’m encountering an issue with the way we detect external HWLOC. If I have a 
directory that includes an hwloc installation in my CPPFLAGS, then we fail to 
build, even if I don’t specify anything with regard to hwloc on my configure 
cmd line. The errors I get look like:

In file included from sec_basic.c:11:0:
../../../../opal/include/opal_config.h:1348:0: note: this is the location of 
the previous definition
 #define HWLOC_SYM_PREFIX opal_hwloc1113_
 ^
In file included from 
../../../../opal/mca/hwloc/hwloc1113/hwloc/include/hwloc.h:53:0,
 from ../../../../opal/mca/hwloc/hwloc1113/hwloc1113.h:24,
 from ../../../../opal/mca/hwloc/hwloc.h:131,
 from ../../../../opal/util/proc.h:21,
 from ../../../../opal/mca/sec/sec.h:25,
 from ../../../../opal/mca/sec/base/base.h:23,
 from sec_basic.c:22:
/Users/rhc/local/include/hwloc/autogen/config.h:200:0: warning: 
"HWLOC_SYM_PREFIX_CAPS" redefined
 #define HWLOC_SYM_PREFIX_CAPS HWLOC_
 ^
In file included from sec_basic.c:11:0:
../../../../opal/include/opal_config.h:1351:0: note: this is the location of 
the previous definition
 #define HWLOC_SYM_PREFIX_CAPS OPAL_HWLOC1113_
 ^
Undefined symbols for architecture x86_64:
  "_hwloc_bitmap_alloc", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_and", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_copy", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_first", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_free", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_intersects", referenced from:
  import-atom in libopen-pal.dylib
  "_hwloc_bitmap_isincluded", referenced from:
...

Lots and lots of repetitions of the warning from many different sources, and 
it’s clear that we somehow picked up the external header and went haywire. This 
isn’t what we intended to have happen - we are supposed to ignore external 
installations unless directed to use them.

Is this the expected behavior? Any way we can clean this up?
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Error in hwloc configury

[r...@open-mpi.org]

Here is the compile line:

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../opal/include 
-I../../ompi/include -I../../oshmem/include 
-I../../opal/mca/hwloc/hwloc1113/hwloc/include/private/autogen 
-I../../opal/mca/hwloc/hwloc1113/hwloc/include/hwloc/autogen 
-I../../ompi/mpiext/cuda/c -I../.. -I../../orte/include 
-I/Users/rhc/local/include 
-I/Users/rhc/openmpi/foobar/opal/mca/hwloc/hwloc1113/hwloc/include 
-I/Users/rhc/openmpi/foobar/opal/mca/event/libevent2022/libevent 
-I/Users/rhc/openmpi/foobar/opal/mca/event/libevent2022/libevent/include 
-L/Users/rhc/local/lib -g -Wall -Wundef -Wno-long-long -Wsign-compare 
-Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic 
-Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing 
-mcx16 -MT proc.lo -MD -MP -MF .deps/proc.Tpo -c proc.c  -fno-common -DPIC -o 
.libs/proc.o
depbase=`echo qsort.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H 
-I. -I../../opal/include -I../../ompi/include -I../../oshmem/include 
-I../../opal/mca/hwloc/hwloc1113/hwloc/include/private/autogen 
-I../../opal/mca/hwloc/hwloc1113/hwloc/include/hwloc/autogen 
-I../../ompi/mpiext/cuda/c   -I../.. -I../../orte/include 
-I/Users/rhc/local/include  
-I/Users/rhc/openmpi/foobar/opal/mca/hwloc/hwloc1113/hwloc/include 
-I/Users/rhc/openmpi/foobar/opal/mca/event/libevent2022/libevent 
-I/Users/rhc/openmpi/foobar/opal/mca/event/libevent2022/libevent/include  
-L/Users/rhc/local/lib -g -Wall -Wundef -Wno-long-long -Wsign-compare 
-Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic 
-Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing 
-mcx16  -MT qsort.lo -MD -MP -MF $depbase.Tpo -c -o qsort.lo qsort.c &&\


It looks to me like my CPPFLAGS are stuck in the middle of the whole mess - I 
wonder if that’s the problem? I see it sandwiched in-between flags for 
different levels of hwloc redirection


> On Sep 22, 2016, at 4:40 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> Is the root cause we append our stuff to CPPFLAGS, instead of prepend ?
> 
> You can retrieve the compile command line with
> make V=1
> 
> If my guess is correct, does someone know the rationale for append vs prepend 
> ?
> 
> Cheers,
> 
> Gilles
> 
> r...@open-mpi.org wrote:
> Hey folks
> 
> I’m encountering an issue with the way we detect external HWLOC. If I have a 
> directory that includes an hwloc installation in my CPPFLAGS, then we fail to 
> build, even if I don’t specify anything with regard to hwloc on my configure 
> cmd line. The errors I get look like:
> 
> In file included from sec_basic.c:11:0:
> ../../../../opal/include/opal_config.h:1348:0: note: this is the location of 
> the previous definition
>  #define HWLOC_SYM_PREFIX opal_hwloc1113_
>  ^
> In file included from 
> ../../../../opal/mca/hwloc/hwloc1113/hwloc/include/hwloc.h:53:0,
>  from ../../../../opal/mca/hwloc/hwloc1113/hwloc1113.h:24,
>  from ../../../../opal/mca/hwloc/hwloc.h:131,
>  from ../../../../opal/util/proc.h:21,
>  from ../../../../opal/mca/sec/sec.h:25,
>  from ../../../../opal/mca/sec/base/base.h:23,
>  from sec_basic.c:22:
> /Users/rhc/local/include/hwloc/autogen/config.h:200:0: warning: 
> "HWLOC_SYM_PREFIX_CAPS" redefined
>  #define HWLOC_SYM_PREFIX_CAPS HWLOC_
>  ^
> In file included from sec_basic.c:11:0:
> ../../../../opal/include/opal_config.h:1351:0: note: this is the location of 
> the previous definition
>  #define HWLOC_SYM_PREFIX_CAPS OPAL_HWLOC1113_
>  ^
> Undefined symbols for architecture x86_64:
>   "_hwloc_bitmap_alloc", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_and", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_copy", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_first", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_free", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_intersects", referenced from:
>   import-atom in libopen-pal.dylib
>   "_hwloc_bitmap_isincluded", referenced from:
> ...
> 
> Lots and lots of repetitions of the warning from many different sources, and 
> it’s clear that we somehow picked up the external header and went haywire. 
> This isn’t what we intended to have happen - we are supposed to ignore 
> external installations unless directed to use them.
> 
> Is this the expected behavior? Any way we can clean this up?
> Ralph
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [MTT devel] reporter error using pyclient

[r...@open-mpi.org]

Okay, I cleaned things up some more and got a little bit further - now hitting 
an error in the server?

<<<<<<< Payload (End  ) -->>>>>>
INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: 
mtt.open-mpi.org
DEBUG:requests.packages.urllib3.connectionpool:"POST /submit/cpy/api/submit 
HTTP/1.1" 500 2535
<<<<<<< Response -->>>>>>
Result: 500: text/html;charset=utf-8
{'content-length': '2535', 'set-cookie': 
'session_id=1b3f3c3df893e673e072430844afe27b4389be71; expires=Sat, 27 Aug 2016 
03:31:28 GMT; Path=/', 'server': 'CherryPy/5.1.0', 'connection': 'close', 
'allow': 'POST', 'date': 'Sat, 27 Aug 2016 02:31:28 GMT', 
'access-control-allow-origin': '*', 'content-type': 'text/html;charset=utf-8'}
Internal Server Error
<<<<<<< Raw Output (Start) >>>>>>
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;>



500 Internal Server Error

#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}

#traceback {
color: red;
}



500 Internal Server Error
The server encountered an unexpected condition which prevented it 
from fulfilling the request.
Traceback (most recent call last):
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cprequest.py",
 line 670, in respond
response.body = self.handler()
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/encoding.py",
 line 217, in __call__
self.body = self.oldhandler(*args, **kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/jsontools.py",
 line 63, in json_handler
value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cpdispatch.py",
 line 60, in __call__
return self.callable(*self.args, **self.kwargs)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/dispatchers.py",
 line 333, in POST
value = self._db.insert_mpi_install(submit_info['submit_id'], 
data['metadata'], entry)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py",
 line 641, in insert_mpi_install
fields, values)
  File 
"/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py",
 line 183, in _select_insert
cursor.execute( select_stmt, values )
ProgrammingError: operator does not exist: text = text[]
LINE 3: ... merge_stdout_stderr = E'false' AND result_stderr = ARRAY[E'...
 ^
HINT:  No operator matches the given name and argument type(s). You might need 
to add explicit type casts.



  
Powered by http://www.cherrypy.org;>CherryPy 5.1.0
  




<<<<<<< Raw Output (End  ) >>>>>>



> On Aug 26, 2016, at 2:01 PM, r...@open-mpi.org wrote:
> 
> That appears to resolve the connection issue:
> 
> <<<<<<< Payload (End  ) -->>>>>>
> INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: 
> mtt.open-mpi.org <http://mtt.open-mpi.org/>
> DEBUG:requests.packages.urllib3.connectionpool:"POST /submit/cpy/api/submit 
> HTTP/1.1" 200 90
> <<<<<<< Response -->>>>>>
> Result: 200: application/json
> {'content-length': '90', 'set-cookie': 
> 'session_id=df54fbc661acdd2cdda5e5ee74de3b62e1fac8e5; expires=Fri, 26 Aug 
> 2016 22:00:12 GMT; Path=/', 'server': 'CherryPy/5.1.0', 'connection': 
> 'close', 'allow': 'POST', 'date': 'Fri, 26 Aug 2016 21:00:12 GMT', 
> 'access-control-allow-origin': '*', 'content-type': 'application/json'}
> OK
> <<<<<<< Raw Output (Start) >>>>>>
> {"status": -2, "status_message": "[DB PG V3] (mpi_install)  Missing field: 
> compiler_name"}
> <<<<<<<---- Raw Output (End  ) ---->>>>>>
> 
> 
> Looks like we are still missing some fields, though...
> 
> 
>> On Aug 26, 2016, at 1:36 PM, Josh Hursey <jjhur...@open-mpi.org 
>> <mailto:jjhur...@open-mpi.org>> 

Re: [MTT devel] reporter error using pyclient

[r...@open-mpi.org]

Hmmm...Hey Josh - is it possible that your server is requiring an MPI_Install 
phase? We don’t have one since we just build and then set the path accordingly 
- is it complaining about missing data for an install phase?


> On Aug 26, 2016, at 7:33 PM, r...@open-mpi.org wrote:
> 
> Okay, I cleaned things up some more and got a little bit further - now 
> hitting an error in the server?
> 
> <<<<<<< Payload (End  ) -->>>>>>
> INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: 
> mtt.open-mpi.org <http://mtt.open-mpi.org/>
> DEBUG:requests.packages.urllib3.connectionpool:"POST /submit/cpy/api/submit 
> HTTP/1.1" 500 2535
> <<<<<<< Response -->>>>>>
> Result: 500: text/html;charset=utf-8
> {'content-length': '2535', 'set-cookie': 
> 'session_id=1b3f3c3df893e673e072430844afe27b4389be71; expires=Sat, 27 Aug 
> 2016 03:31:28 GMT; Path=/', 'server': 'CherryPy/5.1.0', 'connection': 
> 'close', 'allow': 'POST', 'date': 'Sat, 27 Aug 2016 02:31:28 GMT', 
> 'access-control-allow-origin': '*', 'content-type': 'text/html;charset=utf-8'}
> Internal Server Error
> <<<<<<< Raw Output (Start) >>>>>>
>  "-//W3C//DTD XHTML 1.0 Transitional//EN"
> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd 
> <http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd>">
> 
> 
> 
> 500 Internal Server Error
> 
> #powered_by {
> margin-top: 20px;
> border-top: 2px solid black;
> font-style: italic;
> }
> 
> #traceback {
> color: red;
> }
> 
> 
> 
> 500 Internal Server Error
> The server encountered an unexpected condition which prevented it 
> from fulfilling the request.
> Traceback (most recent call last):
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cprequest.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cprequest.py>",
>  line 670, in respond
> response.body = self.handler()
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/encoding.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/encoding.py>",
>  line 217, in __call__
> self.body = self.oldhandler(*args, **kwargs)
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/jsontools.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/lib/jsontools.py>",
>  line 63, in json_handler
> value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cpdispatch.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/env/lib/python2.6/site-packages/cherrypy/_cpdispatch.py>",
>  line 60, in __call__
> return self.callable(*self.args, **self.kwargs)
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/dispatchers.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/dispatchers.py>",
>  line 333, in POST
> value = self._db.insert_mpi_install(submit_info['submit_id'], 
> data['metadata'], entry)
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py>",
>  line 641, in insert_mpi_install
> fields, values)
>   File 
> "/nfs/data/osl/www/mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py
>  
> <http://mtt.open-mpi.org/mtt-modern/server/php/cherrypy/src/webapp/db_pgv3.py>",
>  line 183, in _select_insert
> cursor.execute( select_stmt, values )
> ProgrammingError: operator does not exist: text = text[]
> LINE 3: ... merge_stdout_stderr = E'false' AND result_stderr = ARRAY[E'...
>  ^
> HINT:  No operator matches the given name and argument type(s). You might 
> need to add explicit type casts.
> 
> 
> 
>   
> Powered by http://www.cherrypy.org 
> <http:/

Re: [MTT devel] reporter error using pyclient

[r...@open-mpi.org]

BTW: here is my .ini snippet


[Reporter:IUdatabase]
plugin = IUDatabase

realm = OMPI
username = intel
pwfile = /home/common/mttpwd.txt
platform = bend-rsh
hostname = rhc00[1-2]
url = https://mtt.open-mpi.org/submit/cpy/
email = r...@open-mpi.org


So it looks like the CherryPi server is adding a /submit to the end, and that 
might be the issue?

> On Aug 26, 2016, at 12:07 PM, r...@open-mpi.org wrote:
> 
> Even though I can get there with the browser, I do still hit this error:
> 
> <<<<<<< Raw Output (Start) >>>>>>
> 
> 
> 404 Not Found
> 
> Not Found
> The requested URL /submit/cpy//submit was not found on this server.
> 
> Apache/2.2.15 (Red Hat) Server at mtt.open-mpi.org 
> <http://mtt.open-mpi.org/> Port 443
> 
> 
> <<<<<<< Raw Output (End  ) ---->>>>>>
> 
> 
> It looks to me like the URL isn’t correctly set - yes?
> 
>> On Aug 26, 2016, at 7:10 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> I checked it with my browser and got the expected return message
>> 
>> 
>>> On Aug 25, 2016, at 1:46 PM, Josh Hursey <jjhur...@open-mpi.org 
>>> <mailto:jjhur...@open-mpi.org>> wrote:
>>> 
>>> Can you send me the portion of the ini script that you are using?
>>> 
>>> Can you access this site via the browser (you will need your login 
>>> credentials):
>>>   https://mtt.open-mpi.org/submit/cpy/api/ 
>>> <https://mtt.open-mpi.org/submit/cpy/api/>
>>> It should return:
>>>  {"status": 0, "status_message": "Success"}
>>> 
>>> 
>>> On Thu, Aug 25, 2016 at 3:01 PM, Howard Pritchard <hpprit...@gmail.com 
>>> <mailto:hpprit...@gmail.com>> wrote:
>>> HI Josh,
>>> 
>>> That doesn't seem to help:
>>> 
>>> <<<<<<< Payload (End  ) -->>>>>>
>>> 
>>> INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection 
>>> (1): mtt.open-mpi.org <http://mtt.open-mpi.org/>
>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315:
>>>  SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject 
>>> Name Indication) extension to TLS is not available on this platform. This 
>>> may cause the server to present an incorrect TLS certificate, which can 
>>> cause validation failures. For more information, see 
>>> https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning 
>>> <https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning>.
>>> 
>>>   SNIMissingWarning
>>> 
>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120:
>>>  InsecurePlatformWarning: A true SSLContext object is not available. This 
>>> prevents urllib3 from configuring SSL appropriately and may cause certain 
>>> SSL connections to fail. For more information, see 
>>> https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
>>>  
>>> <https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning>.
>>> 
>>>   InsecurePlatformWarning
>>> 
>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:791:
>>>  InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
>>> certificate verification is strongly advised. See: 
>>> https://urllib3.readthedocs.org/en/latest/security.html 
>>> <https://urllib3.readthedocs.org/en/latest/security.html>
>>>   InsecureRequestWarning)
>>> 
>>> DEBUG:requests.packages.urllib3.connectionpool:"POST /submit/cpy//serial 
>>> HTTP/1.1" 404 300
>>> 
>>> <<<<<<< Response -->>>>>>
>>> 
>>> Result: 404: text/html; charset=iso-8859-1
>>> 
>>> {'Date': 'Thu, 25 Aug 2016 19:58:41 GMT', 'Content-Length': '300', 
>>> 'Content-Type': 'text/html; charset=iso-8859-1', 'Connection': 'close', 
>>> 'Server': 'Apache/2.2.15 (Red Hat)'}
>>> 
>>> Not Found
>>> 
>>> <<<<<<< Raw Output (Start) >>>>>>
>>> 
>>> 
>>> 
>>> 
&g

Re: [MTT devel] reporter error using pyclient

[r...@open-mpi.org]

FWIW: the extra “/“ is inserted in the IUDatabase reporter plugin. Removing it 
didn’t make any difference

Must be something on the server side, I fear


> On Aug 26, 2016, at 12:08 PM, r...@open-mpi.org wrote:
> 
> BTW: here is my .ini snippet
> 
> 
> [Reporter:IUdatabase]
> plugin = IUDatabase
> 
> realm = OMPI
> username = intel
> pwfile = /home/common/mttpwd.txt
> platform = bend-rsh
> hostname = rhc00[1-2]
> url = https://mtt.open-mpi.org/submit/cpy/ 
> <https://mtt.open-mpi.org/submit/cpy/>
> email = r...@open-mpi.org <mailto:r...@open-mpi.org>
> 
> 
> So it looks like the CherryPi server is adding a /submit to the end, and that 
> might be the issue?
> 
>> On Aug 26, 2016, at 12:07 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> Even though I can get there with the browser, I do still hit this error:
>> 
>> <<<<<<< Raw Output (Start) >>>>>>
>> 
>> 
>> 404 Not Found
>> 
>> Not Found
>> The requested URL /submit/cpy//submit was not found on this server.
>> 
>> Apache/2.2.15 (Red Hat) Server at mtt.open-mpi.org 
>> <http://mtt.open-mpi.org/> Port 443
>> 
>> 
>> <<<<<<< Raw Output (End  ) >>>>>>
>> 
>> 
>> It looks to me like the URL isn’t correctly set - yes?
>> 
>>> On Aug 26, 2016, at 7:10 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>> wrote:
>>> 
>>> I checked it with my browser and got the expected return message
>>> 
>>> 
>>>> On Aug 25, 2016, at 1:46 PM, Josh Hursey <jjhur...@open-mpi.org 
>>>> <mailto:jjhur...@open-mpi.org>> wrote:
>>>> 
>>>> Can you send me the portion of the ini script that you are using?
>>>> 
>>>> Can you access this site via the browser (you will need your login 
>>>> credentials):
>>>>   https://mtt.open-mpi.org/submit/cpy/api/ 
>>>> <https://mtt.open-mpi.org/submit/cpy/api/>
>>>> It should return:
>>>>  {"status": 0, "status_message": "Success"}
>>>> 
>>>> 
>>>> On Thu, Aug 25, 2016 at 3:01 PM, Howard Pritchard <hpprit...@gmail.com 
>>>> <mailto:hpprit...@gmail.com>> wrote:
>>>> HI Josh,
>>>> 
>>>> That doesn't seem to help:
>>>> 
>>>> <<<<<<< Payload (End  ) -->>>>>>
>>>> 
>>>> INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS 
>>>> connection (1): mtt.open-mpi.org <http://mtt.open-mpi.org/>
>>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315:
>>>>  SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject 
>>>> Name Indication) extension to TLS is not available on this platform. This 
>>>> may cause the server to present an incorrect TLS certificate, which can 
>>>> cause validation failures. For more information, see 
>>>> https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning 
>>>> <https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning>.
>>>> 
>>>>   SNIMissingWarning
>>>> 
>>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120:
>>>>  InsecurePlatformWarning: A true SSLContext object is not available. This 
>>>> prevents urllib3 from configuring SSL appropriately and may cause certain 
>>>> SSL connections to fail. For more information, see 
>>>> https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
>>>>  
>>>> <https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning>.
>>>> 
>>>>   InsecurePlatformWarning
>>>> 
>>>> /Users/hpp/.virtualenvs/mtt_py3/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:791:
>>>>  InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
>>>> certificate verification is strongly advised. See: 
>>>> https://urllib3.readthedocs.org/en/latest/security.html 
>>>> <https://urllib3.readthedocs.org/en/latest/security.html>
>>>>   InsecureRequestWarning)
>>>> 
>>>> DEBUG:requests.packages.u

Re: [OMPI devel] stdin issue with master

[r...@open-mpi.org]

Yeah, I started working on it earlier this evening - will look some more 
tomorrow

> On Aug 22, 2016, at 7:57 PM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> i made a trivial test
> 
> 
> echo hello | mpirun -np 1 cat
> 
> 
> and with v2.x and v1.10, the output is "hello" as expected
> 
> but with master, there is no output at all (!)
> 
> 
> 
> i was able to fix that with the dirty workaround below.
> 
> 
> the root cause (on master) is that orte_cmd_options.stdin_target is NULL.
> 
> fwiw, i could not see such option in the output of ompi_info --all
> 
> 
> as far as i understand, such option is registered in 
> orte/mca/schizo/ompi/schizo_ompi.c and/or orte/orted/orted_main.c
> 
> i tried various things, but was unable to have this option registered.
> 
> 
> can someone more familiar with this part please have a look ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> diff --git a/orte/orted/orted_submit.c b/orte/orted/orted_submit.c
> index ec5ad04..db2d93d 100644
> --- a/orte/orted/orted_submit.c
> +++ b/orte/orted/orted_submit.c
> @@ -756,6 +756,9 @@ int orte_submit_job(char *argv[], int *index,
> 
> 
> /* check what user wants us to do with stdin */
> +if (NULL == orte_cmd_options.stdin_target) {
> +orte_cmd_options.stdin_target = "0";
> +}
> if (NULL != orte_cmd_options.stdin_target) {
> if (0 == strcmp(orte_cmd_options.stdin_target, "all")) {
> jdata->stdin_target = ORTE_VPID_WILDCARD;
> @@ -1062,6 +1065,8 @@ static int init_globals(void)
> orte_cmd_options.preload_binaries = false;
> orte_cmd_options.preload_files  = NULL;
> 
> +orte_cmd_options.stdin_target = "0";
> +
> /* All done */
> return ORTE_SUCCESS;
> }
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] stdin issue with master

[r...@open-mpi.org]

Fixed in 9210230

> On Aug 22, 2016, at 8:49 PM, r...@open-mpi.org wrote:
> 
> Yeah, I started working on it earlier this evening - will look some more 
> tomorrow
> 
>> On Aug 22, 2016, at 7:57 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
>> 
>> Folks,
>> 
>> 
>> i made a trivial test
>> 
>> 
>> echo hello | mpirun -np 1 cat
>> 
>> 
>> and with v2.x and v1.10, the output is "hello" as expected
>> 
>> but with master, there is no output at all (!)
>> 
>> 
>> 
>> i was able to fix that with the dirty workaround below.
>> 
>> 
>> the root cause (on master) is that orte_cmd_options.stdin_target is NULL.
>> 
>> fwiw, i could not see such option in the output of ompi_info --all
>> 
>> 
>> as far as i understand, such option is registered in 
>> orte/mca/schizo/ompi/schizo_ompi.c and/or orte/orted/orted_main.c
>> 
>> i tried various things, but was unable to have this option registered.
>> 
>> 
>> can someone more familiar with this part please have a look ?
>> 
>> 
>> Cheers,
>> 
>> 
>> Gilles
>> 
>> 
>> diff --git a/orte/orted/orted_submit.c b/orte/orted/orted_submit.c
>> index ec5ad04..db2d93d 100644
>> --- a/orte/orted/orted_submit.c
>> +++ b/orte/orted/orted_submit.c
>> @@ -756,6 +756,9 @@ int orte_submit_job(char *argv[], int *index,
>> 
>> 
>>/* check what user wants us to do with stdin */
>> +if (NULL == orte_cmd_options.stdin_target) {
>> +orte_cmd_options.stdin_target = "0";
>> +}
>>if (NULL != orte_cmd_options.stdin_target) {
>>if (0 == strcmp(orte_cmd_options.stdin_target, "all")) {
>>jdata->stdin_target = ORTE_VPID_WILDCARD;
>> @@ -1062,6 +1065,8 @@ static int init_globals(void)
>>orte_cmd_options.preload_binaries = false;
>>orte_cmd_options.preload_files  = NULL;
>> 
>> +orte_cmd_options.stdin_target = "0";
>> +
>>/* All done */
>>return ORTE_SUCCESS;
>> }
>> 
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] runtime failure on MacOS 10.6

[r...@open-mpi.org]

Hmmm...okay. I guess we’ll replace that call for portability, then. Thx!

> On Aug 22, 2016, at 9:23 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
> 
> I was using /usr/bin/gcc:
>i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646)
> 
> A quick looks suggests that there is no strnlen() on this system.
> It doesn't appear in any header below /usr/include/
> 
> FWIW: strnlen() first appears in IEEE Std 1003.1-2008 (aka POSIX.1-2008) and 
> so could be absent on any system claiming conformance to an earlier revision.
> It was in glic prior to appearing in POSIX.1 and so might be on most any 
> Linux system regardless of age.
> 
> -Paul
> 
> On Mon, Aug 22, 2016 at 9:17 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Hey Paul
> 
> I just checked on my Mac and had no problem. However, I’m at 10.11, and so 
> I’m wondering if the old 10.6 just doesn’t have strnlen on it?
> 
> What compiler were you using?
> 
>> On Aug 22, 2016, at 9:14 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> Huh - I’ll take a look. Thanks!
>> 
>>> On Aug 22, 2016, at 9:11 PM, Paul Hargrove <phhargr...@lbl.gov 
>>> <mailto:phhargr...@lbl.gov>> wrote:
>>> 
>>> On a Mac OSX 10.6 system:
>>> 
>>> $ mpirun -mca btl sm,self -np 2 examples/ring_c'
>>> dyld: lazy symbol binding failed: Symbol not found: _strnlen
>>>   Referenced from: 
>>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>>   Expected in: flat namespace
>>> 
>>> dyld: Symbol not found: _strnlen
>>>   Referenced from: 
>>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>>   Expected in: flat namespace
>>> 
>>> Let me know what additional information is desired and to whom to send.
>>> 
>>> -Paul
>>> 
>>> 
>>> 
>>> -- 
>>> Paul H. Hargrove  phhargr...@lbl.gov 
>>> <mailto:phhargr...@lbl.gov>
>>> Computer Languages & Systems Software (CLaSS) Group
>>> Computer Science Department   Tel: +1-510-495-2352 
>>> <tel:%2B1-510-495-2352>
>>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 
>>> <tel:%2B1-510-486-6900>___
>>> devel mailing list
>>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
> 
> 
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov 
> <mailto:phhargr...@lbl.gov>
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] C89 support

[r...@open-mpi.org]

Just so people don’t spend a lot of time on this: as the release manager for 
the 1.10 series, you are going to have to provide me with a great deal of 
motivation to accept this proposed change. We ended C89 support way back in the 
1.7 series, so reviving it here would really seem odd.

I haven’t yet seen anything on this thread that convinces me to accept it.

> On Aug 29, 2016, at 8:22 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> On Aug 29, 2016, at 11:06 AM, C Bergström  wrote:
>> 
>> If the patches are performance impacting I would never burden
>> upstream, but I do hope that regardless you'll consider them. Based on
>> the patch for 1.x it seems cosmetic. I'll take the most honest and
>> unbiased look at the patches against 2.x and master to see if I feel
>> guilty for asking for review.
> 
> We've used a lot more C99 in master/v2.x (i.e., since we forked for v1.7).  
> It would be a much, much harder sell to remove all the C99 from there.
> 
> Also, if SLES 10 is EOL, that also somewhat detracts from the desire to add a 
> bunch of engineering work to support a 27-year-old version of C.
> 
> As it is, I am surprised that your patches are so small for v1.10 -- that 
> can't possibly remove all the C99 stuff from the entire code base.  Are you 
> are only selectively removing *some* of the C99 from the parts of Open MPI 
> that you are compiling that make it work on your compiler?  If so, that's a 
> bit more of an oddball case: i.e., you're not proposing strict C89 adherence 
> across the entire code base.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] C89 support

[r...@open-mpi.org]

Just to clarify: we primarily use c99 features in our plugins as a means of 
directly specifying which functions are being implemented, and which are not. 
In c89, this can only be done by maintaining positional alignment - c99 allows 
us to do this using the function names. Thus, the c99 method is much more 
maintainable, which was a very large factor in our decision. It wasn’t just 
cosmetic.

We have not found anyone raising an issue with this decision in the four years 
since it was implemented. This includes installations running legacy Fortran 
and C codes, all of which have updated their OMPI installations. So I don’t 
think it is quite the dire situation you are implying.


> On Aug 29, 2016, at 8:32 AM, C Bergström  wrote:
> 
> On Mon, Aug 29, 2016 at 11:22 PM, Jeff Squyres (jsquyres)
> > wrote:
>> On Aug 29, 2016, at 11:06 AM, C Bergström  wrote:
>>> 
>>> If the patches are performance impacting I would never burden
>>> upstream, but I do hope that regardless you'll consider them. Based on
>>> the patch for 1.x it seems cosmetic. I'll take the most honest and
>>> unbiased look at the patches against 2.x and master to see if I feel
>>> guilty for asking for review.
>> 
>> We've used a lot more C99 in master/v2.x (i.e., since we forked for v1.7).  
>> It would be a much, much harder sell to remove all the C99 from there.
>> 
>> Also, if SLES 10 is EOL, that also somewhat detracts from the desire to add 
>> a bunch of engineering work to support a 27-year-old version of C.
>> 
>> As it is, I am surprised that your patches are so small for v1.10 -- that 
>> can't possibly remove all the C99 stuff from the entire code base.  Are you 
>> are only selectively removing *some* of the C99 from the parts of Open MPI 
>> that you are compiling that make it work on your compiler?  If so, that's a 
>> bit more of an oddball case: i.e., you're not proposing strict C89 adherence 
>> across the entire code base.
> 
> I'm not intentionally doing anything oddball. I am however testing
> with clang and our compiler. If upstream clang is doing anything
> oddball (which I'd be a bit surprised) then we probably follow along.
> 
> The features you're using in c99 appear to be cosmetic candy and
> non-performance impacting. 27-year-old standards and older are quite
> frequently still in use for HPC. *cough* Fortran *cough*...
> ---
> Based on the latest response - it seems that we'll just fork OMPI and
> maintain those patches on top. I'll advise our customers not to use
> OMPI and document why.
> 
> Thanks again
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] runtime failure on MacOS 10.6

[r...@open-mpi.org]

Hey Paul

I just checked on my Mac and had no problem. However, I’m at 10.11, and so I’m 
wondering if the old 10.6 just doesn’t have strnlen on it?

What compiler were you using?

> On Aug 22, 2016, at 9:14 PM, r...@open-mpi.org wrote:
> 
> Huh - I’ll take a look. Thanks!
> 
>> On Aug 22, 2016, at 9:11 PM, Paul Hargrove <phhargr...@lbl.gov 
>> <mailto:phhargr...@lbl.gov>> wrote:
>> 
>> On a Mac OSX 10.6 system:
>> 
>> $ mpirun -mca btl sm,self -np 2 examples/ring_c'
>> dyld: lazy symbol binding failed: Symbol not found: _strnlen
>>   Referenced from: 
>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>   Expected in: flat namespace
>> 
>> dyld: Symbol not found: _strnlen
>>   Referenced from: 
>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>   Expected in: flat namespace
>> 
>> Let me know what additional information is desired and to whom to send.
>> 
>> -Paul
>> 
>> 
>> 
>> -- 
>> Paul H. Hargrove  phhargr...@lbl.gov 
>> <mailto:phhargr...@lbl.gov>
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] runtime failure on MacOS 10.6

[r...@open-mpi.org]

Actually, I found that we already dealt with this, but the version in the 2.0.1 
branch didn’t include the update. I’ll see what else is missing and ask that it 
be brought across.

Thanks Paul
Ralph

> On Aug 22, 2016, at 9:25 PM, r...@open-mpi.org wrote:
> 
> Hmmm...okay. I guess we’ll replace that call for portability, then. Thx!
> 
>> On Aug 22, 2016, at 9:23 PM, Paul Hargrove <phhargr...@lbl.gov 
>> <mailto:phhargr...@lbl.gov>> wrote:
>> 
>> I was using /usr/bin/gcc:
>>i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646)
>> 
>> A quick looks suggests that there is no strnlen() on this system.
>> It doesn't appear in any header below /usr/include/
>> 
>> FWIW: strnlen() first appears in IEEE Std 1003.1-2008 (aka POSIX.1-2008) and 
>> so could be absent on any system claiming conformance to an earlier revision.
>> It was in glic prior to appearing in POSIX.1 and so might be on most any 
>> Linux system regardless of age.
>> 
>> -Paul
>> 
>> On Mon, Aug 22, 2016 at 9:17 PM, r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
>> wrote:
>> Hey Paul
>> 
>> I just checked on my Mac and had no problem. However, I’m at 10.11, and so 
>> I’m wondering if the old 10.6 just doesn’t have strnlen on it?
>> 
>> What compiler were you using?
>> 
>>> On Aug 22, 2016, at 9:14 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>> wrote:
>>> 
>>> Huh - I’ll take a look. Thanks!
>>> 
>>>> On Aug 22, 2016, at 9:11 PM, Paul Hargrove <phhargr...@lbl.gov 
>>>> <mailto:phhargr...@lbl.gov>> wrote:
>>>> 
>>>> On a Mac OSX 10.6 system:
>>>> 
>>>> $ mpirun -mca btl sm,self -np 2 examples/ring_c'
>>>> dyld: lazy symbol binding failed: Symbol not found: _strnlen
>>>>   Referenced from: 
>>>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>>>   Expected in: flat namespace
>>>> 
>>>> dyld: Symbol not found: _strnlen
>>>>   Referenced from: 
>>>> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>>>>   Expected in: flat namespace
>>>> 
>>>> Let me know what additional information is desired and to whom to send.
>>>> 
>>>> -Paul
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Paul H. Hargrove  phhargr...@lbl.gov 
>>>> <mailto:phhargr...@lbl.gov>
>>>> Computer Languages & Systems Software (CLaSS) Group
>>>> Computer Science Department   Tel: +1-510-495-2352 
>>>> <tel:%2B1-510-495-2352>
>>>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 
>>>> <tel:%2B1-510-486-6900>___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>>>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>
>> 
>> 
>> 
>> -- 
>> Paul H. Hargrove  phhargr...@lbl.gov 
>> <mailto:phhargr...@lbl.gov>
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] Solaris MPIX failure

[r...@open-mpi.org]

I took a quick glance at this one, and the only way I can see to get that error 
is from this block of code:

#if defined(HAVE_STRUCT_UCRED_UID)
euid = ucred.uid;
gid = ucred.gid;
#else
euid = ucred.cr_uid;
gid = ucred.cr_gid;
#endif

#elif defined(HAVE_GETPEEREID)
pmix_output_verbose(2, pmix_globals.debug_output,
"sec:native checking getpeereid for peer credentials");
if (0 != getpeereid(peer->sd, , )) {
pmix_output_verbose(2, pmix_globals.debug_output,
"sec: getsockopt getpeereid failed: %s",
strerror (pmix_socket_errno));
return PMIX_ERR_INVALID_CRED;
}
#else
return PMIX_ERR_NOT_SUPPORTED;
#endif


I can only surmise, therefore, that Solaris doesn’t pass either of the two #if 
define’d tests. Is there a Solaris alternative?


> On Aug 23, 2016, at 5:55 AM, r...@open-mpi.org wrote:
> 
> Thanks Gilles!
> 
>> On Aug 23, 2016, at 3:42 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>> 
>> Thanks Paul,
>> 
>> at first glance, something is going wrong in the sec module under solaris.
>> I will keep digging tomorrow 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Tuesday, August 23, 2016, Paul Hargrove <phhargr...@lbl.gov 
>> <mailto:phhargr...@lbl.gov>> wrote:
>> On Solaris 11.3 on x86-64:
>> 
>> $ mpirun -mca btl sm,self,openib -np 2 -host pcp-d-3,pcp-d-4 examples/ring_c'
>> [pcp-d-4:25075] PMIX ERROR: NOT-SUPPORTED in file 
>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
>>  at line 529
>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>  at line 983
>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>  at line 199
>> --
>> It looks like MPI_INIT failed for some reason; your parallel process is
>> likely to abort.  There are many reasons that a parallel process can
>> fail during MPI_INIT; some of which are due to configuration or environment
>> problems.  This failure appears to be an internal failure; here's some
>> additional information (which may only be relevant to an Open MPI
>> developer):
>> 
>>   ompi_mpi_init: ompi_rte_init failed
>>   --> Returned "(null)" (-43) instead of "Success" (0)
>> --
>> *** An error occurred in MPI_Init
>> *** on a NULL communicator
>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> ***and potentially your MPI job)
>> [pcp-d-4:25078] Local abort before MPI_INIT completed completed 
>> successfully, but am not able to aggregate error messages, and not able to 
>> guarantee that all other processes were killed!
>> ---
>> Primary job  terminated normally, but 1 process returned
>> a non-zero exit code.. Per user-direction, the job has been aborted.
>> ---
>> --
>> mpirun detected that one or more processes exited with non-zero status, thus 
>> causing
>> the job to be terminated. The first process to do so was:
>> 
>>   Process name: [[25599,1],1]
>>   Exit code:1
>> --
>> 
>> -Paul
>> 
>> -- 
>> Paul H. Hargrove  phhargr...@lbl.gov 
>> <javascript:_e(%7B%7D,'cvml','phhargr...@lbl.gov');>
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] runtime failure on MacOS 10.6

[r...@open-mpi.org]

Huh - I’ll take a look. Thanks!

> On Aug 22, 2016, at 9:11 PM, Paul Hargrove  wrote:
> 
> On a Mac OSX 10.6 system:
> 
> $ mpirun -mca btl sm,self -np 2 examples/ring_c'
> dyld: lazy symbol binding failed: Symbol not found: _strnlen
>   Referenced from: 
> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>   Expected in: flat namespace
> 
> dyld: Symbol not found: _strnlen
>   Referenced from: 
> /Users/paul/OMPI/openmpi-2.0.1rc1-macos10.6-x86-m32/INST/lib/openmpi/mca_pmix_pmix112.so
>   Expected in: flat namespace
> 
> Let me know what additional information is desired and to whom to send.
> 
> -Paul
> 
> 
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov 
> 
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] Solaris MPIX failure

[r...@open-mpi.org]

Looks like Solaris has a “getupeercred” - can you take a look at it, Gilles? 
We’d have to add that to our AC_CHECK_FUNCS and update the native sec component.


> On Aug 23, 2016, at 6:32 AM, r...@open-mpi.org wrote:
> 
> I took a quick glance at this one, and the only way I can see to get that 
> error is from this block of code:
> 
> #if defined(HAVE_STRUCT_UCRED_UID)
> euid = ucred.uid;
> gid = ucred.gid;
> #else
> euid = ucred.cr_uid;
> gid = ucred.cr_gid;
> #endif
> 
> #elif defined(HAVE_GETPEEREID)
> pmix_output_verbose(2, pmix_globals.debug_output,
> "sec:native checking getpeereid for peer 
> credentials");
> if (0 != getpeereid(peer->sd, , )) {
> pmix_output_verbose(2, pmix_globals.debug_output,
> "sec: getsockopt getpeereid failed: %s",
> strerror (pmix_socket_errno));
> return PMIX_ERR_INVALID_CRED;
> }
> #else
> return PMIX_ERR_NOT_SUPPORTED;
> #endif
> 
> 
> I can only surmise, therefore, that Solaris doesn’t pass either of the two 
> #if define’d tests. Is there a Solaris alternative?
> 
> 
>> On Aug 23, 2016, at 5:55 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> Thanks Gilles!
>> 
>>> On Aug 23, 2016, at 3:42 AM, Gilles Gouaillardet 
>>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
>>> wrote:
>>> 
>>> Thanks Paul,
>>> 
>>> at first glance, something is going wrong in the sec module under solaris.
>>> I will keep digging tomorrow 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On Tuesday, August 23, 2016, Paul Hargrove <phhargr...@lbl.gov 
>>> <mailto:phhargr...@lbl.gov>> wrote:
>>> On Solaris 11.3 on x86-64:
>>> 
>>> $ mpirun -mca btl sm,self,openib -np 2 -host pcp-d-3,pcp-d-4 
>>> examples/ring_c'
>>> [pcp-d-4:25075] PMIX ERROR: NOT-SUPPORTED in file 
>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
>>>  at line 529
>>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>>  at line 983
>>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>>  at line 199
>>> --
>>> It looks like MPI_INIT failed for some reason; your parallel process is
>>> likely to abort.  There are many reasons that a parallel process can
>>> fail during MPI_INIT; some of which are due to configuration or environment
>>> problems.  This failure appears to be an internal failure; here's some
>>> additional information (which may only be relevant to an Open MPI
>>> developer):
>>> 
>>>   ompi_mpi_init: ompi_rte_init failed
>>>   --> Returned "(null)" (-43) instead of "Success" (0)
>>> --
>>> *** An error occurred in MPI_Init
>>> *** on a NULL communicator
>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>>> ***and potentially your MPI job)
>>> [pcp-d-4:25078] Local abort before MPI_INIT completed completed 
>>> successfully, but am not able to aggregate error messages, and not able to 
>>> guarantee that all other processes were killed!
>>> ---
>>> Primary job  terminated normally, but 1 process returned
>>> a non-zero exit code.. Per user-direction, the job has been aborted.
>>> ---
>>> --
>>> mpirun detected that one or more processes exited with non-zero status, 
>>> thus causing
>>> the job to be terminated. The first process to do so was:
>>> 
>>>   Process name: [[25599,1],1]
>>>   Exit code:1
>>> --
>>> 
>>> -Paul
>>> 
>>> -- 
>>> Paul H. Hargrove  phhargr...@lbl.gov 
>>> <javascript:_e(%7B%7D,'cvml

[OMPI devel] OMPI v2.0.1rc1 available for test

[r...@open-mpi.org]

Hello folks

Dunno where the head-honcho’s are hiding, but per their request: the newest 
v2.0.1 release candidate has been posted in the usual place:

https://www.open-mpi.org/software/ompi/v2.0/

Beat it up, please!
Ralph

2.0.1 -- 23 August 2016
---

Bug fixes/minor improvements:

- Short message latency and message rate performance improvements for
  all transports.
- Fix shared memory performance when using RDMA-capable networks.
  Thanks to Tetsuya Mishima and Christoph Niethammer for reporting.
- Fix OpenSHMEM crash when running on non-Mellanox MXM-based networks.
  Thanks to Debendra Das for reporting the issue.
- Fix various runtime issues by updating the PMIx internal component
  to v1.15.
- Fix process startup failures on Intel MIC platforms due to very
  large entries in /proc/mounts.
- Fix a problem with use of relative path for specifing executables to
  mpirun/oshrun.  Thanks to David Schneider for reporting.
- Various improvements when running over portals-based networks.
- Fix thread-based race conditions with GNI-based networks.
- Fix a problem with MPI_FILE_CLOSE and MPI_FILE_SET_SIZE.  Thanks
  to Cihan Altinay for reporting.
- Remove all use of rand(3) from within Open MPI so as not to perturb
  applications use of it.  Thanks to Matias Cabral and Noel Rycroft
  for reporting.
- Fix types for MPI_UNWEIGHTED and MPI_WEIGHTS_EMPTY.  Thanks to
  Lisandro Dalcin for reporting.
- Correctly report the name of MPI_INTEGER16.
- Add some missing MPI constants to the Fortran bindings.
- Fixed compile error when configuring Open MPI with --enable-timing.
- Correctly set the shared library version of libompitrace.so.  Thanks
  to Alastair McKinstry for reporting.
- Fix errors in the MPI_RPUT, MPI_RGET, MPI_RACCUMULATE, and
  MPI_RGET_ACCUMULATE Fortran bindings.  Thanks to Alfio Lazzaro and
  Joost VandeVondele for tracking this down.
- Fix problems with use of derived datatypes in non-blocking
  collectives.  Thanks to Yuki Matsumoto for reporting.
- Fix problems with OpenSHMEM header files when using CMake.  Thanks to
  Paul Kapinos for reporting the issue.
- Fix problem with use use of non-zero lower bound datatypes in
  collectives.  Thanks to Hristo Iliev for reporting.
- Fix a problem with memory allocation within MPI_GROUP_INTERSECTION.
  Thanks to Lisandro Dalcin for reporting.
- Fix an issue with MPI_ALLGATHER for communicators that don't consist
  of two ranks.  Thanks to David Love for reporting.
- Various fixes for collectives when used with esoteric MPI datatypes.
- Fixed corner cases of handling DARRAY and HINDEXED_BLOCK datatypes.
- Fix a problem with filesystem type check for OpenBSD.
  Thanks to Paul Hargrove for reporting.
- Fix some debug input within Open MPI internal functions.  Thanks to
  Durga Choudhury for reporting.
- Fix a typo in a configury help message.  Thanks to Paul Hargrove for
  reporting.
- Correctly support MPI_IN_PLACE in MPI_[I]ALLTOALL[V|W] and
  MPI_[I]EXSCAN.

Known issues (to be addressed in v2.0.2):

- See the list of fixes slated for v2.0.2 here:
  https://github.com/open-mpi/ompi/milestone/20, and
  https://github.com/open-mpi/ompi-release/milestone/19
  (note that the "ompi-release" Github repo will be folded/absorbed
  into the "ompi" Github repo at some point in the future)

- ompi-release#1014: Do not return MPI_ERR_PENDING from collectives
- ompi-release#1215: Fix configure to support the NAG Fortran compiler

Known issues (to be addressed in v2.1):

- ompi-release#987: Fix OMPIO performance on Lustre


___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] Solaris MPIX failure

[r...@open-mpi.org]

Thanks Gilles!

> On Aug 23, 2016, at 3:42 AM, Gilles Gouaillardet 
>  wrote:
> 
> Thanks Paul,
> 
> at first glance, something is going wrong in the sec module under solaris.
> I will keep digging tomorrow 
> 
> Cheers,
> 
> Gilles
> 
> On Tuesday, August 23, 2016, Paul Hargrove  > wrote:
> On Solaris 11.3 on x86-64:
> 
> $ mpirun -mca btl sm,self,openib -np 2 -host pcp-d-3,pcp-d-4 examples/ring_c'
> [pcp-d-4:25075] PMIX ERROR: NOT-SUPPORTED in file 
> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
>  at line 529
> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>  at line 983
> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>  at line 199
> --
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
> 
>   ompi_mpi_init: ompi_rte_init failed
>   --> Returned "(null)" (-43) instead of "Success" (0)
> --
> *** An error occurred in MPI_Init
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***and potentially your MPI job)
> [pcp-d-4:25078] Local abort before MPI_INIT completed completed successfully, 
> but am not able to aggregate error messages, and not able to guarantee that 
> all other processes were killed!
> ---
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code.. Per user-direction, the job has been aborted.
> ---
> --
> mpirun detected that one or more processes exited with non-zero status, thus 
> causing
> the job to be terminated. The first process to do so was:
> 
>   Process name: [[25599,1],1]
>   Exit code:1
> --
> 
> -Paul
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov 
> 
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Binding with --oversubscribe in 2.0.0

[r...@open-mpi.org]

Well, that’s a new one! I imagine we could modify the logic to allow a 
combination of oversubscribe and overload flags. Won’t get out until 2.1, 
though you could pull the patch in advance if it is holding you up.


> On Aug 23, 2016, at 11:46 PM, Ben Menadue  wrote:
> 
> Hi,
> 
> One of our users has noticed that binding is disabled in 2.0.0 when
> --oversubscribe is passed, which is hurting their performance, likely
> through migrations between sockets. It looks to be because of 294793c
> (PR#1228).
> 
> They need to use --oversubscribe as for some reason the developers decided
> to run two processes for each MPI task for some reason (a compute process
> and an I/O worker process, I think). Since the second process in the pair is
> mostly idle, there's (almost) no harm in launching two processes per core -
> and it's better than leaving half the cores idle most of the time. In
> previous versions they were binding each pair to a core and letting the
> hyper-threads argue over which of the two processes to run, since this gave
> the best performance.
> 
> I tried creating a rankfile and binding each process to its own hardware
> thread, but it refuses to launch more processes than the number of cores
> (even if all these processes are on the first socket because of the binding)
> unless --oversubscribe is passed, and thus disabling the binding. Is there a
> way of bypassing the disable-binding-if-oversubscribing check introduced by
> that commit? Or can anyone think of a better way of running this program?
> 
> Alternatively, they could leave it with no binding at the mpirun level and
> do the binding in a wrapper.
> 
> Thanks,
> Ben
> 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Binding with --oversubscribe in 2.0.0

[r...@open-mpi.org]

Actually, I stand corrected! Someone must have previously requested it, because 
support already exists.

What you need to do is simply specify the desired binding. If you don’t specify 
one, then we will disable it by default when oversubscribed. This was done to 
protect performance for those who don’t have such kind scenarios, and don’t 
realize we are otherwise binding by default.

So in your case, you’d want something like:

mpirun --map-by core:oversubscribe --bind-to core:overload

HTH
Ralph

> On Aug 24, 2016, at 7:33 AM, r...@open-mpi.org wrote:
> 
> Well, that’s a new one! I imagine we could modify the logic to allow a 
> combination of oversubscribe and overload flags. Won’t get out until 2.1, 
> though you could pull the patch in advance if it is holding you up.
> 
> 
>> On Aug 23, 2016, at 11:46 PM, Ben Menadue <ben.mena...@nci.org.au> wrote:
>> 
>> Hi,
>> 
>> One of our users has noticed that binding is disabled in 2.0.0 when
>> --oversubscribe is passed, which is hurting their performance, likely
>> through migrations between sockets. It looks to be because of 294793c
>> (PR#1228).
>> 
>> They need to use --oversubscribe as for some reason the developers decided
>> to run two processes for each MPI task for some reason (a compute process
>> and an I/O worker process, I think). Since the second process in the pair is
>> mostly idle, there's (almost) no harm in launching two processes per core -
>> and it's better than leaving half the cores idle most of the time. In
>> previous versions they were binding each pair to a core and letting the
>> hyper-threads argue over which of the two processes to run, since this gave
>> the best performance.
>> 
>> I tried creating a rankfile and binding each process to its own hardware
>> thread, but it refuses to launch more processes than the number of cores
>> (even if all these processes are on the first socket because of the binding)
>> unless --oversubscribe is passed, and thus disabling the binding. Is there a
>> way of bypassing the disable-binding-if-oversubscribing check introduced by
>> that commit? Or can anyone think of a better way of running this program?
>> 
>> Alternatively, they could leave it with no binding at the mpirun level and
>> do the binding in a wrapper.
>> 
>> Thanks,
>> Ben
>> 
>> 
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [2.0.1.rc1] Solaris MPIX failure

[r...@open-mpi.org]

When you do, that PR has already been committed, so you can just pull the next 
nightly 2.x tarball and test from there

> On Aug 24, 2016, at 10:39 AM, Paul Hargrove <phhargr...@lbl.gov> wrote:
> 
> I am afraid it might take a day or two before I can get to testing that patch.
> 
> -Paul
> 
> On Tue, Aug 23, 2016 at 10:16 PM, Gilles Gouaillardet <gil...@rist.or.jp 
> <mailto:gil...@rist.or.jp>> wrote:
> Paul,
> 
> 
> you can download a patch at 
> https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1336.patch
>  
> <https://patch-diff.githubusercontent.com/raw/open-mpi/ompi-release/pull/1336.patch>
> (note you need recent autotools in order to use it)
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> On 8/23/2016 10:40 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> Looks like Solaris has a “getupeercred” - can you take a look at it, Gilles? 
>> We’d have to add that to our AC_CHECK_FUNCS and update the native sec 
>> component.
>> 
>> 
>>> On Aug 23, 2016, at 6:32 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>> wrote:
>>> 
>>> I took a quick glance at this one, and the only way I can see to get that 
>>> error is from this block of code:
>>> 
>>> #if defined(HAVE_STRUCT_UCRED_UID)
>>> euid = ucred.uid;
>>> gid = ucred.gid;
>>> #else
>>> euid = ucred.cr_uid;
>>> gid = ucred.cr_gid;
>>> #endif
>>> 
>>> #elif defined(HAVE_GETPEEREID)
>>> pmix_output_verbose(2, pmix_globals.debug_output,
>>> "sec:native checking getpeereid for peer 
>>> credentials");
>>> if (0 != getpeereid(peer->sd, , )) {
>>> pmix_output_verbose(2, pmix_globals.debug_output,
>>> "sec: getsockopt getpeereid failed: %s",
>>>         strerror (pmix_socket_errno));
>>> return PMIX_ERR_INVALID_CRED;
>>> }
>>> #else
>>> return PMIX_ERR_NOT_SUPPORTED;
>>> #endif
>>> 
>>> 
>>> I can only surmise, therefore, that Solaris doesn’t pass either of the two 
>>> #if define’d tests. Is there a Solaris alternative?
>>> 
>>> 
>>>> On Aug 23, 2016, at 5:55 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>>> wrote:
>>>> 
>>>> Thanks Gilles!
>>>> 
>>>>> On Aug 23, 2016, at 3:42 AM, Gilles Gouaillardet 
>>>>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
>>>>> wrote:
>>>>> 
>>>>> Thanks Paul,
>>>>> 
>>>>> at first glance, something is going wrong in the sec module under solaris.
>>>>> I will keep digging tomorrow 
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Gilles
>>>>> 
>>>>> On Tuesday, August 23, 2016, Paul Hargrove <phhargr...@lbl.gov 
>>>>> <mailto:phhargr...@lbl.gov>> wrote:
>>>>> On Solaris 11.3 on x86-64:
>>>>> 
>>>>> $ mpirun -mca btl sm,self,openib -np 2 -host pcp-d-3,pcp-d-4 
>>>>> examples/ring_c'
>>>>> [pcp-d-4:25075] PMIX ERROR: NOT-SUPPORTED in file 
>>>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_listener.c
>>>>>  at line 529
>>>>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>>>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>>>>  at line 983
>>>>> [pcp-d-4:25078] PMIX ERROR: UNREACHABLE in file 
>>>>> /shared/OMPI/openmpi-2.0.1rc1-solaris11-x86-ib-gcc/openmpi-2.0.1rc1/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
>>>>>  at line 199
>>>>> --
>>>>> It looks like MPI_INIT failed for some reason; your parallel process is
>>>>> likely to abort.  There are many reasons that a parallel process can
>>>>> fail during MPI_INIT; some of which are due to configuration or 
>>>>> environment
>>>>> problems.  This failure appears to be an internal failure; here's some
>>>>> additional information (which may only be relevant to an Open MPI
>>>>> developer):
>>>>> 
>>>>&

Re: [OMPI devel] C89 support

[r...@open-mpi.org]

I hadn’t realized we still have a --disable-c99 configure option - that sounds 
bad as we can’t possibly build that way. We need to internally perform the 
configure check, but we shouldn’t be exposing a configure option as that just 
confuses people into thinking it really is an option.

> On Aug 29, 2016, at 7:27 AM, Nathan Hjelm  wrote:
> 
> FYI, C99 has been required since late 2012. Going through the commits there 
> is no way Open MPI could possibly compile with —std=c89 or —std=gnu99. Older 
> compilers require we add —std=c99 so we can not remove the configure check.
> 
> 
> commit aebd1ea43237741bd29878604b742b14cc87d68b
> Author: Nathan Hjelm 
> Date:   Wed Nov 14 04:52:39 2012 +
> 
>Per discussion we will now require a C99 compiant compiler.
> 
>This change will enable the use of C99 features in Open MPI; subobject 
> naming, restricted pointers, etc.
> 
>cmr:v1.7
> 
>This commit was SVN r27604.
> 
> 
> -Nathan
> 
>> On Aug 29, 2016, at 8:18 AM, Gilles Gouaillardet 
>>  wrote:
>> 
>> Thanks Brice !
>> 
>> On Monday, August 29, 2016, Brice Goglin  wrote:
>> s/June 2016/June 2006/ :)
>> 
>> Anyway, it ended on July 31st based on https://www.suse.com/lifecycle/
>> 
>> Brice
>> 
>> 
>> 
>> Le 29/08/2016 16:03, Gilles Gouaillardet a écrit :
>>> According to wikipedia, SLES 10 was released on June 2016, and is supported 
>>> for 10 years.
>>> (SLES 12 is supported for 13 years, and I honestly do not know whether SLES 
>>> 10 support has been extended)
>>> so SLES 10 might already been EOL
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On Monday, August 29, 2016, Jeff Squyres (jsquyres)  
>>> wrote:
>>> The patches for master/v2.x will be considerably larger (we have embraced 
>>> at least a few of the C99 constructs quite a bit).
>>> 
>>> When is the EOL for SLES 10?
>>> 
>>> Can you provide the doc links and an example of the link error that these 
>>> patches are fixing?
>>> 
>>> 
>>> 
 On Aug 29, 2016, at 1:04 AM, C Bergström  wrote:
 
 On Mon, Aug 29, 2016 at 12:59 PM, Gilles Gouaillardet  
 wrote:
> Christopher,
> 
> 
> i made PR #1345 https://github.com/open-mpi/ompi-release/pull/1345
> 
> (there is no copyright in these files, let me know how i should credit
> pathscale (if you want that of course)
 
 I'm not sure that there is anything substantial enough to be
 copyright. If the shoe was on the other foot I'd highly question
 anyone who pushed for attribution on these patches.
 
> 
> 
> this is basically your patch plus a few changes : you need to configure 
> with
> '--disable-c99' if you are using pre C99 compiler.
> 
> i noted these patches are for the v1.10 series. do you also expect v2.x 
> (and
> master too ?) can be built with pre C99 compilers too ?
 
 If these style of changes are acceptable we'll do patches for all.
 
 Thanks for your help on this. I have a couple other small things I'm
 hoping to get upstream after this.
 ___
 devel mailing list
 devel@lists.open-mpi.org
 https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>> 
>>> 
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to: 
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>> 
>>> 
>>> __
>>> _
>>> devel mailing list
>>> 
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Open MPI, PMIx and munge

[r...@open-mpi.org]

OMPI should not build munge support unless specifically requested to do so

> On Oct 2, 2016, at 7:04 PM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> Open MPI policy is to build munge support if it is found, whereas PMIx policy 
> is to build munge support only if it requested
> 
> from a pragmatic point of view, and on a system where munge libs and headers 
> are installed
> 
> ./configure
> 
> will build munge support for Open MPI, but *not* for the embedded PMIx
> 
> /* if you are wondering, this is not typo, see pmix commit 
> https://github.com/pmix/master/commit/765fadb6c3a9a1b3a68d2d551c615db76e3c7dfd
>  */
> 
> 
> should we somehow unify this ?
> 
> - one option is to change Open MPI policy
> 
> - an other option is to change PMIx policy
> 
> - and yet an other option is to have Open MPI pass --with-munge to PMIx 
> configure if not set
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] mtl/psm2 and $PSM2_DEVICES

[r...@open-mpi.org]

2 
>> <https://github.com/open-mpi/ompi/pull/1602>) and should be included in 
>> v2.0.2. HOWEVER, this not the error Juraj is seeing. The root of the 
>> assertion is because the PSM/PSM2 MTLs will check for where the “original” 
>> process are running and, if detects all are local to the node, it will ONLY 
>> initialize the shared memory device (variable PSM2_DEVICES="self,shm” ). 
>> This is to avoid “reserving” HW resources in the HFI card that wouldn’t be 
>> used unless you later on spawn ranks in other nodes.  Therefore, to allow 
>> dynamic process to be spawned on other nodes you need to tell PSM2 to 
>> instruct the HW to initialize all the de devices by making the environment 
>> variable PSM2_DEVICES="self,shm,hfi" available before running the job.
>> Note that setting PSM2_DEVICES (*) will solve the below assertion, you will 
>> most likely still see the transport key issue if PR1602 if is not included.
>>  
>> Thanks,
>>  
>> _MAC
>>  
>> (*)
>> PSM2_DEVICES  -> Omni Path
>> PSM_DEVICES  -> TrueScale
>>  
>> From: users [mailto:users-boun...@lists.open-mpi.org 
>> <mailto:users-boun...@lists.open-mpi.org>] On Behalf Of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org>
>> Sent: Thursday, September 29, 2016 7:12 AM
>> To: Open MPI Users <us...@lists.open-mpi.org> 
>> <mailto:us...@lists.open-mpi.org>
>> Subject: Re: [OMPI users] MPI_Comm_spawn
>>  
>> Ah, that may be why it wouldn’t show up in the OMPI code base itself. If 
>> that is the case here, then no - OMPI v2.0.1 does not support comm_spawn for 
>> PSM. It is fixed in the upcoming 2.0.2
>>  
>> On Sep 29, 2016, at 6:58 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>>  
>> Ralph,
>>  
>> My guess is that ptl.c comes from PSM lib ...
>>  
>> Cheers,
>>  
>> Gilles
>> 
>> On Thursday, September 29, 2016, r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
>> wrote:
>> Spawn definitely does not work with srun. I don’t recognize the name of the 
>> file that segfaulted - what is “ptl.c”? Is that in your manager program?
>>  
>>  
>> On Sep 29, 2016, at 6:06 AM, Gilles Gouaillardet < 
>> <>gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
>> wrote:
>>  
>> Hi,
>>  
>> I do not expect spawn can work with direct launch (e.g. srun)
>>  
>> Do you have PSM (e.g. Infinipath) hardware ? That could be linked to the 
>> failure
>>  
>> Can you please try
>>  
>> mpirun --mca pml ob1 --mca btl tcp,sm,self -np 1 --hostfile my_hosts 
>> ./manager 1
>>  
>> and see if it help ?
>>  
>> Note if you have the possibility, I suggest you first try that without 
>> slurm, and then within a slurm job
>>  
>> Cheers,
>>  
>> Gilles
>> 
>> On Thursday, September 29, 2016,  <>juraj2...@gmail.com 
>> <mailto:juraj2...@gmail.com> < <>juraj2...@gmail.com 
>> <mailto:juraj2...@gmail.com>> wrote:
>> Hello,
>>  
>> I am using MPI_Comm_spawn to dynamically create new processes from single 
>> manager process. Everything works fine when all the processes are running on 
>> the same node. But imposing restriction to run only a single process per 
>> node does not work. Below are the errors produced during multinode 
>> interactive session and multinode sbatch job.
>>  
>> The system I am using is: Linux version 3.10.0-229.el7.x86_64 
>> (buil...@kbuilder.dev.centos.org <mailto:buil...@kbuilder.dev.centos.org>) 
>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) )
>> I am using Open MPI 2.0.1
>> Slurm is version 15.08.9
>>  
>> What is preventing my jobs to spawn on multiple nodes? Does slurm requires 
>> some additional configuration to allow it? Is it issue on the MPI side, does 
>> it need to be compiled with some special flag (I have compiled it with 
>> --enable-mpi-fortran=all --with-pmi)? 
>>  
>> The code I am launching is here: https://github.com/goghino/dynamicMPI 
>> <https://github.com/goghino/dynamicMPI>
>>  
>> Manager tries to launch one new process (./manager 1), the error produced by 
>> requesting each process to be located on different node (interactive 
>> session):
>> $ salloc -N 2
>> $ cat my_hosts
>> icsnode37
>> icsnod

Re: [OMPI devel] use of OBJ_NEW and related calls

[r...@open-mpi.org]

See opal/class/opal_object.h

And your assumption is correct :-)


> On Oct 10, 2016, at 1:18 PM, Emani, Murali  wrote:
> 
> Hi,
> 
> Could someone help me in understanding where the functions OBJ_NEW/ 
> OBJ_CONSTRUCT/ OBJ_DESTRUCT are defined in the source code. Are these 
> specific to OpenMPI code base?
> Is the assumption correct that these calls are wrappers to create new 
> objects, initialize and destroy, similar to any object oriented language.
> 
> — 
> Murali
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] PMIx in 2.x

[r...@open-mpi.org]

Should be handled more gracefully, of course. When a proc departs, we cleanup 
any published storage that wasn’t specifically indicated to be retained for a 
longer period of time (see pmix_common.h for the various supported options). 
You’ve obviously found a race condition, and we’ll have to track it down.

I gather you are working off of the OMPI 2.x repo? The PMIx support over there 
is fairly far behind the OMPI master, though neither branches have seen any 
real testing with the orte-server support. I’m a little buried at the moment, 
but can try to check that out in a bit.


> On Nov 8, 2016, at 10:30 AM, Pieter Noordhuis <piet...@fb.com> wrote:
> 
> ... accidentally sent before finishing.
> 
> This error happened because lookup was returning multiple published entries, 
> N-1 of which had already disconnected. Should this case be handled more 
> gracefully, or is it expected to fail like this?
> 
> Thanks,
> Pieter
> From: Pieter Noordhuis
> Sent: Tuesday, November 8, 2016 10:29:02 AM
> To: OpenMPI Devel
> Subject: Re: [OMPI devel] PMIx in 2.x
>  
> Ah, that's good to know. I'm trying to wire things up through orte-server on 
> 2.x now and am at the point where I can do an MPI_Publish_name on one node 
> and MPI_Lookup_name on another (both started through their own mpirun calls). 
> Then, when the first process does an MPI_Comm_accept and the second process 
> does an MPI_Comm_connect on that port, I'm getting an unreachable error from 
> ompi/mca/bml/r2/bml_r2.c. I don't see the IPs of either process being 
> communicated and with just the PMIx port name they won't be able to connect. 
> Is there anything I'm missing?
> 
> Also, when trying to get this to work, I kept orte-server running between 
> runs, and saw a rather alarming error when doing a lookup:
> [[8602,0],0] recvd lookup returned data ServerName of type 3 from source 
> [[49864,1],0] 
> 
> [[8602,0],0] recvd lookup returned data ServerName of type 3 from source 
> [[10062,1],0] 
> 
> PACK-PMIX-VALUE: UNSUPPORTED TYPE 0   
>   
>
> PMIX ERROR: ERROR in file src/server/pmix_server.c at line 1881   
>       
>            
> 
> 
> 
> 
> From: devel <devel-boun...@lists.open-mpi.org> on behalf of r...@open-mpi.org 
> <r...@open-mpi.org>
> Sent: Monday, November 7, 2016 12:16:50 PM
> To: OpenMPI Devel
> Subject: Re: [OMPI devel] PMIx in 2.x
>  
> I’m not sure your description of the 1.x behavior is entirely accurate. What 
> actually happened in that scenario is that the various mpirun’s would connect 
> to each other, proxying the various MPI dynamic calls across each other. You 
> had to tell each mpirun how to find the other - this was in the “port” string 
> you might have been using. It might be possible to recreate it by adding the 
> PMIx server address of the other mpirun to the string, though it would still 
> be an ugly way to start a job.
> 
> The other option in 1.x was to start the orte-server and let it provide the 
> rendezvous point. This option remains - it just uses PMIx calls to implement 
> the rendezvous.
> 
> 
>> On Nov 7, 2016, at 9:23 AM, Pieter Noordhuis <piet...@fb.com 
>> <mailto:piet...@fb.com>> wrote:
>> 
>> Hi folks,
>> 
>> Question about PMIx in the 2.x tree: on 1.x I used to be able to start N 
>> individual jobs through mpirun with -np1 and have them gradually join a 
>> single intercommunicator through MPI_Comm_accept, MPI_Comm_connect, 
>> MPI_Intercomm_create, and MPI_Intercomm_merge. The port that one of the 
>> processes would listen on included its IP address and others would connect 
>> to that. I tried porting this code to the 2.x tree and found the port is now 
>> just an integer. Reading up on the changelogs and commit history, I found 
>> PMIx replaced DPM starting with 2.x. Reading up on PMIx and OpenMPI, my 
>> understanding is that OpenMPI ships with a PMIx server implementation, and 
>> that all processes in the job have to be connected to this PMIx server at 
>> start. It looks like MPI_Comm_accept and MPI_Comm_connect communicate 
>> through k/v pairs in the PMIx server.
>> 
>> This means it's no longer possible to start jobs through multiple mpirun 
>> executions and then join them into a single interc

Re: [OMPI devel] PMIx in 2.x

[r...@open-mpi.org]

Yes, they should find one another. You do have to pass the server’s URI on the 
mpirun cmd line so it can find it.

> On Nov 8, 2016, at 12:40 PM, Pieter Noordhuis <piet...@fb.com> wrote:
> 
> I'll open an issue on GitHub with more details.
> 
> As far as the accept/connect issue, do you expect the processes to find each 
> other if they do a publish/lookup through the same orte-server if they are 
> started separately?
> 
> I can try with master as well. I'm working off of 2.0.1 now (using stable 
> because I want to rule out any other issues that might be present in master).
> 
> Thanks,
> Pieter
> From: devel <devel-boun...@lists.open-mpi.org> on behalf of r...@open-mpi.org 
> <r...@open-mpi.org>
> Sent: Tuesday, November 8, 2016 12:17:29 PM
> To: OpenMPI Devel
> Subject: Re: [OMPI devel] PMIx in 2.x
>  
> Should be handled more gracefully, of course. When a proc departs, we cleanup 
> any published storage that wasn’t specifically indicated to be retained for a 
> longer period of time (see pmix_common.h for the various supported options). 
> You’ve obviously found a race condition, and we’ll have to track it down.
> 
> I gather you are working off of the OMPI 2.x repo? The PMIx support over 
> there is fairly far behind the OMPI master, though neither branches have seen 
> any real testing with the orte-server support. I’m a little buried at the 
> moment, but can try to check that out in a bit.
> 
> 
>> On Nov 8, 2016, at 10:30 AM, Pieter Noordhuis <piet...@fb.com 
>> <mailto:piet...@fb.com>> wrote:
>> 
>> ... accidentally sent before finishing.
>> 
>> This error happened because lookup was returning multiple published entries, 
>> N-1 of which had already disconnected. Should this case be handled more 
>> gracefully, or is it expected to fail like this?
>> 
>> Thanks,
>> Pieter
>> From: Pieter Noordhuis
>> Sent: Tuesday, November 8, 2016 10:29:02 AM
>> To: OpenMPI Devel
>> Subject: Re: [OMPI devel] PMIx in 2.x
>>  
>> Ah, that's good to know. I'm trying to wire things up through orte-server on 
>> 2.x now and am at the point where I can do an MPI_Publish_name on one node 
>> and MPI_Lookup_name on another (both started through their own mpirun 
>> calls). Then, when the first process does an MPI_Comm_accept and the second 
>> process does an MPI_Comm_connect on that port, I'm getting an unreachable 
>> error from ompi/mca/bml/r2/bml_r2.c. I don't see the IPs of either process 
>> being communicated and with just the PMIx port name they won't be able to 
>> connect. Is there anything I'm missing?
>> 
>> Also, when trying to get this to work, I kept orte-server running between 
>> runs, and saw a rather alarming error when doing a lookup:
>> [[8602,0],0] recvd lookup returned data ServerName of type 3 from source 
>> [[49864,1],0]
>>  
>> [[8602,0],0] recvd lookup returned data ServerName of type 3 from source 
>> [[10062,1],0]
>>  
>> PACK-PMIX-VALUE: UNSUPPORTED TYPE 0  
>>  
>>      
>> PMIX ERROR: ERROR in file src/server/pmix_server.c at line 1881  
>>  
>>  
>> 
>> 
>> 
>> 
>> From: devel <devel-boun...@lists.open-mpi.org 
>> <mailto:devel-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>> Sent: Monday, November 7, 2016 12:16:50 PM
>> To: OpenMPI Devel
>> Subject: Re: [OMPI devel] PMIx in 2.x
>>  
>> I’m not sure your description of the 1.x behavior is entirely accurate. What 
>> actually happened in that scenario is that the various mpirun’s would 
>> connect to each other, proxying the various MPI dynamic calls across each 
>> other. You had to tell each mpirun how to find the other - this was in the 
>> “port” string you might have been using. It might be possible to recreate it 
>> by adding the PMIx server address of the other mpirun to the string, though 
>> it would still be an ugly way to start a job.
>> 
>> The other option in 1.x was to start the orte-server and let it provide the 
>> rendezvous point. This option remains -

Re: [OMPI devel] PMIx in 2.x

[r...@open-mpi.org]

I’m not sure your description of the 1.x behavior is entirely accurate. What 
actually happened in that scenario is that the various mpirun’s would connect 
to each other, proxying the various MPI dynamic calls across each other. You 
had to tell each mpirun how to find the other - this was in the “port” string 
you might have been using. It might be possible to recreate it by adding the 
PMIx server address of the other mpirun to the string, though it would still be 
an ugly way to start a job.

The other option in 1.x was to start the orte-server and let it provide the 
rendezvous point. This option remains - it just uses PMIx calls to implement 
the rendezvous.


> On Nov 7, 2016, at 9:23 AM, Pieter Noordhuis  wrote:
> 
> Hi folks,
> 
> Question about PMIx in the 2.x tree: on 1.x I used to be able to start N 
> individual jobs through mpirun with -np1 and have them gradually join a 
> single intercommunicator through MPI_Comm_accept, MPI_Comm_connect, 
> MPI_Intercomm_create, and MPI_Intercomm_merge. The port that one of the 
> processes would listen on included its IP address and others would connect to 
> that. I tried porting this code to the 2.x tree and found the port is now 
> just an integer. Reading up on the changelogs and commit history, I found 
> PMIx replaced DPM starting with 2.x. Reading up on PMIx and OpenMPI, my 
> understanding is that OpenMPI ships with a PMIx server implementation, and 
> that all processes in the job have to be connected to this PMIx server at 
> start. It looks like MPI_Comm_accept and MPI_Comm_connect communicate through 
> k/v pairs in the PMIx server.
> 
> This means it's no longer possible to start jobs through multiple mpirun 
> executions and then join them into a single intercommunicator at runtime. Is 
> my understanding correct?
> 
> Thank you,
> 
> Pieter
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] RFC: Rename nightly snapshot tarballs

[r...@open-mpi.org]

You make a valid point - I too prefer simplicity to a 50-character-long name. 
There should be some simple way of making it clear which branch the tarball 
came from...your suggestions seem reasonable and easy to do. I’m sure we’ll be 
talking about this on the telecon in the morning.

> On Oct 17, 2016, at 8:56 PM, Gilles Gouaillardet  wrote:
> 
> Jeff,
> 
> 
> i have no strong objections against the new scheme.
> 
> that being said, i think i like the current one better ...
> 
> imho, the issue is the tag was put on a commit that is common to master and 
> branch.
> 
> 
> i guess we did :
> 
> git checkout master
> 
> git tag v2.x
> 
> git branch v2.x
> 
> 
> what if we had done instead :
> git checkout master
> git checkout -b v2.x
> git commit --allow-empty -m "forking the v2.x branch"
> git tag v2.x
> 
> 
> pro: i guess that would have produced the "expected" behavior.
> cons: v2.x would be only on the v2.x branch, so the branch point is really 
> the parent of the tag
> 
> regardless we want to change the way we tag, could we simply sed the git 
> describe output instead ?
> master : git describe | sed -e 's/v2.x-dev/master/g'
> v2.x: git describe | sed -e 's/v2.0.1/v2.x/g'
> v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.x/g'
> or
> v2.x: git describe | sed -e 's/v2.0.1/v2.1.0a/g'
> v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.2a/g'
> 
> 
> Cheers,
> 
> Gilles
> 
> On 10/18/2016 5:17 AM, Jeff Squyres (jsquyres) wrote:
>> SHORT VERSION
>> =
>> 
>> Our nightly snapshot tarball filenames have become confusing.  Here's the 
>> most recent on each branch:
>> 
>> - Master: openmpi-v2.x-dev-3019-gbd1b6fe.tar.bz2
>> - v2.0.x: openmpi-v2.0.1-93-g97511a1.tar.bz2
>> - v2.x:   openmpi-v2.0.1-203-g56991c6.tar.bz2
>> 
>> I propose changing them to the following format:
>> 
>> openmpi-${BRANCHNAME}-${MMDD}-${SHORTHASH}.tar.bz2
>> 
>> Here's what last night's tarballs would have looked like with that scheme:
>> 
>> openmpi-master-20161017-bd1b6fe.tar.bz2
>> openmpi-v2.0.x-20161017-a66f3e2.tar.bz2
>> openmpi-v2.x-20161017-56991c6.tar.bz2
>> 
>> Optionally, we could put a HHMM timestamp in there for finer granularity.
>> 
>> MORE DETAIL
>> ===
>> 
>> We use "git describe" to come up with the version strings for our nightly 
>> snapshot tarballs.  "git describe" shows you three things:
>> 
>> - the first tag that it finds going back along the git commit history
>> - the number of commits it had to traverse to find that tag
>> - the short hast of the HEAD commit
>> 
>> For lack of a longer explanation, the names of the current snapshots *do* 
>> make sense given our commit and tag history; they're just... weird.
>> 
>> After talking this through with a colleague today, I think we have two 
>> choices:
>> 
>> 1. Continue to use the "git describe" output
>> 2. Come up with a different scheme
>> 
>> Using git describe output means that we are reliant on git tags.  Funky 
>> placement of git tags -- including the moving of branches back from 
>> ompi-release to ompi -- is how we ended up in this situation.  This might 
>> constitute empirical evidence that "git describe" is not a good solution for 
>> this community.
>> 
>> Instead, perhaps we should make a new scheme.  We need two properties in the 
>> snapshot filenames:
>> 
>> 1. Name of the branch that the tarball came from.
>> 2. Easily sortable by a human (i.e., know that tarball X came before or 
>> after tarball Y).
>> 
>> Property #1 is self-evident (the branch name is in the filename); property 
>> #2 comes from the timestamp.
>> 
>> NOTE: It may be desirable to add HHMM in there; it's not common, but 
>> *sometimes* we do make more than one snapshot in a day (e.g., if one 
>> snapshot is borked, so we fix it and then generate another snapshot).
>> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Update to Open MPI Administrative Rules

[r...@open-mpi.org]

Hello all

Some of you may have noticed that we have been receiving pull requests on 
Github from contributors who have not signed a formal Contributor’s Agreement. 
This has raised some discussion in the community about how we accept such 
contributions without conflicting with our bylaws.

Accordingly, the Open MPI community has revised its bylaws - you can find the 
full revised text here:

https://github.com/open-mpi/ompi/wiki/Admistrative-rules 


In brief, Contributor Agreements are no longer required to contribute to OMPI. 
Instead, we have adopted the approach taken by the Linux community and are 
asking that _all_ contributions be signed-off by the contributor with a line:

signed-off by:  

You may substitute your Github id for the email, if you prefer, as the Github 
account is directly tied to your email.

It is our hope that this will make contributing to Open MPI easier for many of 
you.
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Update to Open MPI Administrative Rules

[r...@open-mpi.org]

And just to be clear: while it is nice that git makes it so easy to comply, you 
are making a very real legal statement when you use that option! Per the bylaws:

"By making a contribution to this project, I certify that:

The contribution was created in whole or in part by me and I have the right to 
submit it under the Open MPI open source license 
<https://github.com/open-mpi/ompi/blob/master/LICENSE>; or
The contribution is based upon previous work that, to the best of my knowledge, 
is covered under an appropriate open source license and I have the right under 
that license to submit that work with modifications, whether created in whole 
or in part by me, under the Open MPI open source license 
<https://github.com/open-mpi/ompi/blob/master/LICENSE> (unless I am permitted 
to submit under a different license); or
The contribution was provided directly to me by some other person who certified 
(1) or (2) and I have not modified it.
I understand and agree that this project and the contribution are public and 
that a record of the contribution (including all personal information I submit 
with it, including my sign-off) is maintained indefinitely and may be 
redistributed consistent with this project and the open source license(s) 
involved."
PLEASE - do not just blindly use “-s”. Make sure you fully understand the 
above, have checked with your employer, and can legally “sign-off” on your 
contribution

Thanks
Ralph

> On Oct 25, 2016, at 10:50 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
> 
> On Oct 25, 2016, at 12:43 PM, r...@open-mpi.org wrote:
>> 
>> signed-off by:  
> 
> Just to nit-pick: it's "Signed-off-by" (with a capital S and a -).  It's the 
> output you get when you "git commit -s ...".
> 
> We're looking for that specific token, since it's standard / automatically 
> inserted by git for you.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [hwloc-devel] [RFC] applying native OS restrictions to XML imported topology

[r...@open-mpi.org]

Hmmm...I think maybe we are only seeing a small portion of the picture here. 
There are two pieces of the problem when looking at large SMPs:

* time required for discovery - your proposal is attempting to address that, 
assuming that the RM daemon collects the topology and then communicates it to 
each process (which is today’s method)

* memory footprint. We are seeing over 40MBytes being consumed by hwloc 
topologies on fully loaded KNL machines, which is a disturbing number

Where we are headed is to having only one copy of the hwloc topology tree on a 
node, stored in a shared memory segment hosted by the local RM daemon. Procs 
will then access that tree to obtain any required info. Thus, we are less 
interested in each process creating its own tree based on an XML representation 
passed to it by the RM, and more interested in having the hwloc search 
algorithms correctly handle any resource restrictions when searching the RM’s 
tree.

In other words, rather than (or perhaps, in addition to?) filtering the XML, 
we’d prefer to see some modification of the search APIs to allow a proc to pass 
in its resource constraints, and have the search algorithm properly consider 
them when returning the result. This eliminates all the XML conversion 
overhead, and resolves the memory footprint issue.

HTH
Ralph

> On Oct 21, 2016, at 5:16 AM, Brice Goglin  wrote:
> 
> Hello
> 
> Based on recent discussion about hwloc_topology_load() being slow on
> some "large" platforms (almost 1 second on KNL), here's a new feature
> proposal:
> 
> We've been recommending the use of XML to avoid multiple expensive
> discovery: Export to XML once at boot, and reload from XML for each
> actual process using hwloc. The main limitation is cgroups: resource
> managers use cgroups to restrict the processors and memory that are
> actually available to each job. So the topology of different jobs on the
> same machine is actually slightly different from the main XML that
> contained everything when it was created outside of cgroups during boot.
> 
> So we're looking at adding a new topology flag that loads the entire
> machine from XML (or synthetic) and applies restrictions from the
> local/native operating system.
> 
> Details at https://github.com/open-mpi/hwloc/pull/212
> Comments welcome here or there.
> 
> Brice
> 
> ___
> hwloc-devel mailing list
> hwloc-devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-devel

___
hwloc-devel mailing list
hwloc-devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-devel

Re: [hwloc-devel] [RFC] applying native OS restrictions to XML imported topology

[r...@open-mpi.org]

I should add: this does beg the question of how a proc “discovers” its resource 
constraints without having access to the hwloc tree. One possible solution - 
the RM already knows the restrictions, and so it could pass those down at proc 
startup (e.g., as part of the PMIx info). We could pass whatever info hwloc 
would like passed into its calls - doesn’t have to be something 
“understandable” by the proc itself.


> On Oct 21, 2016, at 8:15 AM, r...@open-mpi.org wrote:
> 
> Hmmm...I think maybe we are only seeing a small portion of the picture here. 
> There are two pieces of the problem when looking at large SMPs:
> 
> * time required for discovery - your proposal is attempting to address that, 
> assuming that the RM daemon collects the topology and then communicates it to 
> each process (which is today’s method)
> 
> * memory footprint. We are seeing over 40MBytes being consumed by hwloc 
> topologies on fully loaded KNL machines, which is a disturbing number
> 
> Where we are headed is to having only one copy of the hwloc topology tree on 
> a node, stored in a shared memory segment hosted by the local RM daemon. 
> Procs will then access that tree to obtain any required info. Thus, we are 
> less interested in each process creating its own tree based on an XML 
> representation passed to it by the RM, and more interested in having the 
> hwloc search algorithms correctly handle any resource restrictions when 
> searching the RM’s tree.
> 
> In other words, rather than (or perhaps, in addition to?) filtering the XML, 
> we’d prefer to see some modification of the search APIs to allow a proc to 
> pass in its resource constraints, and have the search algorithm properly 
> consider them when returning the result. This eliminates all the XML 
> conversion overhead, and resolves the memory footprint issue.
> 
> HTH
> Ralph
> 
>> On Oct 21, 2016, at 5:16 AM, Brice Goglin <brice.gog...@inria.fr> wrote:
>> 
>> Hello
>> 
>> Based on recent discussion about hwloc_topology_load() being slow on
>> some "large" platforms (almost 1 second on KNL), here's a new feature
>> proposal:
>> 
>> We've been recommending the use of XML to avoid multiple expensive
>> discovery: Export to XML once at boot, and reload from XML for each
>> actual process using hwloc. The main limitation is cgroups: resource
>> managers use cgroups to restrict the processors and memory that are
>> actually available to each job. So the topology of different jobs on the
>> same machine is actually slightly different from the main XML that
>> contained everything when it was created outside of cgroups during boot.
>> 
>> So we're looking at adding a new topology flag that loads the entire
>> machine from XML (or synthetic) and applies restrictions from the
>> local/native operating system.
>> 
>> Details at https://github.com/open-mpi/hwloc/pull/212
>> Comments welcome here or there.
>> 
>> Brice
>> 
>> ___
>> hwloc-devel mailing list
>> hwloc-devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-devel
> 
> ___
> hwloc-devel mailing list
> hwloc-devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-devel

___
hwloc-devel mailing list
hwloc-devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-devel

Re: [hwloc-devel] [RFC] applying native OS restrictions to XML imported topology

[r...@open-mpi.org]

> On Oct 21, 2016, at 10:09 AM, Brice Goglin <brice.gog...@inria.fr> wrote:
> 
> Le 21/10/2016 17:21, r...@open-mpi.org <mailto:r...@open-mpi.org> a écrit :
>> I should add: this does beg the question of how a proc “discovers” its 
>> resource constraints without having access to the hwloc tree. One possible 
>> solution - the RM already knows the restrictions, and so it could pass those 
>> down at proc startup (e.g., as part of the PMIx info). We could pass 
>> whatever info hwloc would like passed into its calls - doesn’t have to be 
>> something “understandable” by the proc itself.
> 
> Retrieving cgroups info from Linux isn't expensive so my feeling was to
> still have compute processes do it. But, indeed, we could also avoid
> that step by having the caller pass a hwloc_bitmap_t for allowed PUs and
> another one for allowed NUMA nodes. More below.

I think that minimizing the number of procs hitting the file system for info is 
probably a good thing. In fact, the RM doesn’t have to ask Linux at all for the 
cgroup or other binding (and remember, cgroup is only one way of constraining 
resources) as it is the one assigning that constraint. So it can just pass down 
what it is assigning with no need to do a query.

> 
>> 
>>> On Oct 21, 2016, at 8:15 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>> wrote:
>>> 
>>> Hmmm...I think maybe we are only seeing a small portion of the picture 
>>> here. There are two pieces of the problem when looking at large SMPs:
>>> 
>>> * time required for discovery - your proposal is attempting to address 
>>> that, assuming that the RM daemon collects the topology and then 
>>> communicates it to each process (which is today’s method)
> 
> There's actually an easy way to do that. Export to XML during boot,
> export HWLOC_XMLFILE=/path/to/xml to the processes.

Yeah, but that still leaves everyone parsing an XML file, and having several 
hundred procs hitting the same file is not good

> 
>>> 
>>> * memory footprint. We are seeing over 40MBytes being consumed by hwloc 
>>> topologies on fully loaded KNL machines, which is a disturbing number
> 
> I'd be interested in knowing whether these 40MB are hwloc_obj structure,
> or bitmaps, or info strings, etc. Do you have this information already?

Nope - just overall consumption.

> 
>>> Where we are headed is to having only one copy of the hwloc topology tree 
>>> on a node, stored in a shared memory segment hosted by the local RM daemon.
> 
> And would you need some sort of relative pointers so that all processes
> can traverse parent/child/sibling/... pointers from their own mapping at
> a random address in their address space?

Not sure that is necessary - might be a nice optimization

> 
>>> Procs will then access that tree to obtain any required info. Thus, we are 
>>> less interested in each process creating its own tree based on an XML 
>>> representation passed to it by the RM, and more interested in having the 
>>> hwloc search algorithms correctly handle any resource restrictions when 
>>> searching the RM’s tree.
>>> 
>>> In other words, rather than (or perhaps, in addition to?) filtering the 
>>> XML, we’d prefer to see some modification of the search APIs to allow a 
>>> proc to pass in its resource constraints, and have the search algorithm 
>>> properly consider them when returning the result. This eliminates all the 
>>> XML conversion overhead, and resolves the memory footprint issue.
> 
> What do you call "search algorithm"? We have many functions to walk the
> tree, as levels, from top to bottom, etc. Passing such a resource
> constraints to all of them isn't easy. And we have explicit pointers
> between objects too.

Maybe “search” is the wrong term, but you have functions like 
hwloc_get_obj_by_type, hwloc_get_nbobjs_by_type, and hwloc_bitmap_weight that 
might be impacted by constraints. As you say, maybe just creating a wrapper 
that applies the constraints for the caller (like we do in OMPI) would be 
sufficient.

> 
> Maybe define a good basic set of interesting functions for your search
> algorithm and duplicate these in a new hwloc/allowed.h with a new
> allowed_cpuset attribute? Whenever they find an object, they check
> whether hwloc_bitmap_intersects(obj->cpuset, allowed_cpuset). If FALSE,
> ignore that object.

Sure - I’m open on implementation.

> 
> There's also a related change that I wasn't ready/sure to try yet:
> obj->allowed_cpuset is currently just a duplicate of obj->cpuset in the
> default case. When the WHOLE_SYSTEM topology flag is set, it's a binary
> AND between obj

[OMPI devel] Supercomputing 2016: Birds-of-a-Feather meetings

[r...@open-mpi.org]

Hello all

This year, we will again be hosting Birds-of-a-Feather meetings for Open MPI 
and PMIx. 

Open MPI: Wed, Nov 16th, 5:15-7pm

http://sc16.supercomputing.org/presentation/?id=bof103=sess322 



PMIx: Wed, Nov16th, 12:15-1:15pm:

http://sc16.supercomputing.org/presentation/?id=bof104=sess323 


Please plan to attend!

Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] master nightly tarballs stopped on 11/21

[r...@open-mpi.org]

I’ll turn my crontab back on for the holiday, in case Brian isn’t available - 
worst case, the tarball gets pushed upstream twice.

> On Nov 23, 2016, at 7:59 AM, Pritchard Jr., Howard  wrote:
> 
> Hi Brian,
> 
> Could you check what’s going on with the nightly tarball builds?
> Nothing new has been built since 11/21 even though a number of PR’s
> have been merged in since then.
> 
> Thanks,
> 
> Howard
> 
> -- 
> Howard Pritchard
> HPC-DES
> Los Alamos National Laboratory
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] master nightly tarballs stopped on 11/21

[r...@open-mpi.org]

If you want to take a break, I’ve got them running again and have no problem 
letting them run thru the holiday.

> On Nov 23, 2016, at 8:51 AM, Barrett, Brian <bbarr...@amazon.com> wrote:
> 
> Odd, I'm getting the output of crontab still. Let me run by hand and see 
> what's up :/.
> 
> On Nov 23, 2016, at 08:28, "r...@open-mpi.org <mailto:r...@open-mpi.org>" 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> 
>> I’ll turn my crontab back on for the holiday, in case Brian isn’t available 
>> - worst case, the tarball gets pushed upstream twice.
>> 
>>> On Nov 23, 2016, at 7:59 AM, Pritchard Jr., Howard <howa...@lanl.gov 
>>> <mailto:howa...@lanl.gov>> wrote:
>>> 
>>> Hi Brian,
>>> 
>>> Could you check what’s going on with the nightly tarball builds?
>>> Nothing new has been built since 11/21 even though a number of PR’s
>>> have been merged in since then.
>>> 
>>> Thanks,
>>> 
>>> Howard
>>> 
>>> -- 
>>> Howard Pritchard
>>> HPC-DES
>>> Los Alamos National Laboratory
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
>>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/devel>

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [OMPI users] funny SIGSEGV in 'ompi_info'

[r...@open-mpi.org]

The “correct” answer is, of course, to propagate the error upwards so that the 
highest level caller (e.g., MPI_Init or ompi_info) can return it to the user, 
who can then decide what to do.

Disregarding the parameter is not an option as it violates our “do what the 
user said to do, else return an error” policy

> On Nov 21, 2016, at 9:23 PM, Gilles Gouaillardet  wrote:
> 
> Paul,
> 
> 
> SIGSEGV is always a bad idea, even after having displayed a comprehensive and 
> user friendly error message
> 
> 
>> --
>> MCA framework parameters can only take a single negation operator
>> ("^"), and it must be at the beginning of the value.  The following
>> value violates this rule:
>> 
>> ^tcp,^ib
>> 
>> When used, the negation operator sets the "exclusive" behavior mode,
>> meaning that it will exclude all specified components (and implicitly
>> include all others).  If the negation operator is not specified, the
>> "inclusive" mode is assumed, meaning that all specified components
>> will be included (and implicitly exclude all others).
>> 
>> For example, "^a,b" specifies the exclusive behavior and means "use
>> all components *except* a and b", while "c,d" specifies the inclusive
>> behavior and means "use *only* components c and d."
>> 
>> You cannot mix inclusive and exclusive behavior.
>> --
> 
> 
> that raises the question, what should we do when we run into this case ?
> 
> - one option is to propagate the error (currently, functions do not return 
> anything) (and do what after ?)
> - an other option is to brutally exit(1)
> 
> - yet an other option is to disregard the incorrect value of the parameter 
> and continue
> 
> any thoughts anyone ?
> 
> Cheers,
> 
> Gilles
> 
> On 11/14/2016 9:28 PM, Paul Kapinos wrote:
>> Dear developers, 
>> also the following issue is defintely raised by a misconfiguration of Open 
>> MPI, SIGSEGV's in 'ompi_info' isn'n a good thing, thus this one mail. 
>> 
>> Just call: 
>> $ export OMPI_MCA_mtl="^tcp,^ib" 
>> $ ompi_info --param all all --level 9 
>> ... and take a look at the below core dump of 'ompi_info' like below one. 
>> 
>> (yes we know that "^tcp,^ib" is a bad idea). 
>> 
>> Have a nice day, 
>> 
>> Paul Kapinos 
>> 
>> P.S. Open MPI: 1.10.4 and 2.0.1 have the same behaviour 
>> 
>> -- 
>> [lnm001:39957] *** Process received signal *** 
>> [lnm001:39957] Signal: Segmentation fault (11) 
>> [lnm001:39957] Signal code: Address not mapped (1) 
>> [lnm001:39957] Failing at address: (nil) 
>> [lnm001:39957] [ 0] /lib64/libpthread.so.0(+0xf100)[0x2b30f1a79100] 
>> [lnm001:39957] [ 1] 
>> /opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/libopen-pal.so.13(+0x2f11f)[0x2b30f084911f]
>> [lnm001:39957] [ 2] 
>> /opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/libopen-pal.so.13(+0x2f265)[0x2b30f0849265]
>> [lnm001:39957] [ 3] 
>> /opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/libopen-pal.so.13(opal_info_show_mca_params+0x91)[0x2b30f0849031]
>> [lnm001:39957] [ 4] 
>> /opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/libopen-pal.so.13(opal_info_do_params+0x1f4)[0x2b30f0848e84]
>> [lnm001:39957] [ 5] ompi_info[0x402643] 
>> [lnm001:39957] [ 6] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b30f1ca7b15] 
>> [lnm001:39957] [ 7] ompi_info[0x4022a9] 
>> [lnm001:39957] *** End of error message *** 
>> zsh: segmentation fault (core dumped)  ompi_info --param all all --level 9 
>> -- 
>> 
>> 
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@lists.open-mpi.org 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
>> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Developing MPI program without mpirun

[r...@open-mpi.org]

The 2.0.1 NEWS states that the MPI dynamics operations (comm_spawn, connect, 
and accept) do not work on that release. They are being fixed for the 2.0.2 
release.

> On Nov 18, 2016, at 7:48 AM, Rui Liu  wrote:
> 
> Hi Howard,
> 
> 1. I am using a cluster which involves 20 ubuntu 14.04 servers, and each 
> sever is equipped with infiniBand RDMA for communication and data transfer.
> 2. Open MPI v2.0.1
> 3. I just use prefix option to indicate the install directory with 
> ./configure command, and use make all & sudo make install command further.
> 4. the test app program, which has client and server sides, is very simple. I 
> hope to run them by ./server and ./client command 
> 
> server side:
> 
> #include “mpi.h"
> #include 
> using namespace std;
> 
> int main(int argc,  char **argv) {
>   MPI_Comm client;
>   MPI_Status status;
>   MPI_Init(, );
>   int number = 0;
>   char myport[MPI_MAX_PORT_NAME];
>   MPI_Open_port(MPI_INFO_NULL, myport);
>   MPI_Comm_accept(myport, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>   MPI_Recv(, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, client, 
> );
> }   
> 
> client side:
> 
> #include “mpi.h"
> #include 
> using namespace std;
> int main(int argc, char **argv) {
>   MPI_Comm server;
>   MPI_Init(NULL, NULL);
>   char name[] = "3808886785.0:2871653702”;
>   MPI_Comm_connect(name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>   int number = -1 ;
>   MPI_Send(, 1, MPI_INT, 0, 0, server);
> }
> 
> 
> On 18 November 2016 at 11:24:26 PM, Pritchard Jr., Howard (howa...@lanl.gov 
> ) wrote:
> 
>> Hello Rui,
>> 
>> Note there is no standard for the format of the port_name so don’t
>> read much what it looks like when printed out.
>> 
>> Could you provide some more information about your particular setup:
>> 
>> - characteristics of the system you are using, e.g. a Linux cluster,  laptop 
>> running os-x, etc.
>> - what version of Open MPI you are using
>> - if possible, the configure options used to build/install Open MPI 
>> - the client/server test app (if its concise)
>> 
>> Thanks,
>> 
>> Howard
>> 
>> -- 
>> Howard Pritchard
>> HPC-DES
>> Los Alamos National Laboratory
>> 
>> 
>> From: devel > > on behalf of Rui Liu 
>> >
>> Reply-To: Open MPI Developers > >
>> Date: Thursday, November 17, 2016 at 6:58 PM
>> To: "devel@lists.open-mpi.org " 
>> >
>> Subject: [OMPI devel] Developing MPI program without mpirun
>> 
>> Hi folks,
>> 
>> I am working on a simple client/server program but I don’t want to use 
>> mpirun to start the program (In nature, I just want to adopt MPI API into my 
>> own project but cannot start it using mpirun). I mostly followed the 
>> singleton mode of MPI, using APIs like MPI_Open_port, MPI_Comm_connect, and 
>> MPI_Comm_accept. But it doesn’t work, especially the port_name is something 
>> like 4034413554.0:228712872, which really confuse me.
>> 
>> Anyone knows how to achieve that? I attach my server and client key source 
>> code for your reference.
>> 
>> server side:
>> 
>> char myport[MPI_MAX_PORT_NAME];
>> MPI_Open_port(MPI_INFO_NULL, myport);
>> cout << myport << endl; (assume myport is 4034413554.0:228712872, it always 
>> return string like this)
>> MPI_Comm_accept(myport, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>> MPI_Recv(, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, client, );
>> 
>> 
>> client side:
>> 
>> char name[] = "4034413554.0:228712872";
>> MPI_Comm_connect(name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>> int number = -1 ;
>> MPI_Send(, 1, MPI_INT, 0, 0, server);
>> 
>> Best,
>> Rui
>> ___ 
>> devel mailing list 
>> devel@lists.open-mpi.org 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] direct launch problem with master

[r...@open-mpi.org]

I should hope bisecting would be a last resort. The simplest interim solution 
is to set OMPI_MCA_routed=direct in your environment.

I’ll take a look at a more permanent solution in the morning.

> On Oct 30, 2016, at 6:33 PM, Pritchard Jr., Howard  wrote:
> 
> Hi Folks,
> 
> While trying to solve a different problem, I optimistically tried to use
> head-of –master to work on that problem.  Now I’ve found a new problem
> with master when trying to do a direct launch with SLURM, srun:
> 
> [nid00012:09456] [[27960,0],0] ERROR: Failed to identify the local daemon's 
> URI
> [nid00012:09456] [[27960,0],0] ERROR: This is a fatal condition when the 
> radix router
> [nid00012:09456] [[27960,0],0] ERROR: has been selected - either select the 
> unity router
> [nid00012:09456] [[27960,0],0] ERROR: or ensure that the local daemon info is 
> provided
> [nid00012:09456] [[27960,0],0] ORTE_ERROR_LOG: Fatal in file 
> base/ess_base_std_app.c at line 194
> -
> 
> Any ideas what’s going on here?  With my configure pmix/s1 is selected for 
> pmix component,
> but death follows soon thereafter.  I’m confused why with direct launch Open 
> MPI should
> care at all about what the local deamon’s URI is.
> 
> Is there a way to avoid this problem when using direct launch?  I would do a 
> git bisect 
> but I’ve no time for such activities at the moment.
> 
> Thanks for any suggestions,
> 
> Howard
> 
> -- 
> Howard Pritchard
> HPC-DES
> Los Alamos National Laboratory
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] direct launch problem with master

[r...@open-mpi.org]

Fixed in PR https://github.com/open-mpi/ompi/pull/2322 
<https://github.com/open-mpi/ompi/pull/2322>

> On Oct 31, 2016, at 1:20 AM, r...@open-mpi.org wrote:
> 
> I should hope bisecting would be a last resort. The simplest interim solution 
> is to set OMPI_MCA_routed=direct in your environment.
> 
> I’ll take a look at a more permanent solution in the morning.
> 
>> On Oct 30, 2016, at 6:33 PM, Pritchard Jr., Howard <howa...@lanl.gov 
>> <mailto:howa...@lanl.gov>> wrote:
>> 
>> Hi Folks,
>> 
>> While trying to solve a different problem, I optimistically tried to use
>> head-of –master to work on that problem.  Now I’ve found a new problem
>> with master when trying to do a direct launch with SLURM, srun:
>> 
>> [nid00012:09456] [[27960,0],0] ERROR: Failed to identify the local daemon's 
>> URI
>> [nid00012:09456] [[27960,0],0] ERROR: This is a fatal condition when the 
>> radix router
>> [nid00012:09456] [[27960,0],0] ERROR: has been selected - either select the 
>> unity router
>> [nid00012:09456] [[27960,0],0] ERROR: or ensure that the local daemon info 
>> is provided
>> [nid00012:09456] [[27960,0],0] ORTE_ERROR_LOG: Fatal in file 
>> base/ess_base_std_app.c at line 194
>> -
>> 
>> Any ideas what’s going on here?  With my configure pmix/s1 is selected for 
>> pmix component,
>> but death follows soon thereafter.  I’m confused why with direct launch Open 
>> MPI should
>> care at all about what the local deamon’s URI is.
>> 
>> Is there a way to avoid this problem when using direct launch?  I would do a 
>> git bisect 
>> but I’ve no time for such activities at the moment.
>> 
>> Thanks for any suggestions,
>> 
>> Howard
>> 
>> -- 
>> Howard Pritchard
>> HPC-DES
>> Los Alamos National Laboratory
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org <mailto:devel@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] New Open MPI Community Bylaws to discuss

[r...@open-mpi.org]

The OMPI community members have had their respective legal offices review the 
changes, but we decided to provide notice and get input from others prior to 
the formal vote of acceptance. Once approved, there will no longer be a CLA at 
all. The only requirement for contribution will be the sign-off.

Rationale: the open source world has evolved considerably since we first 
initiated the project. The sign-off method has become the most commonly used 
one for accepting contributions. The CLA was intended primarily to protect the 
contributor, not the project, as it ensured that the contributor had discussed 
their contribution with their employer prior to submitting it.

This approach puts more responsibility on the contributor. It doesn’t impact 
the project very much - trying to “relicense” OMPI would be just as problematic 
today as under the revised bylaws, and quite frankly is something we would 
never envision attempting.

The frequency with which OMPI is receiving pull requests from non-members is 
the driving force here. We have traditionally accepted such contributions “if 
they are small”, but that is too arbitrary. We either have to reject all such 
contributions, or move to a model that allows them. We collectively decided to 
pursue the latter approach, and hence the change to the bylaws.

Just to be clear: only official OMPI members have a vote in this matter. If you 
are not a “member” (e.g., you are a “contributor” status), then this is only 
informational. We respect and want your input, but you don’t actually have a 
vote on this matter.

HTH
Ralph


> On Oct 12, 2016, at 7:24 AM, Pavel Shamis  wrote:
> 
> Well, at least on my side I will not be able to provide the answer without 
> legal involvement. 
> 
> On Wed, Oct 12, 2016 at 9:16 AM, Gilles Gouaillardet 
> > wrote:
> My understanding is there will be a vote, and the question will be
> "Do we replace existing CLA with the new one ?"
> If we vote to do so, then everyone will have to sign-off their commits, 
> regardless they previously had (or not) signed a CA
> 
> Cheers,
> 
> Gilles
> 
> 
> On Wednesday, October 12, 2016, Pavel Shamis  > wrote:
> a. As a developer I think it is a good idea to lower barriers for code 
> contribution.
> b. IANAL, but this "signature/certification" is not identical to the existing 
> CLA, which I think has special statement about patents. Seems like the new 
> model is a bit more relaxed. Does it mean that OMPI amends existing CLA ? If 
> not - what is the relation between the two. Most likely existing member would 
> have to take the "new" CLA to the legal for a review.
> 
> -Pasha
> 
> On Wed, Oct 12, 2016 at 8:38 AM, George Bosilca > 
> wrote:
> Yes, my understanding is that unsystematic contributors will not have to sign 
> the contributor agreement, but instead will have to provide a signed patch.
> 
>   George.
> 
> 
> On Wed, Oct 12, 2016 at 9:29 AM, Pavel Shamis > 
> wrote:
> Does it mean that contributors don't have to sign contributor agreement ?
> 
> On Tue, Oct 11, 2016 at 2:35 PM, Geoffrey Paulsen > 
> wrote:
> We have been discussing new Bylaws for the Open MPI Community.  The primary 
> motivator is to allow non-members to commit code.  Details in the proposal 
> (link below).
>  
> Old Bylaws / Procedures:  
> https://github.com/open-mpi/ompi/wiki/Admistrative-rules 
> 
> 
> New Bylaws proposal: 
> https://github.com/open-mpi/ompi/wiki/Proposed-New-Bylaws 
> 
>  
> Open MPI members will be voting on October 25th.  Please voice any comments 
> or concerns.
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org <>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> 

Re: [OMPI devel] New Open MPI Community Bylaws to discuss

[r...@open-mpi.org]

According to the existing bylaws, only those designated as “members” have 
voting rights when it comes to such administrative matters. The CLA solely 
dealt with the right to contribute code - the membership “level” was a separate 
matter.

In the revised bylaws, this tiered membership model has been abandoned. Members 
are still the only ones with voting rights, and membership is still something 
you have to apply for, be nominated, and voted in by existing members. However, 
the other designated community levels have been eliminated.


> On Oct 12, 2016, at 7:46 AM, Pavel Shamis <pasharesea...@gmail.com> wrote:
> 
> Regardless, I would have to notify legal teams about amendment of the 
> existing CLA. If organizations that already signed the agreement don't have 
> any say, then this conversation is pointless. 
> 
> -Pasha
> 
> On Wed, Oct 12, 2016 at 9:29 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> The OMPI community members have had their respective legal offices review the 
> changes, but we decided to provide notice and get input from others prior to 
> the formal vote of acceptance. Once approved, there will no longer be a CLA 
> at all. The only requirement for contribution will be the sign-off.
> 
> Rationale: the open source world has evolved considerably since we first 
> initiated the project. The sign-off method has become the most commonly used 
> one for accepting contributions. The CLA was intended primarily to protect 
> the contributor, not the project, as it ensured that the contributor had 
> discussed their contribution with their employer prior to submitting it.
> 
> This approach puts more responsibility on the contributor. It doesn’t impact 
> the project very much - trying to “relicense” OMPI would be just as 
> problematic today as under the revised bylaws, and quite frankly is something 
> we would never envision attempting.
> 
> The frequency with which OMPI is receiving pull requests from non-members is 
> the driving force here. We have traditionally accepted such contributions “if 
> they are small”, but that is too arbitrary. We either have to reject all such 
> contributions, or move to a model that allows them. We collectively decided 
> to pursue the latter approach, and hence the change to the bylaws.
> 
> Just to be clear: only official OMPI members have a vote in this matter. If 
> you are not a “member” (e.g., you are a “contributor” status), then this is 
> only informational. We respect and want your input, but you don’t actually 
> have a vote on this matter.
> 
> HTH
> Ralph
> 
> 
>> On Oct 12, 2016, at 7:24 AM, Pavel Shamis <pasharesea...@gmail.com 
>> <mailto:pasharesea...@gmail.com>> wrote:
>> 
>> Well, at least on my side I will not be able to provide the answer without 
>> legal involvement. 
>> 
>> On Wed, Oct 12, 2016 at 9:16 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>> My understanding is there will be a vote, and the question will be
>> "Do we replace existing CLA with the new one ?"
>> If we vote to do so, then everyone will have to sign-off their commits, 
>> regardless they previously had (or not) signed a CA
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> 
>> On Wednesday, October 12, 2016, Pavel Shamis <pasharesea...@gmail.com 
>> <mailto:pasharesea...@gmail.com>> wrote:
>> a. As a developer I think it is a good idea to lower barriers for code 
>> contribution.
>> b. IANAL, but this "signature/certification" is not identical to the 
>> existing CLA, which I think has special statement about patents. Seems like 
>> the new model is a bit more relaxed. Does it mean that OMPI amends existing 
>> CLA ? If not - what is the relation between the two. Most likely existing 
>> member would have to take the "new" CLA to the legal for a review.
>> 
>> -Pasha
>> 
>> On Wed, Oct 12, 2016 at 8:38 AM, George Bosilca <bosi...@icl.utk.edu <>> 
>> wrote:
>> Yes, my understanding is that unsystematic contributors will not have to 
>> sign the contributor agreement, but instead will have to provide a signed 
>> patch.
>> 
>>   George.
>> 
>> 
>> On Wed, Oct 12, 2016 at 9:29 AM, Pavel Shamis <pasharesea...@gmail.com <>> 
>> wrote:
>> Does it mean that contributors don't have to sign contributor agreement ?
>> 
>> On Tue, Oct 11, 2016 at 2:35 PM, Geoffrey Paulsen <gpaul...@us.ibm.com <>> 
>> wrote:
>> We have been discussing new Bylaws

Re: [OMPI devel] Errors with CXX=pgc++ (but CXX=pgCC OK)

[r...@open-mpi.org]

Added to 1.10 README - thanks!

> On Dec 16, 2016, at 4:18 PM, Paul Hargrove  wrote:
> 
> With the 1.10.r5c1 tarball on linux/x86-64 and various versions of the PGI 
> compilers I have configured with
> --prefix=[...] --enable-debug CC=pgcc CXX=pgc++ FC=pgfortran
> 
> I see the following with version 14.3 of the PGI compilers:
> 
> /bin/bash ../../../libtool  --tag=CXX   --mode=link pgc++  -g  -version-info 
> 2:3:1  -o libmpi_cxx.la  -rpath 
> /sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/INST/lib 
> mpicxx.lo intercepts.lo comm.lo datatype.lo win.lo file.lo 
> ../../../ompi/libmpi.la  -lrt -lutil
> libtool: link: pgc++  -fPIC -DPIC -shared -nostdlib 
> /usr/lib/x86_64-linux-gnu/crti.o 
> /nfs/software/linux-ubuntu_precise_amd64/com/packages/pgi/143/linux86-64/14.3/libso/trace_init.o
>  /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginS.o 
> /nfs/software/linux-ubuntu_precise_amd64/com/packages/pgi/143/linux86-64/14.3/libso/initmp.o
>   .libs/mpicxx.o .libs/intercepts.o .libs/comm.o .libs/datatype.o .libs/win.o 
> .libs/file.o   -Wl,-rpath 
> -Wl,/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/ompi/.libs
>  -Wl,-rpath 
> -Wl,/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/orte/.libs
>  -Wl,-rpath 
> -Wl,/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/opal/.libs
>  -Wl,-rpath 
> -Wl,/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/INST/lib 
> -L/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/orte/.libs 
> -L/sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/opal/.libs 
> ../../../ompi/.libs/libmpi.so 
> /sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/orte/.libs/libopen-rte.so
>  
> /sandbox/hargrove/OMPI/openmpi-1.10.5rc1-linux-x86_64-pgi-14/BLD/opal/.libs/libopen-pal.so
>  -ldl -lrt -lutil 
> -L/nfs/software/linux-ubuntu_precise_amd64/com/packages/pgi/143/linux86-64/14.3/libso
>  -L/soft/com/packages/pgi/143/14.3/share_objects/lib64 
> -L/nfs/software/linux-ubuntu_precise_amd64/com/packages/pgi/143/linux86-64/14.3/lib
>  -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -lpgatm -lgcc_s -lstdc++ 
> -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lc -lgcc 
> /usr/lib/gcc/x86_64-linux-gnu/4.6/crtendS.o /usr/lib/x86_64-linux-gnu/crtn.o  
> -g   -Wl,-soname -Wl,libmpi_cxx.so.1 -o .libs/libmpi_cxx.so.1.1.3
> pgc++-Error-Unknown switch: -nostdlib
> make[2]: *** [libmpi_cxx.la ] Error 1
> 
> Switching from CXX=pgc++ to CXX=pgCC eliminates the problem.
> 
> This is *possibly* related to the following configure output in which pgc++ 
> has been misidentified as the GNU compiler:
> $ grep 'compiler vendor' configure.log
> checking for the C compiler vendor... portland group
> checking for the C++ compiler vendor... gnu
> checking for the C++ compiler vendor... (cached) gnu
> checking for the C compiler vendor... portland group
> I suspect that pgc++ is intentionally masquerading as a GNU compiler, but 
> falling short of the mark as of their 14.3 release.
> 
> I didn't see this in last night's testing of 2.0.2rc1 because I was 
> configuring with CXX=pgCC on my Linux/x86-64 systems.
> However testing pgc++ today with 2.0.2rc1 I see pretty much the same results 
> with both release candidates:
> 
> With PGI 12.10 and 13.9 configure decides that CC and CXX are not link 
> compatible (but not when CXX=pgCC)
> This is true of both the 1.10 and 2.0 RCs
> 
> With PGI 14.3, v1.10.5rc1 fails as described above, while 2.0.2rc1 (w/ c++ 
> bindings disabled by default) was OK.
> However, enabling the c++ bindings in 2.0.2rc1 leads to the same error shown 
> above.
> 
> With PGI 15.9, pgc++ unfortunately gets an unrelated ICE compiling VT (which 
> also vanishes with CXX=pgCC) for 1.10, and is fine for 2.0.
> 
> With PGI 16.10, pgcc gets an unrelated SEGV compiling mtl_ofi_component.c, 
> but has no problems with pgc++ on either branch once I configure using 
> --without-libfabric.
> 
> 
> Based on my experiences listed above, I would recommend (in the Open MPI 
> README):
> If building C++ bindings or VT, I advise against use of CXX=pgc++ prior to 
> 16.10.
> Otherwise, it appears usable from 14.3 forward.
> 
> -Paul
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov 
> 
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [OMPI users] still segmentation fault with openmpi-2.0.2rc3 on Linux

[r...@open-mpi.org]

Looking at this note again: how many procs is spawn_master generating?

> On Jan 11, 2017, at 7:39 PM, r...@open-mpi.org wrote:
> 
> Sigh - yet another corner case. Lovely. Will take a poke at it later this 
> week. Thx for tracking it down
> 
>> On Jan 11, 2017, at 5:27 PM, Gilles Gouaillardet <gil...@rist.or.jp 
>> <mailto:gil...@rist.or.jp>> wrote:
>> 
>> Ralph,
>> 
>> 
>> 
>> so it seems the root cause is a kind of incompatibility between the --host 
>> and the --slot-list options
>> 
>> 
>> 
>> on a single node with two six cores sockets, 
>> 
>> this works :
>> 
>> mpirun -np 1 ./spawn_master 
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 ./spawn_master
>> mpirun -np 1 --host motomachi --oversubscribe ./spawn_master 
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:12 ./spawn_master 
>> 
>> 
>> this does not work
>>  
>> mpirun -np 1 --host motomachi ./spawn_master # not enough slots available, 
>> aborts with a user friendly error message
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi ./spawn_master # 
>> various errors sm_segment_attach() fails, a task crashes
>> and this ends up with the following error message
>> 
>> At least one pair of MPI processes are unable to reach each other for
>> MPI communications.  This means that no Open MPI device has indicated
>> that it can be used to communicate between these processes.  This is
>> an error; Open MPI requires that all MPI processes be able to reach
>> each other.  This error can sometimes be the result of forgetting to
>> specify the "self" BTL.
>> 
>>   Process 1 ([[15519,2],0]) is on host: motomachi
>>   Process 2 ([[15519,2],1]) is on host: unknown!
>>   BTLs attempted: self tcp
>> 
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:1 ./spawn_master # 
>> same error as above
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:2 ./spawn_master # 
>> same error as above
>> 
>> 
>> for the record, the following command surprisingly works
>> 
>> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:3 --mca btl tcp,self 
>> ./spawn_master
>> 
>> 
>> 
>> bottom line, my guess is that when the user specifies the --slot-list and 
>> the --host options
>> *and* there are no default slot numbers to hosts, we should default to using 
>> the number
>> of slots from the slot list.
>> (e.g. in this case, defaults to --host motomachi:12 instead of (i guess) 
>> --host motomachi:1)
>> 
>> 
>> /* fwiw, i made
>> 
>> https://github.com/open-mpi/ompi/pull/2715 
>> <https://github.com/open-mpi/ompi/pull/2715>
>> https://github.com/open-mpi/ompi/pull/2715 
>> <https://github.com/open-mpi/ompi/pull/2715>
>> but these are not the root cause */
>> 
>> 
>> 
>> Cheers,
>> 
>> 
>> 
>> Gilles
>> 
>> 
>> 
>> 
>>  Forwarded Message 
>> Subject: Re: [OMPI users] still segmentation fault with openmpi-2.0.2rc3 
>> on Linux
>> Date:Wed, 11 Jan 2017 20:39:02 +0900
>> From:Gilles Gouaillardet <gilles.gouaillar...@gmail.com> 
>> <mailto:gilles.gouaillar...@gmail.com>
>> Reply-To:Open MPI Users <us...@lists.open-mpi.org> 
>> <mailto:us...@lists.open-mpi.org>
>> To:  Open MPI Users <us...@lists.open-mpi.org> 
>> <mailto:us...@lists.open-mpi.org>
>> 
>> Siegmar,
>> 
>> Your slot list is correct.
>> An invalid slot list for your node would be 0:1-7,1:0-7
>> 
>> /* and since the test requires only 5 tasks, that could even work with such 
>> an invalid list.
>> My vm is single socket with 4 cores, so a 0:0-4 slot list results in an 
>> unfriendly pmix error */
>> 
>> Bottom line, your test is correct, and there is a bug in v2.0.x that I will 
>> investigate from tomorrow 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Wednesday, January 11, 2017, Siegmar Gross 
>> <siegmar.gr...@informatik.hs-fulda.de 
>> <mailto:siegmar.gr...@informatik.hs-fulda.de>> wrote:
>> Hi Gilles,
>> 
>> thank you very much for your help. What does incorrect slot list
>> mean? My machine has two 6-core processors so that I specified
>> "--slot-list 0:0-5,1:0-5". Does incorrect mean that it isn't
>> allowed to specify more slots than available, to specify fewer
>> slots than available, or to specify more slots than needed for
>> the process

Re: [OMPI devel] Fwd: Re: [OMPI users] still segmentation fault with openmpi-2.0.2rc3 on Linux

[r...@open-mpi.org]

Sigh - yet another corner case. Lovely. Will take a poke at it later this week. 
Thx for tracking it down

> On Jan 11, 2017, at 5:27 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Ralph,
> 
> 
> 
> so it seems the root cause is a kind of incompatibility between the --host 
> and the --slot-list options
> 
> 
> 
> on a single node with two six cores sockets, 
> 
> this works :
> 
> mpirun -np 1 ./spawn_master 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 ./spawn_master
> mpirun -np 1 --host motomachi --oversubscribe ./spawn_master 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:12 ./spawn_master 
> 
> 
> this does not work
>  
> mpirun -np 1 --host motomachi ./spawn_master # not enough slots available, 
> aborts with a user friendly error message
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi ./spawn_master # 
> various errors sm_segment_attach() fails, a task crashes
> and this ends up with the following error message
> 
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
> 
>   Process 1 ([[15519,2],0]) is on host: motomachi
>   Process 2 ([[15519,2],1]) is on host: unknown!
>   BTLs attempted: self tcp
> 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:1 ./spawn_master # same 
> error as above
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:2 ./spawn_master # same 
> error as above
> 
> 
> for the record, the following command surprisingly works
> 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:3 --mca btl tcp,self 
> ./spawn_master
> 
> 
> 
> bottom line, my guess is that when the user specifies the --slot-list and the 
> --host options
> *and* there are no default slot numbers to hosts, we should default to using 
> the number
> of slots from the slot list.
> (e.g. in this case, defaults to --host motomachi:12 instead of (i guess) 
> --host motomachi:1)
> 
> 
> /* fwiw, i made
> 
> https://github.com/open-mpi/ompi/pull/2715 
> <https://github.com/open-mpi/ompi/pull/2715>
> https://github.com/open-mpi/ompi/pull/2715 
> <https://github.com/open-mpi/ompi/pull/2715>
> but these are not the root cause */
> 
> 
> 
> Cheers,
> 
> 
> 
> Gilles
> 
> 
> 
> 
>  Forwarded Message 
> Subject:  Re: [OMPI users] still segmentation fault with openmpi-2.0.2rc3 
> on Linux
> Date: Wed, 11 Jan 2017 20:39:02 +0900
> From: Gilles Gouaillardet <gilles.gouaillar...@gmail.com> 
> <mailto:gilles.gouaillar...@gmail.com>
> Reply-To: Open MPI Users <us...@lists.open-mpi.org> 
> <mailto:us...@lists.open-mpi.org>
> To:   Open MPI Users <us...@lists.open-mpi.org> 
> <mailto:us...@lists.open-mpi.org>
> 
> Siegmar,
> 
> Your slot list is correct.
> An invalid slot list for your node would be 0:1-7,1:0-7
> 
> /* and since the test requires only 5 tasks, that could even work with such 
> an invalid list.
> My vm is single socket with 4 cores, so a 0:0-4 slot list results in an 
> unfriendly pmix error */
> 
> Bottom line, your test is correct, and there is a bug in v2.0.x that I will 
> investigate from tomorrow 
> 
> Cheers,
> 
> Gilles
> 
> On Wednesday, January 11, 2017, Siegmar Gross 
> <siegmar.gr...@informatik.hs-fulda.de 
> <mailto:siegmar.gr...@informatik.hs-fulda.de>> wrote:
> Hi Gilles,
> 
> thank you very much for your help. What does incorrect slot list
> mean? My machine has two 6-core processors so that I specified
> "--slot-list 0:0-5,1:0-5". Does incorrect mean that it isn't
> allowed to specify more slots than available, to specify fewer
> slots than available, or to specify more slots than needed for
> the processes?
> 
> 
> Kind regards
> 
> Siegmar
> 
> Am 11.01.2017 um 10:04 schrieb Gilles Gouaillardet:
> Siegmar,
> 
> I was able to reproduce the issue on my vm
> (No need for a real heterogeneous cluster here)
> 
> I will keep digging tomorrow.
> Note that if you specify an incorrect slot list, MPI_Comm_spawn fails with a 
> very unfriendly error message.
> Right now, the 4th spawn'ed task crashes, so this is a different issue
> 
> Cheers,
> 
> Gilles
> 
> r...@open-mpi.org <> wrote:
> I think there is some relevant discussion here: 
> https://github.com/open-mpi/ompi/issues/1569 
> <https://git

Re: [OMPI devel] OMPI devel] hwloc missing NUMANode object

[r...@open-mpi.org]

Should be fixed here: https://github.com/open-mpi/ompi/pull/2711 
<https://github.com/open-mpi/ompi/pull/2711>

> On Jan 5, 2017, at 6:42 AM, r...@open-mpi.org wrote:
> 
> I can add a check to see if we have NUMA, and if not we can fall back to 
> socket (if present) or just “none”
> 
>> On Jan 5, 2017, at 1:39 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com> wrote:
>> 
>> Thanks Brice,
>> 
>> Right now, the user facing issue is that numa binding is requested, and 
>> there is no numa, so mpirun aborts.
>> 
>> But you have a good point, we could simply not bind at all in this case 
>> instead of aborting, since the numa node would have been the full machine, 
>> which would have been a noop.
>> 
>> Fwiw,
>> - the default binding policy was changed from to socket to numa (for better 
>> out of the box perfs on KNL iirc)
>> - in btl/sm we malloc(0) when there is no numa, which causes some memory 
>> corruption. The fix is trivial and i will push it tomorrow
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> Brice Goglin <brice.gog...@inria.fr> wrote:
>>> 
>>> 
>>> Le 05/01/2017 07:07, Gilles Gouaillardet a écrit :
>>>> Brice,
>>>> 
>>>> things would be much easier if there were an HWLOC_OBJ_NODE object in
>>>> the topology.
>>>> 
>>>> could you please consider backporting the relevant changes from master
>>>> into the v1.11 branch ?
>>>> 
>>>> Cheers,
>>>> 
>>>> Gilles
>>> 
>>> Hello
>>> Unfortunately, I can't backport this to 1.x. This is very intrusive and
>>> would break other things.
>>> However, what problem are you actually seeing? They are no NUMA node in
>>> hwloc 1.x when the machine isn't NUMA (or when there's no NUMA support
>>> in the operating system but that's very unlikely). hwloc master would
>>> show a single NUMA node that is equivalent to the entire machine, so
>>> binding would be a noop.
>>> Regards
>>> Brice
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [OMPI users] still segmentation fault with openmpi-2.0.2rc3 on Linux

[r...@open-mpi.org]

Fix is pending here: https://github.com/open-mpi/ompi/pull/2730 
<https://github.com/open-mpi/ompi/pull/2730>

> On Jan 12, 2017, at 8:57 AM, Howard Pritchard <hpprit...@gmail.com 
> <mailto:hpprit...@gmail.com>> wrote:
> 
> Siegmar,
> 
> Could you confirm that if you use one of the mpirun arg lists that works for 
> Gilles that
> your test case passes.  Something simple like
> 
> mpirun -np 1 ./spawn_master
> 
> ?
> 
> Howard
> 
> 
> 
> 
> 2017-01-11 18:27 GMT-07:00 Gilles Gouaillardet <gil...@rist.or.jp 
> <mailto:gil...@rist.or.jp>>:
> Ralph,
> 
> 
> so it seems the root cause is a kind of incompatibility between the --host 
> and the --slot-list options
> 
> 
> on a single node with two six cores sockets, 
> this works :
> 
> mpirun -np 1 ./spawn_master 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 ./spawn_master
> mpirun -np 1 --host motomachi --oversubscribe ./spawn_master 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:12 ./spawn_master 
> 
> 
> this does not work
>  
> mpirun -np 1 --host motomachi ./spawn_master # not enough slots available, 
> aborts with a user friendly error message
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi ./spawn_master # 
> various errors sm_segment_attach() fails, a task crashes
> and this ends up with the following error message
> 
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
> 
>   Process 1 ([[15519,2],0]) is on host: motomachi
>   Process 2 ([[15519,2],1]) is on host: unknown!
>   BTLs attempted: self tcp
> 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:1 ./spawn_master # same 
> error as above
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:2 ./spawn_master # same 
> error as above
> 
> 
> for the record, the following command surprisingly works
> 
> mpirun -np 1 --slot-list 0:0-5,1:0-5 --host motomachi:3 --mca btl tcp,self 
> ./spawn_master
> 
> 
> 
> bottom line, my guess is that when the user specifies the --slot-list and the 
> --host options
> *and* there are no default slot numbers to hosts, we should default to using 
> the number
> of slots from the slot list.
> (e.g. in this case, defaults to --host motomachi:12 instead of (i guess) 
> --host motomachi:1)
> 
> 
> /* fwiw, i made
> 
> https://github.com/open-mpi/ompi/pull/2715 
> <https://github.com/open-mpi/ompi/pull/2715>
> https://github.com/open-mpi/ompi/pull/2715 
> <https://github.com/open-mpi/ompi/pull/2715>
> but these are not the root cause */
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> 
>  Forwarded Message 
> Subject:  Re: [OMPI users] still segmentation fault with openmpi-2.0.2rc3 
> on Linux
> Date: Wed, 11 Jan 2017 20:39:02 +0900
> From: Gilles Gouaillardet <gilles.gouaillar...@gmail.com> 
> <mailto:gilles.gouaillar...@gmail.com>
> Reply-To: Open MPI Users <us...@lists.open-mpi.org> 
> <mailto:us...@lists.open-mpi.org>
> To:   Open MPI Users <us...@lists.open-mpi.org> 
> <mailto:us...@lists.open-mpi.org>
> 
> 
> Siegmar,
> 
> Your slot list is correct.
> An invalid slot list for your node would be 0:1-7,1:0-7
> 
> /* and since the test requires only 5 tasks, that could even work with such 
> an invalid list.
> My vm is single socket with 4 cores, so a 0:0-4 slot list results in an 
> unfriendly pmix error */
> 
> Bottom line, your test is correct, and there is a bug in v2.0.x that I will 
> investigate from tomorrow 
> 
> Cheers,
> 
> Gilles
> 
> On Wednesday, January 11, 2017, Siegmar Gross 
> <siegmar.gr...@informatik.hs-fulda.de 
> <mailto:siegmar.gr...@informatik.hs-fulda.de>> wrote:
> Hi Gilles,
> 
> thank you very much for your help. What does incorrect slot list
> mean? My machine has two 6-core processors so that I specified
> "--slot-list 0:0-5,1:0-5". Does incorrect mean that it isn't
> allowed to specify more slots than available, to specify fewer
> slots than available, or to specify more slots than needed for
> the processes?
> 
> 
> Kind regards
> 
> Siegmar
> 
> Am 11.01.2017 um 10:04 schrieb Gilles Gouaillardet:
> Siegmar,
> 
> I was able to reproduce the issue on my vm
> (No need for a real heterogeneous cluster here)
> 
> I will keep digging tomorrow.
> No