Jeff,
this is not what I observed
(tcp btl, 2 to 4 nodes with one task per node, cutoff=0)
the add_procs of the tcp btl is invoked once with the 4 tasks.
I checked the sources and found cutoff only controls if the modex is
invoked once for all at init, or on demand.
Cheers,
Gilles
On Monday, May 16, 2016, Jeff Squyres (jsquyres) <jsquy...@cisco.com> wrote:
> We changed the way BTL add_procs is invoked on master and v2.x for
> scalability reasons.
>
> In short: add_procs is only invoked the first time you talk to a given
> peer. The cutoff switch is an override to that -- if the sizeof COMM_WORLD
> is less than the cutoff, we revert to the old behavior of calling add_procs
> for all procs.
>
> As for why one BTL would be chosen over another, be sure to look at not
> only the priority of the component/module, but also the exclusivity level.
> In short, only BTLs with the same exclusivity level will be considered
> (e.g., this is how we exclude TCP when using HPC-class networks), and then
> the BTL modules with the highest priority will be used for a given peer.
>
>
> > On May 16, 2016, at 7:19 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com <javascript:;>> wrote:
> >
> > it seems I misunderstood some things ...
> >
> > add_procs is always invoked, regardless the cutoff value.
> > cutoff is used to retrieve processes info via the modex "on demand" vs
> at init time.
> >
> > Please someone correct me and/or elaborate if needed
> >
> > Cheers,
> >
> > Gilles
> >
> > On Monday, May 16, 2016, Gilles Gouaillardet <gil...@rist.or.jp
> <javascript:;>> wrote:
> > i cannot reproduce this behavior.
> >
> > note mca_btl_tcp_add_procs is invoked once per tcp component (e.g. once
> per physical NIC)
> >
> > so you might want to explicitly select one nic
> >
> > mpirun --mca btl_tcp_if_include xxx ...
> >
> > my printf output are the same and regardless the mpi_add_procs_cutoff
> value
> >
> >
> > Cheers,
> >
> >
> > Gilles
> > On 5/16/2016 12:22 AM, dpchoudh . wrote:
> >> Sorry, I accidentally pressed 'Send' before I was done writing the last
> mail. What I wanted to ask was what is the parameter mpi_add_procs_cutoff
> and why adding it seems to make a difference in the code path but not in
> the end result of the program? How would it help me debug my problem?
> >>
> >> Thank you
> >> Durga
> >>
> >> The surgeon general advises you to eat right, exercise regularly and
> quit ageing.
> >>
> >> On Sun, May 15, 2016 at 11:17 AM, dpchoudh . <dpcho...@gmail.com
> <javascript:;>> wrote:
> >> Hello Gilles
> >>
> >> Setting -mca mpi_add_procs_cutoff 1024 indeed makes a difference to the
> output, as follows:
> >>
> >> With -mca mpi_add_procs_cutoff 1024:
> >> reachable = 0x1
> >> (Note that add_procs was called once and the value of 'reachable is
> correct')
> >>
> >> Without -mca mpi_add_procs_cutoff 1024
> >> reachable = 0x0
> >> reachable = NULL
> >> reachable = NULL
> >> (Note that add_procs() was caklled three times and the value of
> 'reachable' seems wrong.
> >>
> >> The program does run correctly in either case. The program listing is
> as below (note that I have removed output from the program itself in the
> above reporting.)
> >>
> >> The code that prints 'reachable' is as follows:
> >>
> >> if (reachable == NULL)
> >> printf("reachable = NULL\n");
> >> else
> >> {
> >> int i;
> >> printf("reachable = ");
> >> for (i = 0; i < reachable->array_size; i++)
> >> printf("\t0x%llu", reachable->bitmap[i]);
> >> printf("\n\n");
> >> }
> >> return OPAL_SUCCESS;
> >>
> >> And the code for the test program is as follows:
> >>
> >> #include <mpi.h>
> >> #include <stdio.h>
> >> #include <string.h>
> >> #include <stdlib.h>
> >>
> >> int main(int argc, char *argv[])
> >> {
> >> int world_size, world_rank, name_len;
> >> char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
> >>
> >> MPI_Init(&argc, &argv);
> >> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
> >> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
> >> MPI_Get_processor_name(hostname, &name_len);
> >> printf("Hello world from processor %s, rank %d out of %d
> processors\n", hostname, world_rank, world_size);
> >> if (world_rank == 1)
> >> {
> >> MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD,
> MPI_STATUS_IGNORE);
> >> printf("%s received %s, rank %d\n", hostname, buf, world_rank);
> >> }
> >> else
> >> {
> >> strcpy(buf, "haha!");
> >> MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
> >> printf("%s sent %s, rank %d\n", hostname, buf, world_rank);
> >> }
> >> MPI_Barrier(MPI_COMM_WORLD);
> >> MPI_Finalize();
> >> return 0;
> >> }
> >>
> >>
> >>
> >> The surgeon general advises you to eat right, exercise regularly and
> quit ageing.
> >>
> >> On Sun, May 15, 2016 at 10:49 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com <javascript:;>> wrote:
> >> At first glance, that seems a bit odd...
> >> are you sure you correctly print the reachable bitmap ?
> >> I would suggest you add some instrumentation to understand what happens
> >> (e.g., printf before opal_bitmap_set_bit() and other places that
> prevent this from happening)
> >>
> >> one more thing ...
> >> now, master default behavior is
> >> mpirun --mca mpi_add_procs_cutoff 0 ...
> >> you might want to try
> >> mpirun --mca mpi_add_procs_cutoff 1024 ...
> >> and see if things make more sense.
> >> if it helps, and iirc, there is a parameter so a btl can report it does
> not support cutoff.
> >>
> >>
> >> Cheers,
> >>
> >> Gilles
> >>
> >> On Sunday, May 15, 2016, dpchoudh . <dpcho...@gmail.com <javascript:;>>
> wrote:
> >> Hello Gilles
> >>
> >> Thanks for jumping in to help again. Actually, I had already tried some
> of your suggestions before asking for help.
> >>
> >> I have several interconnects that can run both openib and tcp BTL. To
> simplify things, I explicitly mentioned TCP:
> >>
> >> mpirun -np 2 -hostfile ~/hostfile -mca pml ob1 -mca btl self.tcp
> ./mpitest
> >>
> >> where mpitest is a small program that does MPI_Send()/MPI_Recv() on a
> small string, and then does an MPI_Barrier(). The program does work as
> expected.
> >>
> >> I put a printf on the last line of mca_tcp_add_procs() to print the
> value of 'reachable'. What I saw was that the value was always 0 when it
> was invoked for Send()/Recv() and the pointer itself was NULL when invoked
> for Barrier()
> >>
> >> Next I looked at pml_ob1_add_procs(), where the call chain starts, and
> found that it initializes and passes an opal_bitmap_t reachable down the
> call chain, but the resulting value is not used later in the code (the
> memory is simply freed later).
> >>
> >> That, coupled with the fact that I am trying to imitate what the other
> BTL implementations are doing, yet in mca_bml_r2_endpoint_add_btl() by BTL
> is not being picked up, left me puzzled. Please note that the interconnect
> that I am developing for is on a different cluster (than where I ran the
> above test for TCP BTL.)
> >>
> >> Thanks again
> >> Durga
> >>
> >> The surgeon general advises you to eat right, exercise regularly and
> quit ageing.
> >>
> >> On Sun, May 15, 2016 at 10:20 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com <javascript:;>> wrote:
> >> did you check the add_procs callbacks ?
> >> (e.g. mca_btl_tcp_add_procs() for the tcp btl)
> >> this is where the reachable bitmap is set, and I guess this is what you
> are looking for.
> >>
> >> keep in mind that if several btl can be used, the one with the higher
> exclusivity is used
> >> (e.g. tcp is never used if openib is available)
> >> you can simply force your btl and self, and the ob1 pml, so you do not
> have to worry about other btl exclusivity.
> >>
> >> Cheers,
> >>
> >> Gilles
> >>
> >>
> >> On Sunday, May 15, 2016, dpchoudh . <dpcho...@gmail.com <javascript:;>>
> wrote:
> >> Hello all
> >>
> >> I have been struggling with this issue for a while and figured it might
> be a good idea to ask for help.
> >>
> >> Where (in the code path) is the connectivity map created?
> >>
> >> I can see that it is *used* in mca_bml_r2_endpoint_add_btl(), but
> obviously I am not setting it up right, because this routine is not finding
> the BTL corresponding to my interconnect.
> >>
> >> Thanks in advance
> >> Durga
> >>
> >> The surgeon general advises you to eat right, exercise regularly and
> quit ageing.
> >>
> >> _______________________________________________
> >> devel mailing list
> >> de...@open-mpi.org <javascript:;>
> >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/05/18975.php
> >>
> >>
> >> _______________________________________________
> >> devel mailing list
> >> de...@open-mpi.org <javascript:;>
> >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/05/18977.php
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> devel mailing list
> >>
> >> de...@open-mpi.org <javascript:;>
> >>
> >> Subscription:
> >> https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>
> >> Link to this post:
> >> http://www.open-mpi.org/community/lists/devel/2016/05/18979.php
> >
> > _______________________________________________
> > devel mailing list
> > de...@open-mpi.org <javascript:;>
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/05/18981.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com <javascript:;>
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org <javascript:;>
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/05/18982.php
>
