Re: [OMPI users] collective algorithms
Gilles, Thanx for the valuable information. So this solves part of the puzzle. The next thing is know the cost of these algorithms. Some of them seem to be standard, however, I am afraid there could be some modifications that will ultimately alter the cost. Hence I asked for a paper. I will look around..again Thanx --- Daniel Faraj From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gilles Gouaillardet Sent: Monday, November 17, 2014 10:07 PM To: Open MPI Users Subject: Re: [OMPI users] collective algorithms Daniel, you can run $ ompi_info --parseable --all | grep _algorithm: | grep enumerator that will give you the list of supported algo for the collectives, here is a sample output : mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:0:ignore mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:1:basic_linear mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:2:nonoverlapping mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:3:recursive_doubling mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:4:ring mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:5:segmented_ring the decision (which algo is used based on communicator size/message size/...) is made in ompi/mca/coll/tuned/coll_tuned_decision_fixed.c and can be overriden via config file or environment variable i cannot point you to a paper, and hopefully someone else will Cheers, Gilles On 2014/11/18 12:53, Faraj, Daniel A wrote: I am trying to survey the collective algorithms in Open MPI. I looked at the src code but could not make out the guts of the communication algorithms. There are some open mpi papers but not detailed, where they talk about what algorithms are using in certain collectives. Has anybody done this sort of work, or point me to a paper? Basically, for a given collective operation, what are: a) Communication algorithm being used for a given criteria (i.e. message size or np) b) What is theoretical algorithm cost Thanx --- Daniel Faraj ___ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2014/11/25831.php
Re: [OMPI users] collective algorithms
Daniel, Many papers have been published about the performance modeling of different collective communications algorithms (and fortunately these models are implementation independent). I can pinpoint you to our research in collective modeling which is the underlying infrastructure behind the decision function in the tuned collective module in Open MPI ( http://link.springer.com/article/10.1007/s10586-007-0012-0). However, Google Scholar can help with discovering other interesting approaches. George. On Mon, Nov 17, 2014 at 9:53 PM, Faraj, Daniel A wrote: > I am trying to survey the collective algorithms in Open MPI. > > I looked at the src code but could not make out the guts of the > communication algorithms. > > There are some open mpi papers but not detailed, where they talk about > what algorithms are using in certain collectives. > > Has anybody done this sort of work, or point me to a paper? > > > > Basically, for a given collective operation, what are: > > a) Communication algorithm being used for a given criteria (i.e. > message size or np) > > b) What is theoretical algorithm cost > > > > Thanx > > > > > > --- > > Daniel Faraj > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25831.php >
Re: [OMPI users] collective algorithms
Daniel, you can run $ ompi_info --parseable --all | grep _algorithm: | grep enumerator that will give you the list of supported algo for the collectives, here is a sample output : mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:0:ignore mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:1:basic_linear mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:2:nonoverlapping mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:3:recursive_doubling mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:4:ring mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:5:segmented_ring the decision (which algo is used based on communicator size/message size/...) is made in ompi/mca/coll/tuned/coll_tuned_decision_fixed.c and can be overriden via config file or environment variable i cannot point you to a paper, and hopefully someone else will Cheers, Gilles On 2014/11/18 12:53, Faraj, Daniel A wrote: > I am trying to survey the collective algorithms in Open MPI. > I looked at the src code but could not make out the guts of the communication > algorithms. > There are some open mpi papers but not detailed, where they talk about what > algorithms are using in certain collectives. > Has anybody done this sort of work, or point me to a paper? > > Basically, for a given collective operation, what are: > > a) Communication algorithm being used for a given criteria (i.e. message > size or np) > > b) What is theoretical algorithm cost > > Thanx > > > --- > Daniel Faraj > > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25831.php
Re: [OMPI users] collective algorithms
On Dec 10, 2008, at 12:05 AM, Максим Чусовлянов wrote: Hello! I appreciate your help. I received a library of my algorithm's. But I don't know how to use it. Can you explain what you mean / what you did? E.g., did you build your algorithm code inside an OMPI tree and now have an mca_coll_.so file? If so, a "make install" should put that dynamic shared object (DSO) in the Right location in the OMPI installation tree such that an "ompi_info | grep coll" should show your component along with the others that OMPI installs. If all that happens properly, and you set your communicator query priority high enough (from inside the module query function), your component should get called for the back-ends of the various MPI collectives. -- Jeff Squyres Cisco Systems
Re: [OMPI users] collective algorithms
Hello! I appreciate your help. I received a library of my algorithm's. But I don't know how to use it. 2008/12/6 Jeff Squyres > On Dec 6, 2008, at 10:32 AM, Максим Чусовлянов wrote: > > Hello. Thanks for your response. I'm create a my module and component >> into the coll. But when I execute autogen.sh, it's return ../../libtool: >> line 847: X--tag=CC: command not found >> ../../libtool: line 880: libtool: ignoring unknown tag : command not found >> ../../libtool: line 847: X--mode=compile: command not found >> I have a libtool v 1.5.26. Can you get me a some guide about create >> configure.m4. >> > > You might want to upgrade your Libtool; Libtool 2.x has been stable for > quite some time now. See the HACKING file in the top-level Open MPI > directory for how to upgrade your GNU Auto/Libtools. > > If upgrading your Auto tools doesn't work, let's move this conversation to > the devel list and send all relevant information (e.g., your configure.m4, > autogen.sh output, etc.), and we'll diagnose from there. > > Good example configure.m4 files can be found throughout the OMPI code base; > do a "find . -name configure.m4" from the top-level dir and you'll see all > of them. A relatively simple one to template from is > ompi/mca/btl/tcp/configure.m4 (it only has one test in it, but you can see > that you're required to define an M4 macro named > MCA_coll__CONFIG). Just make sure to > s/btl_tcp/coll_/ in your configure.m4. Keep in mind > that per my first mail, you may not need a configure.m4 -- you only need > configure.m4 if you need configure to figure out if your component can build > (e.g., if you need some specific libraries or header files that aren't > generally universally installed on POSIX-like operating systems, such as > support for specific network libraries/hardware like MX, OpenFabrics, etc.). > > -- > Jeff Squyres > Cisco Systems > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] collective algorithms
On Dec 6, 2008, at 10:32 AM, Максим Чусовлянов wrote: Hello. Thanks for your response. I'm create a my module and component into the coll. But when I execute autogen.sh, it's return ../../libtool: line 847: X--tag=CC: command not found ../../libtool: line 880: libtool: ignoring unknown tag : command not found ../../libtool: line 847: X--mode=compile: command not found I have a libtool v 1.5.26. Can you get me a some guide about create configure.m4. You might want to upgrade your Libtool; Libtool 2.x has been stable for quite some time now. See the HACKING file in the top-level Open MPI directory for how to upgrade your GNU Auto/Libtools. If upgrading your Auto tools doesn't work, let's move this conversation to the devel list and send all relevant information (e.g., your configure.m4, autogen.sh output, etc.), and we'll diagnose from there. Good example configure.m4 files can be found throughout the OMPI code base; do a "find . -name configure.m4" from the top-level dir and you'll see all of them. A relatively simple one to template from is ompi/mca/btl/tcp/configure.m4 (it only has one test in it, but you can see that you're required to define an M4 macro named MCA_coll__CONFIG). Just make sure to s/btl_tcp/ coll_/ in your configure.m4. Keep in mind that per my first mail, you may not need a configure.m4 -- you only need configure.m4 if you need configure to figure out if your component can build (e.g., if you need some specific libraries or header files that aren't generally universally installed on POSIX-like operating systems, such as support for specific network libraries/hardware like MX, OpenFabrics, etc.). -- Jeff Squyres Cisco Systems
Re: [OMPI users] collective algorithms
Hello. Thanks for your response. I'm create a my module and component into the coll. But when I execute autogen.sh, it's return ../../libtool: line 847: X--tag=CC: command not found ../../libtool: line 880: libtool: ignoring unknown tag : command not found ../../libtool: line 847: X--mode=compile: command not found I have a libtool v 1.5.26. Can you get me a some guide about create configure.m4. 2008/12/2 Jeff Squyres > On Nov 25, 2008, at 10:29 AM, Максим Чусовлянов wrote: > > Hello! How i can integrated my collective communication algorithm in >> openMPI with MCA? >> > > Sorry for the delay in answering -- SC08 and the US holiday last week got > in the way and I'm way behind on answering the mails in my INBOX. > > Just to make sure we're talking about the same thing -- you have a new > collective algorithm for one of the MPI collective functions, and you want > to include that code in Open MPI so that it can be invoked by MPI_ in > MPI applications, right? > > If so, the right way to do this is to build a new Open MPI "coll" > (collective) component containing the code for your new algorithm. Our coll > components are basically a few housekeeping functions and a bunch of > function pointers for the functions to call that are the back-ends to MPI > collective functions (i.e., MPI_Bcast and friends). > > All the "coll" component code is under the ompi/mca/coll/ directory. The > "base" directory is some "glue" code for the coll framework itself -- it's > not a component. But all other directories are standalone components that > have corresponding dynamic shared objects (DSOs) installed under $pkglibdir > (typically $prefix/lib/openmpi). > > You can build a component inside or outside of the Open MPI tree. If you > build outside of the Open MPI tree, you need to configure OMPI with > --with-devel-headers, which will install all of OMPI's internal headers > under $prefix. That way, you can -I these headers when you compile your > component. Just install your DSO in $pkglibdir; if all goes well, > "ompi_info | grep coll" should show your component. > > If you build inside of the Open MPI tree, you need to make your component > dir under ompi/mca/coll/ and include a configure.params file (look at > ompi/mca/coll/basic/configure.params for a simple example) and a Makefile.am > (see ompi/mca/coll/basic/Makefile.am for an example). Then run the > "autogen.sh" script that is at the top of the tree and then run configure. > You should see your component listed in both the autogen.sh and configure > output; configure should not that it plans to build that component. When > you finish configure, build and install Open MPI. "ompi_info | grep coll" > should show your component. > > But I'm getting ahead of myself... Let's go back a few steps... > > When building inside the OMPI tree, if you need to check for various things > to determine if you can build the component (i.e., some tests during > configure, such as checking for various hardware support libraries), you can > also add a configure.m4 file in your component's directory. This gets a > little tricky if you're not familiar with Autoconf; let me know if you need > some guidance here. > > Now you can add the source code to the component. We have 2 important > abstractions that you need to know about: > > - component: there is only one component instance in an MPI process. It > has global state. > - module: in the coll framework, there is one module instance for every > communicator that uses this component. It has local state relevant to that > specific communicator. > > Think of "component" as a C++ class, and "module" as a C++ object. > > Now read the comments in ompi/mca/coll/coll.h. This file contains the > struct interfaces for both the coll component and module. We basically do > everything by function pointer; the component returns a set of function > pointers and each module returns a struct of function pointers. These > function pointers are invoked by libmpi at various times for various > functions; see coll.h for a description of each. > > During coll module initialization (i.e., when a new communicator has been > created), there's a process called "selection" where OMPI determines which > coll modules will be used on this communicator. Modules can include/exclude > themselves from the selection process. For example, your algorithm may only > be suitable for intracommunicators. So if the communicator in question that > is being created is an intercommunicator, you probably want to exclude your > module from selection. Or if your algorithm can only handle powers-of-two > MPI processes, it should exclude itself if there is a non-power-of-two > number of processes in the communicator. And so on. > > We designed coll modules in OMPI v1.3 to be "mix-n-match"-able such that in > a single communicator, you can use the broadcast function from one module, > but the gather function from a different module. Hence, multiple coll > mod
Re: [OMPI users] collective algorithms
On Nov 25, 2008, at 10:29 AM, Максим Чусовлянов wrote: Hello! How i can integrated my collective communication algorithm in openMPI with MCA? Sorry for the delay in answering -- SC08 and the US holiday last week got in the way and I'm way behind on answering the mails in my INBOX. Just to make sure we're talking about the same thing -- you have a new collective algorithm for one of the MPI collective functions, and you want to include that code in Open MPI so that it can be invoked by MPI_ in MPI applications, right? If so, the right way to do this is to build a new Open MPI "coll" (collective) component containing the code for your new algorithm. Our coll components are basically a few housekeeping functions and a bunch of function pointers for the functions to call that are the back-ends to MPI collective functions (i.e., MPI_Bcast and friends). All the "coll" component code is under the ompi/mca/coll/ directory. The "base" directory is some "glue" code for the coll framework itself -- it's not a component. But all other directories are standalone components that have corresponding dynamic shared objects (DSOs) installed under $pkglibdir (typically $prefix/lib/openmpi). You can build a component inside or outside of the Open MPI tree. If you build outside of the Open MPI tree, you need to configure OMPI with --with-devel-headers, which will install all of OMPI's internal headers under $prefix. That way, you can -I these headers when you compile your component. Just install your DSO in $pkglibdir; if all goes well, "ompi_info | grep coll" should show your component. If you build inside of the Open MPI tree, you need to make your component dir under ompi/mca/coll/ and include a configure.params file (look at ompi/mca/coll/basic/configure.params for a simple example) and a Makefile.am (see ompi/mca/coll/basic/Makefile.am for an example). Then run the "autogen.sh" script that is at the top of the tree and then run configure. You should see your component listed in both the autogen.sh and configure output; configure should not that it plans to build that component. When you finish configure, build and install Open MPI. "ompi_info | grep coll" should show your component. But I'm getting ahead of myself... Let's go back a few steps... When building inside the OMPI tree, if you need to check for various things to determine if you can build the component (i.e., some tests during configure, such as checking for various hardware support libraries), you can also add a configure.m4 file in your component's directory. This gets a little tricky if you're not familiar with Autoconf; let me know if you need some guidance here. Now you can add the source code to the component. We have 2 important abstractions that you need to know about: - component: there is only one component instance in an MPI process. It has global state. - module: in the coll framework, there is one module instance for every communicator that uses this component. It has local state relevant to that specific communicator. Think of "component" as a C++ class, and "module" as a C++ object. Now read the comments in ompi/mca/coll/coll.h. This file contains the struct interfaces for both the coll component and module. We basically do everything by function pointer; the component returns a set of function pointers and each module returns a struct of function pointers. These function pointers are invoked by libmpi at various times for various functions; see coll.h for a description of each. During coll module initialization (i.e., when a new communicator has been created), there's a process called "selection" where OMPI determines which coll modules will be used on this communicator. Modules can include/exclude themselves from the selection process. For example, your algorithm may only be suitable for intracommunicators. So if the communicator in question that is being created is an intercommunicator, you probably want to exclude your module from selection. Or if your algorithm can only handle powers-of- two MPI processes, it should exclude itself if there is a non-power-of- two number of processes in the communicator. And so on. We designed coll modules in OMPI v1.3 to be "mix-n-match"-able such that in a single communicator, you can use the broadcast function from one module, but the gather function from a different module. Hence, multiple coll modules may be active on a single communicator. In your case, you'll need to make sure that your function has a higher priority than the "tuned" coll component (which is the default in many cases). I'd suggest working in the Open MPI v1.3 tree, as we're going to release this version soon and all future work is being done here (vs. the v1.2 tree, which will eventually be deprecated). Hopefully this is enough information to get you going. Please fee
