Hi Ralph, sorry for the late response, but I couldn't find free time to play with this. Finally I've applied the patch you prepared. I've launched my processes in the way you've described and I think it's working as you expected. None of my processes runs the orted daemon and they can perform MPI operations. Unfortunately I'm still hitting the 65 processes issue :( Maybe I'm doing something wrong. I attach my source code. If anybody could have a look on this, I would be grateful.
When I run that code with clients_count <= 65 everything works fine: all the processes create a common grid, exchange some information and disconnect. When I set clients_count > 65 the 66th process crashes on MPI_Comm_connect (segmentation fault). Another thing I would like to know is if it's normal that any of my processes when calling MPI_Comm_connect or MPI_Comm_accept when the other side is not ready, is eating up a full CPU available. Any help would be appreciated, Grzegorz Maj 2010/4/24 Ralph Castain <r...@open-mpi.org>: > Actually, OMPI is distributed with a daemon that does pretty much what you > want. Checkout "man ompi-server". I originally wrote that code to support > cross-application MPI publish/subscribe operations, but we can utilize it > here too. Have to blame me for not making it more publicly known. > The attached patch upgrades ompi-server and modifies the singleton startup > to provide your desired support. This solution works in the following > manner: > 1. launch "ompi-server -report-uri <filename>". This starts a persistent > daemon called "ompi-server" that acts as a rendezvous point for > independently started applications. The problem with starting different > applications and wanting them to MPI connect/accept lies in the need to have > the applications find each other. If they can't discover contact info for > the other app, then they can't wire up their interconnects. The > "ompi-server" tool provides that rendezvous point. I don't like that > comm_accept segfaulted - should have just error'd out. > 2. set OMPI_MCA_orte_server=file:<filename>" in the environment where you > will start your processes. This will allow your singleton processes to find > the ompi-server. I automatically also set the envar to connect the MPI > publish/subscribe system for you. > 3. run your processes. As they think they are singletons, they will detect > the presence of the above envar and automatically connect themselves to the > "ompi-server" daemon. This provides each process with the ability to perform > any MPI-2 operation. > I tested this on my machines and it worked, so hopefully it will meet your > needs. You only need to run one "ompi-server" period, so long as you locate > it where all of the processes can find the contact file and can open a TCP > socket to the daemon. There is a way to knit multiple ompi-servers into a > broader network (e.g., to connect processes that cannot directly access a > server due to network segmentation), but it's a tad tricky - let me know if > you require it and I'll try to help. > If you have trouble wiring them all into a single communicator, you might > ask separately about that and see if one of our MPI experts can provide > advice (I'm just the RTE grunt). > HTH - let me know how this works for you and I'll incorporate it into future > OMPI releases. > Ralph > > > On Apr 24, 2010, at 1:49 AM, Krzysztof Zarzycki wrote: > > Hi Ralph, > I'm Krzysztof and I'm working with Grzegorz Maj on this our small > project/experiment. > We definitely would like to give your patch a try. But could you please > explain your solution a little more? > You still would like to start one mpirun per mpi grid, and then have > processes started by us to join the MPI comm? > It is a good solution of course. > But it would be especially preferable to have one daemon running > persistently on our "entry" machine that can handle several mpi grid starts. > Can your patch help us this way too? > Thanks for your help! > Krzysztof > > On 24 April 2010 03:51, Ralph Castain <r...@open-mpi.org> wrote: >> >> In thinking about this, my proposed solution won't entirely fix the >> problem - you'll still wind up with all those daemons. I believe I can >> resolve that one as well, but it would require a patch. >> >> Would you like me to send you something you could try? Might take a couple >> of iterations to get it right... >> >> On Apr 23, 2010, at 12:12 PM, Ralph Castain wrote: >> >> > Hmmm....I -think- this will work, but I cannot guarantee it: >> > >> > 1. launch one process (can just be a spinner) using mpirun that includes >> > the following option: >> > >> > mpirun -report-uri file >> > >> > where file is some filename that mpirun can create and insert its >> > contact info into it. This can be a relative or absolute path. This process >> > must remain alive throughout your application - doesn't matter what it >> > does. >> > It's purpose is solely to keep mpirun alive. >> > >> > 2. set OMPI_MCA_dpm_orte_server=FILE:file in your environment, where >> > "file" is the filename given above. This will tell your processes how to >> > find mpirun, which is acting as a meeting place to handle the >> > connect/accept >> > operations >> > >> > Now run your processes, and have them connect/accept to each other. >> > >> > The reason I cannot guarantee this will work is that these processes >> > will all have the same rank && name since they all start as singletons. >> > Hence, connect/accept is likely to fail. >> > >> > But it -might- work, so you might want to give it a try. >> > >> > On Apr 23, 2010, at 8:10 AM, Grzegorz Maj wrote: >> > >> >> To be more precise: by 'server process' I mean some process that I >> >> could run once on my system and it could help in creating those >> >> groups. >> >> My typical scenario is: >> >> 1. run N separate processes, each without mpirun >> >> 2. connect them into MPI group >> >> 3. do some job >> >> 4. exit all N processes >> >> 5. goto 1 >> >> >> >> 2010/4/23 Grzegorz Maj <ma...@wp.pl>: >> >>> Thank you Ralph for your explanation. >> >>> And, apart from that descriptors' issue, is there any other way to >> >>> solve my problem, i.e. to run separately a number of processes, >> >>> without mpirun and then to collect them into an MPI intracomm group? >> >>> If I for example would need to run some 'server process' (even using >> >>> mpirun) for this task, that's OK. Any ideas? >> >>> >> >>> Thanks, >> >>> Grzegorz Maj >> >>> >> >>> >> >>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >> >>>> Okay, but here is the problem. If you don't use mpirun, and are not >> >>>> operating in an environment we support for "direct" launch (i.e., >> >>>> starting >> >>>> processes outside of mpirun), then every one of those processes thinks >> >>>> it is >> >>>> a singleton - yes? >> >>>> >> >>>> What you may not realize is that each singleton immediately >> >>>> fork/exec's an orted daemon that is configured to behave just like >> >>>> mpirun. >> >>>> This is required in order to support MPI-2 operations such as >> >>>> MPI_Comm_spawn, MPI_Comm_connect/accept, etc. >> >>>> >> >>>> So if you launch 64 processes that think they are singletons, then >> >>>> you have 64 copies of orted running as well. This eats up a lot of file >> >>>> descriptors, which is probably why you are hitting this 65 process >> >>>> limit - >> >>>> your system is probably running out of file descriptors. You might >> >>>> check you >> >>>> system limits and see if you can get them revised upward. >> >>>> >> >>>> >> >>>> On Apr 17, 2010, at 4:24 PM, Grzegorz Maj wrote: >> >>>> >> >>>>> Yes, I know. The problem is that I need to use some special way for >> >>>>> running my processes provided by the environment in which I'm >> >>>>> working >> >>>>> and unfortunately I can't use mpirun. >> >>>>> >> >>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >> >>>>>> Guess I don't understand why you can't use mpirun - all it does is >> >>>>>> start things, provide a means to forward io, etc. It mainly sits there >> >>>>>> quietly without using any cpu unless required to support the job. >> >>>>>> >> >>>>>> Sounds like it would solve your problem. Otherwise, I know of no >> >>>>>> way to get all these processes into comm_world. >> >>>>>> >> >>>>>> >> >>>>>> On Apr 17, 2010, at 2:27 PM, Grzegorz Maj wrote: >> >>>>>> >> >>>>>>> Hi, >> >>>>>>> I'd like to dynamically create a group of processes communicating >> >>>>>>> via >> >>>>>>> MPI. Those processes need to be run without mpirun and create >> >>>>>>> intracommunicator after the startup. Any ideas how to do this >> >>>>>>> efficiently? >> >>>>>>> I came up with a solution in which the processes are connecting >> >>>>>>> one by >> >>>>>>> one using MPI_Comm_connect, but unfortunately all the processes >> >>>>>>> that >> >>>>>>> are already in the group need to call MPI_Comm_accept. This means >> >>>>>>> that >> >>>>>>> when the n-th process wants to connect I need to collect all the >> >>>>>>> n-1 >> >>>>>>> processes on the MPI_Comm_accept call. After I run about 40 >> >>>>>>> processes >> >>>>>>> every subsequent call takes more and more time, which I'd like to >> >>>>>>> avoid. >> >>>>>>> Another problem in this solution is that when I try to connect >> >>>>>>> 66-th >> >>>>>>> process the root of the existing group segfaults on >> >>>>>>> MPI_Comm_accept. >> >>>>>>> Maybe it's my bug, but it's weird as everything works fine for at >> >>>>>>> most >> >>>>>>> 65 processes. Is there any limitation I don't know about? >> >>>>>>> My last question is about MPI_COMM_WORLD. When I run my processes >> >>>>>>> without mpirun their MPI_COMM_WORLD is the same as MPI_COMM_SELF. >> >>>>>>> Is >> >>>>>>> there any way to change MPI_COMM_WORLD and set it to the >> >>>>>>> intracommunicator that I've created? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Grzegorz Maj >> >>>>>>> _______________________________________________ >> >>>>>>> users mailing list >> >>>>>>> us...@open-mpi.org >> >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> users mailing list >> >>>>>> us...@open-mpi.org >> >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >>>>>> >> >>>>>> >> >>>>> _______________________________________________ >> >>>>> users mailing list >> >>>>> us...@open-mpi.org >> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> users mailing list >> >>>> us...@open-mpi.org >> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >>>> >> >>>> >> >>> >> >> >> >> _______________________________________________ >> >> users mailing list >> >> us...@open-mpi.org >> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
client.c
Description: Binary data
server.c
Description: Binary data
