Re: [OMPI devel] Need help for semaphore in BML
On Apr 19, 2007, at 1:45 PM, po...@cc.gatech.edu wrote: I want to put semaphore in bml.h--- mca_bml_send before and after calling btl_send. SO that when a process call btl_send it first lock a global variable X and then proceeds.Also if an external Tcp function wants to send data it should first lock global variable X and then proceed. Can anyone tell me only changing bml.h is enough or are there any other files where I need to make changes. This is likely to be a complex issue because there's the put and get functions as well. ob1 uses a fairly complex algorithm to decide when to call the bml interface functions -- I doubt that the use of a semaphore in a single location is going to do what you want. (why a semaphore, anyway -- why not a mutex?) (As I tried doing this and run mpi program it gave me ORTE time out error also when I changed file back to normal it was not compiling and giving me error in libmca_bml.la etc...unfortunately I deleted entire folder and downloaded new version.) Changing bml.h should have zero effect on the ORTE layer. ORTE is a whole different abstraction and wholly below the OMPI layer. There are a few places in the OMPI layer that interact with the lower ORTE later, but the bml is not one of them. I'm guessing that you had some other problem. If you're going to be working continually with Open MPI, you might want to get a subversion checkout. Can any one please help me and tell me how should I go about implementing locks/semaphore in bml layer so that all mpi process access lock (of same priority ) and continue working while Tcp acquire only when network is free(or there is lot of serial operation between 2 mpi sends). I want to emphasize again that this won't give you what you have described in previous mails: the PML interface is designed to be asynchronous. So when you call send/put/get, it only (possibly) *starts* the communication transfer. When you unlock upon return, you're allowing the alternate communication mechanism to come in and start another communication method (via a different BTL, perhaps), but it does not change that there may still be activity occurring down in the kernel and/or hardware. Also, this scheme does not account for received message contention -- it only [tries to] account for sending contention. So even if you get the locking working the way that you want, I don't think that you're going to get the overlap and multiplexing that you expect. -- Jeff Squyres Cisco Systems
Re: [OMPI devel] SOS... help needed :(
Are both the IB HCA and the ethernet interfaces on the same physical bus? If they're not, the need for multiplexing them is diminished (but, of course, it depends on what you're trying to do -- if everything is using huge memory transfers, then your bottleneck will be RAM, not the bus that the NICs reside on). That being said, something we have not explored at all is the idea of multiplexing at the MPI layer. Perhaps something like "this is a low priority communicator; I want you to only use the 'tcp' BTL on it" and "this is a high priority communicator; I want you to only use the 'openib' BTL on it". I haven't thought at all about whether that is possible. It would probably take some mucking around in both the bml and the ob1 pml. Hmm. It may or may not be worth it, but I raise the possibility... On Apr 19, 2007, at 9:18 PM, po...@cc.gatech.edu wrote: Hi, Some of our clusters uses Gigabit Ethernet and Infiniband. So we are trying to multiplex them. Thanks and Regards Pooja On Thu, Apr 19, 2007 at 06:58:37PM -0400, po...@cc.gatech.edu wrote: I am Pooja working with chaitali on this project. The idea behind this is while running a parallelized code ,if a huge chunks of serial computation is encountered at that time underlying network infrastructure can be used for some other data transfer. This increases the network utilization. But this (non Mpi) data transfer should not keep Mpi calls blocking. So we need to give them priorities. Also we are trying to predict a behavior of the code (like if there are more MPi calls coming with short interval or if they are coming after large interval ) based on previous calls. As a result we can make this mechanism more efficient. Ok, so you have a Cluster with Infiniband a while the network traffic is low you want to utilize the Infiniband network for other data transfers with a lower priority? What does this have to do with TCP or are you using TCP over Infiniband? Regards Christian Leber -- http://rettetdieti.vde-uni-mannheim.de/ ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] SOS... help needed :(
Hi, Some of our clusters uses Gigabit Ethernet and Infiniband. So we are trying to multiplex them. Thanks and Regards Pooja > On Thu, Apr 19, 2007 at 06:58:37PM -0400, po...@cc.gatech.edu wrote: > >> I am Pooja working with chaitali on this project. >> The idea behind this is while running a parallelized code ,if a huge >> chunks of serial computation is encountered at that time underlying >> network infrastructure can be used for some other data transfer. >> This increases the network utilization. >> But this (non Mpi) data transfer should not keep Mpi calls blocking. >> So we need to give them priorities. >> Also we are trying to predict a behavior of the code (like if there are >> more MPi calls coming with short interval or if they are coming after >> large interval ) based on previous calls. >> As a result we can make this mechanism more efficient. > > Ok, so you have a Cluster with Infiniband a while the network traffic is > low you want to utilize the Infiniband network for other data transfers > with a lower priority? > > What does this have to do with TCP or are you using TCP over Infiniband? > > Regards > Christian Leber > > -- > http://rettetdieti.vde-uni-mannheim.de/ > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
Re: [OMPI devel] SOS... help needed :(
On Thu, Apr 19, 2007 at 06:58:37PM -0400, po...@cc.gatech.edu wrote: > I am Pooja working with chaitali on this project. > The idea behind this is while running a parallelized code ,if a huge > chunks of serial computation is encountered at that time underlying > network infrastructure can be used for some other data transfer. > This increases the network utilization. > But this (non Mpi) data transfer should not keep Mpi calls blocking. > So we need to give them priorities. > Also we are trying to predict a behavior of the code (like if there are > more MPi calls coming with short interval or if they are coming after > large interval ) based on previous calls. > As a result we can make this mechanism more efficient. Ok, so you have a Cluster with Infiniband a while the network traffic is low you want to utilize the Infiniband network for other data transfers with a lower priority? What does this have to do with TCP or are you using TCP over Infiniband? Regards Christian Leber -- http://rettetdieti.vde-uni-mannheim.de/
Re: [OMPI devel] SOS... help needed :(
Hi, I am Pooja working with chaitali on this project. The idea behind this is while running a parallelized code ,if a huge chunks of serial computation is encountered at that time underlying network infrastructure can be used for some other data transfer. This increases the network utilization. But this (non Mpi) data transfer should not keep Mpi calls blocking. So we need to give them priorities. Also we are trying to predict a behavior of the code (like if there are more MPi calls coming with short interval or if they are coming after large interval ) based on previous calls. As a result we can make this mechanism more efficient. Thanks and regards Pooja > On Sun, Apr 15, 2007 at 10:25:06PM -0400, chaitali dherange wrote: > >>schedule MPI and non MPI calls... giving more priority to the MPI >> calls >>over the non >>MPI ones. > > What is the idea behind this and what advantages are expected? > > > Christian Leber > > -- > http://rettetdieti.vde-uni-mannheim.de/ > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
Re: [OMPI devel] SOS... help needed :(
On Sun, Apr 15, 2007 at 10:25:06PM -0400, chaitali dherange wrote: >schedule MPI and non MPI calls... giving more priority to the MPI calls >over the non >MPI ones. What is the idea behind this and what advantages are expected? Christian Leber -- http://rettetdieti.vde-uni-mannheim.de/
[OMPI devel] Fancy ORTE/MPIRUN bugs
Hi, I am experiencing several fancy bugs with ORTE. All bugs occur on Intel 32 bits architecture under Mac OS X using gcc 4.2. The tested version is todays trunk (it also have occured for at least three weeks) First occurs when compiling in "optimized" mode (aka configure --disable-debug --with-platform=optimized) and does not occur in debug mode. ~/ompi$ mpirun -np 1 echo foo [laptop20:22960] *** Process received signal *** [laptop20:22960] Signal: Bus error (10) [laptop20:22960] Signal code: (0) [laptop20:22960] Failing at address: 0x0 [ 1] [0xb698, 0x] (-P-) [ 2] (mca_oob_base_init + 0x26) [0xb6e8, 0x000878a6] [ 3] (orte_rml_oob_init + 0x11) [0xb6f8, 0x00032f21] [ 4] (orte_rml_base_select + 0xc5) [0xb778, 0x0009f415] [ 5] (orte_init_stage1 + 0x20c) [0xb848, 0x000678cc] [ 6] (orte_system_init + 0x1d) [0xb868, 0x0006b03d] [ 7] (orte_init + 0x7d) [0xb888, 0x000674ad] [ 8] (orterun:F(0,1)=r(0,1);-2147483648;2147483647; + 0x220) [0xb938, 0x2008] [ 9] (main:F(0,1)=r(0,1);-2147483648;2147483647; + 0x18) [0xb948, 0x1de6] [10] (_start + 0xd8) [0xb988, 0x1db2] [11] (start + 0x29) [0xb9a0, 0x1cd9] [12] [0x, 0x0005] (FP-) [laptop20:22960] *** End of error message *** Bus error The other one occurs when running MPI program without mpirun (I know this is pretty useless but still ;) ). This bug does not require specific compilation options to occur. Running mpirun -np 1 mympiprogram is fine, but running mympiprogram fails with segfault in MPI_Finalize: ~/ompi$ mpirun -np 1 mpiself ~/ompi$ gdb mpiself (gdb) r Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x77767578 0x90002e46 in szone_malloc () (gdb) bt #0 0x90002e46 in szone_malloc () #1 0x0042b6da in opal_memory_darwin_malloc (zone=0x200, size=48) at memory_darwin_component.c:103 #2 0x90002a2f in malloc () #3 0x00421548 in opal_malloc (size=48, file=0x274fd4 "../../../opal/class/opal_object.h", line=468) at malloc.c:96 #4 0x002218e4 in opal_obj_new (cls=0x27d840) at ../../../opal/class/opal_object.h:468 #5 0x00221851 in opal_obj_new_debug (type=0x27d840, file=0x275424 "base/gpr_base_create_value_keyval.c", line=43) at ../../../opal/class/opal_object.h:247 #6 0x0022147e in orte_gpr_base_create_value (value=0xb8fc, addr_mode=32769, segment=0x510150 "orte-job-0", cnt=2, num_tokens=0) at base/gpr_base_create_value_keyval.c:43 #7 0x00269b79 in orte_smr_base_set_proc_state (proc=0x507d00, state=32, exit_status=0) at base/smr_base_set_proc_state.c:54 #8 0x01035f21 in ompi_mpi_finalize () at runtime/ompi_mpi_finalize.c:145 #9 0x0106ea09 in MPI_Finalize () at finalize.c:44 #10 0x1e5e in main (argc=1, argv=0xbb70) at mpiself.c:44 (gdb)
[OMPI devel] Need help for semaphore in BML
Hi, I want to put semaphore in bml.h--- mca_bml_send before and after calling btl_send. SO that when a process call btl_send it first lock a global variable X and then proceeds.Also if an external Tcp function wants to send data it should first lock global variable X and then proceed. Can anyone tell me only changing bml.h is enough or are there any other files where I need to make changes. (As I tried doing this and run mpi program it gave me ORTE time out error also when I changed file back to normal it was not compiling and giving me error in libmca_bml.la etc...unfortunately I deleted entire folder and downloaded new version.) Can any one please help me and tell me how should I go about implementing locks/semaphore in bml layer so that all mpi process access lock(of same priority ) and continue working while Tcp acquire only when network is free(or there is lot of serial operation between 2 mpi sends). Thanks and Regards Pooja
