maybe it's related to https://svn.open-mpi.org/trac/ompi/ticket/1378 ??
On 12/5/08, Justin <luitj...@cs.utah.edu> wrote: > > The reason i'd like to disable these eager buffers is to help detect the > deadlock better. I would not run with this for a normal run but it would be > useful for debugging. If the deadlock is indeed due to our code then > disabling any shared buffers or eager sends would make that deadlock > reproduceable. In addition we might be able to lower the number of > processors down. Right now determining which processor is deadlocks when we > are using 8K cores and each processor has hundreds of messages sent out > would be quite difficult. > > Thanks for your suggestions, > Justin > Brock Palen wrote: > >> OpenMPI has differnt eager limits for all the network types, on your >> system run: >> >> ompi_info --param btl all >> >> and look for the eager_limits >> You can set these values to 0 using the syntax I showed you before. That >> would disable eager messages. >> There might be a better way to disable eager messages. >> Not sure why you would want to disable them, they are there for >> performance. >> >> Maybe you would still see a deadlock if every message was below the >> threshold. I think there is a limit of the number of eager messages a >> receving cpus will accept. Not sure about that though. I still kind of >> doubt it though. >> >> Try tweaking your buffer sizes, make the openib btl eager limit the same >> as shared memory. and see if you get locks up between hosts and not just >> shared memory. >> >> Brock Palen >> www.umich.edu/~brockp >> Center for Advanced Computing >> bro...@umich.edu >> (734)936-1985 >> >> >> >> On Dec 5, 2008, at 2:10 PM, Justin wrote: >> >> Thank you for this info. I should add that our code tends to post a lot >>> of sends prior to the other side posting receives. This causes a lot of >>> unexpected messages to exist. Our code explicitly matches up all tags and >>> processors (that is we do not use MPI wild cards). If we had a dead lock I >>> would think we would see it regardless of weather or not we cross the >>> roundevous threshold. I guess one way to test this would be to to set this >>> threshold to 0. If it then dead locks we would likely be able to track down >>> the deadlock. Are there any other parameters we can send mpi that will turn >>> off buffering? >>> >>> Thanks, >>> Justin >>> >>> Brock Palen wrote: >>> >>>> When ever this happens we found the code to have a deadlock. users >>>> never saw it until they cross the eager->roundevous threshold. >>>> >>>> Yes you can disable shared memory with: >>>> >>>> mpirun --mca btl ^sm >>>> >>>> Or you can try increasing the eager limit. >>>> >>>> ompi_info --param btl sm >>>> >>>> MCA btl: parameter "btl_sm_eager_limit" (current value: >>>> "4096") >>>> >>>> You can modify this limit at run time, I think (can't test it right >>>> now) it is just: >>>> >>>> mpirun --mca btl_sm_eager_limit 40960 >>>> >>>> I think you can also in tweaking these values use env Vars in place of >>>> putting it all in the mpirun line: >>>> >>>> export OMPI_MCA_btl_sm_eager_limit=40960 >>>> >>>> See: >>>> http://www.open-mpi.org/faq/?category=tuning >>>> >>>> >>>> Brock Palen >>>> www.umich.edu/~brockp >>>> Center for Advanced Computing >>>> bro...@umich.edu >>>> (734)936-1985 >>>> >>>> >>>> >>>> On Dec 5, 2008, at 12:22 PM, Justin wrote: >>>> >>>> Hi, >>>>> >>>>> We are currently using OpenMPI 1.3 on Ranger for large processor jobs >>>>> (8K+). Our code appears to be occasionally deadlocking at random within >>>>> point to point communication (see stacktrace below). This code has been >>>>> tested on many different MPI versions and as far as we know it does not >>>>> contain a deadlock. However, in the past we have ran into problems with >>>>> shared memory optimizations within MPI causing deadlocks. We can usually >>>>> avoid these by setting a few environment variables to either increase the >>>>> size of shared memory buffers or disable shared memory optimizations all >>>>> together. Does OpenMPI have any known deadlocks that might be causing >>>>> our >>>>> deadlocks? If are there any work arounds? Also how do we disable shared >>>>> memory within OpenMPI? >>>>> >>>>> Here is an example of where processors are hanging: >>>>> >>>>> #0 0x00002b2df3522683 in mca_btl_sm_component_progress () from >>>>> /opt/apps/intel10_1/openmpi/1.3/lib/openmpi/mca_btl_sm.so >>>>> #1 0x00002b2df2cb46bf in mca_bml_r2_progress () from >>>>> /opt/apps/intel10_1/openmpi/1.3/lib/openmpi/mca_bml_r2.so >>>>> #2 0x00002b2df0032ea4 in opal_progress () from >>>>> /opt/apps/intel10_1/openmpi/1.3/lib/libopen-pal.so.0 >>>>> #3 0x00002b2ded0d7622 in ompi_request_default_wait_some () from >>>>> /opt/apps/intel10_1/openmpi/1.3//lib/libmpi.so.0 >>>>> #4 0x00002b2ded109e34 in PMPI_Waitsome () from >>>>> /opt/apps/intel10_1/openmpi/1.3//lib/libmpi.so.0 >>>>> >>>>> >>>>> Thanks, >>>>> Justin >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
