Thank you so quick response, I will try it 2010-03-09
马少杰 发件人： Joshua Hursey 发送时间： 2010-03-05 22:10:52 收件人： Open MPI Users 抄送： 主题： Re: [OMPI users] low efficiency when we use --am ft-enable-cr tocheckpoint On Mar 5, 2010, at 3:15 AM, 马少杰 wrote: > Dear Sir: > - What version of Open MPI are you using? > my version is 1.3.4 > - What configure options are you using? > ./configure --with-ft=cr --enable-mpi-threads --enable-ft-thread > --with-blcr=$dir --with-blcr-libdir=/$dir/lib > --prefix=/public/mpi/openmpi134-gnu-cr --enable-mpirun-prefix-by-default > make > make install > - What MCA parameters are you using? > mpirun -np 8 --am ft-enable-cr -machinefile ma xhpl > vim $HOME/.openmpi/mca-params.conf > # Local snapshot directory (not used in this scenario) > crs_base_snapshot_dir=/home/me/tmp > # Remote snapshot directory (globally mounted file system)) > snapc_base_global_snapshot_dir=/home/me/checkpoints > > > - Are you building from a release tarball or a SVN checkout? > building from openmpi-1.3.4.tar.gz > > > Now, I solve the problem successfully. > I found that the mpirun command as > > mpirun -np 8 --am ft-enable-cr --mca opal_cr_use_thread 0 -machinefile ma > ./xhpl > > the time cost is almost equal to the time cost by the command: mpirun -np 8 > -machinefile ma ./xhpl > > I think it should be a bug. Since you have configured Open MPI to use the C/R thread (--enable-ft-thread) then Open MPI will start the concurrent C/R thread when you ask for C/R to be enabled. By default the thread polls very aggressively (waiting only 0 microseconds, or the same as calling sched_yeild() on most systems). By turning it off you eliminate the contention the thread is causing on the system. There are two MCA parameters that control this behavior, links below: http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_check http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_wait I agree that the default behavior is probably too aggressive for most applications. However by increasing these values the user is also increasing the amount of time before a checkpoint can begin. In my setup I usually set: opal_cr_thread_sleep_wait=1000 Which will throttle down the thread when the application is in the MPI library. You might want to play around with these MCA parameters to tune the aggressiveness of the C/R thread to your performance needs. In the mean time I will look into finding better default parameters for these options. Cheers, Josh > > > 2010-03-05 > 马少杰 > 发件人： Joshua Hursey > 发送时间： 2010-03-05 00:07:19 > 收件人： Open MPI Users > 抄送： > 主题： Re: [OMPI users] low efficiency when we use --am ft-enable-cr tocheckpoint > There is some overhead involved when activating the current C/R functionality > in Open MPI due to the wrapping of the internal point-to-point stack. The > wrapper (CRCP framework) tracks the signature of each message (not the > buffer, so constant time for any size MPI message) so that when we need to > quiesce the network we know of all the outstanding messages that need to be > drained. > > So there is an overhead, but it should not be as significant as you have > mentioned. I looked at some of the performance aspects in the paper at the > link below: > http://www.open-mpi.org/papers/hpdc-2009/ > Though I did not look at HPL explicitly in this paper (just NPB, GROMACS, and > NetPipe), I have in testing and the time difference was definitely not 2x > (cannot recall the exact differences at the moment). > > Can you tell me a bit about your setup: > - What version of Open MPI are you using? > - What configure options are you using? > - What MCA parameters are you using? > - Are you building from a release tarball or a SVN checkout? > > -- Josh > > > On Mar 3, 2010, at 10:07 PM, 马少杰 wrote: > > > > > > > 2010-03-04 > > 马少杰 > > Dear Sir: > > I want to use blcr and openmpi to checkpoint, now I can save check > point and restart my work successfully. How erver I find the option "--am > ft-enable-cr" will case large cost . For example , when I run my HPL job > without and with the option "--am ft-enable-cr" on 4 hosts (32 process, IB > network) respectively , the time costed are 8m21.180s and 16m37.732s > respctively. it is should be noted that I did not save the checkpoint when I > run the job, the additional cost is caused by "--am ft-enable-cr" > independently. Why can the optin "--am ft-enable-cr" case so much system > cost? Is it normal? How can I solve the problem. > > I also test other mpi applications, the problem still exists. > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
