Darn I forgot about that bug and I saw it pass on the mailing list too. Sorry for the post then.
I saw your mini-rank posting, I think it's a wonderful idea especially since there are more and more cores per nodes now. Is the infinit loop fixed in the mini-rank codebase? If not should I just apply the patch? Again, thanks for the great work. Louis On 12-11-05 12:17 PM, Sébastien Boisvert wrote: > On 11/05/2012 11:14 AM, Louis Letourneau wrote: >> I have assembled 2 >2.5G genomes (not the same, both mammals) in about >> 48hrs using 2025cores. This works great. >> > > Nice. > >> I'm trying to assemble a fish and I am having issues I don't quite know >> how to debug. >> >> The fish is about 1.9Gb in size and not diploid. >> >> If I run Ray using the paired + mates using k31 I was able to assemble >> it in 168hours (I needed to restart after 120hours because of >> walltime...thanks for the checkpoints :-) ). >> > > That's quite long, what's the latency ? > > We are working on a new programming model called "mini-ranks" to better use > super computer with a lot of nodes, but also with a lot of cores per node. > > Ray uses RayPlatform, and RayPlatform uses MPI. In the new model, RayPlatform > uses "mini-ranks". > > The current model in RayPlatform is to use pure MPI programming model, which > can be really bad on some super computer if there is just one network card > on each node that must serve lots of MPI processes. > > If you are interested, we have a experimental branch called for mini-ranks > that > can use only 1 MPI process per node, and as many IEEE POSIX threads (one for > each > mini-ranks). > > With mini-ranks, the routing code in RayPlatform will become obsolete ! > > Some latency results: > > > Table 1: Comparison of MPI ranks with mini-ranks on the Colosse > super-computer at Laval University. > +-------+---------------------------------------------------+ > | Cores | Average round-trip latency (us) | > +-------+-----------------------+---------------------------+ > | | MPI ranks | mini-ranks | > | | (pure MPI) | (MPI + pthread) | > +-------+-----------------------+---------------------------+ > | 8 | 11.25 +/- 0 | 24.1429 +/- 0 | > | 16 | 35.875 +/- 6.92369 | 43.0179 +/- 8.76275 | > | 32 | 66.3125 +/- 6.76387 | 41.7143 +/- 1.23924 | > | 64 | 90 +/- 16.5265 | 37.75 +/- 6.41984 | > | 128 | 126.562 +/- 25.0116 | 43.0179 +/- 8.76275 | > | 256 | 203.637 +/- 67.4579 | 44.6429 +/- 6.11862 | > +-------+-----------------------+---------------------------+ > > If you want to try that: > > git clone g...@github.com:sebhtml/RayPlatform.git > cd RayPlatform;git checkout minirank-model;cd .. > git clone g...@github.com:sebhtml/ray.git > cd ray;git checkout minirank-model; > make > > then, to run on 100 nodes, with 24 cores per node: > > > mpiexec -n 100 -bynode Ray -mini-ranks-per-rank 23 \ > ... > > Notes: > > 1. the -bynode is necessary in Open-MPI because the default is -byslot. > -byslot will work also if the job scheduler presents the slots in a by-node > round-robin strategy. > > 2. It is important to throw 23 mini-ranks per MPI process and not 24 > because each MPI process has a communication thread too and you > don't want to oversubscribe the folks on the CPU at all. > > 3. The mini-rank code contains 0 (zero) locks, 0 mutexes, 0 spinlocks, 0 > semaphore. > The code is non-blocking, and lock-free which is why it works so well. > > 4. This work should be merged once I have made additional sanity checks. > > 5. If you want to look at the code, the class MessageQueue is particularly > interesting. > >> It worked (although the assembly wasn't great, possibly due to a lot of >> repeats), but took way longer than the bigger genomes. >> >> I'm trying the same without the mates. I also changed the kmer from k31 >> to k61. >> >> I hit walltime 5 times now, 120hours each and it's not finished. >> >> The variables that changed are kmer and no mates. >> >> The first run ran many steps in the log. >> Since the first wall time, the only output I seem to be having >> >> Rank X: gathering scaffold links [Y/2987] [Z/7166] >> >> (X,Y,Z varies of course) >> > > Known bug where the Ray stalls on repeats too long... > > https://github.com/sebhtml/ray/issues/91 > > This is because of a bug (1 month old, actually). I have a patch in the queue, > but I am not satisfied by its impact overall. The patch fixes the problem of > running > time though. I will solves this bug in the scaffolder when I have time. > Meanwhile, you can use the patch, which solves the problem, but it's a dirty > hack. > > You can test this patch: > > wget > http://downloads.sourceforge.net/project/denovoassembler/Ray-v2.1.0.tar.bz2 > tar -xjf Ray-v2.1.0.tar.bz2 > cd Ray-v2.1.0 > wget > https://github.com/sebhtml/patches/raw/master/ray/human-seb-from-13efb22270e4f563c9cafc.patc > patch -p1 < human-seb-from-13efb22270e4f563c9cafc.patch > > make ... > >> I was using a version compiled from sources for the polytope routing. >> > > As I said, "mini-ranks" *will* supercede the virtual routing subsystem. The > problem with > virtual routing is that it increases the number of physical hops. With > mini-ranks, > it is not the case at all. > >> Any ideas? >> > > To wrap-up: > > 1. Try mini-ranks; > 2. Try the patch; > > > p.s.: I should resume the patchwork, branch merging once I am done > implementing > the reviewers' concerns for my Debian package and Fedora package for Ray. > > p.s.2: For your information, our paper about Ray Meta should appear somewhere > in the > near future, it is in re-review (the reviewers are assessing our revised > manuscript). > >> Louis >> >> ------------------------------------------------------------------------------ >> LogMeIn Central: Instant, anywhere, Remote PC access and management. >> Stay in control, update software, and manage PCs from one command center >> Diagnose problems and improve visibility into emerging IT issues >> Automate, monitor and manage. Do more in less time with Central >> http://p.sf.net/sfu/logmein12331_d2d >> _______________________________________________ >> Denovoassembler-users mailing list >> Denovoassembler-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users >> > > ------------------------------------------------------------------------------ LogMeIn Central: Instant, anywhere, Remote PC access and management. Stay in control, update software, and manage PCs from one command center Diagnose problems and improve visibility into emerging IT issues Automate, monitor and manage. Do more in less time with Central http://p.sf.net/sfu/logmein12331_d2d _______________________________________________ Denovoassembler-users mailing list Denovoassembler-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
