Sorry not path, but patch. Louis
On 12-11-08 02:46 PM, Louis Letourneau wrote: > With the path I went from 6*120hrs(walltime was hit each time) to: > resources_used.walltime=47:35:27 > > geez...I should have asked earlier :-) > > > thanks > Louis > > On 12-11-05 04:52 PM, Sébastien Boisvert wrote: >> On 11/05/2012 02:32 PM, Louis Letourneau wrote: >>> Darn I forgot about that bug and I saw it pass on the mailing list too. >>> >>> Sorry for the post then. >>> >> >> Not sure I sent an email about that patch ;-) >> >>> I saw your mini-rank posting, I think it's a wonderful idea especially >>> since there are more and more cores per nodes now. >>> >>> Is the infinit loop fixed in the mini-rank codebase? >> >> It is not an infinite loop, it just that a loop over a k-mer with >> a coverage of 99999 (a large coverage) takes a while with all the messages. >> >>> If not should I just apply the patch? >>> >> >> Yeah, the patch fixes the long running time in the scaffolding. >> >>> Again, thanks for the great work. >>> >> >> Thanks for the testing ! >> >>> Louis >>> >>> On 12-11-05 12:17 PM, Sébastien Boisvert wrote: >>>> On 11/05/2012 11:14 AM, Louis Letourneau wrote: >>>>> I have assembled 2 >2.5G genomes (not the same, both mammals) in about >>>>> 48hrs using 2025cores. This works great. >>>>> >>>> >>>> Nice. >>>> >>>>> I'm trying to assemble a fish and I am having issues I don't quite know >>>>> how to debug. >>>>> >>>>> The fish is about 1.9Gb in size and not diploid. >>>>> >>>>> If I run Ray using the paired + mates using k31 I was able to assemble >>>>> it in 168hours (I needed to restart after 120hours because of >>>>> walltime...thanks for the checkpoints :-) ). >>>>> >>>> >>>> That's quite long, what's the latency ? >>>> >>>> We are working on a new programming model called "mini-ranks" to better use >>>> super computer with a lot of nodes, but also with a lot of cores per node. >>>> >>>> Ray uses RayPlatform, and RayPlatform uses MPI. In the new model, >>>> RayPlatform >>>> uses "mini-ranks". >>>> >>>> The current model in RayPlatform is to use pure MPI programming model, >>>> which >>>> can be really bad on some super computer if there is just one network card >>>> on each node that must serve lots of MPI processes. >>>> >>>> If you are interested, we have a experimental branch called for mini-ranks >>>> that >>>> can use only 1 MPI process per node, and as many IEEE POSIX threads (one >>>> for each >>>> mini-ranks). >>>> >>>> With mini-ranks, the routing code in RayPlatform will become obsolete ! >>>> >>>> Some latency results: >>>> >>>> >>>> Table 1: Comparison of MPI ranks with mini-ranks on the Colosse >>>> super-computer at Laval University. >>>> +-------+---------------------------------------------------+ >>>> | Cores | Average round-trip latency (us) | >>>> +-------+-----------------------+---------------------------+ >>>> | | MPI ranks | mini-ranks | >>>> | | (pure MPI) | (MPI + pthread) | >>>> +-------+-----------------------+---------------------------+ >>>> | 8 | 11.25 +/- 0 | 24.1429 +/- 0 | >>>> | 16 | 35.875 +/- 6.92369 | 43.0179 +/- 8.76275 | >>>> | 32 | 66.3125 +/- 6.76387 | 41.7143 +/- 1.23924 | >>>> | 64 | 90 +/- 16.5265 | 37.75 +/- 6.41984 | >>>> | 128 | 126.562 +/- 25.0116 | 43.0179 +/- 8.76275 | >>>> | 256 | 203.637 +/- 67.4579 | 44.6429 +/- 6.11862 | >>>> +-------+-----------------------+---------------------------+ >>>> >>>> If you want to try that: >>>> >>>> git clone g...@github.com:sebhtml/RayPlatform.git >>>> cd RayPlatform;git checkout minirank-model;cd .. >>>> git clone g...@github.com:sebhtml/ray.git >>>> cd ray;git checkout minirank-model; >>>> make >>>> >>>> then, to run on 100 nodes, with 24 cores per node: >>>> >>>> >>>> mpiexec -n 100 -bynode Ray -mini-ranks-per-rank 23 \ >>>> ... >>>> >>>> Notes: >>>> >>>> 1. the -bynode is necessary in Open-MPI because the default is -byslot. >>>> -byslot will work also if the job scheduler presents the slots in a by-node >>>> round-robin strategy. >>>> >>>> 2. It is important to throw 23 mini-ranks per MPI process and not 24 >>>> because each MPI process has a communication thread too and you >>>> don't want to oversubscribe the folks on the CPU at all. >>>> >>>> 3. The mini-rank code contains 0 (zero) locks, 0 mutexes, 0 spinlocks, 0 >>>> semaphore. >>>> The code is non-blocking, and lock-free which is why it works so well. >>>> >>>> 4. This work should be merged once I have made additional sanity checks. >>>> >>>> 5. If you want to look at the code, the class MessageQueue is particularly >>>> interesting. >>>> >>>>> It worked (although the assembly wasn't great, possibly due to a lot of >>>>> repeats), but took way longer than the bigger genomes. >>>>> >>>>> I'm trying the same without the mates. I also changed the kmer from k31 >>>>> to k61. >>>>> >>>>> I hit walltime 5 times now, 120hours each and it's not finished. >>>>> >>>>> The variables that changed are kmer and no mates. >>>>> >>>>> The first run ran many steps in the log. >>>>> Since the first wall time, the only output I seem to be having >>>>> >>>>> Rank X: gathering scaffold links [Y/2987] [Z/7166] >>>>> >>>>> (X,Y,Z varies of course) >>>>> >>>> >>>> Known bug where the Ray stalls on repeats too long... >>>> >>>> https://github.com/sebhtml/ray/issues/91 >>>> >>>> This is because of a bug (1 month old, actually). I have a patch in the >>>> queue, >>>> but I am not satisfied by its impact overall. The patch fixes the problem >>>> of running >>>> time though. I will solves this bug in the scaffolder when I have time. >>>> Meanwhile, you can use the patch, which solves the problem, but it's a >>>> dirty hack. >>>> >>>> You can test this patch: >>>> >>>> wget >>>> http://downloads.sourceforge.net/project/denovoassembler/Ray-v2.1.0.tar.bz2 >>>> tar -xjf Ray-v2.1.0.tar.bz2 >>>> cd Ray-v2.1.0 >>>> wget >>>> https://github.com/sebhtml/patches/raw/master/ray/human-seb-from-13efb22270e4f563c9cafc.patc >>>> patch -p1 < human-seb-from-13efb22270e4f563c9cafc.patch >>>> >>>> make ... >>>> >>>>> I was using a version compiled from sources for the polytope routing. >>>>> >>>> >>>> As I said, "mini-ranks" *will* supercede the virtual routing subsystem. >>>> The problem with >>>> virtual routing is that it increases the number of physical hops. With >>>> mini-ranks, >>>> it is not the case at all. >>>> >>>>> Any ideas? >>>>> >>>> >>>> To wrap-up: >>>> >>>> 1. Try mini-ranks; >>>> 2. Try the patch; >>>> >>>> >>>> p.s.: I should resume the patchwork, branch merging once I am done >>>> implementing >>>> the reviewers' concerns for my Debian package and Fedora package for Ray. >>>> >>>> p.s.2: For your information, our paper about Ray Meta should appear >>>> somewhere in the >>>> near future, it is in re-review (the reviewers are assessing our revised >>>> manuscript). >>>> >>>>> Louis >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>>>> Stay in control, update software, and manage PCs from one command center >>>>> Diagnose problems and improve visibility into emerging IT issues >>>>> Automate, monitor and manage. Do more in less time with Central >>>>> http://p.sf.net/sfu/logmein12331_d2d >>>>> _______________________________________________ >>>>> Denovoassembler-users mailing list >>>>> Denovoassembler-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users >>>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------------ >>> LogMeIn Central: Instant, anywhere, Remote PC access and management. >>> Stay in control, update software, and manage PCs from one command center >>> Diagnose problems and improve visibility into emerging IT issues >>> Automate, monitor and manage. Do more in less time with Central >>> http://p.sf.net/sfu/logmein12331_d2d >>> _______________________________________________ >>> Denovoassembler-users mailing list >>> Denovoassembler-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users >>> >> >> > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_nov > _______________________________________________ > Denovoassembler-users mailing list > Denovoassembler-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/denovoassembler-users > ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_nov _______________________________________________ Denovoassembler-users mailing list Denovoassembler-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
