Hey,
On 09/10/12 12:30 PM, Rob Egan wrote: > Thanks for the help Sebastien, > > So I think the problem is that I recently refreshed the ray repository, but > not the RayPlatform. > When I pulled from both, my test assembly now goes to completion without > errors. > Excellent! I would be curious to see the outcome of a job with all the raw paired reads and with the Bloom filter. > To answer your other questions, yes the checkpoints wrote out fine, and I > didn't run with any > profiling so I don't know how the IO behaved. If you have a distributed file system, you may see some peaks for the I/O peaks because the caching behavior in Linux is not the same for Lustre or IBM GPFS. I have a ticket to implement buffering for checkpointing to produce a few (5-10) I/O operations per minute or something like that. Speaking of profiling: If you want to run the RayPlatform interruption profiler, you can give these options to Ray: -run-profiler --with-profiler-details *** This will slow down your job by about 5-30 %. This will give you information every 100 milliseconds. > > So I have another question on how to set the routing parameters to properly > scale this up for > our 1.2 Terabase dataset. I saw that you are disabling the Bloom filter. The Bloom filter will just essentially weed out most of the k-mers that occur once. Some of them will go in the de Bruijn graph, but that's not a problem because the strands are fully coherent in the distributed storage engine. Ff your super computer is not a high-end expensive one -- think of a IBM Blue Gene, a IBM iDataPlex, or a Cray XE6, you will want to use the virtual message router implemented in RayPlatform and available in Ray. Those fancy systems have no problem scaling Ray without a virtual router. We did tests on IBM iDataPlex systems and on Cray XE6 systems without any virtual message routing. > I didn't see in the documentation any insight on which routing > shape and degree to use for a given architecture. Basically, the surface (vertices and edges, not the faces) of a regular convex polytope beats any other other models implemented in RayPlatform. It gives you a diameter equal to the one of a de Bruijn graph (not the de Bruijn subgraph used for assembly -- that's another one.). But there is only one shortest path in a de Bruijn (or Kautz) routing graph whereas there are a lot of shortest paths in for the polytope thing. You have one vertex per process. So for the "shape", you want to pick up the polytope. > I'm going to run on hopper > http://www.nersc.gov/users/computational-systems/hopper/configuration/compute-nodes/ > . > Which has 24 cores per physical node, each with 4 NUMA nodes and 6 cores. According to http://i.top500.org/system/10612 , Hopper is a Cray XE6. Therefore, you will be fine without the virtual message router. In collaboration with Cray, Inc. there were some tests done on 4096 processing elements on a Cray XE6 with Ray without any virtual message routing. The round-trip latency was good -- the mode was 29 microseconds. To be sure, run a network test on it: mpiexec -n 4096 Ray -o test-4096-Cray-XE6 -test-network-only For large jobs, I know that Open-MPI 1.6.2 has some new cool features for processor affinity. But if I remember well, the Cray XE6 ships with aprun, something that optimizes a lot of things anyway. For the 4 NUMA nodes shared by 6 cores at the L3 cache level, usually it is the job of the operating system (with NUMA-specific code) or the MPI library (with madvise() for instance) to deal with processor and memory affinity. > 153,216 total cores ;-) > > The nodes themselves are connected in a 3D torus. Yes, probably a projection of something (like a mesh) on a 3D torus. > So, given that shape of the hardware, > how would you recommend that I set the routing and degrees for a large job > of 1000 - 8000 cores? Again, as I said the virtual message router implemented in RayPlatform is for interconnects (or systems) that can not handle the any-to-any communication pattern of Ray. In this case, the virtual message router can transform the any-to-any communication pattern into a very nice communication pattern that uses only edges on the surface of a regular convex polytope. > > Thanks again! My pleasure ! p.s. I will be at Argonne National Laboratory, Illinois from 14 to 18 October 2012. But I think you are in California, right ? > Rob > > > On Tue, Oct 9, 2012 at 7:08 AM, Sébastien Boisvert > <sebastien.boisver...@ulaval.ca <mailto:sebastien.boisver...@ulaval.ca>> > wrote: > > Hello, > > [CC'ed the mailing list.] > > On 08/10/12 05:22 PM, Rob Egan wrote: > > Hello Sebastien, > > > > If you remember, we were both at the ICIS workshop in Utah this year. > > Yes I remember. > > > > > I've been trying to use Ray on our 1.2 Terabase cow rumen metagenomic > dataset but am running into > > trouble getting it to complete successfully, > > and I was wondering if you could give me some advice on how to get it > working. > > Of course. > > > The version I used was from git: > > Ray: d8a8d737481d657ad8979e025eecacb9f400f3b1 > > This is from Thu Sep 27 17:33:06 2012 -0400 > > > RayPlatform: 9a197dc4dd750d0807543dfc2cc394e235df97c6 > > This is from Wed May 2 16:43:10 2012 -0400, before the release of > Ray v2.0.0. > > Are you sure of the git hashes because Ray d8a8d737481d657 should not > compile > properly with RayPlatform 9a197dc4dd750d080. > > Also, if you are using gcc 4.7.x, you need to pull as a bug was fixed. > > See my other answers below. > > > > > I'm running at NERSC's hopper system and even at just a small fraction > of the total data available > > (70GBytes), Ray is segfaulting shortly after it completes the > OptimalMarkers checkpoint > > (after 3 hours on 480 cores). Attached is the last 10000 lines of the > log file. > > > > Are the checkpoints working well for you ? > > Are you seeing I/O peaks on your file system ? > > > This is the not-too-useful message I get from the logfiles: > > > > _pmiu_daemon(SIGCHLD): [NID 00211] [c0-3c1s6n1] [Sat Sep 29 20:25:29 > 2012] PE RANK 308 exit signal Segmentation fault > > [NID 00211] 2012-09-29 20:25:53 Apid 11826156: initiated application > termination > > _pmiu_daemon(SIGCHLD): [NID 00270] [c0-3c1s7n2] [Sat Sep 29 20:25:28 > 2012] PE RANK 180 exit signal Segmentation fault > > _pmiu_daemon(SIGCHLD): [NID 06352] [c1-3c1s7n0] [Sat Sep 29 20:25:28 > 2012] PE RANK 462 exit signal Segmentation fault > > _pmiu_daemon(SIGCHLD): [NID 06355] [c1-3c1s6n1] [Sat Sep 29 20:25:28 > 2012] PE RANK 337 exit signal Segmentation fault > > _pmiu_daemon(SIGCHLD): [NID 00207] [c0-3c0s7n1] [Sat Sep 29 20:25:28 > 2012] PE RANK 100 exit signal Segmentation fault > > > > > > At least we know that some segments were protected by the memory system ! > > > Our data set is a mixture of fragment pairs (2x100 350 insert), jumping > pairs (2x76 3k & 5k insert) > > and overlapping/merged long reads (200-250bp). For the 1/20th view I > only included highly abundant > > (>256 depth by kmer=31) pre-screened reads. > > Nice libraries ! It is nice to see people using long distances for de > novo assemblies. > > > This is the command line options: > > > > "-o ../Ray-151.d2-Ray-2267125.sdb.d -disable-network-test > -bloom-filter-bits 0 -write-contig-paths \ > > -read-write-checkpoints . -k 151 -s > ../kmernator-FR-1521692.sdb-FR-MinDepth2-PartitionDepth \ > > 256-I200.fastq -i > ../kmernator-FR-1521692.sdb-FR-MinDepth2-PartitionDepth256-I300.fastq -i \ > > ../kmernator-FR-1521692.sdb-FR-MinDepth2-PartitionDepth256-I350.fastq -i > > ../kmernator-FR-1521692.sdb-FR-MinDepth2 -PartitionDepth256-I3k.fastq > -i \ > > ../kmernator-FR-1521692.sdb-FR-MinDepth2-PartitionDepth256-I5k.fastq \ > > -s ../med-PD256-I250.fastq" > > > > It's good to see that the interleaved code path is still popular. > > I see that you are using long k-mers. > > We fixed a segmentation fault at this very place recently: > > commit 1e6719eba3a919f32ab7132090e23e67b084df85 > Author: Sébastien Boisvert <sebastien.boisver...@ulaval.ca > <mailto:sebastien.boisver...@ulaval.ca>> > Date: Thu Aug 16 13:20:23 2012 -0400 > > A path with 0 k-mers has 0 nucleotides, not 0-k+1. > > This fixes segmentation faults when using large k-mers for > assemblies. > > Reported-by: Pier-Luc Plante <pier-luc.plant...@ulaval.ca > <mailto:pier-luc.plant...@ulaval.ca>> > Signed-off-by: Sébastien Boisvert <sebastien.boisver...@ulaval.ca > <mailto:sebastien.boisver...@ulaval.ca>> > > code/plugin_KmerAcademyBuilder/Kmer.h | 4 ++++ > code/plugin_Scaffolder/Scaffolder.cpp | 2 +- > code/plugin_SeedExtender/SeedExtender.cpp | 4 +++- > code/plugin_SeedingData/SeedingData.cpp | 13 +++++++++++-- > > > But this git hash you provided indicates that you alrady have that. > > Can you confirm ? > > > Thanks in advance! > > Rob Egan > > > > > > > ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ Denovoassembler-users mailing list Denovoassembler-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
