> [again, please CC the list as Ray is a community effort]
Sorry about that.
>
> How much memory do you have collectively with these 16 MPI ranks ?
512 G.
Christina
>
> On 11/01/2012 03:06 PM, Christina Boucher wrote:
>> Hi Sebastien,
>>
>> I believe that Ray chocked on my data also. In the nohup file I get the
>> following:
>>
>> ...
>> Estimated remaining time for this step: 21 minutes, 17 seconds
>> Rank 14: assembler memory usage: 1312576 KiB
>> Rank 5 is selecting optimal read markers [1560001/8224621]
>> Speed RAY_SLAVE_MODE_INDEX_SEQUENCES 6301 units/second
>> Estimated remaining time for this step: 17 minutes, 37 seconds
>> Rank 5: assembler memory usage: 1312136 KiB
>> --------------------------------------------------------------------------
>> mpiexec noticed that process rank 3 with PID 25953 on node oak exited on
>> signal 1 (Hangup).
>> --------------------------------------------------------------------------
>>
>>
>> It appears although, I ran out of memory again? I ran it with rank 16. Do
>> you have any parameterization
>> suggestions?
>>
>> Best,
>> Christina
>>
>>
>> On 2012-11-01, at 11:27 AM, Sébastien Boisvert
>> <[email protected] <mailto:[email protected]>>
>> wrote:
>>
>>> Regarding the way Ray is run, soon Ray will be able to run on MPI only, MPI
>>> + pthread, or
>>> even pthread only:
>>>
>>> As you may know, Ray ships with a library called RayPlatform, which
>>> abstracts all the parallel stuff from the programmer. In Ray v2.0.0 and
>>> v2.1.0, the associated RayPlatform library (versions 1.0.3 and 1.1.0,
>>> respectively) only utilizes MPI.
>>>
>>> Pure MPI applications work well on some machines, and not so much on
>>> others, usually because the Host Communication Adapter is being used by too
>>> many MPI processes on each node. That where hybrids come in.
>>>
>>> Hybrids are truly the future. I visited Argonne National Laboratory
>>> recently and I discussed with Professor Rick Stevens about hybrid
>>> programming models. Myself, Rick Stevens, and Fangfang Xia devised
>>> something called the "mini-ranks" hybrid programming model.
>>>
>>> The next release of Ray (likely something like 2.1.1) will run on
>>> RayPlatform 7.0.0, which will include support for our newly introduced
>>> "mini-ranks" hybrid programming model.
>>>
>>> So on your hybrid machine, you will be able to run Ray like this, (assuming
>>> 8 nodes and 16 hardware threads per node):
>>>
>>> mpiexec -n 8 -bynode \
>>> Ray -mini-ranks-per-rank 15 \
>>> -k 31 -o MiniRanksAreCool \
>>> -p joe1.fastq.bz2 joe2.fastq.bz2 \
>>> -p thor1.fastq.gz thor2.fastq.gz \
>>>
>>> This will launch 1 MPI process per node. Each MPI process will have exactly
>>> 15 mini-ranks. Each mini-rank will run in 1 IEEE POSIX thread and an
>>> additional thread
>>> (the origin control thread of the process) will do MPI calls.
>>>
>>> If you feel this is interesting for your laboratory, there is a preliminary
>>> implementation of this available for testing.
>>>
>>> (from http://seqanswers.com/forums/showpost.php?p=88306&postcount=200 )
>>>
>>> On 11/01/2012 01:20 PM, Adrian Platts wrote:
>>>> Hi Christina
>>>>
>>>> Here at McGill we have assembled to varying degrees depending on the
>>>> project the selfing crucifers Sisymbrium Irio, Leavenworthia Alabamica
>>>> (recent hexaploid) and Aethionema Arabicum.
>>>> We've also worked on assembly of self incompatible species which tend to
>>>> be highly heterozygous including Capsella bursa pastoris, Capsella
>>>> Grandiflora and a Brassica Rapa
>>>> ecotype ... and are kind of involved in the assembly of a couple of other
>>>> non-crucifers including bean and Cleome.
>>>>
>>>> We only have a small assembly capacity here (3 80 core, 256 GB RAM boxes)
>>>> but I'm keen to compare notes on where things are and are not working well
>>>> in plant assemblies!
>>>>
>>>> We started by using a long Kmer (K>61) approach in Velvet but found the
>>>> chimerism rate around TEs was worryingly high. We then moved on to
>>>> AllpathsLG and Ray-SOAPdeNovo
>>>> (Ray for contiging, Soap for scaffolding). We're also using the
>>>> meraculous assembler for heterozygotes. As I say - I'd be very interested
>>>> in hearing your experiences.
>>>>
>>>> Adrian
>>>> Adrian Platts
>>>> VEGI Project
>>>> McGill
>>>>
>>>> ps. these are our latencies with various params... after some checking
>>>> with Sebastien we're keeping away from tcp based messaging.
>>>>
>>>> mpiexec -n 10 --mca btl sm,self --bind-to-core --bycore --mca maffinity
>>>> libnuma ./Ray -o foo3 -test-network-only
>>>> # AverageForAllRanks: 7
>>>>
>>>> mpiexec -n 40 --mca btl sm,self --bind-to-core --bycore --mca maffinity
>>>> libnuma ./Ray -o foo4 -test-network-only
>>>> # AverageForAllRanks: 18.15
>>>>
>>>> mpiexec -n 32 --mca btl sm,self --bind-to-core --bycore --mca maffinity
>>>> libnuma ./Ray -o foo4 -route-messages -test-network-only
>>>> # AverageForAllRanks: 46.875
>>>> (round-robin)
>>>>
>>>> mpiexec -n 40 --mca btl sm,self --bind-to-core --bycore ./Ray -o foo3
>>>> -test-network-only
>>>> # AverageForAllRanks: 17.9
>>>>
>>>> mpiexec -n 40 --mca btl sm,self ./Ray -o foo3 -test-network-only
>>>> # AverageForAllRanks: 22.425
>>>>
>>>> mpiexec -n 41 --mca btl sm,self ./Ray -o foo3 -test-network-only
>>>> # AverageForAllRanks: 45.6098
>>>>
>>>> mpiexec -n 70 --mca btl sm,self ./Ray -o foo3 -test-network-only
>>>> # AverageForAllRanks: 82.105
>>>>
>>>>
>>>>
>>>> On Nov 1, 2012, at 1:07 PM, Sébastien Boisvert
>>>> <[email protected] <mailto:[email protected]>>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> You should CC the mailing list as I am sure that numerous people in
>>>>> the genomics community would be interested by plant genome de novo
>>>>> assembly !
>>>>>
>>>>> People at McGill University did some work on plant genomes with Ray too.
>>>>> They posted their results on the list I think.
>>>>>
>>>>> So you have something like 1 500 000 000 sequences, right ?
>>>>>
>>>>> On what kind of hardware are you running ?
>>>>>
>>>>> What's the latency reported in NetworkTest.txt ?
>>>>>
>>>>>
>>>>> --
>>>>> Sent from my IBM Blue Gene/Q
>>>>>
>>>>> Sébastien
>>>>>
>>>>>
>>>>>
>>>>> On 11/01/2012 12:57 PM, Christina Boucher wrote:
>>>>>> Thanks. I ended up attaching the .openmpi-setup file in my top-level
>>>>>> home directory and then adding the following line to my .bashrc: source
>>>>>> ~/.openmpi-setup
>>>>>>
>>>>>> After recompiling it seems to be running on my Arabidopsis data. I am
>>>>>> trying it with all 4 lanes and hoping that it works. I don't
>>>>>> necessarily care if I get the *best* assembly but an assembly would be
>>>>>> nice. Other assemblers have been bailing on memory with my 512G server
>>>>>> but I am hopeful about your program.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Best,
>>>>>> Christina
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2012-10-31, at 3:28 PM, Sébastien Boisvert
>>>>>> <[email protected] <mailto:[email protected]>
>>>>>> <mailto:[email protected]>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> On 10/31/2012 03:48 PM, Christina Boucher wrote:
>>>>>>>> MPI is already installed on my server… see:
>>>>>>>> oak # rpm -qa | grep openmpi
>>>>>>>> openmpi-devel-1.5.4-5.fc17.1.x86_64
>>>>>>>> openmpi-1.5.4-5.fc17.1.x86_64
>>>>>>>>
>>>>>>>
>>>>>>> This is something specific to Fedora 17 (which I happen to be using on
>>>>>>> my laptop).
>>>>>>> My answer below is not really related to Ray, but more related to
>>>>>>> Fedora 17.
>>>>>>>
>>>>>>> $ repoquery --list openmpi-1.5.4-5.fc17.1.x86_64 | grep mpiexec$ |grep
>>>>>>> bin
>>>>>>> /usr/lib64/openmpi/bin/mpiexec
>>>>>>>
>>>>>>> $ repoquery --list openmpi-devel-1.5.4-5.fc17.1.x86_64 | grep mpicxx$ |
>>>>>>> grep bin
>>>>>>> /usr/lib64/openmpi/bin/mpicxx
>>>>>>>
>>>>>>>
>>>>>>> However, the default PATH for a user on Fedora 17 is:
>>>>>>>
>>>>>>> [test@panic ~]$ echo $PATH
>>>>>>> /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/test/.local/bin:/home/test/bin
>>>>>>>
>>>>>>>
>>>>>>> You can fix this in Fedora 17 by adding the following 2 lines to your
>>>>>>> $HOME/.bashrc:
>>>>>>>
>>>>>>> export PATH=/usr/lib64/openmpi/bin:$PATH
>>>>>>> export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib/:$LD_LIBRARY_PATH
>>>>>>>
>>>>>>>
>>>>>>> Let me know if that work for you.
>>>>>>>
>>>>>>>
>>>>>>> Sébastien
>>>>>>>
>>>>>>>> oak # rpm -qa | grep openmpi
>>>>>>>> openmpi-devel-1.5.4-5.fc17.1.x86_64
>>>>>>>> openmpi-1.5.4-5.fc17.1.x86_64
>>>>>>>>
>>>>>>>> Still the installation problems persist….
>>>>>>>>
>>>>>>>> Christina
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2012-10-31, at 11:26 AM, Sébastien Boisvert
>>>>>>>> <[email protected]
>>>>>>>> <mailto:[email protected]>
>>>>>>>> <mailto:[email protected]>
>>>>>>>> <mailto:[email protected]>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>> make[1]: mpicxx: Command not found
>>>>>>>>>
>>>>>>>>> To install Ray, you need an MPI library. You don't have one installed.
>>>>>>>>>
>>>>>>>>> For example, on Fedora, the packages are openmpi, openmpi-devel,
>>>>>>>>> gcc-c++.
>>>>>>>>>
>>>>>>>>>> In addition, is the the max kmer length 32? Most people are using
>>>>>>>>>> upwards to 55….?
>>>>>>>>>
>>>>>>>>> The maximum k-mer length is set at compilation. The default is
>>>>>>>>> MAXKMERLENGTH=32.
>>>>>>>>> To change that:
>>>>>>>>>
>>>>>>>>> make MAXKMERLENGTH=64
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sébastien
>>>>>>>>>
>>>>>>>>> On 10/31/2012 12:50 PM, Christina Boucher wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to use your Ray assembler. I've been using Spades
>>>>>>>>>>>> (mainly because I am formerly
>>>>>>>>>>>> from Pavel Pevzner's lab) but running out of memory on a large
>>>>>>>>>>>> dataset.
>>>>>>>>>>>
>>>>>>>>>>> Maybe processing your large dataset is more amenable with a
>>>>>>>>>>> distributed assembler.
>>>>>>>>>>
>>>>>>>>>> Spades group released a new version yesterday that's supposed to use
>>>>>>>>>> less memory. I am trying that and the Ray assembler.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> To get it and install it:
>>>>>>>>>>>
>>>>>>>>>>> $ wget
>>>>>>>>>>> http://downloads.sourceforge.net/project/denovoassembler/Ray-v2.1.0.tar.bz2
>>>>>>>>>>> $ sha1sum Ray-v2.1.0.tar.bz2
>>>>>>>>>>> 4c09f2731445852857af53b65aa47e444792eeb0 Ray-v2.1.0.tar.bz2
>>>>>>>>>>>
>>>>>>>>>>> $ tar xjf Ray-v2.1.0.tar.bz2
>>>>>>>>>>> $ cd Ray-v2.1.0/
>>>>>>>>>>> $ make
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The problem is this compilation error: After those steps I get the
>>>>>>>>>> following error:
>>>>>>>>>>
>>>>>>>>>> eggs:~/Ray-v2.1.0$ make
>>>>>>>>>>
>>>>>>>>>> Compilation options (you can change them of course)
>>>>>>>>>>
>>>>>>>>>> PREFIX = install-prefix
>>>>>>>>>> MAXKMERLENGTH = 32
>>>>>>>>>> FORCE_PACKING = n
>>>>>>>>>> ASSERT = n
>>>>>>>>>> HAVE_LIBZ = n
>>>>>>>>>> HAVE_LIBBZ2 = n
>>>>>>>>>> INTEL_COMPILER = n
>>>>>>>>>> MPICXX = mpicxx
>>>>>>>>>> GPROF = n
>>>>>>>>>> OPTIMIZE = y
>>>>>>>>>> DEBUG = n
>>>>>>>>>>
>>>>>>>>>> Compilation and linking flags (generated automatically)
>>>>>>>>>>
>>>>>>>>>> CXXFLAGS = -Wall -std=c++98 -O3 -D MAXKMERLENGTH=32 -D
>>>>>>>>>> RAY_VERSION=\"2.1.0\"
>>>>>>>>>> LDFLAGS =
>>>>>>>>>>
>>>>>>>>>> make[1]: Entering directory
>>>>>>>>>> `/s/parsons/f/fac/cboucher/Ray-v2.1.0/RayPlatform'
>>>>>>>>>> mpicxx -Wall -std=c++98 -O3 -D MAXKMERLENGTH=32 -D
>>>>>>>>>> RAY_VERSION=\"2.1.0\" -D RAYPLATFORM_VERSION=\"1.1.0\" -I. -c -o
>>>>>>>>>> memory/ReusableMemoryStore.o memory/ReusableMemoryStore.cpp
>>>>>>>>>> make[1]: mpicxx: Command not found
>>>>>>>>>> make[1]: *** [memory/ReusableMemoryStore.o] Error 127
>>>>>>>>>> make[1]: Leaving directory
>>>>>>>>>> `/s/parsons/f/fac/cboucher/Ray-v2.1.0/RayPlatform'
>>>>>>>>>> make[1]: Entering directory
>>>>>>>>>> `/s/parsons/f/fac/cboucher/Ray-v2.1.0/code'
>>>>>>>>>> mpicxx -Wall -std=c++98 -O3 -D MAXKMERLENGTH=32 -D
>>>>>>>>>> RAY_VERSION=\"2.1.0\" -I ../RayPlatform -I. -c -o
>>>>>>>>>> application_core/ray_main.o application_core/ray_main.cpp
>>>>>>>>>> make[1]: mpicxx: Command not found
>>>>>>>>>> make[1]: *** [application_core/ray_main.o] Error 127
>>>>>>>>>> make[1]: Leaving directory
>>>>>>>>>> `/s/parsons/f/fac/cboucher/Ray-v2.1.0/code'
>>>>>>>>>> mpicxx code/TheRayGenomeAssembler.a RayPlatform/libRayPlatform.a
>>>>>>>>>> -o Ray
>>>>>>>>>> make: mpicxx: Command not found
>>>>>>>>>> make: *** [Ray] Error 127
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any thoughts?
>>>>>>>>>>
>>>>>>>>>> In addition, is the the max kmer length 32? Most people are using
>>>>>>>>>> upwards to 55….?
>>>>>>>>>>
>>>>>>>>>> Christina
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> $ mpiexec -n 1 ./Ray -version
>>>>>>>>>>> $ mpiexec -n 999 ./Ray -k 31 -p 1.left.fastq 1.right.fastq -p
>>>>>>>>>>> 2.left.fastq 2.right.fastq -o Test
>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Christina
>>>>>>>>>>>>
>>>>>>>>>>>> *
>>>>>>>>>>>> ------------------------------------------------*
>>>>>>>>>>>> *Christina Boucher*
>>>>>>>>>>>> **Department of Computer Science
>>>>>>>>>>>> Colorado State University
>>>>>>>>>>>> Fort Collins, CO 80523
>>>>>>>>>>>> +1.970.491.8063
>>>>>>>>>>>> [email protected] <mailto:[email protected]>
>>>>>>>>>>>> <mailto:[email protected]>
>>>>>>>>>>>> <mailto:[email protected]>
>>>>>>>>>>>> <mailto:[email protected]>
>>>>>>>>>>>> <mailto:[email protected]>
>>>>>>>>>>>> www.christinaboucher.com <http://www.christinaboucher.com>
>>>>>>>>>>>> <http://www.christinaboucher.com>
>>>>>>>>>>>> <http://www.christinaboucher.com>
>>>>>>>>>>>> <http://www.christinaboucher.com> <http://www.christinaboucher.com>
>>>>>>>>>>>> *------------------------------------------------*
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *
>>>>>>>>>> ------------------------------------------------*
>>>>>>>>>> *Christina Boucher*
>>>>>>>>>> **Department of Computer Science
>>>>>>>>>> Colorado State University
>>>>>>>>>> Fort Collins, CO 80523
>>>>>>>>>> +1.970.491.8063
>>>>>>>>>> [email protected] <mailto:[email protected]>
>>>>>>>>>> <mailto:[email protected]>
>>>>>>>>>> <mailto:[email protected]> <mailto:[email protected]>
>>>>>>>>>> www.christinaboucher.com <http://www.christinaboucher.com>
>>>>>>>>>> <http://www.christinaboucher.com> <http://www.christinaboucher.com>
>>>>>>>>>> <http://www.christinaboucher.com>
>>>>>>>>>> *------------------------------------------------*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ***
>>>>>>>>> Sébastien Boisvert
>>>>>>>>> http://boisvert.info
>>>>>>>>> Sent from a PC (Linux panic 3.6.2-4.fc17.x86_64).
>>>>>>>>
>>>>>>>> *
>>>>>>>> ------------------------------------------------*
>>>>>>>> *Christina Boucher*
>>>>>>>> **Department of Computer Science
>>>>>>>> Colorado State University
>>>>>>>> Fort Collins, CO 80523
>>>>>>>> +1.970.491.8063
>>>>>>>> [email protected] <mailto:[email protected]>
>>>>>>>> <mailto:[email protected]> <mailto:[email protected]>
>>>>>>>> www.christinaboucher.com <http://www.christinaboucher.com>
>>>>>>>> <http://www.christinaboucher.com> <http://www.christinaboucher.com>
>>>>>>>> *------------------------------------------------*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ***
>>>>>>> Sébastien Boisvert
>>>>>>> http://boisvert.info
>>>>>>> Sent from a PC (Linux panic 3.6.2-4.fc17.x86_64).
>>>>>>
>>>>>> *
>>>>>> ------------------------------------------------*
>>>>>> *Christina Boucher*
>>>>>> **Department of Computer Science
>>>>>> Colorado State University
>>>>>> Fort Collins, CO 80523
>>>>>> +1.970.491.8063
>>>>>> [email protected] <mailto:[email protected]>
>>>>>> <mailto:[email protected]>
>>>>>> www.christinaboucher.com <http://www.christinaboucher.com>
>>>>>> <http://www.christinaboucher.com>
>>>>>> *------------------------------------------------*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Everyone hates slow websites. So do we.
>>>>> Make your web apps faster with AppDynamics
>>>>> Download AppDynamics Lite for free today:
>>>>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>>>>> _______________________________________________
>>>>> Denovoassembler-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
>>>>
>>>
>>>
>>> --
>>> Sent from my IBM Blue Gene/Q
>>
>> *
>> ------------------------------------------------*
>> *Christina Boucher*
>> **Department of Computer Science
>> Colorado State University
>> Fort Collins, CO 80523
>> +1.970.491.8063
>> [email protected] <mailto:[email protected]>
>> www.christinaboucher.com <http://www.christinaboucher.com>
>> *------------------------------------------------*
>>
>
>
> --
> Sent from my IBM Blue Gene/Q
------------------------------------------------
Christina Boucher
Department of Computer Science
Colorado State University
Fort Collins, CO 80523
+1.970.491.8063
[email protected]
www.christinaboucher.com
------------------------------------------------
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Denovoassembler-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
