On 23/09/13 11:48 AM, VJR VJR wrote: > I am using the latest 2.2 as in the attached log I think.
Hi Ranjeev, "-80285528 bytes" is an integer overflow. The problem was in part fixed in the git repository. But basically, 145745448 reads is a lot of reads for a single MPI rank. Here are 2 possible workarounds that I can offer: 1. Set the number of bits manually with -bloom-filter-bits. For example, to use 512 MiB of memory for the Bloom filter on each MPI rank, use -bloom-filter-bits 4398046511104. 2. Use more MPI ranks (more processor cores). If you go from 20 to 200 MPI ranks, everything will be faster etc. > > Ranjeev > > > Sent from my Windows 8 phone > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > From: Sébastien Boisvert <mailto:sebastien.boisver...@ulaval.ca> > Sent: 23/9/2013 9:41 PM > To: Ranjeev <mailto:ranj...@um.edu.my> > Cc: s...@boisvert.info <mailto:s...@boisvert.info>; > denovoassembler-users@lists.sourceforge.net > <mailto:denovoassembler-users@lists.sourceforge.net>; choo siew woh > <mailto:csw1...@hotmail.com>; Shi Yang Tan <mailto:shiyang...@gmail.com> > Subject: Re: Ray network test takes forever > > On 23/09/13 12:37 AM, Ranjeev wrote: > > I managed to get Ray running however, I get the some memory problem which > I cant really understand. I ran the pbs on three nodes without multiple > processes and still the error shows up. > > Hello, > > It seems to me that 145745448 reads is a lot of reads for a single > MPI rank. > > > However, "-80285528 bytes" is definitely a bug in Ray. > > Which version of Ray are you using ? > > > > > qsub -l nodes=compute-0-0+compute-0-1+compute-0-2 ray_p.pbs > > > > Attached is the log. Below the summary > > > > *** > > Step: Sequence loading > > Date: Mon Sep 23 10:50:19 2013 > > Elapsed time: 15 minutes, 57 seconds > > Since beginning: 27 minutes, 2 seconds > > *** > > > > > > Rank 2 has 145745448 sequence reads (completed) > > Critical exception: The system is out of memory, returned NULL. > > Requested -80285528 bytes of type RAY_MALLOC_TYPE_BLOOM_FILTER > > Critical exception: The system is out of memory, returned NULL. > > Requested -80285528 bytes of type RAY_MALLOC_TYPE_BLOOM_FILTER > > Critical exception: The system is out of memory, returned NULL. > > Requested -80285528 bytes of type RAY_MALLOC_TYPE_BLOOM_FILTER > > -------------------------------------------------------------------------- > > mpiexec has exited due to process rank 0 with PID 8831 on > > node compute-0-0.local exiting improperly. There are two reasons this > could occur: > > > > 1. this process did not call "init" before exiting, but others in > > the job did. This can cause a job to hang indefinitely while it waits > > for all processes to call "init". By rule, if one process calls "init", > > then ALL processes must call "init" prior to termination. > > > > 2. this process called "init", but exited without calling "finalize". > > By rule, all processes that call "init" MUST call "finalize" prior to > > exiting or it will be considered an "abnormal termination" > > > > This may have caused other processes in the application to be > > terminated by signals sent by mpiexec (as reported here). > > -------------------------------------------------------------------------- > > > > *Hope you can help* > > > > > > Thanks. > > > > > > > > > > On Sun, Sep 22, 2013 at 2:28 PM, Ranjeev <ranj...@um.edu.my > <mailto:ranj...@um.edu.my>> wrote: > > > > Btw, my assembly of the panda genome terminated with PBS: job killed: > node 2 (compute-0-4) requested job terminate, 'EOF' (code 1099) - received > SISTER_EOF attempting to communicate with sister MOM's. > > > > > > On Sun, Sep 22, 2013 at 12:21 PM, Ranjeev <ranj...@um.edu.my > <mailto:ranj...@um.edu.my>> wrote: > > > > Hi > > > > Thanks I was using torque and specified the nodes only with > Infiniband and it seemed to have added those n parameters by itself. Btw, if > I have a node with 192Gb RAM and 24 processors, how many processes do you > recommend per node? I notice each process consuming considerable RAM and > start to use swap when I use 24 processes per node. > > > > > > > > On Fri, Sep 20, 2013 at 7:08 PM, Sébastien Boisvert > <sebastien.boisver...@ulaval.ca <mailto:sebastien.boisver...@ulaval.ca>> > wrote: > > > > On 19/09/13 11:26 PM, Ranjeev wrote: > > > > HI, > > > > I'm using openmpi and torque - I managed to test Ray once > and worked fine (for a single node). > > The next time I run the network test takes forever. > Doesn't go beyond the last line > > > > Ray command: > > mpiexec -n 2 Ray \ > > -n \ > > 40 \ > > > > > > > > Do you want 2 ranks or 40 ranks ? > > > > > > > > -p \ > > /home/ranjeev/NTM/L007_R1.fq \ > > /home/ranjeev/NTM/L007_R2.fq \ > > -o \ > > /home/ranjeev/temp/Ray/test3 > > > > Rank 0 wrote /home/ranjeev/temp/Ray/test3/__RayCommand.txt > > > > k-mer length: 21 > > Rank 1: assembler memory usage: 24972 KiB > > Rank 0: assembler memory usage: 24976 KiB > > Rank 1: assembler memory usage: 90796 KiB > > Rank 1: Rank= 1 Size= 2 ProcessIdentifier= 12290 > > Rank 0: assembler memory usage: 90804 KiB > > Rank 0: Rank= 0 Size= 2 ProcessIdentifier= 18150 > > Rank 0: testing the network, please wait... > > > > Rank 0 is testing the network [0/1000] > > > > > > Regards, > > Ranjeev > > > > > > > > -- > > " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan > bersamanya ("Mesej") adalah ditujukan hanya untuk kegunaan > penerima(-penerima) yang termaklum di atas dan mungkin mengandungi maklumat > sulit. Anda dengan ini dimaklumkan bahawa mengambil apa jua tindakan > bersandarkan kepada, membuat penilaian, mengulang hantar, menghebah, > mengedar, mencetak, atau menyalin Mesej ini atau sebahagian daripadanya oleh > sesiapa selain daripada penerima(-penerima) yang termaklum di atas adalah > dilarang. Jika anda telah menerima Mesej ini kerana kesilapan, anda mesti > menghapuskan Mesej ini dengan segera dan memaklumkan kepada penghantar Mesej > ini menerusi balasan e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang > maklumat lain di dalam Mesej ini yang tidak berkait dengan urusan rasmi > Universiti Malaya adalah difahami sebagai bukan dikeluar atau diperakui oleh > mana-mana pihak yang disebut. > > > > > > DISCLAIMER: This e-mail and any files transmitted with it > ("Message") is intended only for the use of the recipient(s) named above and > may contain confidential information. You are hereby notified that the taking > of any action in reliance upon, or any review, retransmission, dissemination, > distribution, printing or copying of this Message or any part thereof by > anyone other than the intended recipient(s) is strictly prohibited. If you > have received this Message in error, you should delete this Message > immediately and advise the sender by return e-mail. Opinions, conclusions and > other information in this Message that do not relate to the official business > of University of Malaya shall be understood as neither given nor endorsed by > any of the forementioned. " > > > > > > > > > > > -- > " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya ("Mesej") > adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang termaklum di > atas dan mungkin mengandungi maklumat sulit. Anda dengan ini dimaklumkan > bahawa mengambil apa jua tindakan bersandarkan kepada, membuat penilaian, > mengulang hantar, menghebah, mengedar, mencetak, atau menyalin Mesej ini > atau sebahagian daripadanya oleh sesiapa selain daripada > penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah > menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini > dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan > e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di > dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya > adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak > yang disebut. > > > DISCLAIMER: This e-mail and any files transmitted with it ("Message") is > intended only for the use of the recipient(s) named above and may contain > confidential information. You are hereby notified that the taking of any > action in reliance upon, or any review, retransmission, dissemination, > distribution, printing or copying of this Message or any part thereof by > anyone other than the intended recipient(s) is strictly prohibited. If you > have received this Message in error, you should delete this Message > immediately and advise the sender by return e-mail. Opinions, conclusions > and other information in this Message that do not relate to the official > business of University of Malaya shall be understood as neither given nor > endorsed by any of the forementioned. " ------------------------------------------------------------------------------ LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk _______________________________________________ Denovoassembler-users mailing list Denovoassembler-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
