Depending on the k-mer length, I suppose a distributed amount of memory between 300 and 600 GiB is required.
I say distributed because you don't need one big machine with all the memory, Ray can run on 10 machines for instance thanks to message passing. On 02/19/2013 09:29 AM, frederic texier wrote: > Thank your for your feedback. > The genome length id estimated at 2 Gb. > > Do you know the minimum of memory required for this analysis? > > Frederic. > > > -----Message d'origine----- > De : Sébastien Boisvert [mailto:[email protected]] > Envoyé : mardi 19 février 2013 14:39 > À : frederic texier > Cc : [email protected] > Objet : Re: Ray bad termination > > You have too many sequences (695 318 412 sequences) for the available memory > (49554300 kB). > > The git version of Ray provides a better adaptive Bloom filter that reduces > memory usage caused by erroneous k-mers, but in your case I think you need to > stack up more machines. > > Also important, what is the estimated genome length ? > > On 02/19/2013 07:44 AM, frederic texier wrote: >> I don't know if it's that. I can't see more. >> >> Rank 5: assembler memory usage: 6190616 KiB Rank 1 has 39700000 >> vertices Rank 1: assembler memory usage: 6190616 KiB Rank 5 is >> counting k-mers in sequence reads [1860001/86914801] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1546 units/second Estimated remaining time >> for this step: 15 hours, 16 minutes, 56 seconds Rank 4 is counting >> k-mers in sequence reads [1830001/86914801] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1567 units/second Estimated remaining time >> for this step: 15 hours, 4 minutes, 57 seconds Rank 7 is counting >> k-mers in sequence reads [1900001/86914805] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1584 units/second Estimated remaining time >> for this step: 14 hours, 54 minutes, 30 seconds Rank 2 has 39800000 >> vertices Rank 2: assembler memory usage: 6194904 KiB Rank 3 has >> 39800000 vertices Rank 3: assembler memory usage: 6194904 KiB Rank 0 >> has 39800000 vertices Rank 0: assembler memory usage: 6194904 KiB Rank >> 7 has 39800000 vertices Rank 7: assembler memory usage: 6198988 KiB >> Rank 4 has 39800000 vertices Rank 4: assembler memory usage: 6194904 >> KiB Rank 6 has 39800000 vertices Rank 6: assembler memory usage: >> 6199000 KiB Rank 5 has 39800000 vertices Rank 5: assembler memory >> usage: 6194904 KiB Rank 1 has 39800000 vertices Rank 1: assembler >> memory usage: 6194904 KiB Rank 6 is counting k-mers in sequence reads >> [1860001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1516 units/second >> Estimated remaining time for this step: 15 hours, 35 minutes, 4 >> seconds Rank 2 has 39900000 vertices Rank 2: assembler memory usage: >> 6199192 KiB Rank 3 has 39900000 vertices Rank 3: assembler memory >> usage: 6199192 KiB Rank 0 has 39900000 vertices Rank 0: assembler >> memory usage: 6199192 KiB Rank 1 is counting k-mers in sequence reads >> [1860001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1529 units/second >> Estimated remaining time for this step: 15 hours, 27 minutes, 7 >> seconds Rank 7 has 39900000 vertices Rank 7: assembler memory usage: >> 6203276 KiB Rank 6 has 39900000 vertices Rank 6: assembler memory >> usage: 6203288 KiB Rank 4 has 39900000 vertices Rank 4: assembler >> memory usage: 6199192 KiB Rank 5 has 39900000 vertices Rank 5: >> assembler memory usage: 6199192 KiB Rank 1 has 39900000 vertices Rank >> 1: assembler memory usage: 6199192 KiB Rank 3 is counting k-mers in >> sequence reads [1830001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1484 units/second Estimated remaining time for this step: 15 hours, 55 >> minutes, 34 seconds Rank 0 is counting k-mers in sequence reads >> [1820001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1540 units/second >> Estimated remaining time for this step: 15 hours, 20 minutes, 56 >> seconds Rank 2 is counting k-mers in sequence reads [1850001/86914801] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1580 units/second Estimated >> remaining time for this step: 14 hours, 57 minutes, 18 seconds Rank 2 >> has 40000000 vertices Rank 2: assembler memory usage: 6203480 KiB Rank >> 3 has 40000000 vertices Rank 3: assembler memory usage: 6199192 KiB >> Rank 0 has 40000000 vertices Rank 0: assembler memory usage: 6199192 >> KiB Rank 7 has 40000000 vertices Rank 7: assembler memory usage: >> 6207564 KiB Rank 6 has 40000000 vertices Rank 6: assembler memory >> usage: 6207576 KiB Rank 4 has 40000000 vertices Rank 4: assembler >> memory usage: 6203480 KiB Rank 5 has 40000000 vertices Rank 5: >> assembler memory usage: 6203480 KiB Rank 1 has 40000000 vertices Rank >> 1: assembler memory usage: 6199192 KiB Rank 5 is counting k-mers in >> sequence reads [1870001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1520 units/second Estimated remaining time for this step: 15 hours, 32 >> minutes, 30 seconds Rank 4 is counting k-mers in sequence reads >> [1840001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1563 units/second >> Estimated remaining time for this step: 15 hours, 7 minutes, 10 >> seconds Rank 7 is counting k-mers in sequence reads [1910001/86914805] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1602 units/second Estimated >> remaining time for this step: 14 hours, 44 minutes, 21 seconds Rank 2 >> has 40100000 vertices Rank 2: assembler memory usage: 6203480 KiB Rank >> 3 has 40100000 vertices Rank 3: assembler memory usage: 6203480 KiB >> Rank 0 has 40100000 vertices Rank 0: assembler memory usage: 6203480 >> KiB Rank 7 has 40100000 vertices Rank 7: assembler memory usage: >> 6207564 KiB Rank 4 has 40100000 vertices Rank 4: assembler memory >> usage: 6203480 KiB Rank 6 has 40100000 vertices Rank 6: assembler >> memory usage: 6207576 KiB Rank 5 has 40100000 vertices Rank 5: >> assembler memory usage: 6203480 KiB Rank 1 has 40100000 vertices Rank >> 1: assembler memory usage: 6207768 KiB Rank 6 is counting k-mers in >> sequence reads [1870001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1517 units/second Estimated remaining time for this step: 15 hours, 34 >> minutes, 21 seconds Rank 2 has 40200000 vertices Rank 2: assembler >> memory usage: 6207768 KiB Rank 3 has 40200000 vertices Rank 3: >> assembler memory usage: 6207768 KiB Rank 0 has 40200000 vertices Rank >> 0: assembler memory usage: 6207768 KiB Rank 1 is counting k-mers in >> sequence reads [1870001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1497 units/second Estimated remaining time for this step: 15 hours, 46 >> minutes, 50 seconds Rank 7 has 40200000 vertices Rank 7: assembler >> memory usage: 6211852 KiB Rank 6 has 40200000 vertices Rank 6: >> assembler memory usage: 6211864 KiB Rank 4 has 40200000 vertices Rank >> 4: assembler memory usage: 6207768 KiB Rank 5 has 40200000 vertices >> Rank 5: assembler memory usage: 6207768 KiB Rank 1 has 40200000 >> vertices Rank 1: assembler memory usage: 6207768 KiB Rank 3 is >> counting k-mers in sequence reads [1840001/86914801] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1466 units/second Estimated remaining time >> for this step: 16 hours, 7 minutes, 11 seconds Rank 0 is counting >> k-mers in sequence reads [1830001/86914801] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1485 units/second Estimated remaining time >> for this step: 15 hours, 54 minutes, 56 seconds Rank 2 is counting >> k-mers in sequence reads [1860001/86914801] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1516 units/second Estimated remaining time >> for this step: 15 hours, 35 minutes, 4 seconds Rank 2 has 40300000 >> vertices Rank 2: assembler memory usage: 6212056 KiB Rank 3 has >> 40300000 vertices Rank 3: assembler memory usage: 6212056 KiB Rank 0 >> has 40300000 vertices Rank 0: assembler memory usage: 6212056 KiB Rank >> 7 has 40300000 vertices Rank 7: assembler memory usage: 6216140 KiB >> Rank 5 has 40300000 vertices Rank 5: assembler memory usage: 6212056 >> KiB Rank 6 has 40300000 vertices Rank 6: assembler memory usage: >> 6216152 KiB Rank 4 has 40300000 vertices Rank 4: assembler memory >> usage: 6212056 KiB Rank 1 has 40300000 vertices Rank 1: assembler >> memory usage: 6212056 KiB Rank 5 is counting k-mers in sequence reads >> [1880001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1495 units/second >> Estimated remaining time for this step: 15 hours, 47 minutes, 59 >> seconds Rank 4 is counting k-mers in sequence reads [1850001/86914801] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1436 units/second Estimated >> remaining time for this step: 16 hours, 27 minutes, 17 seconds Rank 7 >> is counting k-mers in sequence reads [1920001/86914805] Speed >> RAY_SLAVE_MODE_ADD_VERTICES 1489 units/second Estimated remaining time >> for this step: 15 hours, 51 minutes, 21 seconds Rank 2 has 40400000 >> vertices Rank 2: assembler memory usage: 6216344 KiB Rank 3 has >> 40400000 vertices Rank 3: assembler memory usage: 6216344 KiB Rank 0 >> has 40400000 vertices Rank 0: assembler memory usage: 6216344 KiB Rank >> 7 has 40400000 vertices Rank 7: assembler memory usage: 6220428 KiB >> Rank 6 has 40400000 vertices Rank 6: assembler memory usage: 6220440 >> KiB Rank 5 has 40400000 vertices Rank 5: assembler memory usage: >> 6216344 KiB Rank 4 has 40400000 vertices Rank 4: assembler memory >> usage: 6216344 KiB Rank 1 has 40400000 vertices Rank 1: assembler >> memory usage: 6216344 KiB Rank 6 is counting k-mers in sequence reads >> [1880001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1522 units/second >> Estimated remaining time for this step: 15 hours, 31 minutes, 10 >> seconds Rank 1 is counting k-mers in sequence reads [1880001/86914801] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1474 units/second Estimated >> remaining time for this step: 16 hours, 1 minutes, 29 seconds Rank 2 >> has 40500000 vertices Rank 2: assembler memory usage: 6216344 KiB Rank >> 3 has 40500000 vertices Rank 3: assembler memory usage: 6216344 KiB >> Rank 0 has 40500000 vertices Rank 0: assembler memory usage: 6216344 >> KiB Rank 7 has 40500000 vertices Rank 7: assembler memory usage: >> 6220428 KiB Rank 6 has 40500000 vertices Rank 6: assembler memory >> usage: 6224728 KiB Rank 4 has 40500000 vertices Rank 4: assembler >> memory usage: 6216344 KiB Rank 5 has 40500000 vertices Rank 5: >> assembler memory usage: 6216344 KiB Rank 1 has 40500000 vertices Rank >> 1: assembler memory usage: 6216344 KiB Rank 3 is counting k-mers in >> sequence reads [1850001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1342 units/second Estimated remaining time for this step: 17 hours, 36 >> minutes, 26 seconds Rank 0 is counting k-mers in sequence reads >> [1840001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1322 units/second >> Estimated remaining time for this step: 17 hours, 52 minutes, 33 >> seconds Rank 2 is counting k-mers in sequence reads [1870001/86914801] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1399 units/second Estimated >> remaining time for this step: 16 hours, 53 minutes, 9 seconds Rank 2 >> has 40600000 vertices Rank 2: assembler memory usage: 6220632 KiB Rank >> 3 has 40600000 vertices Rank 3: assembler memory usage: 6220632 KiB >> Rank 0 has 40600000 vertices Rank 0: assembler memory usage: 6220632 >> KiB Rank 7 has 40600000 vertices Rank 7: assembler memory usage: >> 6224716 KiB Rank 6 has 40600000 vertices Rank 6: assembler memory >> usage: 6224728 KiB Rank 5 has 40600000 vertices Rank 5: assembler >> memory usage: 6220632 KiB Rank 4 has 40600000 vertices Rank 4: >> assembler memory usage: 6220632 KiB Rank 1 has 40600000 vertices Rank >> 1: assembler memory usage: 6220632 KiB Rank 5 is counting k-mers in >> sequence reads [1890001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES >> 1290 units/second Estimated remaining time for this step: 18 hours, 18 >> minutes, 30 seconds Rank 4 is counting k-mers in sequence reads >> [1860001/86914801] Speed RAY_SLAVE_MODE_ADD_VERTICES 1324 units/second >> Estimated remaining time for this step: 17 hours, 50 minutes, 40 >> seconds Rank 7 is counting k-mers in sequence reads [1930001/86914805] >> Speed RAY_SLAVE_MODE_ADD_VERTICES 1353 units/second Estimated >> remaining time for this step: 17 hours, 26 minutes, 52 seconds Rank 2 >> has 40700000 vertices Rank 2: assembler memory usage: 6224920 KiB Rank >> 3 has 40700000 vertices Rank 3: assembler memory usage: 6224920 KiB >> Rank 0 has 40700000 vertices Rank 0: assembler memory usage: 6224920 >> KiB Rank 7 has 40700000 vertices Rank 7: assembler memory usage: >> 6229004 KiB Rank 6 has 40700000 vertices Rank 6: assembler memory >> usage: 6229016 KiB Rank 5 has 40700000 vertices Rank 5: assembler >> memory usage: 6224920 KiB Rank 4 has 40700000 vertices Rank 4: >> assembler memory usage: 6224920 KiB Rank 1 has 40700000 vertices Rank >> 1: assembler memory usage: 6224920 KiB >> >> >> -----Message d'origine----- >> De : Sébastien Boisvert [mailto:[email protected]] >> Envoyé : mardi 19 février 2013 13:29 >> À : frederic texier >> Cc : [email protected] >> Objet : Re: Ray bad termination >> >> On 02/19/2013 07:28 AM, frederic texier wrote: >>> I have 49554300 kB physical memory. >>> 695 318 412 sequences are in those files. >>> >>> You think I don't have enough physical memory for this number of sequences. >>> >> >> In the standard output, what was the memory usage reported by Ray ? >> >>> >>> Frederic. >>> >>> >>> -----Message d'origine----- >>> De : Sébastien Boisvert [mailto:[email protected]] >>> Envoyé : mardi 19 février 2013 13:19 >>> À : frederic texier >>> Cc : [email protected] >>> Objet : Re: Ray bad termination >>> >>> How much physical memory does your system have (head -n 1 /proc/meminfo) ? >>> >>> How many sequences are in those files (tail -n 3 >>> ray_hiseq/NumberOfSequences.txt) ? >>> >>> On 02/19/2013 04:13 AM, frederic texier wrote: >>>> Hi, >>>> >>>> I have an error when I launch an analysis with Ray. >>>> >>>> The following message appears: >>>> >>>> ===================================================================================== >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = EXIT CODE: 9 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> ==================================================================== >>>> = = =============== APPLICATION TERMINATED WITH THE EXIT STRING: >>>> Killed (signal 9) >>>> >>>> Could you help me , please? >>>> I don't know if it's a problem of memory or another problem. >>>> >>>> Best regards, >>>> Frédéric. >>>> >>>> >>> >> > ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb _______________________________________________ Denovoassembler-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
