On 09/12/13 06:21 AM, MacManes, Matthew wrote: > Another datapoint here- this seems like it is specific to architecture- I > have the error on Cray using their MPI implementation, but >not on Stampede (https://www.tacc.utexas.edu/stampede/) which runs OpenMPI and >the “normal” compilers, e.g. gcc. >
Hi, Are you using the same command on both machines ? > Matt > _____________________________________ > *Matthew MacManes, Ph.D. * > University of New Hampshire I Assistant Professor > Department of Molecular, Cellular, & Biomedical Sciences > Durham, NH 03824 > Phone: 603-862-4052I Twitter: @PeroMHC <https://twitter.com/PeroMHC> > Web: genomebio.org <http://genomebio.org/> > Office: 189 Rudman Hall > Laboratory: 145 Rudman Hall > > On Dec 7, 2013, at 10:19 AM, MacManes, Matthew <matthew.macma...@unh.edu > <mailto:matthew.macma...@unh.edu>> wrote: > >> Though the same thing happens with the regular release Ray-2.3.0. >> >> Seb- please let me know what other debugging info I can send you. >> >> Matt >> _____________________________________ >> *Matthew MacManes, Ph.D. * >> University of New Hampshire I Assistant Professor >> Department of Molecular, Cellular, & Biomedical Sciences >> Durham, NH 03824 >> Phone: 603-862-4052I Twitter: @PeroMHC <https://twitter.com/PeroMHC> >> Web: genomebio.org <http://genomebio.org/> >> Office: 189 Rudman Hall >> Laboratory: 145 Rudman Hall >> >> On Dec 6, 2013, at 11:03 AM, MacManes, Matthew <matthew.macma...@unh.edu >> <mailto:matthew.macma...@unh.edu>> wrote: >> >>> This is Ray version 2.3.1-devel. I pulled from github a couple of days ago. >>> >>> Mat >>> >>> _____________________________________ >>> *Matthew MacManes, Ph.D. * >>> University of New Hampshire I Assistant Professor >>> Department of Molecular, Cellular, & Biomedical Sciences >>> Durham, NH 03824 >>> Phone: 603-862-4052I Twitter: @PeroMHC <https://twitter.com/PeroMHC> >>> Web: genomebio.org <http://genomebio.org/> >>> Office: 189 Rudman Hall >>> Laboratory: 145 Rudman Hall >>> >>> On Dec 6, 2013, at 10:42 AM, Sébastien Boisvert >>> <sebastien.boisver...@ulaval.ca <mailto:sebastien.boisver...@ulaval.ca>> >>> wrote: >>> >>>> On 05/12/13 05:19 PM, MacManes, Matthew wrote: >>>>> Hi All, >>>>> >>>>> Having problems with segfault on a Cray XE6m-200 using 480 ranks: Looks >>>>> like 'Selection of optimal read markers’ had just finished. >>>> >>>> Which Ray version are you using ? >>>> >>>>> >>>>> error code: >>>>>> _pmiu_daemon(SIGCHLD): [NID 00285] [c1-0c2s1n3] [Thu Dec 5 16:34:07 >>>>>> 2013] PE RANK 139 exit signal Segmentation fault >>>>>> [NID 00285] 2013-12-05 16:34:07 Apid 54698: initiated application >>>>>> termination >>>>> >>>>> mpiexec -n 480 Ray \ >>>>> -show-memory-usage \ >>>>> -read-write-checkpoints \ >>>>> checkpoints \ >>>>> -k \ >>>>> 51 \ >>>>> -p \ >>>>> L1.notCombined_1.fastq \ >>>>> L1.notCombined_2.fastq \ >>>>> -p \ >>>>> L2.notCombined_1.fastq \ >>>>> L2.notCombined_2.fastq \ >>>>> -p \ >>>>> L3.notCombined_1.fastq \ >>>>> L3.notCombined_2.fastq \ >>>>> -s \ >>>>> L3.extendedFrags.fastq \ >>>>> -s \ >>>>> L2.extendedFrags.fastq \ >>>>> -s \ >>>>> L3.extendedFrags.fastq \ >>>>> -s \ >>>>> 360.unpaired.fastq >>>>> >>>>> >>>>> Here are last 100 lines of output: >>>>> >>>>> Rank 314: assembler memory usage: 3958204 KiB >>>>> Rank 105: memory usage for optimal read markers= 69632 KiB >>>>> Rank 105 is creating seeds [1/13097800] >>>>> Rank 105: assembler memory usage: 3958204 KiB >>>>> Rank 157: memory usage for optimal read markers= 69632 KiB >>>>> Rank 157 is creating seeds [1/13097174] >>>>> Rank 157: assembler memory usage: 3962332 KiB >>>>> Rank 324: memory usage for optimal read markers= 69632 KiB >>>>> Rank 324 is creating seeds [1/13103516] >>>>> Rank 324: assembler memory usage: 3962332 KiB >>>>> Rank 185: memory usage for optimal read markers= 69632 KiB >>>>> Rank 185 is creating seeds [1/13089676] >>>>> Rank 185: assembler memory usage: 3962332 KiB >>>>> Rank 385: memory usage for optimal read markers= 69632 KiB >>>>> Rank 385 is creating seeds [1/13101020] >>>>> Rank 385: assembler memory usage: 3958204 KiB >>>>> Rank 85: memory usage for optimal read markers= 69632 KiB >>>>> Rank 85 is creating seeds [1/13092560] >>>>> Rank 85: assembler memory usage: 3958204 KiB >>>>> Rank 158: memory usage for optimal read markers= 65536 KiB >>>>> Rank 158 is creating seeds [1/13093210] >>>>> Rank 158: assembler memory usage: 3958204 KiB >>>>> Rank 332: memory usage for optimal read markers= 69632 KiB >>>>> Rank 332 is creating seeds [1/13094478] >>>>> Rank 332: assembler memory usage: 3962332 KiB >>>>> Rank 309: memory usage for optimal read markers= 65536 KiB >>>>> Rank 309 is creating seeds [1/13102758] >>>>> Rank 309: assembler memory usage: 3958204 KiB >>>>> Rank 136: memory usage for optimal read markers= 69632 KiB >>>>> Rank 136 is creating seeds [1/13094218] >>>>> Rank 136: assembler memory usage: 3962332 KiB >>>>> Rank 293: memory usage for optimal read markers= 73728 KiB >>>>> Rank 293 is creating seeds [1/13090794] >>>>> Rank 293: assembler memory usage: 3962332 KiB >>>>> Rank 271: memory usage for optimal read markers= 73728 KiB >>>>> Rank 271 is creating seeds [1/13093488] >>>>> Rank 271: assembler memory usage: 3962332 KiB >>>>> Rank 176: memory usage for optimal read markers= 73728 KiB >>>>> Rank 176 is creating seeds [1/13107640] >>>>> Rank 176: assembler memory usage: 3964384 KiB >>>>> Rank 264: memory usage for optimal read markers= 69632 KiB >>>>> Rank 264 is creating seeds [1/13109980] >>>>> Rank 264: assembler memory usage: 3962332 KiB >>>>> Rank 138: memory usage for optimal read markers= 69632 KiB >>>>> Rank 138 is creating seeds [1/13093866] >>>>> Rank 138: assembler memory usage: 3958204 KiB >>>>> Rank 395: memory usage for optimal read markers= 69632 KiB >>>>> Rank 395 is creating seeds [1/13091210] >>>>> Rank 395: assembler memory usage: 3958204 KiB >>>>> Rank 335: memory usage for optimal read markers= 73728 KiB >>>>> Rank 335 is creating seeds [1/13095456] >>>>> Rank 335: assembler memory usage: 3966460 KiB >>>>> Rank 329: memory usage for optimal read markers= 73728 KiB >>>>> Rank 329 is creating seeds [1/13106094] >>>>> Rank 329: assembler memory usage: 3962332 KiB >>>>> Rank 328: memory usage for optimal read markers= 69632 KiB >>>>> Rank 328 is creating seeds [1/13100648] >>>>> Rank 328: assembler memory usage: 3958204 KiB >>>>> Rank 303: memory usage for optimal read markers= 69632 KiB >>>>> Rank 303 is creating seeds [1/13085492] >>>>> Rank 303: assembler memory usage: 3962332 KiB >>>>> Rank 290: memory usage for optimal read markers= 65536 KiB >>>>> Rank 290 is creating seeds [1/13091084] >>>>> Rank 290: assembler memory usage: 3958204 KiB >>>>> Rank 65: memory usage for optimal read markers= 69632 KiB >>>>> Rank 65 is creating seeds [1/13092946] >>>>> Rank 65: assembler memory usage: 3958204 KiB >>>>> Rank 119: memory usage for optimal read markers= 69632 KiB >>>>> Rank 119 is creating seeds [1/13096562] >>>>> Rank 119: assembler memory usage: 3958204 KiB >>>>> Rank 454: memory usage for optimal read markers= 69632 KiB >>>>> Rank 454 is creating seeds [1/13100860] >>>>> Rank 454: assembler memory usage: 3958204 KiB >>>>> Rank 114: memory usage for optimal read markers= 69632 KiB >>>>> Rank 114 is creating seeds [1/13091328] >>>>> Rank 114: assembler memory usage: 3958204 KiB >>>>> Rank 305: memory usage for optimal read markers= 69632 KiB >>>>> Rank 305 is creating seeds [1/13101714] >>>>> Rank 305: assembler memory usage: 3958204 KiB >>>>> Rank 338: memory usage for optimal read markers= 69632 KiB >>>>> Rank 338 is creating seeds [1/13099074] >>>>> Rank 338: assembler memory usage: 3958204 KiB >>>>> Rank 318: memory usage for optimal read markers= 65536 KiB >>>>> Rank 318 is creating seeds [1/13099310] >>>>> Rank 318: assembler memory usage: 3958204 KiB >>>>> Rank 342: memory usage for optimal read markers= 69632 KiB >>>>> Rank 342 is creating seeds [1/13096148] >>>>> Rank 342: assembler memory usage: 3958204 KiB >>>>> Rank 460: memory usage for optimal read markers= 69632 KiB >>>>> Rank 460 is creating seeds [1/13106978] >>>>> Rank 460: assembler memory usage: 4127836 KiB >>>>> Rank 74: memory usage for optimal read markers= 69632 KiB >>>>> Rank 74 is creating seeds [1/13087278] >>>>> Rank 74: assembler memory usage: 3958204 KiB >>>>> Rank 461: memory usage for optimal read markers= 73728 KiB >>>>> Rank 461 is creating seeds [1/13100888] >>>>> Rank 461: assembler memory usage: 4131964 KiB >>>>> Application 54698 exit codes: 139 >>>>> Application 54698 exit signals: Killed >>>>> Application 54698 resources: utime ~11381s, stime ~63s, Rss ~809788, >>>>> inblocks ~77421751, outblocks ~17677754 >>>>> >>>>> Any ideas? >>>>> >>>>> _____________________________________ >>>>> *Matthew MacManes, Ph.D. * >>>>> University of New Hampshire I Assistant Professor >>>>> Department of Molecular, Cellular, & Biomedical Sciences >>>>> Durham, NH 03824 >>>>> Phone: 603-862-4052I Twitter: @PeroMHC <https://twitter.com/PeroMHC> >>>>> Web:genomebio.org <http://genomebio.org/><http://genomebio.org/> >>>>> Office: 189 Rudman Hall >>>>> Laboratory: 145 Rudman Hall >>> >>> ------------------------------------------------------------------------------ >>> Sponsored by Intel(R) XDK >>> Develop, test and display web and hybrid apps with a single code base. >>> Download it for free now! >>> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk_______________________________________________ >>> Denovoassembler-users mailing list >>> Denovoassembler-users@lists.sourceforge.net >>> <mailto:Denovoassembler-users@lists.sourceforge.net> >>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users >> >> ------------------------------------------------------------------------------ >> Sponsored by Intel(R) XDK >> Develop, test and display web and hybrid apps with a single code base. >> Download it for free now! >> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk_______________________________________________ >> Denovoassembler-users mailing list >> Denovoassembler-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users > ------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk _______________________________________________ Denovoassembler-users mailing list Denovoassembler-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
