Hi Ole, I'm in the middle of optimising the processing of such file. While I'm at it I have a quick question:
I'm processing my input file with -N 2500 as it seems to give me the best processing time. My CPU usage and disk IO are well below their maximum capacity and wondered how GNU parallel processes/submits new jobs as others are completed? I'm thinking that GNU parallel is somehow stalling my pipeline. Could you provide some information on this aspect of GNU parallel? Cheers, Nathan > Nathan Watson-Haigh Senior Bioinformatician | The Australian Wine Research Institute Waite Precinct, Hartley Grove cnr Paratoo Road, Urrbrae (Adelaide) SA 5064 | http://www.awri.com.au/contact/map.asp PO Box 197, Glen Osmond SA 5064, Australia T: +61 8 83136836 (direct) | T: +61 8 83136600 | F: +61 8 83136601 8 www: http://www.awri.com.au/ | http://www.awri.com.au/events/calendar/ This communication, including attachments, is intended only for the addressee(s) and contains information which might be confidential and/or the copyright of The Australian Wine Research Institute (AWRI) or a third party. If you are not the intended recipient of this communication please immediately delete and destroy all copies and contact the sender. If you are the intended recipient of this communication you should not copy, disclose or distribute any of the information contained herein without the consent of the AWRI and the sender. Any views expressed in this communication are those of the individual sender except where the sender specifically states them to be the views of the AWRI. No representation is made that this communication, including attachments, is free of viruses. Virus scanning is recommended and is the responsibility of the recipient. -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Ole > Tange > Sent: Saturday, 20 August 2011 11:49 PM > To: Nathan Watson-Haigh > Cc: [email protected] > Subject: Re: Parallel Merge > > On Sat, Aug 20, 2011 at 12:54 AM, Nathan Watson-Haigh > <[email protected]> wrote: > > > > What I'm actually doing is using the ABySS genome assembler. Part of > the pipeline is: > > > > KAligher | ParseAligns | sort | DistanceEst > > > > KAligner takes sequences from one file (queries) and finds alignments > agianst sequences in another file (targets), outputting these in > Sequence Alignment/Map (SAM) format. ParseAligns takes the SAM format > and filters out some alignments. It is the ParseAligns step which is > slowest and I'm looking at how best to split up the work to make use of > more cores. A job for early next week! > > The most obvious way seems to be: > > cat queries | parallel --pipe --files 'KAligher | ParseAligns | > sort' | parallel -Xj1 sort -m {}\;rm {} | DistanceEst > > Would that work? > > /Ole
