I was finally able to parse these nextseq runs.
The key was to allow more reads per tile in RAM.
I used:
MAX_READS_IN_RAM_PER_TILE=12000000 \
I think it’s supposed to work with default settings, using disk if necessary,
but i get these kinds of back traces If i use the default setting (i’m
certainly not out of disk space).
INFO 2014-11-12 16:05:11 IlluminaBasecallsConverter Read
139,000,000 records. Elapsed time: 00:36:56s. Time for last 1,000,000: 15s.
Last read position: */*
INFO 2014-11-12 16:05:15 IlluminaBasecallsConverter Write
93,000,000 records. Elapsed time: 00:37:00s. Time for last 1,000,000: 14s.
Last read position: */*
Exception in thread "pool-1-thread-26" ERROR 2014-11-12 16:05:16
IlluminaBasecallsConverter Failure encountered in worker thread;
attempting to shut down remaining worker threads and terminate ...
net.sf.samtools.util.RuntimeIOException: java.io.IOException: No space left on
device
at
net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:232)
at
net.sf.samtools.util.SortingCollection.add(SortingCollection.java:152)
at
net.sf.picard.illumina.IlluminaBasecallsConverter$TileProcessingRecord.addRecord(IlluminaBasecallsConverter.java:372)
at
net.sf.picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:482)
at
net.sf.picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:577)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at
org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:101)
at
org.xerial.snappy.SnappyOutputStream.close(SnappyOutputStream.java:137)
at
net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:223)
... 7 more
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
net.sf.picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:628)
at
net.sf.picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:248)
at
net.sf.picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:157)
at
net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:179)
at
net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:120)
at
net.sf.picard.illumina.IlluminaBasecallsToFastq.main(IlluminaBasecallsToFastq.java:304)
INFO 2014-11-12 16:06:03 IlluminaBasecallsConverter Read
140,000,000 records. Elapsed time: 00:37:48s. Time for last 1,000,000: 52s.
Last read position: */*
On Nov 20, 2014, at 5:19 PM, Nils Homer
<nho...@broadinstitute.org<mailto:nho...@broadinstitute.org>> wrote:
Hey Brad,
it looks like someone or something stopped the execution midstream. Is there a
time limit on your job, or some other resource manager that would shut this
down? This tool supports NextSeq data so the machine type should not be an
issue.
N
On Mon, Nov 10, 2014 at 3:56 PM, Langhorst, Brad
<langho...@neb.com<mailto:langho...@neb.com>> wrote:
[Mon Nov 10 11:07:48 EST 2014] picard.illumina.IlluminaBasecallsToFastq
BASECALLS_DIR=/mnt/ngswork/langhorst/141029_NS500355_0002_AH15U8BGXX/Data/Intensities/BaseCalls
LANE=2 RUN_BARCODE=141029_NS500355_0002_AH15U8BGXX MACHINE_NAME=NS500355
FLOWCELL_BARCODE=H15U8BGXX READ_STRUCTURE=151T7B151T
MULTIPLEX_PARAMS=/mnt/ngswork/langhorst/141029_NS500355_0002_AH15U8BGXX/lane2_multiplex_params.txt
NUM_PROCESSORS=32 ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2,
FLUIDIGM] APPLY_EAMSS_FILTER=true FORCE_GC=true
MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true
READ_NAME_FORMAT=CASAVA_1_8 COMPRESS_OUTPUTS=false VERBOSITY=INFO QUIET=false
VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000
CREATE_INDEX=false CREATE_MD5_FILE=false
[Mon Nov 10 11:07:48 EST 2014] Executing as langhorst@seq-himem01-i on Linux
3.13.0-32-generic amd64; OpenJDK 64-Bit Server VM 1.7.0_65-b32; Picard version:
1.124(69ecf101f612fdc0f3d555aa2d3cc0b1ea193c68_1415030499) IntelDeflater
...
INFO 2014-11-10 11:25:11 IlluminaBasecallsConverter Write
62,000,000 records. Elapsed time: 00:17:15s. Time for last 1,000,000: 6s.
Last read position: */*
ERROR 2014-11-10 11:25:11 IlluminaBasecallsConverter Failure
encountered in worker thread; attempting to shut down remaining worker threads
and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:695)
at
picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:311)
at
picard.illumina.IlluminaBasecallsToFastq.doWork(IlluminaBasecallsToFastq.java:186)
at
picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
at
picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:89)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:99)
Anybody have experience parsing nextseq data will these tools?
The code does refer to nextseq, but I'm not sure how well tested it is.
Brad
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Samtools-help mailing list
Samtools-help@lists.sourceforge.net<mailto:Samtools-help@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/samtools-help
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Samtools-help mailing list
Samtools-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/samtools-help
