Sorry for reporting back late...
Robert and Tom, thanks for all the hints/pointers. We've tried the latest
version 1.6 and got the same problem. After much more thorough investigation,
we've finally figured out the root cause, the excessive memory usage is not
cause by samtools, as we all expected before, but due to some bug more upfront.
Anyway thank you all for your help!
Best regards,
Jing
________________________________
发件人: Robert Davies <r...@sanger.ac.uk>
发送时间: 2018年1月2日 13:40
收件人: samtools-help@lists.sourceforge.net
主题: Re: [Samtools-help] samtools view error: convert SAM to BAM fail silently
(fwd)
Sorry, forgot to CC this to the list...
---------- Forwarded message ----------
Date: Tue, 26 Dec 2017 13:18:06 +0000
From: rmd <r...@sanger.ac.uk>
To: Hu Jing <hj_hanna2...@hotmail.com>
Subject: Re: [Samtools-help] samtools view error: convert SAM to BAM fail
silently
On 2017-12-26 00:56, Hu Jing wrote:
> Recently we ran into an issue with converting SAM to BAM using samtools view
> command called in python, please see the commands below,
>
> cmd = 'samtools view -b -o {} {}'.format(output_BAM, input_SAM)
> os.system(cmd)
>
> The SAM was generated from the Bowtie2, and appended with a few more tags
> defined by our own, such as sequence of UMI (unique molecular index). The SAM
> file is ~4Gb in size. The samtools we're using is version 1.3.1.
Samtools 1.3.1 is very out of date. Do you see the same problem if you
use the latest release (1.6)? If nothing else, it may be better at
telling you that something went wrong.
> When we tried to run the above command in a VM with 32Gb RAM, it failed
> silently, and we couldn't see the BAM file generated. After we increased to
> 64Gb RAM, the BAM was generated.
>
> Wonder why increasing the RAM could help the conversion of SAM to BAM (64Gb
> is really big compared to the SAM size 4Gb)? Is there any option that we may
> use to print the logs for easier troubleshooting?
samtools view shouldn't need anywhere near this much memory. I suspect
something in your file is triggering a bug that caused it to allocate
much more memory than it should have, but it's difficult to tell without
having the actual file. A number of these bugs have been fixed since
version 1.3.1, so it would be very useful to know what happens with the
latest release. You could also try using `head` to limit the number of
SAM file lines passed to `samtools view`. If you get no output at all
then it's very likely that the failure happens when reading the header,
so you may be able to reproduce the problem with a much smaller file.
Please let us know how you get on,
Rob Davies r...@sanger.ac.uk
The Sanger Institute http://www.sanger.ac.uk/
[http://www.sanger.ac.uk/sites/default/files/logo-front-wgc.png]<http://www.sanger.ac.uk/>
Home Page | Wellcome Sanger Institute<http://www.sanger.ac.uk/>
www.sanger.ac.uk
Wellcome Trust Genome Campus, Hinxton
Hinxton, Cambs., Tel. +44 (1223) 834244
CB10 1SA, U.K. Fax. +44 (1223) 494919
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Samtools-help mailing list
Samtools-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/samtools-help
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Samtools-help mailing list
Samtools-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/samtools-help
