SAMtools, BCFtools and HTSlib version 1.25 are now available from GitHub and SourceForge. The main changes are listed below.
The HTSlib release includes fixes for crashes and other undefined behaviour that could happen when reading malformed VCF, BCF and index files (for any format). It also fixes crashes in the HTSlib preload:// URL handler, and in the http authentication token refresh code. These fixes have also been back-ported to HTSlib versions 1.21.3, 1.22.4, 1.23.3, and 1.24.1. Due to the risk that these bugs could be exploited to take control of programs reading such files, we strongly recommend upgrading to one of the patched versions. https://github.com/samtools/htslib/releases/tag/1.25 https://github.com/samtools/samtools/releases/tag/1.25 https://github.com/samtools/bcftools/releases/tag/1.25 https://sourceforge.net/projects/samtools/ ------------------------------------------------------------------------------ htslib - changes v1.25 ------------------------------------------------------------------------------ * NEW. Added environment variable HTS_GCS_HOST, analogous to the existing HTS_S3_HOST, to redirect gs:// URLs to a caller-specified endpoint. (PR #2051. Thanks to Peter Dowdy) Updates ------- * Various code quality improvements. Added memory and error checks. Some changes to better data types. (PR #2054, PR #2059, PR #2064, PR #2066, PR #2070, PR #2073, PR #2075, PR #2077, PR #2078) * Add a new format option to strip UR tags from CRAM. (PR #2056, partial fix for samtools/samtools#2327. Requested by Florian Plaza Oñate) * Report when khash can no longer expand. (PR #2063) * Use multi iterators in the synced reader. This adds bcf_itr_regions() and tbx_itr_regions() interfaces as BCF and tabix equivalents to sam_itr_regions(). This can speed up `bcftools view`. (PR #2052, fixes samtools/bcftools#2557. Reported by Carsten Erickson) * Apply CURL_CA_BUNDLE environment variable in the S3 backend. (PR #2069. Thanks to John Marshall) * Cram embed_ref=2 and huffman optimisation. (PR #2082) * API fix. Protect against out of bounds ref id in cram_put_bam_seq. (PR #2087) * Adds validation to options specified using hts_opt_add or cram_set_voption. Mostly for overflow detection. (PR #2095. Reported by Sn0wyDay) * Return an error instead of calling abort in hts_getline(). (PR #2093, fixes #2091. Reported by an) * bcf_set_variant_type() now recognises single break-ends as described in section 5.4.9 of the VCF 4.5 specification. (PR #2115, fixes #2108) Build Changes ------------- * Add more CI tests for a variety of Linux and non-Linux platforms. (PR #2053) * Update build badge, remove .cirrus.yml, spelling corrections. (PR #2071. Thanks to John Marshall) * Make check_copyright.pl better at skipping auto-generated files. (PR #2080) * Use bash and pipefail for tests using simple_test_driver.sh (PR #2083) * Add S3 integration tests. (PR #2067. Thanks to Peter Dowdy) Bug fixes --------- * Fix a use-after-free bug when renewing http authentication tokens. This could happen if the file used to supply a replacement token was not present at renewal time. (PR #2119 Reported by Xuqing Yang. https://github.com/samtools/htslib/security/advisories/GHSA-x79c-vxf6-6j9c) * Fix use-after-free and double free when handling file close errors in the preload:// file reader. (PR #2119 Reported by Xuqing Yang. https://github.com/samtools/htslib/security/advisories/GHSA-5cvj-m89h-h3cw) * Fix inconsistent region list clean-up on hts_itr_regions() failure. This function takes ownership of the region list passed to it, even if it fails due to out-of-memory conditions. Not accounting for this correctly led to a possible double free in the functions used to build multi-region iterators. (PR #2119. Reported by Xuqing Yang. https://github.com/samtools/htslib/security/advisories/GHSA-2r8w-wxh5-c7r8) * Fix a double free on out-of-memory conditions in the CRAM BYTE_ARRAY_LEN encoder. (PR #2119. Reported by Xuqing Yang. https://github.com/samtools/htslib/security/advisories/GHSA-2r8w-wxh5-c7r8) * Fix possible free of an uninitialised pointer if read errors occurred when loading .bai, .csi, or .tbx indexes. (PR #2119. Reported by maxmlz. https://github.com/samtools/htslib/security/advisories/GHSA-38qp-vpxq-r6xc) * Properly validate index metadata before attempting to access it. (PR #2119. https://github.com/samtools/htslib/security/advisories/GHSA-vhg8-qwm7-cv9m) * Fix use-after-free in bcf_sr_destroy1() if it tries to report an error on file close. (PR #2119. Reported by Junki Lee. https://github.com/samtools/htslib/security/advisories/GHSA-644m-mq4p-c7c6) * Reject VCF records containing ASCII NUL bytes in FORMAT data. (PR #2119. Reported by haehet. https://github.com/samtools/htslib/security/advisories/GHSA-6qq4-hj39-g6ph) * Validate tid value passed to hts_idx_get_stat() to prevent out-of-bounds reads. (PR #2119. Reported by haehet. https://github.com/samtools/htslib/security/advisories/GHSA-vhg8-qwm7-cv9m) * Fix an undefined behaviour in os/rand.c (PR #2050) * Fix heapsort bug and add ksort tests and improved documentation in comments. (PR #2055) * Fix possible leak of bgzidx_t::offs and changes to bgzidx_t::noffs and bgzidx_t::moffs data types. (PR #2057) * Remove an assert in cram_index_build and have it fail more gracefully. (PR #2061, fixes #2060. Reported and co-authored by Shujie Xiang) * Remove an infinite loop in bam_mods_at_qpos(). (PR #2068) * Avoid calling libcurl functions if curl_global_init() failed. (Part of PR #2069. Thanks to John Marshall) * Make sam_format_aux1 'A' type be nul terminated. (PR #2079) * Fix multiple VCF header parsing bugs. (PR #2074, fixes #2065 Reported by Mark Woon) * Prevent integer overflow in resolve_cigar2(). (PR #2072) * Improved code testing with the addition of a fuzzer for the expression filters (hts_expr). This resulted in the detection and correction of several serious expression parsing bugs. (PR #2076) * Fix leak of region list in hts_itr_multi_bam(). (PR #2084) * Improve checking of SEQ/CIGAR consistency when processing records with CG tags. (PR # 2089. Fixes oss-fuzz issue 560967311) * Stop the CRAM zlib decoder from getting stuck in an endless loop. (PR #2085, fixes #2081. Reported by Sewon Ahn) * Fix hts_set_filter_expression(fp, NULL) so that it properly disables filtering without leaving a dangling pointer. (PR #2094. Reported and code fix suggestion by Sewon Ahn) * Tighten version number limits in bcf_get_version() (PR #2092, fixes #2088. Reported by an) * Fix CRAM query of unaligned ('*') reads. (PR #2100, fixes samtools/samtools#2396. Reported by Matt Sexton) * Fix hfile_s3 reads failing at EOF on a part boundary. (PR #2098, fixes #2097. Thanks to Peter Dowdy) * Reduce CRAM decode memory usage when multi-threading. Previous improvements to threaded throughput added greater memory usage, especially with long reads. This change mitigates that. (PR #2102, fixes samtools/samtools#2393. Reported by Matthieu Muffato) * Prevent negative range start in multi-region iterators, which could in some cases cause the index look-up to become stuck in an endless loop. (PR #2110. Reported by haehet) * Fix SQ line removal using sam_hdr_remove_lines(). It failed to update some internal sam_hdr_t structures, leading to deleted SQ lines reappearing, notably when writing BAM files. (PR #2105) * Fix the wildly incorrect length value returned by hfile_mem_get_buffer() and hfile_mem_steal_buffer(). (PR #2109) * Fix a number of bugs that could be triggered by removing all samples with bcf_hdr_set_samples(). (PR #2114) * Fix fai_load3() crash if a .fai index was present but it attempted a rebuild because it also needed a .gzi one, and the rebuild failed. (PR #2116) ------------------------------------------------------------------------------ samtools - changes v1.25 ------------------------------------------------------------------------------ New work and changes: * CHANGE. `samtools checksum` reported version set to 2.0. Checksum calculations differed between samtools 1.24 and previous releases. In earlier releases, the --check-mate option produced incorrect checksums. This was fixed in v1.24 but made already existing checksums invalid. Therefore, the fixed checksum is now designated Checksum 2.0 while the previous one is left at Checksum 1.0 in order differentiate the versions. N.B.: Samtools version 1.24 produced corrected checksums but erroneously labelled them as version 1.0. If you used this release with the --check-mate option then we advise recomputing or manually editing the version numbers in the output files. (PR #2365, fixes #2354. Reported by Ruben Vorderman) * CHANGE. When the HTSlib version in use cannot be deduced from the SAMtools version, the `@PG` `VN` field now records it too, for example `VN:1.24.1 (with HTSlib 1.25)`. (PR #2350. Thanks to Benjamin Demaille) * In `mpileup`, added an ENDPOS field to --output-extra. Combined with POS, this gives the left and right extents on the reference for this alignment, meaning we can observe how close to either end this pileup base is. (PR #2351, fixes #2343. Requested by Devang Thakkar) * Strip UR tags from header SQ lines in `samtools view`. (PR #2357, fixes #2327. Requested by Florian Plaza Oñate) * Make `samtools view` --unmap and --unoutput work together. (PR #2358, fixes #2356. Requested by Eric Boyden) * Make `samtools import` accept empty FASTQ input. This avoids halting a pipeline when one file has no data. (PR #2395. Thanks to Clint Valentine) * `samtools stats` changed to use hts_crc32() instead of zlib's crc32() to take advantage of libdeflate (if used). Plus some minor fixes. (PR #2406, thanks to John Marshall) Documentation: * Update values in embed_ref and multi_seq_per_slice options. (PR #2386) Bug fixes: * Fix reuse-after-free in bam_plcmd ref cache. (PR #2364, fixes #2363. Reported by James Smagala. https://github.com/samtools/samtools/security/advisories/GHSA-xj7m-w8mc-xvf2 https://github.com/samtools/samtools/security/advisories/GHSA-vh3p-9c59-j3hv) * Check string aux tag types before accessing data. (PR #2366 https://github.com/samtools/samtools/security/advisories/GHSA-3j6q-r48h-wvm3) * Fix `samtools fixmates` `-M` and `-z cigar` options. Fixing base modifications failed on reads without base modifications tags. Trimming CIGAR strings to the length of the reference sequence failed on strings containing 0M. (PR #2407 https://github.com/samtools/samtools/security/advisories/GHSA-f423-v554-fwqr) * In `samtools stats` grow the coverage ring buffer instead of wrapping around. (PR #2379, fixes #2378. Thanks to Cynthia Krafft) * Fix --move-umi-to-tag in `samtools markdup`. (PR #2382, fixes #2380. Reported by mrubio-chavarria) * Relax the --barcode-name regex in `samtools markdup`. (PR #2387, fixes #2385. Reported by fjvorhoelter) * Make `phase` and `targetcut` work with records where seq is '*'. (PR #2393, fixes #2390. Reported by babayaga) * Check the number of temporary files is above 0 in `samtools collate`. (PR #2394. Thanks to Pierre Lindenbaum) * Fix optical duplicates being set in `markdup` even when read name prefixes differ. (PR #2400, fixes #2399. Thanks to Kris Alavattam) * Fix `samtools cat` exit code when unable to use CRAM file. cram_cat() now distinguishes between failure and EOF. (PR #2403, fixes #2401. Reported by Matthieu Muffato) * FIX. `samtools stats` counted secondary and supplementary records in *reads duplicated* and *bases duplicated*, while *sequences* and *total length* exclude them. Non-primary duplicates are now not included in the counts. Also, `samtools stats` counted pairs with insert size 0 (TLEN unknown) in the denominator of *insert size average* and *insert size standard deviation*, and in the mean but not in the sum of squares of the standard deviation. They are now left out of both statistics and of the `-m` main-bulk cut-off. (PR #2388. Addresses a question in #696. Thanks to Cindy Krafft) * Fix `samtools calmd` when correcting MD tags that aren't aux type `Z`. (PR #2408) Non user-visible changes and build improvements: * Add more CI targets (various Linux releases and other operating systems). Fixed warnings that occurred on some platforms. (PR #2353) * Add --norc --noprofile to bash invocations in test.pl. (PR #2362) * Remove unnecessary strdup(optarg) calls in CLI parsing. (PR #2361) * Fix memory leaks when samtools view CLI bails. Minimal impact but helps with enabling memory leak detection in fuzzing. (PR #2372) * Various error checking and reporting changes. (PR #2368, PR #2369, PR #2370, PR #2371, PR #2373) * Switch containers to ones built by samtools/c-maint. This reduces the dependence on upstream repositories. (PR #2381) ------------------------------------------------------------------------------ bcftools - changes v1.25 ------------------------------------------------------------------------------ Changes affecting the whole of bcftools, or multiple commands: * New option `--progress[=INT]` in `mpileup`, `norm` and `view` to periodically print the current position (CHR:POS) to stderr, indicating how far a long-running command has progressed (#2559) * Fix allele merging in `norm` and `merge` to preserve symbolic IDs and breakend contig names and respect their case during matching, as required by VCF specification. * The `-i/-e` filtering expressions - Fix negated regular-expression (!~) filtering of string fields, which could incorrectly evaluate to true without testing the pattern (#2609) Changes affecting specific commands: * bcftools call - To comply with the VCF specification, the INFO/MQ output tag is now declared as Type=Float rather than Type=Integer * bcftools csq - New option `--allow` to selectively relax validation checks (#2584) - New option `-F, --format` to extend the consequence annotation with cds_position (position within the coding sequence and CDS length), exon (exon index and the number of exons), and exon_position (position within the exon and the exon length) - Fix a segfault caused by incorrect handling of symbolic ALT alleles. Support symbolic alleles at multiallelic sites. * bcftools consensus - Fix -M masking of symbolic deletions and mixed multiallelic sites, including out-of-bounds accesses with overlapping variants and at FASTA region boundaries. - Fix REF span handling when applying sequence alleles at sites where INFO/END describes a longer symbolic allele. * bcftools +fill-tags - Add two tags to annotate variant sequence context: INFO/CTX .. k-nt reference context centered at POS INFO/CpG .. CpG change relative to REF considering all ALTs; C=created, D=destroyed, B=both; N=CpG present and unchanged - Fix a bug where any sample of one or two characters is rejected (#1931, #1933) * bcftools index - Make `--nrecords` with non-standard file extensions (#2588) * bcftools +mendelian - Support sites with more than 64 alleles (#2603) * bcftools mpileup - Fix INFO/MQBZ, BQBZ, RPBZ, SCBZ, MQSBZ and NMBZ at sites where more than 1290 reads share one quality or position bin, reachable at a single deep site or across the samples of a multi-sample call. The Mann-Whitney U tie correction was accumulated in 32-bit int and overflowed, which shrank the Z scores towards zero. * bcftools norm - Left-align <DUP> allele no matter which of INFO/SVLEN or INFO/END is present - Skip records whose alleles cannot safely be realigned together (#2605) - Realign long indels in linear time (#2604) - Recalculate INFO/AN,AC from genotype fields unless the new option `--no-update` is given (#2137, #548). - Extend the `-c, --check-ref` option for explicit control of output case, report the number of records changed (#2606) - Make `-d exact` consistent for sequence alleles differing only in case. - Fix incorrect allele mappings and associated annotations when atomizing (`-a`) multiallelic records after a compound variant (#2607) * bcftools reheader - Better error reporting on invalid --samples file, don't segfault (#2538) * bcftools sort - Add --threads and, in --verbose 4 mode or higher, print elapsed time ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA. _______________________________________________ Samtools-help mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/samtools-help
