SAMtools, BCFtools and HTSlib version 1.24 are now available from GitHub and SourceForge. The main changes are listed below.
The HTSlib release includes fixes for crashes and other undefined behaviour that could happen when reading malformed CRAM or BCF files, or when reading invalid fasta or fastq index files. These fixes have also been back-ported to HTSlib versions 1.21.2, 1.22.3, and 1.23.2. Due to the risk that these bugs could be exploited to take control of programs reading such files, we strongly recommend upgrading to one of the patched versions. https://github.com/samtools/htslib/releases/tag/1.24 https://github.com/samtools/samtools/releases/tag/1.24 https://github.com/samtools/bcftools/releases/tag/1.24 https://sourceforge.net/projects/samtools/ ------------------------------------------------------------------------------ htslib - changes v1.24 ------------------------------------------------------------------------------ * NEW. Contribution guidelines, Including sections on complexity, completeness, signing and our AI policy. (PR #2003) Updates ------- * Remove CRAM v4.0 support. The experimental CRAM v4 code was large, complex and insufficiently tested. With no real move toward v4 adoption we have removed the code in order to reduce the likelihood of security issues. (PR #2020 and PR #2031. Reported by Trail of Bits and Anthropic: https://github.com/samtools/htslib/security/advisories/GHSA-6q7c-m967-9v37 Reported by Team Atlanta: https://github.com/samtools/htslib/security/advisories/GHSA-hg3g-v57p-459q https://github.com/samtools/htslib/security/advisories/GHSA-rjqv-xg3q-g423) * Ensure indirect function calls have the correct type. Some HTSlib interfaces have function callbacks that are intended to be generic, so the function signatures include a void * for data to be passed in. While this mostly works it is strictly undefined behaviour. Some new wrapper functions and interfaces are added to address this. (PR #1994) * Add wrappers for malloc, realloc with a calloc-like interface. This helps avoid bugs due to integer wrap-around when calculating memory sizes. (PR #2006) * Make faidx work with very long (>4 Gbyte!) lines. Although faidx should support very long references, writing one longer than 4Gbases on a single line broke it because it used a uint32_t field to store the line length. (PR #2008. Fixes samtools/samtools#2331. Reported by Ying Chen) * Parallel cram2bam. Move more of the work into worker threads to speed everything up. (PR #2015) * Improve synced reader error checking. Ensure memory failures are caught and that error are propagated to callers. (PR #2024) * Add tbx and bcf multi-region iterators. (PR #2030, fixes #1930. Requested by Adam Novak. Replaces the alternative PRs #1997 and #2022) * Remove references to the non-existent FAI_CACHE. (PR #2033, fixes #2032. Reported by John Marshall) * The underlying causes of errors when accessing files in S3 buckets are now reported more accurately. (PR #2036. Thanks to John Marshall) * In VCF, improve the "not defined in the header" messages. (PR #2007) Build Changes ------------- * Add Github actions builds for Linux and Mac OS. This replaces the Cirrus CI tests that were discontinued. (PR #2000) * Ensure PACKAGE_VERSION is set in the Makefile. (PR #2038. See also samtools/samtools#2337) Bug fixes --------- * Better enforce QNAME length restrictions in the CRAM reader. HTSlib limits the length of QNAMEs (and mate QNAMEs) to 254 characters to match the maximum allowed by the BAM format. To avoid problems with handling very long names, enforcement of this limit has been moved to an earlier stage in the CRAM decoder. (PR #2046. Reported by Claude and Ada Logics. https://github.com/samtools/htslib/security/advisories/GHSA-2jx2-wm7f-rv9m) * Bounds-check refid when loading a CRAM .crai index. (PR #2029. Thanks to Sidhartha Kumar. https://github.com/samtools/htslib/security/advisories/GHSA-28r7-prwc-hf5c) * Improve checks of record consitency in the BCF reader, and disallow excessively large FORMAT entries. (PR #2047. Reported by VulnSeeker Security Research. https://github.com/samtools/htslib/security/advisories/GHSA-2wm6-8hgm-7g92) * Improve validation of faidx index entries. (PR #2008. https://github.com/samtools/htslib/security/advisories/GHSA-4hjq-r829-8c8v) * Several fixes co-authored by Team Atlanta. - Possible integer overflow when checking input size for the beta codec. - Oversized shift in cram_subexp_decode. (PR #2025, #2046. Thanks to Team Atlanta. https://github.com/samtools/htslib/security/advisories/GHSA-rjqv-xg3q-g423) * Fix s3_seek returning wrong offset on cache-hit. This caused unreliability reading of S3 data when using the multi-region iterator. The bug affects 1.23 and 1.23.1. Releases prior to 1.23 were unaffected. (PR #2012. Thanks to Nick Edwards. Also fixes #2043 reported by Andrew J. Tock.) * Protect against uninitialised variable read with X_NOSZ codecs. (PR #2023. Fixes #2021. Reported by Jiami Lin) * Fix a possible crash when using nonsensical embed_ref=2,no_ref CRAM encode options. (PR #2048) * Fix a read buffer overflow and improve SQ LN field checking. The buffer overflow was in development code only and never in a release. (PR #1999. Fixes oss-fuzz issue 499447432) * Small updates to fix misplaced NULL checks, remove unused variables, and ensure the package builds without warnings on newer compilers. (PR #1992, PR #1993, PR #2004, PR #2010) * Remove a signed overflow bug in bgzf_read_small. (PR #2013) * In bgzip and tabix, when handling errors, don't try to access the BGZF::errcode field after calling bgzf_close() as it may no longer be valid. (PR #2035, #2042) * Speed up hts_parse_decimal() handling of oversized exponents (PR #2045) * Fix some memory leaks on failing to build or load an index (PR #2049) Documentation updates --------------------- * Clarify how defaults work when building an index with tabix. (PR #2002, addresses #1995. Query made by Dario Beraldi) ------------------------------------------------------------------------------ samtools - changes v1.24 ------------------------------------------------------------------------------ New work and changes: * NEW. Contribution guidelines, Including sections on complexity, completeness, signing and our AI policy. (PR #2335) * CHANGE. The default seed for `samtools view --subsample` is now based on the input file header and not zero. This means that repeated subsampling of the same file will work as expected. The previous behaviour can be replicated by setting the seed to zero. (PR #2320, fixes #1201. Thanks to Tim Fennell. Reported by Stathis) * Add --move-umi-to-tag option to `samtools markdup` to extract UMI from QNAME and add to RX tag. (PR #2316, fixes #2181. Thanks to Matthias De Smet) * Ensure indirect function calls have the correct type. This uses the new HTSlib interface introduced in samtools/htslib#1994. (PR #2322) * Changes to make `checksum` merging compatible with biobambam2 checksums. (PR #2329, fixes #2323. Reported by Keith James) * Speed improvement for `samtools stats`. (PR #2326) * Changed --customized-index-file to --customized-index in `samtools stats` to match other subcommands. As this option had never worked (see #2315) this should not cause any compatibility problems. (PR #2334, fixes #2333. Thanks to Alexis Lucattini) * Fill in mate information on supplementary alignments in `samtools fixmate`. Takes mate information from the primary and adds it to the supplementary alignment. Does not affect secondary alignments. (PR #2347. Thanks to Nils Homer) Bug fixes: * Various option fixes. Removed the mistaken required argument from --customized-index-file in `stats`. Also removed some unused and undocumented options from bam_plcmd.c. (PR #2317, fixes #2315. Reported by Alexis Lucattini) * For `samtools stats` check before and after region with -target-regions. Previously doing only one or the other. (PR #2328, fixes #2303. Reported by Yuan Tian) Non user-visible changes and build improvements: * Add Github actions builds for Linux and Mac OS. This replaces the Cirrus CI tests that were discontinued. (PR #2330) * Ensure PACKAGE_VERSION is set in the Makefile. (PR #2342, fixes #2337. Reported by James Ferguson) * Make version.sh work when .git is present but the git executable is not. (PR #2345) * Adjust tail commands so they work on Solaris. (PR #2346) ------------------------------------------------------------------------------ bcftools - changes v1.24 ------------------------------------------------------------------------------ Changes affecting the whole of bcftools, or multiple commands: * Calculation of MAC and MAF in `-i/-e` filtering expressions - Minor allele count and minor allele frequency were calculated incorrectly at multiallelic sites. Changes affecting specific commands: * bcftools +af-dist - New `-s, --samples` to print per-sample HWE probability, geometric mean * bcftools annotate - Fix a bug in annotation renaming via `--rename-annots` and `-c NEW:=OLD`, where renaming INFO/TAG would simultaneously rename also FORMAT/TAG (#2489) - Allow annotation of Number=A,R INFO tags from tab-delimited files when alleles are split across multiple records. * bcftools cnv - Fix off-by-one error in reporting HMM qualities * bcftools convert - Make --gvcf2vcf work around malformed gVCF records where INFO/END overlaps the next record * bcftools csq - Fix a start_lost issue in haplotype mode: suppress formatting of amino-acid changes of variants that follow on the same haplotype, but still print DNA changes (#2543) - Add a new -G, --greedy option which forces the program to check intronic consequences after feature hits (#2548) * bcftools +fill-tags - Fix custom functions to use rounding instead of truncation when converting to integer, so that -t 'MAC:1=float(MAF*AN)' and -t 'MAC:1=int(MAF*AN)' return consistent values. - Support per-population annotation for embedded functions, such as in -t 'MAC:1=int(MAF*AN)' -S groups.txt (#2514) * bcftools +mendelian2 - Number of fixes and improvements, including a fix in assigning 1X vs 2X inheritance rules with -p (#2545) * bcftools merge - Fixes in memory access with localized alleles, it could result in printing printing trailing garbage in localized FORMAT fields (#2520) - Add new `-m exact` mode to merge only records with the exact same alleles (#2498) * bcftools mpileup - Remove unused experimental INFO/MIN_PL_SUM annotation - Add new FORMAT/QM annotation, to be used with the new `bcftools +trio-dnm3 --use-ALM` model. * bcftools norm - Fix a bug in splitting Type=String FORMAT fields via -a and -m (#2476) - Fix erroneous INFO/END recalculation of symbolic alleles (#2500). - Added `--no-realign[=NUM]` to skip realignment during normalization; when NUM is given, only events longer than NUM bp are skipped. This deprecates the option `--do-not-normalize` (equivalent to `--no-realign=0`). See also #2500. - Convert ambiguous IUPAC bases to their lexicographically first concrete base, as required by the VCF specification (#2551) * bcftools +prune - When pruning by window length in base pairs, one out-of-window record would be incorrectly accessed. Not only this by-one error would calculate LD in unwanted pair of records, it also led to unexpected behavior when the first out-of-window record would change by filtering (#2475) * bcftools roh - Fix off-by-one error in reporting HMM qualities * bcftools +split-vep - Add support for SnpEff output, add a new option --csq-field, --consequence-field which allows to override the default consequence field name 'Consequence' (#2566) - Stop converting consequence severity to lower case * bcftools stats - Fix per-sample stats when samples provided in different order via -s/-S (#2534) * bcftools +tag2tag - Make the advertised default for `-t, --threshold` match the implementation: 0, not 0.1. Also clarify that the value is the maximum allowed GP error (`call if GP >= 1 - FLOAT`), not a minimum GP threshold (#2564) * bcftools +trio-dnm3 - Add a new TrioDNM model and make it the new default. To prevent confusion, the old +trio-dnm2 plugin was removed and replaced with +trio-dnm3. The original +trio-dnm3 model can be run as bcftools +trio-dnm3 --use-ALM * bcftools view - Fix the -v/V, --types/--exclude-types option to work with ref and extend for all types supported by htslib: snps,indels,mnps,ref,bnd,other,overlap (#2567) ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA. _______________________________________________ Samtools-help mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/samtools-help
