Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Hi Aleksandar, thanks a lot for your explanation of the patch! That's really appreciated and your help is really needed here. I think we whould wait until upstream has updated their sources before we upload to make sure they are happy with the patch. (Any member of Debian Med team feel free to override my suggestion!) Thanks again Andreas. On Mon, Jul 07, 2014 at 01:11:48PM +, Aleksandar Zlicic wrote: Hi, Dominique System endianness issue refers to a bug in function swap_endian_data (bam.c:160). Function swap_endian_data() is used on big-endian architectures when data is read from input BAM files and/or written to output BAM files, for converting multibyte integer values from little endian to big endian byte order and vice versa (depending on the operation). Additional argument to the function is_host specifies whether the input data is in host byte order or not (reading or writing), since swapping of BAM auxiliary array length values has to be done at different times for reading and writing. -static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data) +static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data, int is_host) +if(!is_host) bam_swap_endian_4p(s+1); -bam_swap_endian_4p(s+1); +if(is_host) bam_swap_endian_4p(s+1); +s += n * Bsize + 4; The last line in this change updates pointer 's' so that it points to the next uxiliary data tag-value pair (there are 'n' elements, each element of the array is Bsize bytes long; additional 4 bytes are for the 4-byte integer where length of array is stored). The following fixes 'for' loop limits (there are 'n' elements and each element is Bsize bytes long): -for (i = 0; i n; i += 2) +for (i = 0; i n*Bsize; i += 2) -for (i = 0; i n; i += 4) +for (i = 0; i n*Bsize; i += 4) Changes in fix_alignment.patch are to avoid unaligned memory access for architectures that don't allow it. I am currently adapting changes to the latest version of samtools. When I'm done I will contact upstream. Best Regards Aleksandar Zlicic ___ Debian-med-packaging mailing list debian-med-packag...@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-packaging -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Hi, Dominique System endianness issue refers to a bug in function swap_endian_data (bam.c:160). Function swap_endian_data() is used on big-endian architectures when data is read from input BAM files and/or written to output BAM files, for converting multibyte integer values from little endian to big endian byte order and vice versa (depending on the operation). Additional argument to the function is_host specifies whether the input data is in host byte order or not (reading or writing), since swapping of BAM auxiliary array length values has to be done at different times for reading and writing. -static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data) +static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data, int is_host) +if(!is_host) bam_swap_endian_4p(s+1); -bam_swap_endian_4p(s+1); +if(is_host) bam_swap_endian_4p(s+1); +s += n * Bsize + 4; The last line in this change updates pointer 's' so that it points to the next uxiliary data tag-value pair (there are 'n' elements, each element of the array is Bsize bytes long; additional 4 bytes are for the 4-byte integer where length of array is stored). The following fixes 'for' loop limits (there are 'n' elements and each element is Bsize bytes long): -for (i = 0; i n; i += 2) +for (i = 0; i n*Bsize; i += 2) -for (i = 0; i n; i += 4) +for (i = 0; i n*Bsize; i += 4) Changes in fix_alignment.patch are to avoid unaligned memory access for architectures that don't allow it. I am currently adapting changes to the latest version of samtools. When I'm done I will contact upstream. Best Regards Aleksandar Zlicic
Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Package: samtools
Version: 0.1.19-1
Tags: sid patch
Severity: important
User: debian-mips-dev-disc...@lists.alioth.debian.org
Usertags: mips-patch
'python-pysam' package FTBFS on mips/mipsel due to issues in 'samtools' package.
Build log for 'python-pysam' on mips:
https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipsver=0.7.7-1stamp=1397894446
Build log for 'python-pysam' on mipsel:
https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipselver=0.7.7-1stamp=1397887877
Output of samtools executable from samtools package is used as a reference
during tests in python-pysam (its output is compared with output from
pysam python module), but samtools has both unaligned memory access
and system endianness issues.
I have attached patches that fix this. With these patches applied to samtools,
python-pysam builds fine.
Best Regards
Aleksandar Zlicic
Index: samtools-0.1.19/bam_import.c
===
--- samtools-0.1.19.orig/bam_import.c
+++ samtools-0.1.19/bam_import.c
@@ -21,6 +21,10 @@ void bam_init_header_hash(bam_header_t *
void bam_destroy_header_hash(bam_header_t *header);
int32_t bam_get_tid(const bam_header_t *header, const char *seq_name);
+#if defined(__i386__) || defined(__i386) || defined(__amd64__) || defined(__amd64) || defined(__x86_64__) || defined(__x86_64) || defined(__i686__) || defined(__i686)
+# define ALLOW_UAC
+#endif
+
unsigned char bam_nt16_table[256] = {
15,15,15,15, 15,15,15,15, 15,15,15,15, 15,15,15,15,
15,15,15,15, 15,15,15,15, 15,15,15,15, 15,15,15,15,
@@ -365,6 +369,10 @@ int sam_read1(tamFile fp, bam_header_t *
if (dret != '\n' dret != '\r') { // aux
while (ks_getuntil(ks, KS_SEP_TAB, str, dret) = 0) {
uint8_t *s, type, key[2];
+#ifndef ALLOW_UAC
+ uint8_t tmpData[8];
+ int j;
+#endif
z += str-l + 1;
if (str-l 6 || str-s[2] != ':' || str-s[4] != ':')
parse_error(fp-n_lines, missing colon in auxiliary data);
@@ -410,14 +418,30 @@ int sam_read1(tamFile fp, bam_header_t *
}
}
} else if (type == 'f') {
+#ifndef ALLOW_UAC
+float *ptmpData = (float*)tmpData;
+#endif
s = alloc_data(b, doff + 5) + doff;
*s++ = 'f';
+#ifdef ALLOW_UAC
*(float*)s = (float)atof(str-s + 5);
+#else
+*ptmpData = (float)atof(str-s + 5);
+for(j=0;j4;j++) s[j] = tmpData[j];
+#endif
s += 4; doff += 5;
} else if (type == 'd') {
+#ifndef ALLOW_UAC
+float *ptmpData = (float*)tmpData;
+#endif
s = alloc_data(b, doff + 9) + doff;
*s++ = 'd';
+#ifdef ALLOW_UAC
*(float*)s = (float)atof(str-s + 9);
+#else
+*ptmpData = (float)atof(str-s + 9);
+for(j=0;j4;j++) s[j] = tmpData[j];
+#endif
s += 8; doff += 9;
} else if (type == 'Z' || type == 'H') {
int size = 1 + (str-l - 5) + 1;
Index: samtools-0.1.19/bam.c
===
--- samtools-0.1.19.orig/bam.c
+++ samtools-0.1.19/bam.c
@@ -157,7 +157,7 @@ int bam_header_write(bamFile fp, const b
return 0;
}
-static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data)
+static void swap_endian_data(const bam1_core_t *c, int data_len, uint8_t *data, int is_host)
{
uint8_t *s;
uint32_t i, *cigar = (uint32_t*)(data + c-l_qname);
@@ -174,16 +174,18 @@ static void swap_endian_data(const bam1_
else if (type == 'Z' || type == 'H') { while (*s) ++s; ++s; }
else if (type == 'B') {
int32_t n, Bsize = bam_aux_type2size(*s);
+ if(!is_host) bam_swap_endian_4p(s+1);
memcpy(n, s + 1, 4);
if (1 == Bsize) {
} else if (2 == Bsize) {
-for (i = 0; i n; i += 2)
+for (i = 0; i n*Bsize; i += 2)
bam_swap_endian_2p(s + 5 + i);
} else if (4 == Bsize) {
-for (i = 0; i n; i += 4)
+for (i = 0; i n*Bsize; i += 4)
bam_swap_endian_4p(s + 5 + i);
}
- bam_swap_endian_4p(s+1);
+ if(is_host) bam_swap_endian_4p(s+1);
+ s += n * Bsize + 4;
}
}
}
@@ -217,7 +219,7 @@ int bam_read1(bamFile fp, bam1_t *b)
}
if (bam_read(fp, b-data, b-data_len) != b-data_len) return -4;
b-l_aux = b-data_len - c-n_cigar * 4 - c-l_qname - c-l_qseq - (c-l_qseq+1)/2;
- if (bam_is_be) swap_endian_data(c, b-data_len, b-data);
+ if (bam_is_be) swap_endian_data(c, b-data_len, b-data, 0);
if (bam_no_B) bam_remove_B(b);
return 4 + block_len;
}
@@ -240,11 +242,11 @@ inline int bam_write1_core(bamFile fp, c
for (i = 0; i 8; ++i) bam_swap_endian_4p(x + i);
y = block_len;
bam_write(fp, bam_swap_endian_4p(y), 4);
- swap_endian_data(c, data_len, data);
+ swap_endian_data(c, data_len, data, 1);
} else bam_write(fp, block_len, 4);
bam_write(fp, x, BAM_CORE_SIZE);
bam_write(fp, data, data_len);
- if (bam_is_be) swap_endian_data(c, data_len, data);
+ if (bam_is_be) swap_endian_data(c, data_len, data, 0);
return 4 + block_len;
}
Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Hi Aleksandar, Thanks for the patches. Could you please explain what the issue is and how the patch fixes it? I don't see any comments in the patch. Does upstream accept your patch? Thanks -Dominique
Bug#753485: [Debian-med-packaging] Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Le Wed, Jul 02, 2014 at 12:30:24PM +, Aleksandar Zlicic a écrit : Package: samtools Version: 0.1.19-1 Tags: sid patch Severity: important User: debian-mips-dev-disc...@lists.alioth.debian.org Usertags: mips-patch 'python-pysam' package FTBFS on mips/mipsel due to issues in 'samtools' package. Build log for 'python-pysam' on mips: https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipsver=0.7.7-1stamp=1397894446 Build log for 'python-pysam' on mipsel: https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipselver=0.7.7-1stamp=1397887877 Output of samtools executable from samtools package is used as a reference during tests in python-pysam (its output is compared with output from pysam python module), but samtools has both unaligned memory access and system endianness issues. I have attached patches that fix this. With these patches applied to samtools, python-pysam builds fine. Hi Aleksandar, I admit that I am a bit worried of applying a patch that I am not able to review. Could you ask upstream (https://github.com/samtools/samtools) to have a look ? Have a nice day, -- Charles Plessy Debian Med packaging team, http://www.debian.org/devel/debian-med Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#753485: [Debian-med-packaging] Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools
Ok, I will contact upstream. Best Regards Aleksandar Zlicic From: Charles Plessy [ple...@debian.org] Sent: Wednesday, July 02, 2014 3:16 PM To: Aleksandar Zlicic; 753...@bugs.debian.org Subject: Re: [Debian-med-packaging] Bug#753485: samtools: python-pysam FTBFS on mips/mipsel due to issues in samtools Le Wed, Jul 02, 2014 at 12:30:24PM +, Aleksandar Zlicic a écrit : Package: samtools Version: 0.1.19-1 Tags: sid patch Severity: important User: debian-mips-dev-disc...@lists.alioth.debian.org Usertags: mips-patch 'python-pysam' package FTBFS on mips/mipsel due to issues in 'samtools' package. Build log for 'python-pysam' on mips: https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipsver=0.7.7-1stamp=1397894446 Build log for 'python-pysam' on mipsel: https://buildd.debian.org/status/fetch.php?pkg=python-pysamarch=mipselver=0.7.7-1stamp=1397887877 Output of samtools executable from samtools package is used as a reference during tests in python-pysam (its output is compared with output from pysam python module), but samtools has both unaligned memory access and system endianness issues. I have attached patches that fix this. With these patches applied to samtools, python-pysam builds fine. Hi Aleksandar, I admit that I am a bit worried of applying a patch that I am not able to review. Could you ask upstream (https://github.com/samtools/samtools) to have a look ? Have a nice day, -- Charles Plessy Debian Med packaging team, http://www.debian.org/devel/debian-med Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
