Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Mattia Rizzolo]

On Thu, Nov 09, 2017 at 10:11:30PM -0500, Afif Elghraoui wrote:
> On November 9, 2017 6:20:03 PM EST, Diane Trout  wrote:
> >
> [...]
> >
> >Anyone want to review? or should I go ahead and release?
> 
> I think you can release.

Yes please release it, and IMHO close this bug with that upload.
I've tested building python-pysam against it, and it gets a dependency
on 'libhts2 (>= 1.5)' instead of just 'libhts2' :)

> However, it appears the changelog wasn't updated in git since the last upload 
> [1] you'll need to go for a 1.5-3.

I've fixed it.
I force pushed a tag change for debian/1.5-2, as it pointed to something
that was not what was uploaded (bad tille!)

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Afif Elghraoui]

On November 9, 2017 6:20:03 PM EST, Diane Trout  wrote:
>
[...]
>
>Anyone want to review? or should I go ahead and release?
>

I think you can release. However, it appears the changelog wasn't updated in 
git since the last upload [1] you'll need to go for a 1.5-3.

regards
Afif

1. 
http://metadata.ftp-master.debian.org/changelogs/main/h/htslib/htslib_1.5-2_changelog

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Mattia Rizzolo]

On Wed, Nov 08, 2017 at 11:32:56PM -0800, Diane Trout wrote:
> I think we'd need to use the Built-Using tag? I haven't used that
> before.

No, that's needed when doing static linking for GPL compliance (and
other kind of things, but all related to static linking that thanks god
is not a topic here...

> (Though doing that would push htslib into NEW).

Please worry not about NEW, I've got contacts with ftp-master and can
get stuff out fairly quick (after a respectful time passed, like a week
at least).

> > And there is another action item--
> > TODO update the htslib package to the latest release.
> 
> Very true I did try building 1.6 and there was a problem with
> running tests that I haven't investigated yet.

Let me recommend adding the symbols file and getting it right before
doing 1.6, so symbols new in 1.6 are correctly versioned.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Afif Elghraoui]

On November 9, 2017 2:32:56 AM EST, Diane Trout  wrote:
>On Thu, 2017-11-09 at 02:03 -0500, Afif Elghraoui wrote:
>> > - TODO Split private cram headers off into a new libhts-private-dev
>> > package
>> 
>> I'd rather be in favor of restoring the bundled htslib to seqlib as
>> the short term solution. Putting a private package in the archive may
>> exacerbate the problem and is odd nevertheless.
>
>The no convenience copies of libraries is a pretty strong rule of
>Debian, and there are good maintenance reasons for it.

Yes, I understand that, but we don't have good options right now. I don't see 
the maintenance advantages for seqlib as worth requiring a transition for every 
htslib update to make sure it doesn't break--in addition to putting a package 
in the archive and telling users/developers not to install it.

> Although I'm not
>opposed to it I'd like several people to agree that its the best option
>first.
>
>On the plus side overriding it would allow us to drop the patch that is
>making the cram symbols public, on the downside we'd have to remember
>that bugs involving htslib also impact libseqlib.

If this whole situation is primarily seqlib's problem, I think it's only fair 
for it to bear the kludges required for its approach. Otherwise, we have the 
kludge in htslib and the need for a htslib transition with every update, right?

In fact, lofreq never entered Debian because it needs to use samtools as a 
library and we were not going to bundle it [1]. I felt that would be an RC bug.

>
>I think we'd need to use the Built-Using tag? I haven't used that
>before.
>
>On the other hand upstream did suggest that the private-dev library was
>a viable temporary solution. (Though doing that would push htslib into
>NEW).

Well, I think they were saying that if we were going to go so far as to 
misrepresent htslib, we should at the very least make a division and distribute 
htslib proper as such. I read that as saying anything would be better than the 
current situation--not necessarily that they're equally better.

>
>> 
>> And there is another action item--
>> TODO update the htslib package to the latest release.
>
>Very true I did try building 1.6 and there was a problem with
>running tests that I haven't investigated yet.
>

Regards
Afif

1. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808895

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Diane Trout]

On Thu, 2017-11-09 at 02:03 -0500, Afif Elghraoui wrote:
> > - TODO Split private cram headers off into a new libhts-private-dev
> > package
> 
> I'd rather be in favor of restoring the bundled htslib to seqlib as
> the short term solution. Putting a private package in the archive may
> exacerbate the problem and is odd nevertheless.

The no convenience copies of libraries is a pretty strong rule of
Debian, and there are good maintenance reasons for it. Although I'm not
opposed to it I'd like several people to agree that its the best option
first.

On the plus side overriding it would allow us to drop the patch that is
making the cram symbols public, on the downside we'd have to remember
that bugs involving htslib also impact libseqlib.

I think we'd need to use the Built-Using tag? I haven't used that
before.

On the other hand upstream did suggest that the private-dev library was
a viable temporary solution. (Though doing that would push htslib into
NEW).

> 
> And there is another action item--
> TODO update the htslib package to the latest release.

Very true I did try building 1.6 and there was a problem with
running tests that I haven't investigated yet.

Diane

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Afif Elghraoui]

Hi, Diane,

Thanks for working on this.


On November 8, 2017 7:58:49 PM EST, Diane Trout  wrote:
>One of the htslib developers filed a new bug,
>
>https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881170
>
>asking us to not make their private libraries public. His suggestions
>are fairly similar to whats Charles proposed.
>
>What I'm thinking is:
>
>- TODO Recommit symbols file

+1

>- TODO Split private cram headers off into a new libhts-private-dev
>package

I'd rather be in favor of restoring the bundled htslib to seqlib as the short 
term solution. Putting a private package in the archive may exacerbate the 
problem and is odd nevertheless.

And there is another action item--
TODO update the htslib package to the latest release.

>- WAITING Try to talk htslib & SeqLib developers to agree on a public
>cram api so we can drop the private-dev package.
>
>
> 
>[...]
>
>> 
>> > I was wondering if we should split the cram headers into a
>> > libhts-private-dev so we can at least track what is depending on
>> > the
>> > non-public api.

We would find this out anyway because the packages woukd break until either a 
dependency on such a package had to be added (most likely by our team anyway), 
or the library had to be rebundled.

>> 
>[...]
>
>> 
>> > I did realize that my thought about updating the SOVERSION might be
>> > wrong because I was just looking in the source tree for the removed
>> > functions but I should have been checking the public header files.
>> 
>> Indeed, packages using private functions need to have a tight
>> dependency
>> on the htslib (unless we are very confident that there are regression
>> tests that cover this area of the code).  Packages that are more
>> well-behaved can infer their dependency through the (to be re-added)
>> symbols file.
>
>
>
>So that implies packaging that uses -private-dev that implies they have
>an == dependency on a specific binary version of libhts?
>
>That should probably probably be documented in a README.Debian for the
>private-dev package.
>

Does this imply a transition for each htslib update?

Many thanks and regards
Afif

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Andreas Tille]

Dear Diane,

On Wed, Nov 08, 2017 at 04:58:49PM -0800, Diane Trout wrote:
> 
> Should I push these proposed changes to a clone of the htslib packaging
> repository? A branch of the alioth repository, or just push it to the
> alioth master?

Thanks for your sane considerations and pleas push to alioth master.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Diane Trout]

One of the htslib developers filed a new bug,

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881170

asking us to not make their private libraries public. His suggestions
are fairly similar to whats Charles proposed.

What I'm thinking is:

- TODO Recommit symbols file
- TODO Split private cram headers off into a new libhts-private-dev
package
- WAITING Try to talk htslib & SeqLib developers to agree on a public
cram api so we can drop the private-dev package.


> 
> Symbols file are strange to work with because their update usually
> goes
> through a build failure that outputs a patch, which is not very
> intuitive.  And then the patched symbols file has to be edited to
> remove
> the Debian minor version, otherwise it complicates backports etc.
> Perhaps it can be simplified, better explained and streamlined.  In
> any
> case, I think that for the htslib it is worth the effort.

The KDE team had some nice utilities that downloaded the symbols files
for all the architectures and could do batch patches.

Unfortunately I think they're KDE specific.

I'll commit my rebuilt symbols files the next time I'm not spending my
day writing emails to everyone else. I need a chance to look more
carefully if the missing symbols were actually not part of the private
api.


> 
> > I was wondering if we should split the cram headers into a
> > libhts-private-dev so we can at least track what is depending on
> > the
> > non-public api.
> 
> An ideal solution, and I understand that it may not be easy, would be
> to
> make the upstream users of htslib talk with the htslib developers, so
> that they can implement what they want to without needing to access
> private functions.  I think that it would fit the aims of both sides.

I wonder if I can forward one Debian bug to multiple upstreams

But I tried to get SeqLb & htslib to talk to each other.

SeqLib issue: https://github.com/walaj/SeqLib/issues/21
htslib issue: https://github.com/samtools/htslib/issues/619

> 
> > I did realize that my thought about updating the SOVERSION might be
> > wrong because I was just looking in the source tree for the removed
> > functions but I should have been checking the public header files.
> 
> Indeed, packages using private functions need to have a tight
> dependency
> on the htslib (unless we are very confident that there are regression
> tests that cover this area of the code).  Packages that are more
> well-behaved can infer their dependency through the (to be re-added)
> symbols file.



So that implies packaging that uses -private-dev that implies they have
an == dependency on a specific binary version of libhts?

That should probably probably be documented in a README.Debian for the
private-dev package.

Should I push these proposed changes to a clone of the htslib packaging
repository? A branch of the alioth repository, or just push it to the
alioth master?

Diane
> 

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Charles Plessy]

Hi Diane and everybody,

Le Tue, Nov 07, 2017 at 05:09:34PM -0800, Diane Trout a écrit :
> 
> I do think we should bring back the symbols file

I think so too.

Symbols file are strange to work with because their update usually goes
through a build failure that outputs a patch, which is not very
intuitive.  And then the patched symbols file has to be edited to remove
the Debian minor version, otherwise it complicates backports etc.
Perhaps it can be simplified, better explained and streamlined.  In any
case, I think that for the htslib it is worth the effort.

> I was wondering if we should split the cram headers into a
> libhts-private-dev so we can at least track what is depending on the
> non-public api.

An ideal solution, and I understand that it may not be easy, would be to
make the upstream users of htslib talk with the htslib developers, so
that they can implement what they want to without needing to access
private functions.  I think that it would fit the aims of both sides.

> I did realize that my thought about updating the SOVERSION might be
> wrong because I was just looking in the source tree for the removed
> functions but I should have been checking the public header files.

Indeed, packages using private functions need to have a tight dependency
on the htslib (unless we are very confident that there are regression
tests that cover this area of the code).  Packages that are more
well-behaved can infer their dependency through the (to be re-added)
symbols file.

Have a nice day,

Charles

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

Bug#879886: [Debian-med-packaging] Bug#879886: libhts2: libhts2 needs to handle ABI changes

[Mattia Rizzolo]

On Thu, Oct 26, 2017 at 12:12:05PM -0700, Diane Trout wrote:
> libhts2 introduced an ABI change which broke python-pysam, and a new
> version of python-pysam needed to be released to update to the new ABI.

FTR, this is what changed between the symbols of the version 1.4.1-5 and
1.5-1:

--- 1.4.1-5
+++ dpkg-gensymbolsIEaASY
@@ -178,6 +178,7 @@
  bgzf_flush_try@Base 1.4.1
  bgzf_getc@Base 1.4.1
  bgzf_getline@Base 1.4.1
+ bgzf_hfile@Base 1.5
  bgzf_hopen@Base 1.4.1
  bgzf_index_add_block@Base 1.4.1
  bgzf_index_build_init@Base 1.4.1
@@ -395,7 +396,7 @@
  file_size@Base 1.4.1
  find_file_url@Base 1.4.1
  find_path@Base 1.4.1
- flen@Base 1.4.1
+#MISSING: 1.5# flen@Base 1.4.1
  grp_create_key@Base 1.4.1
  hclose@Base 1.4.1
  hclose_abruptly@Base 1.4.1
@@ -411,6 +412,7 @@
  hfile_plugin_init_libcurl@Base 1.4.1
  hfile_plugin_init_net@Base 1.4.1
  hfile_plugin_init_s3@Base 1.4.1
+ hfile_set_blksize@Base 1.5
  hflush@Base 1.4.1
  hgetc2@Base 1.4.1
  hgetdelim@Base 1.4.1
@@ -436,6 +438,7 @@
  hts_format_file_extension@Base 1.4.1
  hts_get_bgzfp@Base 1.4.1
  hts_get_format@Base 1.4.1
+ hts_get_log_level@Base 1.5
  hts_getline@Base 1.4.1
  hts_hopen@Base 1.4.1
  hts_idx_destroy@Base 1.4.1
@@ -460,6 +463,7 @@
  hts_json_fskip_value@Base 1.4.1
  hts_json_snext@Base 1.4.1
  hts_json_sskip_value@Base 1.4.1
+ hts_log@Base 1.5
  hts_md5_destroy@Base 1.4.1
  hts_md5_final@Base 1.4.1
  hts_md5_hex@Base 1.4.1
@@ -480,6 +484,7 @@
  hts_realloc_or_die@Base 1.4.1
  hts_set_cache_size@Base 1.4.1
  hts_set_fai_filename@Base 1.4.1
+ hts_set_log_level@Base 1.5
  hts_set_opt@Base 1.4.1
  hts_set_thread_pool@Base 1.4.1
  hts_set_threads@Base 1.4.1
@@ -588,7 +593,7 @@
  mfgets@Base 1.4.1
  mfmmap@Base 1.4.1
  mfopen@Base 1.4.1
- mfprintf@Base 1.4.1
+#MISSING: 1.5# mfprintf@Base 1.4.1
  mfread@Base 1.4.1
  mfrecreate@Base 1.4.1
  mfreopen@Base 1.4.1
@@ -704,13 +709,13 @@
  vcf_read@Base 1.4.1
  vcf_write@Base 1.4.1
  vcf_write_line@Base 1.4.1
- vflen@Base 1.4.1
- zfclose@Base 1.4.1
- zfeof@Base 1.4.1
- zfgets@Base 1.4.1
- zfopen@Base 1.4.1
- zfpeek@Base 1.4.1
- zfputs@Base 1.4.1
- zfseeko@Base 1.4.1
- zftello@Base 1.4.1
+#MISSING: 1.5# vflen@Base 1.4.1
+#MISSING: 1.5# zfclose@Base 1.4.1
+#MISSING: 1.5# zfeof@Base 1.4.1
+#MISSING: 1.5# zfgets@Base 1.4.1
+#MISSING: 1.5# zfopen@Base 1.4.1
+#MISSING: 1.5# zfpeek@Base 1.4.1
+#MISSING: 1.5# zfputs@Base 1.4.1
+#MISSING: 1.5# zfseeko@Base 1.4.1
+#MISSING: 1.5# zftello@Base 1.4.1
  zlib_mem_inflate@Base 1.4.1

> libhts2 probably needs a proper symbols file to make it easier to see
> when the ABI is changing. https://wiki.debian.org/UsingSymbolsFiles
> 
> Mattia Rizzolo, also suggests other methods of dealing with managing ABI
> changes.

In pysam's case, it is looking for hts_log, if you had properly checked
the symbols of your package before uploading you should have at least
tweaked the dh_shlibdeps invocation to force a version, or introduced a
.symbols file (very recommended for a case like hstlib where the symbols
seems to be simple).

But then, the ABI is broken, so just nicest symbols handling is not
enough for your case, you need
> Such as bumping soname, changing the package name and use Conflicts, or
> adding versioned Breaks against broken packages

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature