Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
On Wed, Nov 23, 2016 at 11:59:16AM +0400, Stepan Golosunov wrote: > 22.11.2016 в 22:57:30 +0200 Adrian Bunk написал: > > 23:14 < bunk> Q_: If you come up with a better patch than mine in #844018, > > please post to that bug. I know that my patch is not pretty, > > but > > I did not find any better short-term solution. > > Search for CURLOPT_SSL_CTX_FUNCTION on codesearch.debian.net produces > the following list of potentially affected packages: > cargo > chromium-browser > cmake > criticalmass > curl > curlpp > firefox > firefox-esr > fpc > hhvm > icedove > lastpass-cli > libapache2-mod-auth-cas > libwww-curl-perl > lua-curl > netcdf > netsurf > openjfx > r-cran-curl > r-cran-rcurl > ruby-curb > slcurl > sx > tclcurl > wpa > xmltooling > zurl > > So the the alternative to you patch looks like: >... > fixing #828608 or removing xmltooling out of testing, >... "removing xmltooling out of testing" is not an option. Either *all* packages in this cluster get fixes to work with 1.1, or the whole cluster must stay at 1.0.2 in stretch. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
22.11.2016 в 22:57:30 +0200 Adrian Bunk написал: > 23:14 < bunk> Q_: If you come up with a better patch than mine in #844018, > please post to that bug. I know that my patch is not pretty, > but > I did not find any better short-term solution. Search for CURLOPT_SSL_CTX_FUNCTION on codesearch.debian.net produces the following list of potentially affected packages: cargo chromium-browser cmake criticalmass curl curlpp firefox firefox-esr fpc hhvm icedove lastpass-cli libapache2-mod-auth-cas libwww-curl-perl lua-curl netcdf netsurf openjfx r-cran-curl r-cran-rcurl ruby-curb slcurl sx tclcurl wpa xmltooling zurl So the the alternative to you patch looks like: fixing #828564 (fixed-upstream, new upstream version available), fixing #828608 or removing xmltooling out of testing, checking whether last apache2 upload fixed #844799, fixing or ignoring #828259 (not in testing, fixed upstream version available), fixing #828371 (untested patch available) or removing lastpass-cli, removing 3 characters from zurl's debian/control, binnmu-ing affected packages that still depend on libssl1.0.2, no need to ensure that applications (even if they are linked with libcurl3 indirectly) are linked with the same libssl as libcurl3 (unlike with the patch), more compatibility with applications from jessie than with the patch.
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
20.11.2016 в 22:35:26 +0100 Jan Niehusmann написал:
> On Mon, Nov 21, 2016 at 01:03:19AM +0400, Stepan Golosunov wrote:
> > So far I do not know why using libssl1.1 together with a
> > libssl1.0.2-using Qt wouldn't work.
>
> Well I don't know enough about the dynamic linker and about the internals
> of openssl to know if (indirectly) linking to both libraries at the same
> time is fine.
Indirectly linking with both libraries is supposed to work due to
symbol versioning. If it wasn't than OpenSSL transition wouldn't have
started. And parts of Qt are already linked with libssl1.1 (via
libpq5). Problems could (and did in previous transition) arise if
code (Qt or application) is initializing one libssl while parts of
application are using another but this is supposed to be fixed for
libssl1.1.
And the problem with Qt is that it is not actually linked with OpenSSL
but uses it via dlopen. My reading of dlsym(3) suggests that it
should work but I could be misreading the documentation or it can be
incorrect.
> If it was, that would be great news. Many mails in the thread "OpenSSL
> 1.1.0" on debian-devel seem to be based on the assumption that such
> linking could cause bugs, and therefore packages can only transition in
> clusters of packages linking to the same version of openssl.
If application passes OpenSSL objects between the two libraries than
it won't work.
> Still, qt is only an example - the same holds true for other libraries
> linking to openssl1.0-dev. There may be cases where your 2nd case
> ('Application passes OpenSSL objects from libssl1.1 to ...') is more
> probable than with qt.
I expect that if libcurl3 is linked with libssl1.0.2 there would be more
problematic applications (as applications need to link with
libssl1.0.2 and run initialization in this case) than if curl is
linked with libssl1.1 (as applications not using
CURLOPT_SSL_CTX_FUNCTION-like functionality are free to link with
whatever libssl they want or not to link with it at all).
> The safest way to avoid hidden bugs would still be changing SONAME and
> package name, so package maintainers would be aware of the change and
> could check their packages for compatibility.
>
> But yes, it may be less work to somehow identify affected packages and
> handle them directly instead of forcing all packages depending on curl
> through a transition. Identifying those packages in a reliable way could
> be difficult, though.
If libcurl3 is linked with libssl1.1 then maintainer needs to
explicitly break his application (like happened with zurl 1.7.1-1).
The rest just need to be recompiled (and they need to be recompiled in
any case).
The part of the bug where dependencies are insufficiently tough and
libcurl does not change its SONAME exists independently of which
libssl it is linked to (as long as that libssl is different from the
one used in jessie).
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
On Mon, Nov 21, 2016 at 01:03:19AM +0400, Stepan Golosunov wrote:
> So far I do not know why using libssl1.1 together with a
> libssl1.0.2-using Qt wouldn't work.
Well I don't know enough about the dynamic linker and about the internals
of openssl to know if (indirectly) linking to both libraries at the same
time is fine.
If it was, that would be great news. Many mails in the thread "OpenSSL
1.1.0" on debian-devel seem to be based on the assumption that such
linking could cause bugs, and therefore packages can only transition in
clusters of packages linking to the same version of openssl.
Still, qt is only an example - the same holds true for other libraries
linking to openssl1.0-dev. There may be cases where your 2nd case
('Application passes OpenSSL objects from libssl1.1 to ...') is more
probable than with qt.
The safest way to avoid hidden bugs would still be changing SONAME and
package name, so package maintainers would be aware of the change and
could check their packages for compatibility.
But yes, it may be less work to somehow identify affected packages and
handle them directly instead of forcing all packages depending on curl
through a transition. Identifying those packages in a reliable way could
be difficult, though.
Jan
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
20.11.2016 в 21:29:35 +0100 Jan Niehusmann написал: > On Mon, Nov 21, 2016 at 12:02:45AM +0400, Stepan Golosunov wrote: > > It would be a grave bug in such application if it does not have a > > working version, yes. Whether or not it would be a serious bug in > > lubcurl3 depends on how many and how important such applications are. > > (And how difficult they are to fix.) > > As there are applications which depend on libcurl and qt, and it seems > like qt can't be ported to openssl 1.1 in time for stretch, fixing such > applications would mean uploading a version of curl linked to openssl > 1.0. This could be a new package, or a statically linked version etc. - > IMHO all worse than just linking the curl package with openssl 1.0 until > (close to) all packages are ready for a transition to openssl 1.1. So far I do not know why using libssl1.1 together with a libssl1.0.2-using Qt wouldn't work. So far I can imagine the following reasons: 1. Application relies on Qt initializing the library. That was the cause of breakage during libssl1.0.0 -> libssl1.0.2 transition, but is not applicable now as libssl1.1 does not require explicit initialization. 2. Application passes OpenSSL objects from libssl1.1 to Qt (or vice versa). Why one would do it? 3. dlsym(3) is used with RTLD_DEFAULT or RTLD_NEXT. Does not seem to be the case. 4. dlsym(3) finds wrong library. Manual page suggests that should not be the case. … Is any of the above applicable?
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
On Mon, Nov 21, 2016 at 12:02:45AM +0400, Stepan Golosunov wrote: > It would be a grave bug in such application if it does not have a > working version, yes. Whether or not it would be a serious bug in > lubcurl3 depends on how many and how important such applications are. > (And how difficult they are to fix.) As there are applications which depend on libcurl and qt, and it seems like qt can't be ported to openssl 1.1 in time for stretch, fixing such applications would mean uploading a version of curl linked to openssl 1.0. This could be a new package, or a statically linked version etc. - IMHO all worse than just linking the curl package with openssl 1.0 until (close to) all packages are ready for a transition to openssl 1.1. Jan
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
20.11.2016 в 20:01:25 +0100 Jan Niehusmann написал: > On Sun, Nov 20, 2016 at 07:40:22PM +0400, Stepan Golosunov wrote: > > If SONAME change is needed it is needed when linking libcurl with > > libssl1.0.2 too. (But when linking libcurl with libssl1.0.2 more > > applications are affected due to the need of explicit initialization.) > > Upstream documentation suggests that this is not true: > > "Minor releases that change the last digit, e.g. 1.0.1 vs. 1.0.2, can > and are likely to contain new features, but in a way that does not break > binary compatibility. This means that an application compiled and > dynamically linked with 1.0.0 does not need to be recompiled when the > shared library is updated to 1.0.2." > (https://www.openssl.org/policies/releasestrat.html) > > But of course, in practice, there may have been unintended ABI changes > - so yes, may be a SONAME change is nessary for 1.0.2 as well. I just > don't know. SONAME for OpenSSL 1.0.2 in Debian was changed because it was recompiled with different options to remove some deprecated (but widely used) functions. And if application initializes libssl1.0.0 then libssl1.0.2 is left uninitialized and does not work correctly, while libssl1.1 does not need explicit initialization at all. OpenSSL libraries have lots of global state, so passing pointers created with libssl1.0.2 to libssl1.0.0 isn't safe. Different features and implementation details of the two libraries are likely to cause problems too in that case. > Still, for 1.1 the SONAME change is definitely necessary. Breaks: against applications using CURLOPT_SSL_CTX_FUNCTION and compiled with OpenSSL 1.0 will probably suffice in that case.
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
20.11.2016 в 23:38:48 +0400 Stepan Golosunov написал: > 20.11.2016 в 20:01:25 +0100 Jan Niehusmann написал: > > Still, for 1.1 the SONAME change is definitely necessary. > > Breaks: against applications using CURLOPT_SSL_CTX_FUNCTION and > compiled with OpenSSL 1.0 will probably suffice in that case. No, that won't be sufficient. One must also ensure that applications compiled with libssl1.1-using libcurl3 also depend on such libcurl3 (both for the sake of applications using CURLOPT_SSL_CTX_FUNCTION and applications dropping OpenSSL initialization).
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
20.11.2016 в 20:48:44 +0100 Jan Niehusmann написал: > On Sun, Nov 20, 2016 at 11:38:49PM +0400, Stepan Golosunov wrote: > > > Still, for 1.1 the SONAME change is definitely necessary. > > > > Breaks: against applications using CURLOPT_SSL_CTX_FUNCTION and > > compiled with OpenSSL 1.0 will probably suffice in that case. > > Wouldn't this still be a serious bug as it would - explicitly - break > those applications? It would be a grave bug in such application if it does not have a working version, yes. Whether or not it would be a serious bug in lubcurl3 depends on how many and how important such applications are. (And how difficult they are to fix.)
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
On Sun, Nov 20, 2016 at 11:38:49PM +0400, Stepan Golosunov wrote: > > Still, for 1.1 the SONAME change is definitely necessary. > > Breaks: against applications using CURLOPT_SSL_CTX_FUNCTION and > compiled with OpenSSL 1.0 will probably suffice in that case. Wouldn't this still be a serious bug as it would - explicitly - break those applications? Jan
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
On Sun, Nov 20, 2016 at 07:40:22PM +0400, Stepan Golosunov wrote: > If SONAME change is needed it is needed when linking libcurl with > libssl1.0.2 too. (But when linking libcurl with libssl1.0.2 more > applications are affected due to the need of explicit initialization.) Upstream documentation suggests that this is not true: "Minor releases that change the last digit, e.g. 1.0.1 vs. 1.0.2, can and are likely to contain new features, but in a way that does not break binary compatibility. This means that an application compiled and dynamically linked with 1.0.0 does not need to be recompiled when the shared library is updated to 1.0.2." (https://www.openssl.org/policies/releasestrat.html) But of course, in practice, there may have been unintended ABI changes - so yes, may be a SONAME change is nessary for 1.0.2 as well. I just don't know. Still, for 1.1 the SONAME change is definitely necessary. Jan
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
11.11.2016 в 21:30:07 +0100 Jan Niehusmann написал: > the curl ABI contains structs inherited from OpenSSL, e.g. in calls > like: > > curl_easy_setopt(easy, CURLOPT_SSL_CTX_FUNCTION, _cb); > > Here, sslCtxFunction_cb is a function which takes an SSL_CTX * as a > parameter. > > (This is from zurl, one example of a package affected by this bug.) > > Since 7.51.0-1, curl links against OpenSSL 1.1 instead of OpenSSL 1.0 > (implicitly caused by an update of libssl-dev, not by a change to the > curl package). This changes the structure of SSL_CTX, which in turn > changes the above mentioned ABI and breaks zurl (and possibly other > packages). And libcurl3 in testing links with libssl1.0.2 while libcurl3 in jessie links with libssl1.0.0. And while structure of SSL_CTX is probably identical in this case, global state of libssl is different, as well as code and supported features are different. Also, when libcurl is linked with OpenSSL older then 1.1 many applications need to be linked with the same OpenSSL libraries as curl as they need to initialize libssl as described in https://lists.debian.org/debian-devel/2016/11/msg00718.html With OpenSSL 1.1 this requirement goes away, as newer OpenSSL initializes itself automatically (and if application initializes old version of OpenSSL no harm is done). > Such ABI changes require a SONAME change, according to policy 8.1, > exactly to avoid breaking other packages which use the library. > > Therefore, please consider changing the SONAME (and the name of the > binary package). Alternatively, build-depend on libssl1.0-dev, to link > against OpenSSL 1.0 and keep the old ABI. If SONAME change is needed it is needed when linking libcurl with libssl1.0.2 too. (But when linking libcurl with libssl1.0.2 more applications are affected due to the need of explicit initialization.)
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
Control: tags -1 patch
On Fri, Nov 11, 2016 at 09:30:07PM +0100, Jan Niehusmann wrote:
> Package: libcurl3
> Version: 7.51.0-1
> Severity: serious
> Justification: Policy 8.1
>
> Dear Maintainer,
>
> the curl ABI contains structs inherited from OpenSSL, e.g. in calls
> like:
>
> curl_easy_setopt(easy, CURLOPT_SSL_CTX_FUNCTION, _cb);
>
> Here, sslCtxFunction_cb is a function which takes an SSL_CTX * as a
> parameter.
>
> (This is from zurl, one example of a package affected by this bug.)
>
> Since 7.51.0-1, curl links against OpenSSL 1.1 instead of OpenSSL 1.0
> (implicitly caused by an update of libssl-dev, not by a change to the
> curl package). This changes the structure of SSL_CTX, which in turn
> changes the above mentioned ABI and breaks zurl (and possibly other
> packages).
>
> Such ABI changes require a SONAME change, according to policy 8.1,
> exactly to avoid breaking other packages which use the library.
>
> Therefore, please consider changing the SONAME (and the name of the
> binary package).
For 3rd party software doing anything Debian-specific here would be bad.
Debian should really follow whatever upstream and other distributions
are doing when they switch to 1.1 (hopefully a new soname from upstream).
For stretch it would also not solve the problem that there is
libcurl-using software that is using OpenSSL 1.0.2
> Alternatively, build-depend on libssl1.0-dev, to link
> against OpenSSL 1.0 and keep the old ABI.
>...
Unfortunately this is not sufficient.
libcurl4-openssl-dev also has to depend on libssl1.0-dev,
otherwise you will have the same problem the other way around.
I know it is not pretty, but the following seems to be required:
--- debian/control.old 2016-11-20 08:19:55.0 +
+++ debian/control 2016-11-20 08:20:32.0 +
@@ -16,7 +16,7 @@
libnss3-dev,
librtmp-dev (>= 2.4+20131018.git79459a2-3~),
libssh2-1-dev,
- libssl-dev,
+ libssl1.0-dev,
libtool,
openssh-server ,
python:native,
@@ -121,14 +121,13 @@
libcurl4-dev,
libcurl3-dev
Conflicts: libcurl4-gnutls-dev, libcurl4-nss-dev
-Depends: ${misc:Depends}, libcurl3 (= ${binary:Version})
+Depends: ${misc:Depends}, libcurl3 (= ${binary:Version}), libssl1.0-dev
Suggests: libcurl4-doc, libcurl3-dbg,
libidn11-dev,
libkrb5-dev,
libldap2-dev,
librtmp-dev,
libssh2-1-dev,
- libssl-dev,
pkg-config,
zlib1g-dev
Multi-Arch: same
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Bug#844018: libcurl3: Building with OpenSSL 1.1 breaks packages using both OpenSSL 1.0 and curl
Package: libcurl3 Version: 7.51.0-1 Severity: serious Justification: Policy 8.1 Dear Maintainer, the curl ABI contains structs inherited from OpenSSL, e.g. in calls like: curl_easy_setopt(easy, CURLOPT_SSL_CTX_FUNCTION, _cb); Here, sslCtxFunction_cb is a function which takes an SSL_CTX * as a parameter. (This is from zurl, one example of a package affected by this bug.) Since 7.51.0-1, curl links against OpenSSL 1.1 instead of OpenSSL 1.0 (implicitly caused by an update of libssl-dev, not by a change to the curl package). This changes the structure of SSL_CTX, which in turn changes the above mentioned ABI and breaks zurl (and possibly other packages). Such ABI changes require a SONAME change, according to policy 8.1, exactly to avoid breaking other packages which use the library. Therefore, please consider changing the SONAME (and the name of the binary package). Alternatively, build-depend on libssl1.0-dev, to link against OpenSSL 1.0 and keep the old ABI. (Set the severity to serious, to keep the package with the broken ABI from entering testing. IMHO this bug is a policy violation, but policy is not 100% clear here, as it only says 'the SONAME should change'. If you don't agree, please don't just lower the severity, but discuss the issue on debian-devel.) Thanks, Jan -- System Information: Debian Release: stretch/sid APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-rc4 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages libcurl3 depends on: ii libc6 2.24-5 ii libcomerr21.43.3-1 ii libgssapi-krb5-2 1.15~beta1-1 ii libk5crypto3 1.15~beta1-1 ii libkrb5-3 1.15~beta1-1 ii libldap-2.4-2 2.4.42+dfsg-2+b3 ii libnghttp2-14 1.16.0-1 ii librtmp1 2.4+20151223.gitfa8646d.1-1 ii libssh2-1 1.7.0-1 ii libssl1.1 1.1.0c-1 ii zlib1g1:1.2.8.dfsg-2+b3 Versions of packages libcurl3 recommends: ii ca-certificates 20161102 libcurl3 suggests no packages. -- no debconf information
