Re: [R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()
On Tue, Jan 11, 2022 at 10:12 PM Simon Urbanek wrote: > > Petře, > > thanks, for the detailed analysis. It is rather curious that the issue > appears only on _newer_ systems - we are more used to issues due to older CA > chains and similar. It looks like an Apple bug on specific systems, so > hopefully it will be fixed eventually. In general I was trying to avoid > having to supply our own SSL library since that opens a whole can of worms - > on one hand due the dependency issues (which libraries get compiled against > what) and on the other hand we become responsible for security updates. > > Thanks to Jeroen for the work-around (CURL_SSL_BACKEND=SecureTransport), > using the native API is certainly preferred, there have been several issues > with both OpenSSL and LibreSSL before. It seems that Apple has been > flip-flopping with libcurl a lot - on El Capitan it was shipped with > SecureTransport, on High-Sierra with LibreSSL, on Catalina and higher with > both, but Libre the default. > > I am somewhat less apprehensive to use static libcurl for R than SSL > libraries as the fallout is a bit smaller. As a trial I have added static > curl[2] which is close to the Apple build minus MultiSSL to big-sur nightly > builds of R[3] and as expected that solves the problem. It may not be > entirely unproblematic for package space, because packages often forget to > prepend --static when using static builds of libraries, and so do other > dependencies that may use curl, but I'll see what comes out of it. I would much recommend to stick with the apple version of libcurl; perhaps override the default ssl-backend if you like. There is some example code to do this in the curl package that you could adapt for base r: https://github.com/jeroen/curl/blob/master/src/ssl.c The benefit of dynamically linking to apple's libcurl is that we automatically get a version of libcurl+deps+certs that is tuned and maintained for that version of macos, including future ones. If you ship a version of base-R with a static libcurl now, that version of R may not work anymore a few years from now or on a future version of macos, when things have moved on (for example, when servers start to require TLS1.3). ___ R-SIG-Mac mailing list R-SIG-Mac@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-mac
Re: [R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()
> On Jan 13, 2022, at 12:47 AM, Jeroen Ooms wrote:
>
> On Tue, Jan 11, 2022 at 10:12 PM Simon Urbanek
> wrote:
>>
>> Petře,
>>
>> thanks, for the detailed analysis. It is rather curious that the issue
>> appears only on _newer_ systems - we are more used to issues due to older CA
>> chains and similar. It looks like an Apple bug on specific systems, so
>> hopefully it will be fixed eventually. In general I was trying to avoid
>> having to supply our own SSL library since that opens a whole can of worms -
>> on one hand due the dependency issues (which libraries get compiled against
>> what) and on the other hand we become responsible for security updates.
>>
>> Thanks to Jeroen for the work-around (CURL_SSL_BACKEND=SecureTransport),
>> using the native API is certainly preferred, there have been several issues
>> with both OpenSSL and LibreSSL before. It seems that Apple has been
>> flip-flopping with libcurl a lot - on El Capitan it was shipped with
>> SecureTransport, on High-Sierra with LibreSSL, on Catalina and higher with
>> both, but Libre the default.
>>
>> I am somewhat less apprehensive to use static libcurl for R than SSL
>> libraries as the fallout is a bit smaller. As a trial I have added static
>> curl[2] which is close to the Apple build minus MultiSSL to big-sur nightly
>> builds of R[3] and as expected that solves the problem. It may not be
>> entirely unproblematic for package space, because packages often forget to
>> prepend --static when using static builds of libraries, and so do other
>> dependencies that may use curl, but I'll see what comes out of it.
>
> I would much recommend to stick with the apple version of libcurl; perhaps
> override the default ssl-backend if you like. There is some example code to
> do this in the curl package that you could adapt for base r:
> https://github.com/jeroen/curl/blob/master/src/ssl.c
>
> The benefit of dynamically linking to apple's libcurl is that we
> automatically get a version of libcurl+deps+certs that is tuned and
> maintained for that version of macos, including future ones. If you ship a
> version of base-R with a static libcurl now, that version of R may not work
> anymore a few years from now or on a future version of macos, when things
> have moved on (for example, when servers start to require TLS1.3).
>
Yes, but if you are using an old version of R on a new system, you have a lot
of other worries - you can't expect new technologies to work with old software.
CURL itself has fewer evolution issues than SSL libraries. As I said, I am a
big proponent of re-using system libraries as much as possible, but, for
example, High Sierra doesn't ship with ST back-end support, so using a static
version that does is better there as Apple doesn't not maintain the curl CAs
but it does the system ones so it's arguably better. The current issue is quite
curious since breaking on the latest system is quite unusual, just preferring
ST works only because it is the latest system that breaks and it has the ST
option.
As Brian pointed out static curl has its own issues since its pkg-config flags
are broken - that's why I have not activated the add-on recipes by default, I
have seen those issues before.
For R itself there are thee options:
a) add CURL_SSL_BACKEND=${CURL_SSL_BACKEND-'SecureTransport'} to
$R_HOME/etc/Renviron of the distribution
b) add something like your
https://github.com/r-devel/r-svn/pull/75/commits/79b22b461e527e8a46de84c145e8e5fb59e75d14
to R
c) build against static libcurl
The big advantage of the first one is that it applies to all processes, so even
command line curl will then work and so will all packages.
The drawback of the second one is that it only applies the R itself. The third
one could be done both for R and packages, but causes headaches resp. requires
slight patching of libcurl.pc. The advantage is that it can bring more recent
curl to all older systems.
I don't have a strong opinion. I am not thrilled with option b) because that is
a hack just to react to something which is never a good idea from maintenance
point of view (we would require all curl-based code to use it). So I think a)
and c) are more palatable with a) having the benefit of handling non-R cases. A
slight benefit of c) is that some dependencies require more recent curl version
than provided by older systems, so that would cover it at the cost of
maintaining the curl binaries. Finally, the real benefit of c) is that if Apple
screws things up even more we don't care - we may not be at that point yet,
though.
Cheers,
Simon
___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
Re: [R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()
On Wed, Jan 12, 2022 at 10:05 PM Simon Urbanek
wrote:
> Yes, but if you are using an old version of R on a new system, you have a lot
> of other worries - you can't expect new technologies to work with old
> software. CURL itself has fewer evolution issues than SSL libraries. As I
> said, I am a big proponent of re-using system libraries as much as possible,
> but, for example, High Sierra doesn't ship with ST back-end support, so using
> a static version that does is better there as Apple doesn't not maintain the
> curl CAs but it does the system ones so it's arguably better. The current
> issue is quite curious since breaking on the latest system is quite unusual,
> just preferring ST works only because it is the latest system that breaks and
> it has the ST option.
>
> As Brian pointed out static curl has its own issues since its pkg-config
> flags are broken - that's why I have not activated the add-on recipes by
> default, I have seen those issues before.
>
> For R itself there are thee options:
>
> a) add CURL_SSL_BACKEND=${CURL_SSL_BACKEND-'SecureTransport'} to
> $R_HOME/etc/Renviron of the distribution
>
> b) add something like your
> https://github.com/r-devel/r-svn/pull/75/commits/79b22b461e527e8a46de84c145e8e5fb59e75d14
> to R
>
> c) build against static libcurl
>
>
> The big advantage of the first one is that it applies to all processes, so
> even command line curl will then work and so will all packages.
>
> The drawback of the second one is that it only applies the R itself. The
> third one could be done both for R and packages, but causes headaches resp.
> requires slight patching of libcurl.pc. The advantage is that it can bring
> more recent curl to all older systems.
>
> I don't have a strong opinion. I am not thrilled with option b) because that
> is a hack just to react to something which is never a good idea from
> maintenance point of view (we would require all curl-based code to use it).
> So I think a) and c) are more palatable with a) having the benefit of
> handling non-R cases. A slight benefit of c) is that some dependencies
> require more recent curl version than provided by older systems, so that
> would cover it at the cost of maintaining the curl binaries. Finally, the
> real benefit of c) is that if Apple screws things up even more we don't care
> - we may not be at that point yet, though.
I don't think apple screwed up per se; they probably tested several
configurations and picked this one to be the safest default. TLS is a
complex protocol with many versions and implementations; if some weird
server uses some non-standard cipher or unusual response, it may just
depend on the TLS library if it can handle that. I'm sure you'll be
able to find counter examples where libre/openssl works and
SecureTransport does not. For example, a case that we often encounter
on Windows are corporate networks which require connecting via
authenticated proxy servers or using a TLS client cert, which only
works on certain back-ends, see the table in:
https://cran.r-project.org/web/packages/curl/vignettes/windows.html
So I much favor of option A. This introduces the least complexity, and
keeps the ability for users to undo our change and switch back to
CURL_SSL_BACKEND=openssl in their .Renviron. Also it is a big benefit
in practice that curl in R behaves the same as command line curl on
that same machine, in order to narrow down if a connection problem is
a bug in our R code, or if it also exists outside of R.
___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
Re: [R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()
I am not an expert, but it seems to me that switching the backend is a
runtime setting. Couldn't we detect which version of OS X we're running and
then select the backend conditionally on that test?
Best,
Kasper
On Wed, Jan 12, 2022 at 5:12 PM Jeroen Ooms wrote:
> On Wed, Jan 12, 2022 at 10:05 PM Simon Urbanek
> wrote:
> > Yes, but if you are using an old version of R on a new system, you have
> a lot of other worries - you can't expect new technologies to work with old
> software. CURL itself has fewer evolution issues than SSL libraries. As I
> said, I am a big proponent of re-using system libraries as much as
> possible, but, for example, High Sierra doesn't ship with ST back-end
> support, so using a static version that does is better there as Apple
> doesn't not maintain the curl CAs but it does the system ones so it's
> arguably better. The current issue is quite curious since breaking on the
> latest system is quite unusual, just preferring ST works only because it is
> the latest system that breaks and it has the ST option.
> >
> > As Brian pointed out static curl has its own issues since its pkg-config
> flags are broken - that's why I have not activated the add-on recipes by
> default, I have seen those issues before.
> >
> > For R itself there are thee options:
> >
> > a) add CURL_SSL_BACKEND=${CURL_SSL_BACKEND-'SecureTransport'} to
> $R_HOME/etc/Renviron of the distribution
> >
> > b) add something like your
> https://github.com/r-devel/r-svn/pull/75/commits/79b22b461e527e8a46de84c145e8e5fb59e75d14
> to R
> >
> > c) build against static libcurl
> >
> >
> > The big advantage of the first one is that it applies to all processes,
> so even command line curl will then work and so will all packages.
> >
> > The drawback of the second one is that it only applies the R itself. The
> third one could be done both for R and packages, but causes headaches resp.
> requires slight patching of libcurl.pc. The advantage is that it can bring
> more recent curl to all older systems.
> >
> > I don't have a strong opinion. I am not thrilled with option b) because
> that is a hack just to react to something which is never a good idea from
> maintenance point of view (we would require all curl-based code to use it).
> So I think a) and c) are more palatable with a) having the benefit of
> handling non-R cases. A slight benefit of c) is that some dependencies
> require more recent curl version than provided by older systems, so that
> would cover it at the cost of maintaining the curl binaries. Finally, the
> real benefit of c) is that if Apple screws things up even more we don't
> care - we may not be at that point yet, though.
>
> I don't think apple screwed up per se; they probably tested several
> configurations and picked this one to be the safest default. TLS is a
> complex protocol with many versions and implementations; if some weird
> server uses some non-standard cipher or unusual response, it may just
> depend on the TLS library if it can handle that. I'm sure you'll be
> able to find counter examples where libre/openssl works and
> SecureTransport does not. For example, a case that we often encounter
> on Windows are corporate networks which require connecting via
> authenticated proxy servers or using a TLS client cert, which only
> works on certain back-ends, see the table in:
> https://cran.r-project.org/web/packages/curl/vignettes/windows.html
>
> So I much favor of option A. This introduces the least complexity, and
> keeps the ability for users to undo our change and switch back to
> CURL_SSL_BACKEND=openssl in their .Renviron. Also it is a big benefit
> in practice that curl in R behaves the same as command line curl on
> that same machine, in order to narrow down if a connection problem is
> a bug in our R code, or if it also exists outside of R.
>
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>
--
Best,
Kasper
[[alternative HTML version deleted]]
___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
