[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily  added the comment:

As of 2.7.15 (finally!) all current python.org macOS installers now provide 
their own copies of OpenSSL: 1.0.2x for 3.6.x and 2.7.15. 1.1.0x for the 
upcoming 3.7.0 release.  Until a better solution for root certificates is 
implemented, all of the installers also now provide a "Install 
Certificates.command', a script which will download and install the latest 
third-party `certifi` bundle of certificates from PyPI and configure the ssl 
module to use them.  Declaring this issue resolved.  Thanks everyone!

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily  added the comment:


New changeset 93c91ac8f67ef0819ddacdcca12ef0ae88e5802e by Ned Deily in branch 
'2.7':
[2.7] Fix errant NEWS item: bpo-19019 -> bpo-17128 (GH-6470)
https://github.com/python/cpython/commit/93c91ac8f67ef0819ddacdcca12ef0ae88e5802e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Change by Ned Deily :


--
pull_requests: +6169
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes  added the comment:

Ned, is this issue still relevant? It targets 3.5 and hasn't seen much activity 
since 2016.

--
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Cosimo Lupo]

Cosimo Lupo  added the comment:

Hello,

I see that the official Python.org OSX 10.6+ installers are still linking with 
OSX outdated OpenSSL (0.9.8zh 14 Jan 2016; I'm using macOS High Sierra 10.13.2).

In the installer's README, they motivates this because:

> Apple's 0.9.8 version includes an important additional feature: if a 
> certificate cannot be verified using the manually administered certificates 
> in /System/Library/OpenSSL, the certificates managed by the system security 
> framework in the user and system keychains are also consulted (using Apple 
> private APIs)

However, because of this outdated OpenSSL version, I cannot use pip to install 
from the TestPyPI server https://test.pypi.org.

When I try (even with --trusted-host) I get this error:

```
$ pip install --trusted-host --index-url https://test.pypi.org/simple afdko
Collecting https://test.pypi.org/simple
Exception:
Traceback (most recent call last):
...
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version 
(_ssl.c:661)
```

A similar issue on pypa/pip repository was closed as

> There's no actionable item here. People with old versions of openssl that 
> don't support sha1 SSL certificates need to upgrade or else they are 
> insecure. If they wish to be insecure they can continue using pip 1.2

https://github.com/pypa/pip/issues/829#issuecomment-20931050

~~~

Well, I find it particularly odd that the official binary distribution for the 
latest Python 2.7.14 has a broken (or insecure) _ssl module, even when running 
the latest macOS version.

Of course, using pyenv or homebrew fixes the problem (as they require and link 
with the latest openssl 1.0.2), but I would like to recommend installing python 
from the official binaries to my less technically-skilled colleagues of mine.

Please consider embedding the latest openssl on the 10.6+ installers like you 
already do on 10.5 32bit ones and the Windows ones, thank you.

Cosimo Lupo

--
nosy: +Cosimo Lupo2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Changes by Christian Heimes :


--
components: +SSL

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Roundup Robot]

Roundup Robot added the comment:

New changeset bfd0a73cf907 by Ned Deily in branch 'default':
Issue #17128: Build OS X installer for 3.6 with private copy of OpenSSL.
https://hg.python.org/cpython/rev/bfd0a73cf907

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Brian Curtin]

Changes by Brian Curtin :


--
nosy:  -brian.curtin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Kevin Ollivier]

Kevin Ollivier added the comment:

The OpenSSL included with OS X, still at 0.9.8, has become very dated and a 
growing number of servers (including openssl.org) now fail the handshake 
because they no longer support any of the protocols and ciphers included with 
that build. It is reaching the point where for some projects the system OpenSSL 
on OS X is no longer viable to use.

Here's a ticket for a Python project showing an example of this problem:

https://github.com/kivy/kivy-ios/issues/198

I have run into this issue myself on this and a couple other projects I am 
working on.

Would it be possible to have the fix for 10.5 builds get applied to all the 
Python OS X installers, perhaps for 3.6, and ideally also for 2.7 if there are 
plans to make a new release for that?

--
nosy: +Kevin Ollivier

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Roundup Robot]

Roundup Robot added the comment:

New changeset 288b03b5c60d by Ned Deily in branch '3.4':
Issue #17128: Use private version of OpenSSL for 3.x OS X 10.5+ installers.
https://hg.python.org/cpython/rev/288b03b5c60d

New changeset 1c249d0cab5d by Ned Deily in branch 'default':
Issue #17128: Merge / update from 3.4
https://hg.python.org/cpython/rev/1c249d0cab5d

New changeset 961f988eaa6a by Ned Deily in branch '2.7':
Use rtf format files for legacy OS X installer builds.
https://hg.python.org/cpython/rev/961f988eaa6a

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Roundup Robot]

Roundup Robot added the comment:

New changeset 31f506f4e2d2 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/31f506f4e2d2

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2b5fa8651bd0 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/2b5fa8651bd0

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

The link below contains a script for building fat binaries for openssl. There's 
nothing surprising in the script, just building multiple times and then merging 
the result using lipo.

https://gist.github.com/foozmeat/5154962

BTW. I'm not proposing to use that script, at most it could be used for 
inspiration to enhance the build-installer script.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Piotr Dobrogost]

Changes by Piotr Dobrogost p...@bugs.python.dobrogost.net:


--
nosy: +piotr.dobrogost

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes added the comment:

Thanks to Hynek we were able to dig deeper into Apple's modifications. OpenSSL 
on OSX uses TEA (TrustEvaluationAgent) to verify cert chains. TEA is pretty 
much undocumented on the internet but perhaps we can use it to verify certs 
with OpenSSL 1.x, too?

http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.c

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

Crys, as Ronald noted above: Now that I look at that code again: we can't 
extract that code and use it to patch upstream OpenSSL, the 
TrustEvaluationAgent framework is a private framework and hence off limits.  
It doesn't seem like a good idea to be trying to base security on a private, 
undocumented framework and one that can change from OS X release to OS X 
release: our binary installers for OS X are designed to support multiple OS X 
versions.  I think the certsync approach is safer and more robust.  The other 
approach would be to directly use Apple's crypto APIs rather than OpenSSL but 
that would be a lot of work and a lot of testing and would also be more coupled 
to specific OS X releases.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

AFAIK OpenSSL has hooks that can be called when a certificate needs to be 
validated. If I my memory is correct this could be used to validate 
certificates using a public API (basically doing the same as Apple's patch, but 
using public APIs for the system and OpenSSL).

This has one significant risk though: as we've found at a couple of times (such 
as with the _scproxy extension) Apple's API don't necessary play along nicely 
when you use execv without fork or fork without execv :-(. I have no idea if 
Apple's preferred crypto APIs suffer from this problem.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

Update: the MacPorts certsync daemon has matured and is now included as
an optional MacPorts port.

It's not a perfect solution as noted in the macports-devel thread:

The only catch is that custom added certificates or trust anchors need
to be in the system keychain to be picked up by certsync by default.
Yeah, this was an unfortunate trade-off; since certsync is a system-wide 
daemon, and the resulting CA certs file is also system-wide, it seemed to
be the most appropriate course of action. Most of the alternatives involve
patching OpenSSL and some of the software that depends on it, which is a
road I'm personally wary of committing to.

http://comments.gmane.org/gmane.os.apple.macports.devel/22653

It works by registering a launchd agent that is run whenever any of the
system keychain files or trust setting files are modified.  That raises the
issues of when and how a Python install should register the agent (will
likely need admin/root privileges to do that) and how to delete the agent
(we currently don't have a formal uninstall procedure on OS X).  It would
be easier to manage these things with the binary installer-based Pythons,
as provided by python.org, in which case Pythons built by users from source
would still use the deprecated Apple-supplied libssl and libcrypto.  But
I'd like to separate out the building of third-party libraries, like
openssl, from the installer build process so that user source builds can
take advantage of features like this.

--
versions: +Python 3.5 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Donald Stufft]

Changes by Donald Stufft donald.stu...@gmail.com:


--
nosy: +dstufft

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Mateusz Lenik]

Mateusz Lenik added the comment:

I finally managed to find some time to clean up the script I used to compile 
multiarch openssl.

The diff is available at 
https://bitbucket.org/_mlen/cpython/commits/319f10362eb4b947b12750a6eb66fb622bbb0079

For 10.8 I had to comment out Tk related part and use the command below to 
build:
./build-installer.py 
--sdk-path=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/
 --third-party=/Users/m/Documents/Code/sources --universal-archs=intel 
--dep-target=10.8

The test results are below:
test-osx ➤ arch -i386 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.
test-osx ➤ arch -x86_64 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.

--
hgrepos: +204

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

The file not even empty, it doesn't even exist in default installations.

As Ned mentioned the CA roots on OSX are stored in a system database (the 
keychain). The situation is more or less the same as on Windows: their either 
needs to be code that queries the system keychain to get the CA roots, or we 
need to use some other CA list (like the mozilla one). 

I'd prefer the former because that's easier for the end user (add the 
company-wide CA cert to the system database to configure it system wide instead 
of hunting down where every app stores it CA list).

One glitch: a lot of Apple API's above the Unix layer are unsafe when used in a 
process started with os.fork (without calling execv), I don't know yet if the 
keychain APIs are also affected by this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes added the comment:

How about we include certsync [1] as an external program then? It's not as 
elegant as an internal API but it's not going to cause trouble with forking 
servers.

The first time a root CA cert is require, some new code in Python's SSL module 
runs certsync and grabs the list of PEM encoded certs from its stdout. The 
output is cached in a module variable and loaded with the new features from 
#16487 and #18138.

How does that sound to you?

[1] 
https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Mateusz Lenik]

Mateusz Lenik added the comment:

The other way to solve problem with CA certs may be this script[1] that can be 
used to generate pem files from mozilla CA cert bundle[2] during compilation or 
installation.

[1]: http://curl.haxx.se/docs/caextract.html
[2]: 
http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes added the comment:

Mateusz,
we would like to avoid that path and use the operating system's root CAs list 
when the OS provides one. It gives better user experience and the OS' root CA 
list gets updates through the OS.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Mateusz Lenik]

Mateusz Lenik added the comment:

I managed to build Python with OpenSSL 1.0.1e on Mac OS 10.8.4 using 
build-installer.py script during Europython sprint. I'll attach patches in few 
days, as I'll try to clean the code a bit. Currently the build works only for 
intel.

Here is an example output:
test-osx ~ ➤ uname -a
Darwin test-osx.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May  1 17:57:12 
PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
test-osx ~ ➤ arch -i386 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:34)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type help, copyright, credits or license for more information.
 import sys
 sys.maxsize
2147483647
 import ssl
 ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
 ^D
test-osx ~ ➤ arch -x86_64 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:35)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type help, copyright, credits or license for more information.
 import sys
 sys.maxsize
9223372036854775807
 import ssl
 ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
 ^D
test-osx ~ ➤

--
nosy: +mlen

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes added the comment:

Can you please run the unit test to see if it actually works?

 arch -i386 python -m test -unetwork test_ssl

That commands runs the unit test suite and tries to connect to a couple of 
HTTPS servers, too.

By the way do you know how well supported and up to date 
/System/Library/OpenSSL/certs/cert.pem is? The file suppose to contain all CA 
certs.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

Chrys, /System/Library/OpenSSL/certs/cert.pem is empty on current OS X systems. 
As mentioned above, cert management is handled through Apple APIs.  See 
http://developer.apple.com/library/ios/#DOCUMENTATION/Security/Conceptual/CertKeyTrustProgGuide/01introduction/introduction.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

The RVM issue is wrong, ML still includes OpenSSL. Apple has deprecated the use 
of the system install of OpenSSL, but the library and include files are still 
there.

There are two paths for avoiding the deprecated library: either ship your own 
build of OpenSSL, or (and that's probably what Apple would prefer) use the 
Apple specific frameworks (common crypto and/or security transforms). 

The latter has the advantage of using the security infrastructure, such as the 
CA chain, as provided by Apple, but would require significant code changes in 
Python, might take even more work to get to work properly on OSX 10.6 or 
earlier (and cannot work on 10.4), and might cause problems with scripts that 
use os.fork() because a number of core Apple frameworks won't work properly in 
the child process and cause a hard crash when they were initialized in the 
parent and then used in a child.

All in all it would be better to ship a recent version of OpenSSL with the 
binary installers on OSX.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

The one difference between the system openssl and a separately compiled one is 
that the former can use the CA root from the KeyChain (and uses a private API 
to do that, as noted earlier).

I just stumbled across a utility that can sync the KeyChain to an OpenSSL CA 
file: 
https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m,
 and a blog message at 
http://landonf.bikemonkey.org/code/macosx/certsync.20130514.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Stefan Drees]

Changes by Stefan Drees ste...@drees.name:


--
nosy: +dilettant

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Senthil Kumaran]

Senthil Kumaran added the comment:

Ronald:

Sorry for the delay in response. I was referring to this:

https://github.com/capistrano/capistrano/issues/250#issuecomment-8055991

 Mountain Lion no longer includes OpenSSL as it uses a different SSL
implementation. So, you need to have RVM install OpenSSL, then
reinstall rubies. This also fixes the SSL error with bundler updating
gems.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Senthil Kumaran]

Senthil Kumaran added the comment:

It should be noted that latest OSX Mountain Lion has caused problems for other 
language libraries too (specifically ruby, which I use at work). Ease the 
support of correct openssl in OSX may help a long way in all versions of python.

--
nosy: +orsenthil

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

What other problems? Do you have more information on that?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

After spending some time on this, I'm downgrading this from release blocker 
status.  First, no one has yet identified any immediate need for openssl 1.0.x 
features to support possible PyPI enhancements, which was my original concern.  
Second, since the openssl build system does not support OS X universal builds 
or SDKs and is not autoconf-based, it does not fit well into the current OS X 
installer build process.  I have a working first cut of building the libs but 
there is more to do.  Third, there is the open issue of how to make root certs 
available.  Ronald, I'm probably missing something obvious here but I don't see 
which Apple patch you are referring to.  Can you elaborate?

There is also the issue of government export restrictions that seems to always 
come up with crypto software.  AFAICT, as of a couple of years ago, there is no 
longer any restriction on shipping openssl binaries with any encryption 
algorithm from the US to any other country.  There are still a few well-known 
patent issue which seem easy to avoid.  But I am not a lawyer.
 
Unless someone objects, I'm going to treat this as a new feature for now and, 
once ready, we can re-examine backporting.

--
priority: release blocker - high
versions:  -Python 2.7, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

See also: issue 15740


A version of OpenSSL as included in some versions of OSX can be downloaded from 
http://opensource.apple.com/tarballs/OpenSSL098/, as mentioned in issue 15740 
the versions as included in the most recent OS updates doesn't seem to be there.

I've downloaded OpenSSL098-35.1 and that includes files 
./src/crypto/x509/x509_vfy_apple.h and ./src/crypto/x509/x509_vfy_apple.c which 
implement the behavior I mentioned earlier: first try to verify using the 
default OpenSSL mechanism, then verify using the TrustEvaluationAgent. 

Now that I look at that code again: we can't extract that code and use it to 
patch upstream OpenSSL, the TrustEvaluationAgent framework is a private 
framework and hence off limits.

It is probably possible to reimplement the same feature using public APIs, but 
that's new development and should be off-limits for a bugfix release (and isn't 
something that can be done very soon without risking to introduce new bugs in 
security-related code).

Direct link to the source code I mentioned: 
http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.c,
 
http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.h


A blog about this feature by the one of the curl developers: 
http://daniel.haxx.se/blog/2011/11/05/apples-modified-ca-cert-handling-and-curl/


P.S. Apple doesn't exactly make it easy to find this information.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Changes by Christian Heimes li...@cheimes.de:


--
nosy: +christian.heimes

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Christian Heimes]

Christian Heimes added the comment:

On Windows urllib.request.urlopen(http://www.google.com;, cadefault=True) 
fails with certificate verify failed. (tested with Python 3.3 64bit)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

It doesn't raise an exception on OSX (close to the tip of the default branch), 
both for http://www.google.com/ and https://www.google.com/

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

New submission from Ned Deily:

Apple has deprecated use of openssl in OS X due to its unstable API between 
versions:

If your app depends on OpenSSL, you should compile OpenSSL yourself and 
statically link a known version of OpenSSL into your app

https://developer.apple.com/library/mac/#documentation/security/Conceptual/cryptoservices/GeneralPurposeCrypto/GeneralPurposeCrypto.html

Currently OS X ships with patched versions of libssl 0.9.7 and 0.9.8.  The 
32-bit python.org installer links with and dynamically loads 0.9.7 and the 
64-/32-bit installer with 0.9.8.

build-installer.py should be enhanced to build and link with its own universal 
more up-to-date static libssl, as is done for several other OS X-supplied 
libraries.  Since apparently the openssl upstream builds do not support OS X 
universal builds, build-installer.py will need to learn how to build each arch 
separately and lipo them together.

With the current discussion around security issues, are there features in 
openssl 1.x.x that warrant making this a release blocker for 2.7.4 and 3.2.4?  
I should be able to implement and test this over the next few days if so.

Setting to release blocker for release managers' decision.

--
assignee: ned.deily
components: Build, Macintosh
messages: 181366
nosy: benjamin.peterson, georg.brandl, larry, ned.deily, ronaldoussoren
priority: release blocker
severity: normal
stage: needs patch
status: open
title: OS X system openssl deprecated - installer should build local libssl
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ed Campbell]

Changes by Ed Campbell drescampb...@gmail.com:


--
nosy: +esc24

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Benjamin Peterson]

Benjamin Peterson added the comment:

As you are the MacOSX expert, I'm going to defer your judgement (and/or 
Ronald's). I don't think the release will be for several days at least, so you 
should have time to test.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

I'm not sure if it is worthwhile to switch right now.  Apple does deprecate the 
use of OpenSSL, but there version does offer a feature that's not in the 
default tree: it verifies SSL certificates against the CA list in the system 
keychain.   

This means that users that verify certificates (cert_reqs=CERT_REQUIRED in the 
ssl module) could see a regression when they don't specificy a custom CA list. 
Not having to maintain such a list manually is very convenient.

In the longer run I'd like to try if it is possible to implement the SSL module 
(and other extensions linking with openssl) using Apple's crypto APIs.

(Note that a clear disadvantage of the latter is that those APIs are above 
the unix layer and likely cause problems when you use fork(2) without exec(2)).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

Yes, as we've discussed, using the Apple Crypto APIs would be nice longer-term 
assuming the compatibility issues can be managed: the set of available APIs 
appear to have been evolving over the past several OS X releases.  But moving 
away from openssl seems out of scope for maintenance releases.  The reason why 
I brought this up now is to try to determine if there are any newer ssl 
features that may be need to be batteries-included to deal with possible 
changes to Distutils and/or its users (pip, et al).   

Managing the CA certificates certainly is another issue.  We should investigate 
what Apple and third-party distributors of openssl on OS X do.  It would be 
best to be able to use the system-supplied ones since they are actively managed 
by Apple and can be by the user.

Also, it would be good to know how the Python Windows distribution handles 
openssl and certificates.  Martin?  Brian?

--
nosy: +brian.curtin, loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Éric Araujo]

Éric Araujo added the comment:

Using the CA bundle from the OS sounds great, not only for Macs :)

--
nosy: +eric.araujo, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ned Deily]

Ned Deily added the comment:

Somewhat coincidentally, Issue17129 addresses the topic of certificate 
management across multiple platforms.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17128] OS X system openssl deprecated - installer should build local libssl

[Ronald Oussoren]

Ronald Oussoren added the comment:

Replacing openssl by the supported crypto api's is something for 3.4 or even 
3.5.

There is a way to keep the current functionality while still shipping a build 
of openssl: apply the patch that implements the feature to the upstream version 
when building it (the patch is available on opensource.apple.com, that's how I 
know that they do this in the first place).

Something that should be tested before this gets merged: what happens when a 
user installs pyOpenSSL with python 2.7.3 install (linked to system openssl) 
and then upgrades to 2.7.4 linked to a custom build of openssl without changing 
pyOpenSSL.  

I wouldn't expect problems when looking at the documentation (there doesn't 
seem to be a way to transfer SSL state at the C level), and something similar 
can already happen: python is linked with a fairly old version of OpenSSL, and 
you get a later version when linking on a newer OSX release (hence a lot of 
users that download the binary installer and then install pyOpenSSL already 
have a version mismatch between the two extensions using openssl).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17128
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com