[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

Many thanks Christian, that resolved the issue! I really appreciate your 
efforts here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Christian Heimes]

Christian Heimes  added the comment:

pyenv uses default value for ./configure --with-ssl-default-suites. You have to 
use --with-ssl-default-suites=openssl so your build uses the system's crypto 
policy correctly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

Many thanks Christian, see the attached for the output of the commands on 
Python 3.9.10 and 3.10.2, along with a diff removing version numbers and memory 
addresses. 

I've run the commands on the Ubuntu distribution, we can also run the same for 
the Centos VM, if helpful.

There are a few differences in the outputs but nothing that appears obviously 
the cause.

--
Added file: https://bugs.python.org/file50654/python_details.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Christian Heimes]

Christian Heimes  added the comment:

Could you please provide the outputs of the following commands:

python3 -c "import _ssl; print(_ssl.__file__)"

ldd $(python3.10 -c "import _ssl; print(_ssl.__file__)")

strace -e openat python3.10 -c "from urllib.request import urlopen; 
urlopen('https://www.python.org')"


The outputs look like this on my computer:

$ python3 -c "import _ssl; print(_ssl.__file__)"
/usr/lib64/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so
$ ldd $(python3.10 -c "import _ssl; print(_ssl.__file__)")
linux-vdso.so.1 (0x7ffd1a10e000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x7ff838e17000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x7ff838b29000)
libc.so.6 => /lib64/libc.so.6 (0x7ff83891f000)
libz.so.1 => /lib64/libz.so.1 (0x7ff838905000)
/lib64/ld-linux-x86-64.so.2 (0x7ff838f0b000)

$ strace -e openat python3.10 -c "from urllib.request import urlopen; 
urlopen('https://www.python.org')"
...
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) 
= 4
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/pki/tls/cert.pem", O_RDONLY) = 3
...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

Update, the Pyenv team confirmed that they do not install OpenSSL in linux, its 
only installed for MacOS, and it should be built using the system OpenSSL 
within Linux.

We're investigating further to attempt to debug the issue. Interestingly the 
OpenSSL build flags for both Python versions appear to be the same.

`Trying link with OPENSSL_LDFLAGS=; OPENSSL_LIBS=-lssl -lcrypto; 
OPENSSL_INCLUDES=`

I've attached the build logs for both the Python 3.9.10 and 3.10.2 build, in 
case you're able to review. Many thanks.

https://github.com/pyenv/pyenv/issues/2257

--
Added file: https://bugs.python.org/file50653/python_builds.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

Yes agreed, it may well be a Pyenv issue. Interestingly we can demonstrate that 
the global OpenSSL crypto policies is respected with the 3.9.10 version, 
through adjusting the policy. The ssl error occurs with the default policy 
setting and is resolved with the legacy policy setting. With 3.10.2 this is no 
longer the case. I can’t see any obvious changes to the build recipe that would 
cause this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Christian Heimes]

Christian Heimes  added the comment:

This is a pyenv issue, not a Python issue. Custom builds of OpenSSL typically 
do not and cannot use global settings like crypto policies. They are missing 
distro downstream patches and use different config files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

I found the Python build recipes and Pyenv does appear to install OpenSSL from 
source. The only difference I can see, aside from the Python version, is an 
update on the OpenSSL versions; openssl-1.1.1l (3.9.10) to openssl-1.1.1k 
(3.10.2). The OpenSSL release notes do not appear to suggest anything relevant.

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/3.10.2

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/3.9.10

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/bin/python-build

https://www.openssl.org/news/openssl-1.1.1-notes.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam]

Adam  added the comment:

Thanks for the quick reply. On both Ubuntu and Centos, I’m installing Python 
using Pyenv, testing with 3.9.10 and 3.10.2. Pyenv provides a verbose install 
flag, I can rebuild the Python versions and review the build commands, if 
helpful? I’m testing with clean Linux distributions and I believe there is only 
one OpenSSL installed and available. I don’t know if it’s possible to gain more 
details from the Python ssl module to confirm? I did confirm the OpenSSL 
versions aligns using ssl.OPENSSL_VERSION.

Command: pyenv install 3.10.2 --verbose

https://github.com/pyenv/pyenv

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Christian Heimes]

Christian Heimes  added the comment:

How did you build Python 3.10? Neither CentOS 8 nor Ubuntu 20.04 come with 
Python 3.10. Does your build of Python use system's OpenSSL build?

--
assignee: christian.heimes -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46863] Python 3.10 OpenSSL Configuration Issues

[Adam Pinckard]

New submission from Adam Pinckard :

Python 3.10 does not appear to respecting the OpenSSL configuration within 
linux. Testing completed using Pyenv on both Ubuntu 20.04.4 and Centos-8. Note 
PEP 644 which requires OpenSSL >= 1.1.1 is released in Python 3.10.

We operate behind a corporate proxy / firewall which causes an SSL error where 
the Diffie-Hellman key size is too small. In previous Python versions this is 
resolved by updating the OpenSSL configuration, e.g. downgrading the linux 
crypto policies `sudo update-crypto-policies --set LEGACY`. 

The issue is reproducible in both Ubuntu 20.04.4 and Centos-8. In both linux 
distributions the SSL error is resolvable in earlier Python version, using the 
OpenSSL configurations, but the configuration is not respected with Python 
3.10.2.

See the details below on the kernel versions, linux distributions, and Openssl 
versions, many thanks in advance.

1. Python 3.10.2 Error:
(py_3_10_2) ➜  py_3_10_2 pip install --upgrade pip
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, 
status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: 
DH_KEY_TOO_SMALL] dh key too small (_ssl.c:997)'))': /simple/pip/

2. Ubuntu details
uname -a
Linux Horatio 5.13.0-30-generic #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 
2022 x86_64 x86_64 x86_64 GNU/Linux

lsb_release  -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 20.04.4 LTS
Release:20.04
Codename:   focal

openssl version -a
OpenSSL 1.1.1f  31 Mar 2020
built on: Wed Nov 24 13:20:48 2021 UTC
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g 
-O2 -fdebug-prefix-map=/build/openssl-dnfdFp/openssl-1.1.1f=. 
-fstack-protector-strong -Wformat -Werror=format-security 
-DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC 
-DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT 
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM 
-DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM 
-DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time 
-D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"
Seeding source: os-specific

2. Centos-8 details
uname -a
Linux localhost.localdomain 5.4.181-1.el8.elrepo.x86_64 #1 SMP Tue Feb 22 
10:00:15 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/centos-release
CentOS Stream release 8

openssl version -a
OpenSSL 1.1.1k  FIPS 25 Mar 2021
built on: Thu Dec  2 16:40:48 2021 UTC
platform: linux-x86_64
options:  bn(64,64) md2(char) rc4(16x,int) des(int) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe 
-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS 
-fexceptions -fstack-protector-strong -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic 
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection 
-Wa,--noexecstack -Wa,--generate-missing-build-notes=yes 
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -DOPENSSL_USE_NODELETE -DL_ENDIAN 
-DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT 
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM 
-DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM 
-DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG 
-DPURIFY -DDEVRANDOM="\"/dev/urandom\"" 
-DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config"
OPENSSLDIR: "/etc/pki/tls"
ENGINESDIR: "/usr/lib64/engines-1.1"
Seeding source: os-specific
engines:  rdrand dynamic

--
assignee: christian.heimes
components: SSL
messages: 414072
nosy: adam, christian.heimes
priority: normal
severity: normal
status: open
title: Python 3.10 OpenSSL Configuration Issues
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com