[Python-Dev] Re: How To Get Started

[Christian Heimes]

On 28/07/2022 18.34, kiddkod...@gmail.com wrote:

New coding bootcamp graduate here looking actively for my first programming 
job. In the meantime, I want to start contributing to open source which brought 
me here. But after consulting the Python's Developer Guide, I'm still faced 
with some confusion and reluctance to get started. I tried looking at some of 
the current active issues and it has only caused more confusion.

Any advice, tips, or other resources that can help get me started?


I have a great idea how you could get started and help us:

Please write down which parts of the dev guide confuses you and which 
section you had a hard time to understand. This would help us greatly to 
identify problems with the dev guide and help other people that want to 
contribute.


Regards,
Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJNU4TP7DRTJCO4PQKMJHX7DIZWL2LGN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install

[Christian Heimes]

On 16/06/2022 16.29, Xavier de Gaye wrote:

Victor Stinner wrote:
 > https://github.com/python/cpython/issues/66913 
 doesn't explain how to 
reproduce the issue, it only gives some info about what doesn't work.

 > I don't know how to reproduce the issue.

Yes this issue does explain how to reproduce the problem. It even shows 
the backtrace printed when attempting to cross-build Android x86_64 on a 
linux x86_64.
To reproduce the issue, just cross-build python for android x86_64 
following the well documented process by the Android team.

It is not clear how this backtrace can be missed when reading the issue !

There is also a patch provided in this issue that is straightforward and 
that does not involve any change on distutils, only the Makefile and 
configure.


What Victor means that we would like to have clear instructions how to 
reproduce the problem ourselves. Could you please provide step by step 
instructions how I could set up a build environment on a X86_64 PC with 
a standard Linux distro (Fedora, Debian/Ubuntu)?


What packages have to be installed? Do I have to download any extra 
packages? How do I have to set up my build environment? Which commands 
do I have to execute? Is there a container image available that comes 
with everything pre-installed?


You mentioned well-documented process by the Android team. Could you 
please provide links to the relevant documents?


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WNEFUJZH32LSNBX75QBLP6X6XZQVAHNK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Christian Heimes]

On 10/05/2022 13.18, Victor Stinner wrote:

On Tue, May 10, 2022 at 11:28 AM Christian Heimes  wrote:

Right now, Python still uses distutils internally for multiple use
cases. I propose to start with renaming the distutils package to
_distutils in the stdlib:

* https://github.com/python/cpython/issues/92584
* https://github.com/python/cpython/pull/92585


Erlend and I got most extension modules ported to autoconf and
Modules/Setup.stdlib. The remaining modules should be done in a couple
of weeks. I recommend that we do not rename distutils and instead remove
it entirely.


test_peg_generator and test_cppext build C extensions with distutils.
Until these tests are modified to use something else, we still need to
keep distutils. So I propose to rename it to _distutils to remove it
from the stdlib. Maybe these tests can use ensurepip to install
setuptools which provides distutils.

There is also the c-analyzer tool which uses distutils to run a C preprocessor.


We can easily take care of test_cppext and add the build step to 
Makefile. How does test_peg_generator depend on distutils? I don't see 
an import of distutils in the directory.


I would prefer to fix and remove all imports of distutils before we 
resort to renaming the package. Please give us time until alpha 1. There 
is no need to rush it *now*.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5FOZXCSYPFJGWVFI7S6QNUKKICNPU5AQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Christian Heimes]

On 10/05/2022 09.53, Victor Stinner wrote:

Hi,

Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
distutils package was deprecated in Python 3.10 and scheduled for
removal in Python 3.12. Questions.

* Is the Python ecosystem ready for the distutils removal? How many
projects are expected to be broken by this removal?

* Is setuptools a drop-in replacement of distutils for most cases? Are
there tools and documentation explaining how to replace distutils with
setuptools?

* Is there a tool to migrate from setup.py (distutils) to
pyproject.toml (setuptools)? The dephell project can convert a
setup.py script to pyproject.toml using poetry as the build system.

* Can we simply suggest installing setuptools to users who still use
"import distutils"? setuptools now provides the a "distutils" package.

* Should we keep distutils in Python stdlib a few more Python releases
if the removal causes too many practical issues?



Setuptools now provides _distutils_hack meta importer that injects a 
facade distutils package. "from distutils import setup" will keep 
working with modern setuptools.




Right now, Python still uses distutils internally for multiple use
cases. I propose to start with renaming the distutils package to
_distutils in the stdlib:

* https://github.com/python/cpython/issues/92584
* https://github.com/python/cpython/pull/92585


Erlend and I got most extension modules ported to autoconf and 
Modules/Setup.stdlib. The remaining modules should be done in a couple 
of weeks. I recommend that we do not rename distutils and instead remove 
it entirely.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FYRFMWETDJXU3SRAOTZJIQRN6QDLAMXT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Christian Heimes]

On 10/05/2022 09.53, Victor Stinner wrote:

Hi,

Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
distutils package was deprecated in Python 3.10 and scheduled for
removal in Python 3.12. Questions.

* Is the Python ecosystem ready for the distutils removal? How many
projects are expected to be broken by this removal?

* Is setuptools a drop-in replacement of distutils for most cases? Are
there tools and documentation explaining how to replace distutils with
setuptools?

* Is there a tool to migrate from setup.py (distutils) to
pyproject.toml (setuptools)? The dephell project can convert a
setup.py script to pyproject.toml using poetry as the build system.

* Can we simply suggest installing setuptools to users who still use
"import distutils"? setuptools now provides the a "distutils" package.

* Should we keep distutils in Python stdlib a few more Python releases
if the removal causes too many practical issues?



Setuptools now provides _distutils_hack meta importer that injects a 
facade distutils package. "from distutils import setup" will keep 
working with modern setuptools.




Right now, Python still uses distutils internally for multiple use
cases. I propose to start with renaming the distutils package to
_distutils in the stdlib:

* https://github.com/python/cpython/issues/92584
* https://github.com/python/cpython/pull/92585


Erlend and I got most extension modules ported to autoconf and 
Modules/Setup.stdlib. The remaining modules should be done in a couple 
of weeks. I recommend that we do not rename distutils and instead remove 
it entirely.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7R5DKP5NZI5MVYQ7H44LFIGDHHHO3FOM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Supported make implementations

[Christian Heimes]

On 12/03/2022 06.27, Lincoln Auster wrote:

Hi,

When attempting to build and run doctests for cpython on FreeBSD, I got
a vague error from the default BSD make. gmake, of course, ran fine. Is
GNU make the only supported make implementation? If so, did I miss that
in the devguide, or should that be in there somewhere?


Hi,

the top-level Makefile works with gmake (GNU) and bmake (BSD). Buildbot 
CI tests that Python builds with standard compiler and make on FreeBSD.


The Makefile in Doc/ subdirectory is different. It is based on Sphinx's 
Makefile template. It looks like Sphinx uses GNU make features that are 
not supported by BSD make. Could you please open a bug at 
https://github.com/sphinx-doc/sphinx ?


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C465NHESPDGTZZMVZO4DPW6N4DBUSW7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Defining tiered platform support

[Christian Heimes]

On 07/03/2022 18.02, Petr Viktorin wrote:
Why the devguide? I view the list of platforms as important for public 
consumption as for the core dev team to know what to (not) accept PRs 
for.


So, let's put it in the main docs?
Yes, I guess the devguide is a weird place to check for this kind of 
info. But a Python enhancement proposal is even weirder.



+1 for our main docs (cpython/Doc/)

Platform support is Python versions specific. Python 3.10 may support 
different version than 3.11 or 3.12. It makes sense to keep the support 
information with the code.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJID7Y7RFSCRUYLJS3E56WBGJU2R44E4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Defining tiered platform support

[Christian Heimes]

On 04/03/2022 21.41, Brett Cannon wrote:

Therefore I propose that we target the oldest manylinux standard
accepted by PyPI, for which the operating system has not reached its
EOL. At the moment this is manylinux2014, aka CentOS 2024 with EOL June
2024. We could also state that we aim for compatibility with oldest
Debian Stable and Ubuntu LTS with standard, free security updates.
As of
today Debian 10 Buster Ubuntu 18.04 Bionic are the oldest versions with
regular updates.


So does that mean you want to list the Linux distros explicitly? Or you 
want to explicitly list the glibc version?


I want to explicitly mention glibc and Linux distro for manylinux binary 
wheels standards that are supported by PyPI. For the rest we should be a 
bit more hand-wavy. Maybe we can point to our list of stable buildbots?




We should say something about compilers. I wouldn't list compiler
versions, though. Compiler features like C99 support should be
sufficient.


Then what more would you want than what's listed in PEP 7 already?


Nothing in particular other than a link to the PEP, so people can 
discover the requirement more easily.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TM3NTO652FLS4XUX7QTMTP7GSX6DQRRH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Defining tiered platform support

[Christian Heimes]

Hi Brett,

thanks for starting the discussion! Much appreciated.

On 04/03/2022 00.30, Brett Cannon wrote:
Tier 1 is the stuff we run CI against: latest Windows, latest macOS, 
Linux w/ the latest glibc (I don't know of a better way to define Linux 
support as I don't know if a per-distro list is the right abstraction). 
These are platforms we won't even let code be committed for if they 
would break; they block releases if they don't work. These platforms we 
all implicitly promise to support.

>
Tier 2 is the platforms we would revert a change within 24 hours if they 
broke: latest FeeBSD, older Windows, older macOS, Linux w/ older 
glibc.This is historically the "stable buildbot plus a core dev" group 
of platforms. The change I would like to see is two core devs (in case 
one is on vacation), and a policy as to how a platform ends up here 
(e.g. SC must okay it based on consensus of everyone). The stable 
buildbot would still be needed to know if a release is blocked as we 
would hold a release up if they were red. The platform and the core devs 
supporting these platforms would be listed in PEP 11.


I would like to see an explicit statement about glibc compatibility. 
glibc's API and ABI is very stable. We have autoconf feature checks for 
newer glibc features, so I'm not overly concerned with breaking 
compatibility with glibc. Anyhow we should also ensure that we are 
backwards compatible with older glibc releases that are commonly used in 
the community.


Therefore I propose that we target the oldest manylinux standard 
accepted by PyPI, for which the operating system has not reached its 
EOL. At the moment this is manylinux2014, aka CentOS 2024 with EOL June 
2024. We could also state that we aim for compatibility with oldest 
Debian Stable and Ubuntu LTS with standard, free security updates. As of 
today Debian 10 Buster Ubuntu 18.04 Bionic are the oldest versions with 
regular updates.



Apropos libc, what is our plan concerning musl libc (Alpine)? It's a 
popular distro for containers. CPython's test suite is failing on latest 
Alpine (https://bugs.python.org/issue46390). Some of the problems seem 
to be caused by issues or missing features in musl libc. I like to see 
the problems fixed before we claim basic support for Alpine.



I would expect PEP 11 to list the appropriate C symbol that's set for 
that platform, e.g. __linux__.


For POSIX-like OS I would rather follow the example of Rust and use 
platform target triplet. The triplet encodes machine (CPU arch), vendor, 
and operating system. The OS part can encode libc. For example 
x86_64-*-linux-gnu for "x84_64 arch", "any vendor", and "Linux with GNU 
libc (glibc)". Commands like ./config.guess or gcc -dumpmachine return 
the current triplet.


The target triplet is used by autoconf's ./configure script a lot.


I don't know if we want to bother listing CPU architectures since we are 
a pure C project which makes CPU architecture less of a thing, but I'm 
personally open to the idea of CPU architectures being a part of the 
platform definition.


I strongly recommend that we include machine architecture, too. We have 
some code that uses machine specific instructions or features, e.g. 
unaligned memory access. We cannot debug problems on CPU archs unless we 
have access to the hardware.



I don't think we should cover C compilers here as that's covered by PEP 
7. Otherwise PEP 7 should only list C versions/features and PEP 11 lists 
compilers and their versions.


We should say something about compilers. I wouldn't list compiler 
versions, though. Compiler features like C99 support should be sufficient.


Do we target the platform's default compiler or are we targeting the 
latest compiler that is officially supported for the platform? CentOS 7 
comes with an old GCC, but has newer GCC versions in SCL (Developer 
Toolset 8). I'm asking because CentOS 7's default gcc does not support 
stdatomic.h. The official manylinux2014 OSCI container image ships GCC 
from devtoolset-8.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5LDLSPQVYLVKNEJADOFBQQ3TKBCFEDEV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Christian Heimes]

On 06/02/2022 15.08, Victor Stinner wrote:

Hi,

I propose to deprecate the urllib module in Python 3.11. It would emit
a DeprecationWarning which warn users, so users should consider better
alternatives like urllib3 or httpx: well known modules, better
maintained, more secure, support HTTP/2 (httpx), etc.

I don't propose to schedule its removal. Let's discuss the removal in
1 or 2 years.

--

urllib has many abstraction to support a wide range of protocols with
"handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP
authentication, HTTP Cookie, etc. A simple HTTP request using Basic
Authentication requires 10-20 lines of code, whereas it should be a
single line.

Users (me included) don't like urllib API which was too complicated
for common tasks.

--



[...]



urllib is a package made of 4 parts:

* urllib.request for opening and reading URLs
* urllib.error containing the exceptions raised by urllib.request
* urllib.parse for parsing URLs
* urllib.robotparser for parsing robots.txt files

I propose to deprecate all of them. Maybe the deprecation can be
different for each sub-module?


Thanks for bringing this topic forward, Victor!

Disclaimer: I proposed the removal of urllib today in Python core's 
internal chat.


The urllib package -- and to some degree also the http package -- are 
constant source of security bugs. The code is old and the parsers for 
HTTP and URLs don't handle edge cases well. Python core lacks a true 
maintainer of the code. To be honest, we have to admit defeat and be up 
front that urllib is not up to the task for this decade. It was designed 
written during a more friendly, less scary time on the internet.


If I had the power and time, then I would replace urllib with a simpler, 
reduced HTTP client that uses platform's HTTP library under the hood 
(WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten, 
maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or 
aiohttp are much better suited than urllib.


The second best option is to reduce the feature set of urllib to core 
HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter, 
more standard conform parsers for urls, query strings, and RFC 2822 
instead of RFC 822 for headers.


Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WYVETVHMGRS4CI47GTFY6W7B43YLSJH2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

[Christian Heimes]

On 02/02/2022 09.19, dw-...@d-woods.co.uk wrote:

Guido van Rossum wrote:

My question for you is if you're willing to write up a list of things in
CPython that you depend on. Or is this just something you're not willing to
commit to? It would be nice to know which it is, just so the CPython team
knows what we're up against.


I'm happy to prepare a list of CPython internals that Cython uses. Obviously 
there's no guarantee that it'll be complete (just because it involves lots of 
manually finding things in code so is full of human error) or that it doesn't 
change in future. But a list would at least let everyone know where they stand.


You should be able to automate much of the task and avoid human errors:

1) dump all exported symbols from libpython.so, e.g. readelf -Ws 
/usr/lib64/libpython3.10.so or readelf -Ws /usr/lib64/libpython3.10.so

2) look for each symbol in Cython sources

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HBKPPVXXCX7NDIZQ3LDGUVGQOPULRNY3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python

[Christian Heimes]

On 01/02/2022 16.08, Victor Stinner wrote:

--

I would prefer to introduce C API incompatible changes differently:
first fix Cython, and *then* introduce the change.

- (1) Propose a Cython PR and get it merged
- (2) Wait until a new Cython version is released
- (3) If possible, wait until numpy is released with regenerated Cython code
- (4) Introduce the incompatible change in Python

Note: Fedora doesn't need (3) since we always regenerated Cython code in numpy.


Hi,

this is a reasonable request for beta releases, but IMHO it is not 
feasible for alphas. During alphas we want to innovate fast and play 
around. Your proposal would slow down innovation and impose additional 
burden on core developers.


There are more code binding generators than just Cython. Shouldn't we 
work with cffi, SWIG, sip, pybind11, and PyO3 developers as well? I care 
for cffi and PyO3, too...


I would prefer if we can get Cython and all the other code generator and 
bindings library off the unstable C-API. They should use the limited API 
instead. If they require any C-APIs outside the limited API, then we 
should investigate and figure something out.


Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7UD4FZC7ANLR646CNP4HJ2WNWLFRYL7I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dropping AC_C_CHAR_UNSIGNED from configure.ac?

[Christian Heimes]

On 24/01/2022 14.34, Miro Hrončok wrote:

Hello Pythonistas.

In (development branch of) Fedora, we have juts upgraded to GCC 12.

It seems that the presence of AC_C_CHAR_UNSIGNED in Python's autotools 
files (configure.ac?) is causing the __CHAR_UNSIGNED__ symbol to be 
defined in pyconfig.h and that breaks some other packages with GCC 12.


The GCC maintainers told us it is a reserved symbol,
see https://bugzilla.redhat.com/2043555 for details.

It seems that using AC_C_CHAR_UNSIGNED is not recommended and also not 
required, but I must confess that I am pretty much horrified by 
autotools and I don't really know if we can get rid of 
AC_C_CHAR_UNSIGNED or not. I can test this in Fedora and I am quite sure 
it'll work, but I don't know the impact on all the other environments 
where CPython can be compiled.


Is there anybody on this list who knows a reason we need to keep 
AC_C_CHAR_UNSIGNED around in 2022? Or is it safe to get rid of it?


Let's find out!

I have created draft PR https://github.com/python/cpython/pull/30851 and 
scheduled the PR on our buildbot fleet.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VJMJBKCOQ5AX5I4C7M4MQECRSE5HIXMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [PSA] OpenSSL 3.0 support is preliminary and experimental

[Christian Heimes]

On 10/01/2022 17.01, Miro Hrončok wrote:

On 09. 01. 22 19:39, Christian Heimes wrote:

Hi,

I would like to remind everybody that Python's support for OpenSSL 3.0 
is preliminary [1]. Python compiles with OpenSSL 3.0.0 and simple code 
kinda works. However there are known performance regressions, missing 
features (e.g. usedforsecurity flag), and potential bugs cause by API 
incompatibilities.


Due to the experimental state I advise against using Python with 
OpenSSL 3.0 in production.


It may take a while until Python gains full support for the next 
version of OpenSSL. I have shifted my personal OSS time to more fun 
topics like performance and WASM. My work time is currently limited, too.


Hello Christian.

Do you think we should switch Python in Fedora 36 to OpenSSL 1.1.1? 
Python was naturally rebuilt with OpenSSL 3.0 when the distro upgraded 
OpenSSL. But the older version is still available.


Note that Fedora 36 is also "preliminary" so we still have time to make 
this decision until +- the beta freeze/release (end of February, early 
March this year).



Hi Miro,

I suggest to wait and re-evaluate the situation in a month from now. The 
situation might improve by then. OpenSSL Upstream is working on 
performance improvements. I have a pending fix for the hashlib 
usedforsecurity feature.


For the other issues, somebody has to put in the work and review all 
differences between OpenSSL 1.1.1 and 3.0.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C5WOH56A3MZUTL5Y3ARSNPGGUUFYEZ3G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [PSA] OpenSSL 3.0 support is preliminary and experimental

[Christian Heimes]

On 11/01/2022 12.02, Antoine Pitrou wrote:


Hi Christian,

On Sun, 9 Jan 2022 19:39:06 +0100
Christian Heimes  wrote:

Hi,

I would like to remind everybody that Python's support for OpenSSL 3.0
is preliminary [1]. Python compiles with OpenSSL 3.0.0 and simple code
kinda works. However there are known performance regressions, missing
features (e.g. usedforsecurity flag), and potential bugs cause by API
incompatibilities.

Due to the experimental state I advise against using Python with OpenSSL
3.0 in production.

It may take a while until Python gains full support for the next version
of OpenSSL. I have shifted my personal OSS time to more fun topics like
performance and WASM.


Sounds reasonable :-)

Out of curiosity, what are the performance regressions about?


OpenSSL 3.0 introduced a new context, provider, and algorithm fetch 
system. It is not optimized yet and has an impact on performance.


I noticed that hashlib.sha512(b'12345678') is more than 20% slower with 
new EVP_MD_fetch() API. Impact on multi threading applications is even 
bigger due to lock congestion.


https://github.com/openssl/openssl/issues/17064
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CWM32SJHRXIPM2YZDCTUHZRT5ETE2FGH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [PSA] OpenSSL 3.0 support is preliminary and experimental

[Christian Heimes]

Hi,

I would like to remind everybody that Python's support for OpenSSL 3.0 
is preliminary [1]. Python compiles with OpenSSL 3.0.0 and simple code 
kinda works. However there are known performance regressions, missing 
features (e.g. usedforsecurity flag), and potential bugs cause by API 
incompatibilities.


Due to the experimental state I advise against using Python with OpenSSL 
3.0 in production.


It may take a while until Python gains full support for the next version 
of OpenSSL. I have shifted my personal OSS time to more fun topics like 
performance and WASM. My work time is currently limited, too.


Christian

[1] https://docs.python.org/3/whatsnew/3.10.html#ssl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ATO4DM6QYZGLSGGDZ3TRN5X3QDD5OHOE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

[Christian Heimes]

On 14/12/2021 19.19, Eric Snow wrote:

A while back I concluded that neither approach would work for us.  The
approach I had taken would have significant cache performance
penalties in a per-interpreter GIL world.  The approach that modifies
Py_INCREF() has a significant performance penalty due to the extra
branch on such a frequent operation.


Would it be possible to write the Py_INCREF() and Py_DECREF() macros in 
a way that does not depend on branching? For example we could use the 
highest bit of the ref count as an immutable indicator and do something like


ob_refcnt += !(ob_refcnt >> 63)

instead of

ob_refcnt++

The code performs "ob_refcnt += 1" when the highest bit is not set and 
"ob_refcnt += 1" when the bit is set. I have neither tested if the 
approach actually works nor it's performance.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBTHSOI2XRWRO6WQOLUW3X7S5DUXFAOV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python release announcement format

[Christian Heimes]

On 14/12/2021 11.56, Yann Droneaud wrote:

Hi,

I'm not familiar with the Python release process, but looking at the latest 
release

https://www.python.org/downloads/release/python-3101/

we can see MD5 is still used ... which doesn't sound right in 2021 ...
especially since we proved it's possible to build different .tar.gz that have
the same MD5

https://twitter.com/ydroneaud/status/1448659749604446211
https://twitter.com/angealbertini/status/1449736035110461443

You would reply there's OpenPGP / GnuPG signature. But then I would like to 
raise
another issue regarding the release process:

As the announcement on comp.lang.python.announce 
/python-announce-l...@python.org
doesn't record the release digest / release signature, the operator behind
https://www.python.org/downloads/release/python-3101/  are free to change the 
release
content at any time, provided there's a valid signature. And there will no way 
for
us to check the release wasn't modified after the announcement.

It would be great ifhttps://www.python.org/dev/peps/pep-0101/  would be improved
from the naive:

  "Write the announcement for the mailing lists.  This is the fuzzy bit because 
not
   much can be automated.  You can use an earlier announcement as a template, 
but
   edit it for content!"

to require the release announcement to record release archives digests as SHA-2 
256
(added point if the announcement is signed), or the armored OpenPGP signatures 
(but's
that a lot of base64 characters).



I would also argue that OpenPGP signatures are a bad solution in 2021. 
PGP has not aged well and GnuPG tool has flaws. Better, more modern 
options like sigstore are still under development, though.


We could (and maybe should) provide a SHA256 tag file (sha256sum --tag) 
and sign it with OpenGPG. The signature of a sha256 checksum file is as 
good as signing the files directly.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FEQAD752SIWTOBMLVOP2JJV3RFPRJBD4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Explicit markers for special C-API situations

[Christian Heimes]

On 10/12/2021 03.08, Jim J. Jewett wrote:

Christian Heimes wrote:

On 09/12/2021 19.26, Petr Viktorin wrote:



If the code is the authoritative source of truth, we need a proper
parser to extract the information.  ... unfortunately I don't trust it
enough to let it define the API. Bugs in the parser could result in
the API definition silently changing.



There are other options than writing a new parser. GCC and Clang are
flexible. For example GCC can be extended with plugins and custom
attributes.


But they have the same problem ... it can be difficult to know if there is a 
subtle bug in someone's understanding of how the plugin interacts with, for 
example, nested ifndef.

The failure mode for an explicitly manually maintained text file is that 
something doesn't get added when it should, and the more conservative API 
consumers wait an extra release before using it.



Macros and ifndefs are not a problem. A GCC plugin for user-defined 
attributes hooks into the build process at a late stage. By the time the 
plugin hook is invoked, the precompiler has resolved all macros and 
ifdefs, and the C code has been parsed. The plugin operates on the same 
intermediate code as the compiler.



The approach would allow us to make the headers the authoritative source 
for most API and ABI symbols. I don't think that we can use it for 
macros. We can even include additional metadata in the custom attribute, 
e.g. version added


  PyAPI_ABI_FUNC("3.2", PyObject *) PyLong_FromLong(long);


We can convert Misc/stable_abi.txt into an auto-generated file. The file 
should still stay in git, so we can use it to verify the stable ABI in CI.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5GSBMLBCXC3THYZYXC5E627CGQELE6JI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Explicit markers for special C-API situations (re: Clarification regarding Stable ABI and _Py_*)

[Christian Heimes]

On 09/12/2021 19.26, Petr Viktorin wrote:

I'll not get back to CPython until Tuesday, but I'll add a quick note
for now. It's a bit blunt for lack of time; please don't be offended.

If the code is the authoritative source of truth, we need a proper
parser to extract the information. But we can't really use an existing
parser (e.g. we need to navigate various #ifdef combinations), and
writing a correct (=tested) custom C parser is pretty expensive. C
declarations being "deterministically discoverable by tools" is a
myth.
I know you wrote a parser (kudos!), but unfortunately I don't trust it
enough to let it define the API. Bugs in the parser could result in
the API definition silently changing.


There are other options than writing a new parser. GCC and Clang are 
flexible. For example GCC can be extended with plugins and custom 
attributes. We could extend the header files with custom attributes and 
then use a plugin to create an ABI file from the attributes.


I created a quick n' hack 
https://github.com/python/cpython/compare/main...tiran:gcc-pythonapi-plugin?expand=1 
as proof of concept.


The plugin takes

   PyAPI_ABI_FUNC(PyObject *) PyLong_FromLong(long);

and dumps the declaration as:

  extern struct PyObject * PyLong_FromLong (long int); "abi_func"

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N5ZTLGPLWHVI5UECVGVND43MBI4FAQN7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Christian Heimes]

On 07/12/2021 19.28, Guido van Rossum wrote:
I assume it would be insensitive to ask whether we could just get rid of 
the stable ABI altogether and focus on the limited API? Just tell 
everyone they have to rebuild binary wheels for every Python feature 
release. Presumably the deprecation of the stable ABI itself would 
require a two-release waiting period. But maybe it would be worth it, 
given how subtle it is to do the historical research about even a single 
function.


The stable ABI is useful for Python packages that ship binary wheels.

Take PyCA cryptography [1] as an example. Alex and Paul already build, 
upload, and ship 12 abi3 wheels for each release and combinations of CPU 
arch, platform, and libc ABI. Without a stable ABI they would have to 
create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The 
number will only increase over time. Python 3.6 is very common on 
LTS/Enterprise Linux distros.


If the current stable ABI makes performance improvements too complex 
then we should consider to define a new stable ABI with less symbols.


Christian

[1] https://pypi.org/project/cryptography/#files
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PQ7GNBTFAHIC6HWNZ62MQJJOV75N7UAT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: compiled python3.10 is unable to find _ssl

[Christian Heimes]

On 20/10/2021 09.43, Robin Becker wrote:

On 19/10/2021 16:45, Christian Heimes wrote:
We use the standard AX_CHECK_OPENSSL() m4 macro from autoconf-archive 
to detect OpenSSL. The macro uses pkg-config to detect OpenSSL. It 
doesn't check for specific version, though. We don't want to prevent 
people with 
outdated OpenSSL or LibreSSL from building Python without ssl support.


Christian
I would not suggest that, but if the test at the end of the configure 
had said it can compile against openssl, but that a wrong or 
insufficient information would prevent building _ssl then a lot of 
cycles could be saved.


I suppose that might be impossible as the test might need to actually 
try and build _ssl.so.


That's an excellent idea!

I can add an autoconf test that checks the most important and newish 
API. The check won't catch all problems, but it will detect most common 
cases of missing APIs in headers and libraries.


PR: https://github.com/python/cpython/pull/29088

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DNXIGO5FNCEQH527UK2SFLHTI6THLVQX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: compiled python3.10 is unable to find _ssl

[Christian Heimes]

On 19/10/2021 17.26, Robin Becker wrote:

On 19/10/2021 11:21, Christian Heimes wrote:

On 19/10/2021 11.57, Robin Becker wrote:

..
For PEP 644 I added new instructions how to build Python 3.10 with 
custom OpenSSL builds. The instructions should work on all major Linux 
distributions. They have been tested on Debian-like and Fedora-like 
platforms:


https://docs.python.org/3/using/unix.html?highlight=openssl#custom-openssl 





Unfortunately I don't have a custom openssl installation although it may 
not appear/behave as the python configuration wants.


I am using the officially approved installed version of openssl so far 
as I know. I'm aware that Ubuntu 18.04 is now somewhat out of date, but 
I would expect a simple configure & make dance to succeed.


I'm working with linux for dummies(self) and need to 
install/update/upgrade openssl, libssl-dev. After that _ssl _hashlib are 
compiled and importable. I suppose the configure 'compiling with 
openssl' test is a bit naive.


We use the standard AX_CHECK_OPENSSL() m4 macro from autoconf-archive to 
detect OpenSSL. The macro uses pkg-config to detect OpenSSL. It doesn't 
check for specific version, though. We don't want to prevent people with 
outdated OpenSSL or LibreSSL from building Python without ssl support.


Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ARM25SOTLX7TU6TMCWLUBQZ5HDQHJQVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: compiled python3.10 is unable to find _ssl

[Christian Heimes]

On 19/10/2021 11.57, Robin Becker wrote:

On 18/10/2021 18:50, Senthil Kumaran wrote:

Your configure script did pick up openssl as the support version was not
found.

What is your operating system? Make sure you have supported version of
ssl. Python requires openssl 1.1.1 or higher.


...
I tried to build this on ubuntu 18.04, but got the _ssl problem; it 
definitely has openssl 1.1.1 installed. On updated Arch linux no issue.


during configure I see that compiling and linking against openssl works.

with the simplest ./configure --prefix=$HOME/LOCAL/3.10.0 I find _ssl is 
not importable (_socket is).


If I try setting --with-openssl-rpath I get _ssl importable, but it 
fails because _socket is not importable.


It's not a big deal as I can install python 3.10.0 using 
deadsnakes-ubuntu-ppa-bionic ppa.


I think the problem here is that I don't seem to have a single openssl root

/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/usr/include/openssl/

This all used to work in Python 3.9.x, but I suppose some improvement(s) 
have been made.


For PEP 644 I added new instructions how to build Python 3.10 with 
custom OpenSSL builds. The instructions should work on all major Linux 
distributions. They have been tested on Debian-like and Fedora-like 
platforms:


https://docs.python.org/3/using/unix.html?highlight=openssl#custom-openssl

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NHTDMNB2W76J43E2JV77RQ37DXWHIGPH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: python3.10 compilation on OpenBSD: running into ssl issues

[Christian Heimes]

On 06/10/2021 09.06, Sandeep Gupta wrote:
Tried with openssl. Some progress but no success.  The configure checks 
went through find.
 >configure:17536: checking for openssl/ssl.h in 
/home/kabira/DrivingRange//project_versa/Build

 >s/openssl-1.1.1l
 >configure:17543: result: yes
 >configure:17559: checking whether compiling and linking against 
OpenSSL works
 >Trying link with 
OPENSSL_LDFLAGS=-L/home/kabira/DrivingRange//project_versa/Builds/openssl-1.
 >1.1l/lib; OPENSSL_LIBS=-lssl -lcrypto; 
OPENSSL_INCLUDES=-I/home/kabira/DrivingRange//project_

 >versa/Builds/openssl-1.1.1l/include

But for some reason module could not be imported. I could find any 
errors related to import.

There were no compilation errors:

 >Following modules built successfully but were removed because they 
could not be imported:

 >_hashlib              _ssl                  readline


[...]

*** WARNING: renaming "_ssl" since importing it failed: Cannot load 
specified object
*** WARNING: renaming "_hashlib" since importing it failed: Cannot load 
specified object


OpenBSD uses clang C compiler. The new --with-openssl-rpath=auto option 
was only tested with GCC. It turned out that Python's distutils package 
didn't support rpath with clang. The bugfix 
https://bugs.python.org/issue45371 will be available in upcoming release 
Python 3.10.1.


In the mean time you either need to apply the patch from the issue 
manually or figure out the right environment variables to add correct 
rpath yourself.


I'm sorry for the inconvenience. We don't have any CI for OpenBSD. 
Apparently this feature was never tested on OpenBSD during the release 
candidate phase either.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RAKDWHTH7TMUYEWJJFNOPLKLBBDXZBED/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: python3.10 compilation on OpenBSD: running into ssl issues

[Christian Heimes]

On 05/10/2021 16.40, Sandeep Gupta wrote:
Trying to compile python3.10 on openbsd 7.0 on Pi4. It seems to run into 
several openssl issue.  I have installed openssl as I couldn't find 
libreSSL in the package manager.


Your installation is picking up header files from LibreSSL. Python 3.10 
requires a fully OpenSSL 1.1.1 API compliant OpenSSL. LibreSSL is 
missing a bunch of required features.


Please follow the instruction 
https://docs.python.org/3.10/using/unix.html#custom-openssl to compile, 
install, and use a custom installation of OpenSSL. I recommend that you 
use the latest OpenSSL 1.1.1 version.


Regards,
Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BK22LJ5LICXUMIU6MYB746FSZ57KJX2T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Christian Heimes]

On 27/09/2021 16.32, Ronald Oussoren via Python-Dev wrote:



On 26 Sep 2021, at 19:03, Christian Heimes <mailto:christ...@python.org>> wrote:


On 26/09/2021 13.07, jack.jan...@cwi.nl <mailto:jack.jan...@cwi.nl> wrote:
The problem with the stable ABI is that very few developers are 
targeting it. I’m not sure why not, whether it has to do with 
incompleteness of the ABI, or with issues targeting it easily and 
your builds and then having pip/PyPI do the right things with wheels 
and all that. I’ve been on the capi-sig mailing list since its 
inception in 2007, but the discussions are really going over my head. 
I don’t understand what the problems are that keep people from 
targeting the stable ABI (or the various other attempts at 
standardising extensions over Python versions).


It takes some effort to port old extensions to stable ABI. Several old 
APIs are not supported in stable ABI extensions. For example 
developers have to port static type definitions to heap types. It's 
not complicated, but it takes some effort.


The stable ABI is also not complete, although it should be complete 
enough for a lot of projects.  A, fairly esoteric, issue I ran into is 
that it is currently not possible to define a class with a non-default 
meta class using the type-spec API (AFAIK), see #15870.


And as you write “it takes some effort”, that alone likely reduces the 
amount of projects that migrate to the stable ABI esp. for projects that 
already have a CI/CD setup that creates binary wheels for you (for 
example using cibuildwheel).


Indeed, the stable ABI is not complete. I just figured out that limited 
API < 3.9 cannot define objects with weakref support or dict offset. The 
__weaklistoffset__ and  __dictoffset__ PyMemberDefs were added in Python 
3.9.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G7YTJ62XB4QJJY43RDZTUFIYBQ5P5W57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Christian Heimes]

On 26/09/2021 13.07, jack.jan...@cwi.nl wrote:
The problem with the stable ABI is that very few developers are 
targeting it. I’m not sure why not, whether it has to do with 
incompleteness of the ABI, or with issues targeting it easily and your 
builds and then having pip/PyPI do the right things with wheels and all 
that. I’ve been on the capi-sig mailing list since its inception in 
2007, but the discussions are really going over my head. I don’t 
understand what the problems are that keep people from targeting the 
stable ABI (or the various other attempts at standardising extensions 
over Python versions).


It takes some effort to port old extensions to stable ABI. Several old 
APIs are not supported in stable ABI extensions. For example developers 
have to port static type definitions to heap types. It's not 
complicated, but it takes some effort.


The other issue is Cython. Stable releases of Cython do not support 
stable ABI yet. It's an experimental feature in Cython 3.0.0 alpha.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O33XTH37NGAFHHPNV5653JVBCC4NXRHE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why list.sort() uses mergesort and not timsort?

[Christian Heimes]

On 06/06/2021 11.42, Marco Sulla wrote:
> As title. Is it faster for inplace sorting, or simply the
> implementation of list.sort() was done before the implementation of
> timsort?

list.sort() uses timsort. What makes you think that Python uses mergesort?

Tim Peters invented timsort for Python about twenty years ago. Tim a
first generation Python core dev. Other languages like Java adopted
timsort from Python later.

Christian



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P7XFRAU6GL2CH6RBXPLGXEK4AL5KF2HV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] IMPORTANT: Python 3.10b2 release blockers

[Christian Heimes]

On 25/05/2021 00.45, Pablo Galindo Salgado wrote:
> Hi,
> 
> Tomorrow is the scheduled release of Python 3.10 beta 2 but
> unfortunately we have several release blockers:
>
> https://bugs.python.org/issue44043 :
> 3.10 b1 armhf Bus Error in hashlib test: test_gil

The problem is already fixed. I forgot to close the release blocker bug
after Greg and I took care of https://bugs.python.org/issue36515.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GSGRKOGDZICAX7RJRBWNVKHFQAE5EWVQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: March Steering Council update.

[Christian Heimes]

On 18/05/2021 16.19, Guido van Rossum wrote:
> There are a few mentions of Debian, but no explanation of what the issue
> is about. Can you elaborate on that?

Debian and Debian-based distros like Ubuntu are applying downstream
packages and split CPython interpreter and stdlib into multiple
packages. For example venv and distutils is split out and ensurepip is
patched out. Debian packages were also missing dependencies on
ca-certificates and tzdata, which are required for Python's ssl and
zoneinfo modules. The changes caused usability issues for new and
sometimes even for seasoned users of Python.

I have contacted the SC in the beginning of this year and asked them to
work with Debian maintainers. Some issues have been addressed and will
be available in future releases. Matthias' talk at the language summit
is related to the effort of improving Debian packaging.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VV5IVOSIKWV5END64PXNAFEQ2SNGP3VS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer

[Christian Heimes]

On 30/03/2021 13.46, Pablo Galindo Salgado wrote:
> Hi Christian,
> 
> Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating
> the situation and discussing the PEP, the Steering Council is happy with
> the PEP,
> and hereby accepts it. The SC is of the opinion that this change will
> make it considerable
> easier to maintain the extension modules that depend on OpenSSL while
> allowing us
> to leverage the new APIs and improvements in the future. As indicated in
> the PEP,
> OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost
> all supported
> platforms and distributions almost all known Major CI providers provide
> images with
> OpenSSL 1.1.1, so we believe this has the right balance to improve the
> situation without
> causing major impact or breakage.
> 
> Nevertheless, the Steering Council would like to see this change
> reflected properly in the
> documentation, What's New (linking against the new instructions here: 
> https://docs.python.org/3.10/using/unix.html#custom-openssl
> ) and
> releasemanager notices
> so this is properly communicated to our users. 
> 
> The Steering Council also thanks Christian for his patience explaining
> different aspects
> of the PEP, including in the PEP some extra requested information and
> for considering
> the concerts raised.
> 
> Congratulations, Christian!
> 
> With thanks from the whole Python Steering Council,
> Pablo Galindo Salgado

Hi Pablo,

I have implemented PEP 644 in https://bugs.python.org/issue43669 and
marked the PEP as final.

Thanks!

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4BXOPVK5P3QIHTMFANKG2FLGI2U5OU4C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Christian Heimes]

On 16/04/2021 19.14, redrad...@gmail.com wrote:
> My personal stop of contributing in CPython is that it is written in pure C !!
> I wrote code in both: pure C and C++, but I like writing code in C++, because 
> it simplifies things without losing perfomance

There are plenty of Open Source projects that could use more capable C++
developers. :)

I'm not a fan of C++. It has its use cases, e.g. in UI. Python core
isn't the best fit. AFAIK most core devs are not fluent in C++. Despite
it's name, C++ is really a different language than C. It has a different
ABI and stdlib than C, too. In my personal opinion C++ won't give us any
net benefits. I'd much rather go for Rust than C++ to gain memory safety.

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2HTFWK3JXOTCENWOVW6TYAZPIBP3HR76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Christian Heimes]

On 16/04/2021 19.39, Victor Stinner wrote:
> A *fresh* build (after make clean) of CPython on my laptop (8 threads,
> 4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh
> build in release mode (make -j10 and gcc -O3) takes 44 seconds on the
> same laptop.

$ make clean
$ time make -j10
...
real0m2,072s
user0m4,715s
sys 0m2,333s

./configure -C and ccache are fantastic.

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LLDVGXNH32KDVVTT7NXIG7UOXHMW4WXB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup

[Christian Heimes]

On 30/03/2021 19.01, Barry Warsaw wrote:
> Hello Mario,
> 
> Thank you for your submission of PEP 648 (Extensible customizations of the 
> interpreter at startup).  The Python Steering Council has reviewed the PEP 
> and before we can pronounce on it, we have some additional questions and 
> comments we’d like you to address.  Once these questions are settled, we are 
> requesting that you post the PEP to python-dev for another round of comments.

Hi Mario,

could you please include a security analysis of the feature, too? I
would like to avoid new ways to exploit Python.

In particular I don't think that -S (no site module) is the right way to
disable __sitecustomize__. It disables too much useful features. It
might be a good idea to disable __sitecustomize__ with -I (isolated mode).

There should be a new audit event, too.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UFADYZGCTF6ZJUX66VAK4XYMJBL6TMMW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] OpenSSL 1.1.1k CVE fixes

[Christian Heimes]

Hi,

OpenSSL released 1.1.1k today with two high severity CVEs,
https://www.openssl.org/news/vulnerabilities.html


The ssl module is not affected by CVE-2021-3450 in its default
configuration. Python does not set X509_V_FLAG_X509_STRICT on
SSLContext. Only applications that that use ssl.VERIFY_X509_STRICT
verify flag are affected.

It looks like Python's ssl module is vulnerable to CVE-2021-3449. The
crash does not affect pip, requests, or any other client-side socket.
Only server-side SSL/TLS sockets are vulnerable (ssl.PROTOCOL_TLS_SERVER
and server_side=True).


I haven't had time to reproduce and verify any of the CVE bugs yet. That
means I'm not entirely sure how the CVEs affect CPython. I strongly
recommend that you update OpenSSL through your vendor and restart your
services. If you cannot update OpenSSL (e.g. for Python.org installers),
then you can apply workarounds:


To disable X509_V_FLAG_X509_STRICT flag either remove any lines that set
the flag or unset the flag with:

ctx.verify_flags &= ~ssl.VERIFY_X509_STRICT

(That's bitwise AND and unary bitwise invert operator)


To work around CVE-2021-3449 either set disable TLS 1.0, 1,1, and 1.2 with

ctx.minimum_version = ssl.TLSVersion.TLSv1_3

or disable renegotiation with

ctx.options |= ssl.OP_NO_RENEGOTIATION

NOTE: Renegotiation is required for TLS 1.2 rekeying, optional TLS
client cert authention with TLS 1.2 and possible other features. TLS 1.3
is not supported by older clients and servers.


Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2GULUR43MNEW3IJM44LS5ZY2TOUANPNT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Christian Heimes]

On 25/03/2021 18.39, Antoine Pitrou wrote:
> On Thu, 25 Mar 2021 20:22:55 +0300
> Ivan Pozdeev via Python-Dev  wrote:
>> On 24.03.2021 19:58, Antoine Pitrou wrote:
>>> On Wed, 24 Mar 2021 19:45:49 +0300
>>> Ivan Pozdeev via Python-Dev  wrote:  
 How does C++ fare in binary compatibility? Last time I checked it out 
 (about 10 years ago), there was completely none, every compiler's ABI
 was a black box without any guarantees whatsoever.
 For any software that's going to dynamically link and exchange binary 
 types with other independently produced software, that's a deal breaker.  
>>> That depends if you use C++ internally or expose C++ bits in the public
>>> API.
>>> If you only use C++ internally, binary compatibility is presumably less
>>> of an issue.  
>>
>> Python produces and accepts its internal types in API calls and allows 
>> extension modules to derive from them -- so it cannot "only use C++ 
>> internally" if those are going to become C++ types. (And if not, the point 
>> of using C++ is unclear.)
> 
> You can use C++ without exposing C++ types in the API.  For example,
> C++ templates could improve the maintainability of the generic
> "stringlib" routines that are currently based on C macros.  Another
> example is using RAII in function bodies to help cleanup owned
> references.

C has ways to handle cleanup and ownership better -- at least some
compilers do. For example gcc has
__attribute__(cleanup(cleanup_function)) for automatic cleanup on scope
boundaries. If I recall correctly it works something like this:

void
Py_auto_decref(PyObject **o)
{
if (!o || !*o)
return;

Py_DECREF(*o);
*o = NULL;
}


PyObject *
func(PyObject self)
{
PyObject *spam  __attribute__((cleanup(Py_auto_decref)));
...
Py_RETURN_NONE;
// spam gets automatically decrefed
}

It's too bad that the feature is not universally available in C99.

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JGONDY5TRVTSCRMDUPPQH6OMEOAQ37Y2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council update for February

[Christian Heimes]

On 11/03/2021 00.38, Mike Miller wrote:
> 
> On 2021-03-10 13:45, David Mertz wrote:
>> In contrast, the "master" used in version control directly borrows
>> from so-called "master/slave network architecture."
> 
> 
> It was shown upthread that this isn't the case.  Do you have more
> accurate documentation to refute the claim?
> 
> - https://twitter.com/xpasky/status/1272280760280637441
> - https://en.wikipedia.org/wiki/Mastering_(audio)

Petr Baudis (xpasky) also wrote:

---
(But as noted in a separate thread, it is possible it stems from
bitkeeper's master/slave terminology. I hoped to do some historical
research but health emergency in my family delayed that.)

Regardless, the impression words form in the reader is more important
than their intent.
---

https://twitter.com/xpasky/status/1272817929046962176

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S7PXQS7OIE245MGFIIMCMIQNFBJWYLOI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council update for February

[Christian Heimes]

On 10/03/2021 10.30, Antoine Pitrou wrote:
> On Wed, 10 Mar 2021 10:30:43 +0900
> Inada Naoki  wrote:
>> On Wed, Mar 10, 2021 at 10:10 AM Ivan Pozdeev via Python-Dev
>>  wrote:
>>>
>>> On 10.03.2021 3:53, Chris Angelico wrote:  
 On Wed, Mar 10, 2021 at 11:47 AM Damian Shaw
  wrote:  
>> Does 'master' confuse people?  
> There's a general movement to replace language from common programming 
> practises that derive from, or are associated with, the dehumanization of 
> people. Such as master and slave, as well as whitelist and blacklist.
>  
 Is that *actually* the origin of the term in this context, or is it
 the "master", the pristine, the original from which copies are made?
 There's no "slave" branch anywhere in the git repository.  
>>>
>>> It is, actually, the ultimate origin of the term.
>>>
>>> A more immediate origin is the master-slave architecture (the master agent 
>>> initiates some operation and slave agents respond to it and/or
>>> carry it out).
>>>  
>>
>> Petr Baudis (who named "master" branch) says its origin is "master
>> recording". So it is unrelated to master-slave.
>> https://twitter.com/xpasky/status/1272280760280637441
> 
> And the origin of the English word is the latin noun "magister":
> https://en.wiktionary.org/wiki/magister#Latin

Words change meaning.

For example take the word "dumb". It used to mean "mute" or
"speechless". Ableism turned its meaning into "stupid" /
"nonintellectual" as people equated speaking/hearing disability with
mental disability.

Fun fact: German language adopted the new meaning while Dutch language
kept the original meaning of the word. In German phrase "Bist du doof?"
translates to "Are you dumb?" while the Dutch sentence "Ben je doof"
means "Are you deaf?". I grew up close to the border of the Netherlands
and had a friend who's little brother was deaf. This caused some
confusing on my side when he told a Dutch person "mijn broer is doof".



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R7LMRZLOHEBFVQWV3CLJOGY4T3VQIZ3C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council update for February

[Christian Heimes]

On 10/03/2021 01.53, Chris Angelico wrote:
> On Wed, Mar 10, 2021 at 11:47 AM Damian Shaw
>  wrote:
>>
>>> Does 'master' confuse people?
>>
>> There's a general movement to replace language from common programming 
>> practises that derive from, or are associated with, the dehumanization of 
>> people. Such as master and slave, as well as whitelist and blacklist.
>>
> 
> Is that *actually* the origin of the term in this context, or is it
> the "master", the pristine, the original from which copies are made?
> There's no "slave" branch anywhere in the git repository.

There is some historical evidence that the developers of git adopted the
term from Bitkeeper. After all git was created when Bitkeeper changed
its pricing model back in mid of 2000s. Bitkeeper uses master and slave
terminology,
https://github.com/bitkeeper-scm/bitkeeper/blob/master/doc/HOWTO.ask#L232

> We are then going to modify the file on both
> the master and slave repository and then merge the work. For the sake
> of simplicity, we are doing work in the master repository.

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VZEIRCSPYKMBR2UQFHH7XVZBKWZMTEYH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Need help to fix known Python security vulnerabilities

[Christian Heimes]

On 08/03/2021 22.02, Victor Stinner wrote:

Thanks Victor!

> == XML ==
> 
> Python XML parsers have at least two known vulnerabilities: "billion
> laughs" and "quadratic blowup" which are documented:
> https://docs.python.org/dev/library/xml.html#xml-vulnerabilities
> 
> The third party defusedxml module address these vulnerabilities:
> https://pypi.org/project/defusedxml/
> 
> But Python remains unsafe by default, issue reported 8 years ago:
> https://bugs.python.org/issue17239

I still maintain defusedxml and just released a new version earlier this
week. A couple of years ago I also worked on fixing libexpat (the parser
used by Python's stdlib), https://github.com/libexpat/libexpat/issues/46
. To move forward somebody could finish my patch for libexpat and then
hook it up in Python's stdlib.

I have no interest to work on the matter. My days of XML processing are
long gone. Fixing it for "fame and glory" doesn't motivate me either.

Christian



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6WL35XOFMNKDVMRSTQHQQTLOSBBMIARR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 20.03, Christian Heimes wrote:
> On 24/02/2021 19.17, Steve Dower wrote:
>> On 2/24/2021 4:26 PM, Christian Heimes wrote:
>>> On 24/02/2021 15.16, Random832 wrote:
>>>> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>>>>> Separate directories don't prevent clashes and system breakage. But
>>>>> they
>>>>> provide an easy way to *recover* from a broken system.
>>>>
>>>> I think it could be turned into a way to prevent them by A) having
>>>> site-packages always take precedence over dist-packages [i believe
>>>> this is already the case] in normal usage and B) providing an option
>>>> to the interpreter, used by system scripts, to exclude site-packages
>>>> entirely from the path.
>>>>
>>>> Basically, site-packages would effectively be layered on top of "Lib
>>>> + dist-packages" in a similar way to how a venv is layered on top of
>>>> the main python installation - the inverse of the suggestion someone
>>>> else in the thread made for the system python to be a venv. This
>>>> wouldn't *exactly* be a venv because it wouldn't imply the other
>>>> things that entering a venv does such as "python" [and script names
>>>> such as pip] being an alias for the correct version of python, but it
>>>> would provide the same kind of one-way isolation, whereby the "system
>>>> environment" can influence the "normal environment" and not
>>>> vice-versa, in the same way that packages installed in the main
>>>> installation affect a venv [unless system-site-packages is disabled]
>>>> but the venv obviously has no effect on the main installati
>> on.
>>>
>>> Yes, you are describing one major aspect of my idea for a system Python
>>> interpreter. I'm happy to read that other users are coming to similar
>>> conclusions. Instead of an option I'd create a new executable to lock
>>> down additional things (e.g. isolated mode, code verification hook). A
>>> separate executable would also allow distros to provide a stripped down
>>> interpreter that does not cause bad user experience.
>>
>> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
>> option and PYTHONNOUSERSITE env variable to provide that one-way
>> isolation).
>>
>> Is the problem that pip doesn't use it by default? Or that the distros
>> went and made patches for the runtime rather than patching pip? (For
>> Windows installs from the Store, where even admin rights can't do an
>> all-users package install, I added a new config file location for pip to
>> change this default, but a patch would also have worked.)
>>
>> Maybe we need an easier way to patch the location of user site packages?
>> I also had to do this for the Store install on Windows, and it's a
>> little bit of a hack... but maybe having an official recommendation
>> would help encourage distributors to use the mechanism?
> 
> (sorry for terse reply, I'm heading out.)
> 
> Linux distros want an additional layer so the can differentiate between
> Python packages installed by their package manager and Python packages
> installed with "sudo pip install".
> 
> Python has two default site-package directories (from highest to lowest
> precedence):
> 
> 1. ~/.local/lib/python3.9/site-packages
> 2. /usr/lib/python3.9/site-packages
> 
> (1) is defined by site.getusersitepackages(), (2) site.getsitepackages()
> 
> Linux distro have additional site-package directories. My Fedora system
> X86_64 system with multiarch has three additional candidates.
> 
> 1. ~/.local/lib/python3.9/site-packages
> 2. /usr/local/lib64/python3.9/site-packages
> 3. /usr/local/lib/python3.9/site-packages
> 4. /usr/lib64/python3.9/site-packages
> 5. /usr/lib/python3.9/site-packages
> 
> The "lib" directories are for pure Python packages whiel the "lib64"
> directories contain X86_64 native code extensions (aka shared
> libraries). The "/usr/lib*" directories are used for distro packages.
> The downstream patch [1] ensures that "sudo pip install" installs
> packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.
> 
> Debian has a similar mechanism to provide
> "/usr/lib/python3/dist-packages" [2].
> 
> A hypothetical /usr/bin/system-python3 interpreter would only use (4)
> and (5) as site-package directories. A user-facing /usr/bin/python3
> would use (1) to (5).
> 
> Christian
> 
> [1]
&g

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 19.17, Steve Dower wrote:
> On 2/24/2021 4:26 PM, Christian Heimes wrote:
>> On 24/02/2021 15.16, Random832 wrote:
>>> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>>>> Separate directories don't prevent clashes and system breakage. But
>>>> they
>>>> provide an easy way to *recover* from a broken system.
>>>
>>> I think it could be turned into a way to prevent them by A) having
>>> site-packages always take precedence over dist-packages [i believe
>>> this is already the case] in normal usage and B) providing an option
>>> to the interpreter, used by system scripts, to exclude site-packages
>>> entirely from the path.
>>>
>>> Basically, site-packages would effectively be layered on top of "Lib
>>> + dist-packages" in a similar way to how a venv is layered on top of
>>> the main python installation - the inverse of the suggestion someone
>>> else in the thread made for the system python to be a venv. This
>>> wouldn't *exactly* be a venv because it wouldn't imply the other
>>> things that entering a venv does such as "python" [and script names
>>> such as pip] being an alias for the correct version of python, but it
>>> would provide the same kind of one-way isolation, whereby the "system
>>> environment" can influence the "normal environment" and not
>>> vice-versa, in the same way that packages installed in the main
>>> installation affect a venv [unless system-site-packages is disabled]
>>> but the venv obviously has no effect on the main installati
> on.
>>
>> Yes, you are describing one major aspect of my idea for a system Python
>> interpreter. I'm happy to read that other users are coming to similar
>> conclusions. Instead of an option I'd create a new executable to lock
>> down additional things (e.g. isolated mode, code verification hook). A
>> separate executable would also allow distros to provide a stripped down
>> interpreter that does not cause bad user experience.
> 
> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
> option and PYTHONNOUSERSITE env variable to provide that one-way
> isolation).
> 
> Is the problem that pip doesn't use it by default? Or that the distros
> went and made patches for the runtime rather than patching pip? (For
> Windows installs from the Store, where even admin rights can't do an
> all-users package install, I added a new config file location for pip to
> change this default, but a patch would also have worked.)
> 
> Maybe we need an easier way to patch the location of user site packages?
> I also had to do this for the Store install on Windows, and it's a
> little bit of a hack... but maybe having an official recommendation
> would help encourage distributors to use the mechanism?

(sorry for terse reply, I'm heading out.)

Linux distros want an additional layer so the can differentiate between
Python packages installed by their package manager and Python packages
installed with "sudo pip install".

Python has two default site-package directories (from highest to lowest
precedence):

1. ~/.local/lib/python3.9/site-packages
2. /usr/lib/python3.9/site-packages

(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()

Linux distro have additional site-package directories. My Fedora system
X86_64 system with multiarch has three additional candidates.

1. ~/.local/lib/python3.9/site-packages
2. /usr/local/lib64/python3.9/site-packages
3. /usr/local/lib/python3.9/site-packages
4. /usr/lib64/python3.9/site-packages
5. /usr/lib/python3.9/site-packages

The "lib" directories are for pure Python packages whiel the "lib64"
directories contain X86_64 native code extensions (aka shared
libraries). The "/usr/lib*" directories are used for distro packages.
The downstream patch [1] ensures that "sudo pip install" installs
packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.

Debian has a similar mechanism to provide
"/usr/lib/python3/dist-packages" [2].

A hypothetical /usr/bin/system-python3 interpreter would only use (4)
and (5) as site-package directories. A user-facing /usr/bin/python3
would use (1) to (5).

Christian

[1]
https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-user-install-location.patch
[2]
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RZACFNIDET7FZJG6LC25EAPWCR7ADFUE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 15.16, Random832 wrote:
> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>> Separate directories don't prevent clashes and system breakage. But they
>> provide an easy way to *recover* from a broken system.
> 
> I think it could be turned into a way to prevent them by A) having 
> site-packages always take precedence over dist-packages [i believe this is 
> already the case] in normal usage and B) providing an option to the 
> interpreter, used by system scripts, to exclude site-packages entirely from 
> the path.
> 
> Basically, site-packages would effectively be layered on top of "Lib + 
> dist-packages" in a similar way to how a venv is layered on top of the main 
> python installation - the inverse of the suggestion someone else in the 
> thread made for the system python to be a venv. This wouldn't *exactly* be a 
> venv because it wouldn't imply the other things that entering a venv does 
> such as "python" [and script names such as pip] being an alias for the 
> correct version of python, but it would provide the same kind of one-way 
> isolation, whereby the "system environment" can influence the "normal 
> environment" and not vice-versa, in the same way that packages installed in 
> the main installation affect a venv [unless system-site-packages is disabled] 
> but the venv obviously has no effect on the main installation.

Yes, you are describing one major aspect of my idea for a system Python
interpreter. I'm happy to read that other users are coming to similar
conclusions. Instead of an option I'd create a new executable to lock
down additional things (e.g. isolated mode, code verification hook). A
separate executable would also allow distros to provide a stripped down
interpreter that does not cause bad user experience.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F5NCGF7EWYDMBAHPNDFNWWKSE2GRKHLA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 11.52, Stéfane Fermigier wrote:
> I love pipx and I'm glad it exists at this point because it make 
> 
> The main issue is that each virtualenv takes space, lots of space.
> 
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.
> 
>  ~  cd .local/pipx/venvs/
>  ~/.l/p/venvs  ls
> abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/
> ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/
> assertize/      cruft/          isort/          pylint/         tlv/
> autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/
> black/          dephell/        lektor/         pytype/         tox/
> borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/
> borgmatic/      flake8/         medikit/        radon/          virtualenv/
> bpytop/         flit/           mypy/           re-ver/         virtualfish/
> check-manifest/ flynt/          nox/            sailboat/       vulture/
> clone-github/   gh-clone/       pdoc3/          salvo/
> cloneall/       ghtop/          pdocs/          shed/
> com2ann/        gitchangelog/   pybetter/       sixer/
>  ~/.l/p/venvs  du -sh .
> 990M.
>  ~/.l/p/venvs  ls | wc
>       57      57     475
> 
> There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations. 

There are tools like https://rdfind.pauldreik.se/rdfind.1.html that
create hard links to deduplicate files. Some files systems have
deduplicated baked in, too.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CO6GV2CRDKBJMLE7DZVVQ4AMIPSKPMCJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 08.03, Michał Górny wrote:
> On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
>> I was reading a discussion thread 
>> <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about 
>> various issues with the Debian packaged version of Python, and the following 
>> statement stood out for me as shocking:
>>
>> Christian Heimes wrote:
>>> Core dev and PyPA has spent a lot of effort in promoting venv because we 
>>> don't want users to break their operating system with sudo pip install.
>>
>> I don't think sudo pip install should break the operating system. And I 
>> think if it does, that problem should be solved rather than merely advising 
>> users against using it. And why is it, anyway, that distributions whose 
>> package managers can't coexist with pip-installed packages don't ever seem 
>> to get the same amount of flak for "damaging python's brand" as Debian is 
>> getting from some of the people in the discussion thread? Why is it that 
>> this community is resigned to recommending a workaround when distributions 
>> decide the site-packages directory belongs to their package manager rather 
>> than pip, instead of bringing the same amount of fiery condemnation of that 
>> practice as we apparently have for *checks notes* splitting parts of the 
>> stdlib into optional packages? Why demand that pip be present if we're not 
>> going to demand that it works properly?
>>
>> I think that installing packages into the actual python installation, both 
>> via distribution packaging tools and pip [and using both simultaneously - 
>> the Debian model of separated dist-packages and site-packages folders seems 
>> like a reasonable solution to this problem] can and should be a supported 
>> paradigm, and that virtual environments [or more extreme measures such as 
>> shipping an entire python installation as part of an application's 
>> deployment] should ideally be reserved for the rare corner cases where that 
>> doesn't work for some reason.
> 
> The problem is a little deeper and the Debian solution doesn't really
> solve all of it.
> 
> Yes, pip installing into the same directory as the package manager
> is a problem.  It's a problem to the point that I'm patching pip
> in Gentoo to explicitly block that.  We've gotten far too many bug
> reports about people's systems suddenly being horribly broken after they
> used pip.
> 
> While using two directories can prevent pip from directly overwriting
> system packages, you still can't expect two independent package managers
> to work simultaneously unless they can communicate with each other to
> prevent conflicts.  If pip installs a different version of the same
> package as the package manager, which one is supposed to be used? 
> Whichever choice you make, you'll bound to eventually break dependency
> graph of some package.

Thanks Michał, you have explained the situation well.

Most Linux distros have two separate directories for distro packages and
"sudo pip install" packages. This is a great thing!

Separate directories don't prevent clashes and system breakage. But they
provide an easy way to *recover* from a broken system. The package
manager of several Linux distributions is written in Python. If your
system Python is broken, your package manager is likely busted as well.
The split directories allow ysers to fix their system with a single
command like "sudo rm -rf /usr/local/lib/python*" (paths may vary).

The issue is: Split distro/pip directories are not baked into Python or
pip. Distros have their own set of downstream patches that modify the
behavior of Python. Some of the patches have caused issues in the past.

I would like to see this feature implemented in Python's core, so distro
packages have to do less patching and upstream pip can test against it.

Christian

PS: I wrote PEP 370 for "install --user" 13 years ago before we had
virtual envs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZYXP5FARQIYQQTW46UFWVAXQRTFNFRFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Christian Heimes]

On 21/02/2021 13.47, glaub...@debian.org wrote:
> Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. 
> There is no separate configure guard. All platforms that Rust can build for, 
> are always enabled by default. No one in Rust keeps anyone from 
> cross-compiling code for sparc64 or powerpcspe, for example.
> 
> So if you want to copy Rust's mechanism, you should just leave it as is and 
> not claim that users are being confused because "m68k" shows up in 
> configure.ac.

A --enable-unstable-platforms configure flag is my peace offer to meet
you half way. You get a simple way to enable builds on untested
platforms and we can clearly communicate that some OS and hardware
platforms are not supported.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W3L2RISXFKHRWPYQB232XH7PDIOPKNDY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Christian Heimes]

On 21/02/2021 13.13, Victor Stinner wrote:
> Hi,
> 
> I propose to actively remove support for *legacy* platforms and
> architectures which are not supported by Python according to PEP 11
> rules: hardware no longer sold and end-of-life operating systems. The
> removal should be discussed on a case by case basis, but I would like
> to get an agreement on the overall idea first. Hobbyists wanting to
> support these platforms/archs can continue to support them with
> patches maintained outside Python. For example, I consider that the
> 16-bit m68k architecture is legacy, whereas the OpenBSD platform is
> still actively maintained.
[...]
> Python has different kinds of platform and architecture supports. In
> practice, I would say that we have:
> 
> * (1) Fully supported. Platform/architecture used by core developers
> and have at least one working buildbot worker: fully supported. Since
> core developers want to use Python on their machine, they fix issues
> as soon as they notice them. Examples: x86-64 on Linux, Windows and
> macOS.
> 
> * (2) Best effort. Platform/architecture which has a buildbot worker
> usually not used by core developers. Regressions (buildbot failures)
> are reported to bugs.python.org, if someone investigates and provides
> a fix, the fix is merged. But there is usually no "proactive" work to
> ensure that Python works "perfectly" on these platforms. Example:
> FreeBSD/x86-64.
> 
> * (3) Not (officially) supported. We enter the blurry grey area. There
> is no buildbot worker, no core dev use it, but Python contains code
> specific to these platforms/architectures. Example: 16-bit m68k and
> 31-bit s390 architectures, OpenBSD.
> 
> The Rust programming language has 3 categories of Platform Support,
> the last one is :

Thanks Victor!

(short reply, I'm heading out)

I'm +1 in general for your proposal. I also like the idea to adopt
Rust's platform support definition.

For 3.10 I propose to add a configure option to guard builds on
unsupported / unstable platforms. My draft PR
https://github.com/python/cpython/pull/24610/commits/f8d2d56757a9cec7ae4dc721047336eaba097125
implements a checker for unsupported platforms and adds a
--enable-unstable-platforms flag. Configuration on unsupported platforms
fails unless users explicitly opt-in.

The checker serves two purposes:

1) It gives users an opportunity to provide full PEP 11 support
(buildbot, engineering time) for a platform.
2) It gives us the leverage to remove a flagged platform in the future
or refuse support on BPO.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKSE43ASVGV4DURGOEWVTQIJJM4EBOEI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Christian Heimes]

On 19/02/2021 23.22, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases
> are too hard, and there’s a risk of not having enough volunteers as a
> result.
> 
> How hard is it to fix that? 

Actually it's easy to fix!

The PSF needs needs sufficient money to hire a couple of people, so the
PSF can turn release management and security maintenance from unpaid
volunteers into paid fulltime jobs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4INCHKPOGKMTDQ4IDS4RDR33KJ4RR2H5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 624: Remove Py_UNICODE encoder APIs

[Christian Heimes]

On 01/02/2021 17.39, M.-A. Lemburg wrote:
>> Can you explain where wchar_t* type is appropriate and how two
>> conversions is a performance bottleneck?
> 
> If an extension has a wchar_t* string, it should be easy
> to convert this in to a Python bytes object for use in Python.

How much software actually uses wchar_t these days and interfaces with
Python? Do you have examples for software that uses wchar_t and would
benefit from wchar_t support in Python?

I did a quick search for wcslen in all shared libraries and binaries on
my system. It's a good indicator how many programs actually use wchar_t.
126 out of more than 9,000 shared libraries and binaries contain the
string "wcslen". The only hit for PyUnicode_AsWideCharString was
libpypy3-c.so...

(Fedora has unified /usr and /lib64, e.g. /bin -> /usr/bin)

$ ls /usr/bin/ /usr/sbin/ | grep -v python | wc -l
4264
$ grep -R wcslen /usr/bin/ /usr/sbin/ | grep -v python | wc -l
92

$ find /usr/lib64/ -name '*.so' -not -name '*python*' | wc -l
5478
$ find /usr/lib64/ -name '*.so' -not -name '*python*' | xargs grep
wcslen | wc -l
34

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M6JC5XCXL4ENTMTFR7SUKM7PDQO5KZPT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Upcoming 3.7.10 and 3.6.13 Security Releases

[Christian Heimes]

On 20/01/2021 13.06, Miro Hrončok wrote:
> On 10. 01. 21 21:15, Ned Deily wrote:
>> We are planning to produce the next security-fix rollup releases for
>> Python 3.7.x and 3.6.x on 2021-01-15. The most recent releases for
>> these versions were on 2020-08-17.  There has not been a lot of
>> activity for either branch since then.
>>
>> Core developers: if you know of any additional security issues that
>> should be addressed in these releases, please mark the relevant bpo
>> issues as "release blocker" and, if possible, submit PRs for review
>> prior to the end of 2021-01-14 AOE.
> 
> Hi Ned. I am not a core developer, but can
> https://bugs.python.org/issue42938 please be included? I know it is past
> the deadline, but I have not seen the releases.

Ned has postponed the upcoming security releases for the issue.
Benjamin's fix has landed two days ago. The fixes will be included in
3.7.10 and 3.6.13.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EMSDC5PS5EU2W77SRJAON4BOHNCD7N76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Where is the SQLite module maintainer

[Christian Heimes]

On 27/12/2020 21.20, Erlend Aasland wrote:
> Hi, everyone.
> 
> I'm trying to find a reviewer for this trivial
> PR: https://github.com/python/cpython/pull/20530
> 
> (The PR fixes /CheckTraceCallbackContent/ (in the sqlite3 test suite)
> for SQLite pre 3.7.15.)
> 
> 
> I've given up hunting for alternative reviewers (a process I find very
> uncomfortable, since I feel I'm just bugging people who have too much to
> do with things they're not interested in), so as a last resort, I'm
> trying the mailing list:
> 
> Who can help me review code that touches the sqlite3 module code base?
> 
> 
> I've received a lot of constructive reviews from Victor Stinner, Dong-he
> Na and Pablo Galindo on my past sqlite3 PR's; thank you so much! I still
> feel uncomfortable requesting their review, as none of them are sqlite3
> module maintainers.

Hi Erlend,

as far as I know we don't have an active module owner and maintainer any
more. Gerhard Häring, the original author of pysqlite, is still listed
as expert. But he hasn't been active in many years. I haven't seen him
in quite some time, too.

How about you put your name in the expert index instead of him? :)

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L2IKTBKEOVZIMXAMH2ZFPZJKESXHDYXK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Distro packagers: PEP 615 and the tzdata dependency

[Christian Heimes]

On 16/11/2020 15.32, Paul Ganssle wrote:
> Hi all,
> 
> One part of PEP 615 (IANA time zones) that I expected to be easily
> overlooked (a point about which I seem to have been right) is the
> recommendation that starting with Python 3.9, distros should add a
> dependency on the system time zone data to the Python package. Quoting
> the PEP:
> 
>> Python distributors are encouraged to ensure that time zone data is
>> installed alongside Python whenever possible (e.g. by declaring tzdata
>> as a dependency for the python package).
> 
> https://www.python.org/dev/peps/pep-0615/#system-time-zone-information
> 
> I am not sure what is the best way to reach the largest number of distro
> packagers to suggest that they add a dependency on tzdata for Python
> 3.9+, so I figured I'd send a message here. If you know a distro
> packager who does not follow this list, or you use a distro that doesn't
> have a dependency on tzdata for Python 3.9, please forward this e-mail
> to them and/or open a bug on their issue tracker.
> 
> So far I know conda-forge got the dependency right from the first Python
> 3.9 release, Fedora has plans to fix this
> <https://src.fedoraproject.org/rpms/python3.10/pull-request/13>, and
> Christian Heimes has opened a bug on the Ubuntu launchpad
> <https://bugs.launchpad.net/ubuntu/+source/python3.9/+bug/1904271> for
> this. I will figure out how best to notify Arch Linux and do that (since
> that's the distro I use).
> 
> I suspect this will be most important for distros that are heavily used
> in containers, since tzdata is a common enough dependency that it's
> likely to be satisfied accidentally in most full-featured user
> environments anyway.
> 
> Thanks!
> 
> Paul
> 
> P.S. If there's a mailing list or other community for Python distro
> packagers, please let me know, because I often find that we're making
> decisions that affect them that they don't hear about until way down the
> line. I'd love to have an easy way to keep them in the loop.

Thanks Paul,

would it make sense to add a packaging section to our documentation or
to write an informational PEP? There is already PEP 394, which defines
the "python" command on Unix-like systems. A new informational PEP could
1) assist packagers with best practices, 2) state how the CPython core
team envisions packaging of Python on Unix distributions.

After all there are more dependencies than "tzdata". Python has
dependencies on /etc/mime.types and system trust store for the ssl
module. Users also have been complaining about usability issues and bugs
caused by downstream patches and packaging decisions (like venv,
distutils, tkinter split into optional packages).

By the way Fedora isn't affected by lack of tzdata database. Fedora's
Python package has a transient dependency on tzdata. The package is
always shipped in Fedora's minimal container images, too. Miro's PR
makes the dependency explicit.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YMQEPZTDY643DIZGIWZEA5QEFJU5ZS2E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

[Christian Heimes]

On 30/10/2020 08.58, Serhiy Storchaka wrote:
> It would make life of Illumos and OpenIndiana developers much harder,
> that can be seen hostile to open source community. It would make the
> code of CPython more rigid, virtually Linux-only with Windows and MacOS
> patches, and as a side effect can make harder porting it to other
> platforms. This is not great.
> 
> If you want to remove platform specific code from CPython source, do you
> consider the idea of removing support of MacOS and maintain it as
> "downstream patches"?

That's an unfair and polarizing comparison. You know as good as I that
macOS is a major platform which is actively used and maintained by
several core developers. I don't recall any core devs that maintain
Solaris port.

PEP-11 lists clear rules about platform support. As long as we don't
have a core developer for Solaris support and stable build bots, I'm all
in favor to remove the platforms.

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGIE4UKLVRMUOW6LU55QMKNQVJKSRUEN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: When to remove BytesWarning?

[Christian Heimes]

On 24/10/2020 05.19, Inada Naoki wrote:
> Hi, all.
> 
> To avoid BytesWarning, the compiler needs to do some hack when they
> need to store bytes and str constants in one dict or set.
> BytesWarning has maintenance costs. It is not huge, but significant.
> 
> When can we remove it? My idea is:
> 
> 3.10: Deprecate the -b option.
> 3.11: Make the -b option no-op. Bytes warning never emits.
> 3.12: Remove the -b option.
> 
> BytesWarning will be deprecated in the document, but not to be removed.
> Users who want to use the -b option during 2->3 conversion need to use
> Python ~3.10 for a while.

In my experience it would be useful to keep the bytes warning for
implicit representation of bytes in string formatting. It's still a
common source of issues in code. Bytes / str comparison or dict lookup
is a less common issue.

>>> b = b'bytes'
>>> f"{b}"
Traceback (most recent call last):
  File "", line 1, in 
BytesWarning: str() on a bytes instance
>>> str(b)
Traceback (most recent call last):
  File "", line 1, in 
BytesWarning: str() on a bytes instance

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M2NJZTGLUA5IAGYDJZOZCPOL4AIXKMY6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 20.55, Larry Hastings wrote:
> On 10/21/20 5:58 AM, Petr Viktorin wrote:
>> At the risk of going off topic: That's for GCC. As far as I know, MSVC
>> uses something like __declspec( thread ).
>> What are the options for generic C99 compilers, other than staying slow?
> 
> 
> As a practical matter: does CPython even support "generic C99
> compilers"?  AFAIK we support three specific compilers: GCC, Clang, and
> MSVC.
> 
> (Maybe we also support icc?  I think mostly because it supports GCC
> language extensions.)

We don't prohibit users to use exotic compilers. Some users maintain
Python on platforms like AIX and Solaris with closed source compilers.

In my opinion it would fine to focus on X86_64 and GCC first. That will
cover the majority of servers and consumer PCs. Clang and GCC have a
similar feature set and extensions, so clang should be doable with
manageable amount of effort, too. After X86_64 I'd consider AArch64
(ARM64) and MSVC next.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N4OKG5Y4TY2VMOJRTTVMLJ3U2LDK3DJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 11.37, Steven D'Aprano wrote:
> On Wed, Oct 21, 2020 at 09:06:58AM +0200, Christian Heimes wrote:
>> On 21/10/2020 00.14, Steven D'Aprano wrote:
>>> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
>>>
>>>> What I don't see is where the money's coming from. It's fine to ask,
>>>> but will anyone come up with that sort of funding?
>>>
>>> I don't think Mark is asking for you or I to fund the exercise. He's 
>>> asking for the PSF to fund it.
>>
>> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
>> half the money.
> 
> I think that's inaccurate. The funding.md document doesn't mention "fund 
> raiser", it doesn't specify how the PSF is to collect the funds, and I 
> wouldn't expect it to.

You are spreading FUD. Stop speculating and give Mark the benefit of a
doubt. This kind of negative attitude is a main cause of contributor
burnout.

If you are unsure about Mark's intentions and goals then please open an
issue on his repository.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NKJN6BPR737NENM4F4AGV6DZ7FTPPMBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 09.35, Paul Moore wrote:
> On Wed, 21 Oct 2020 at 08:14, Christian Heimes  wrote:
>>
>> On 21/10/2020 00.14, Steven D'Aprano wrote:
>>> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
>>>
>>>> What I don't see is where the money's coming from. It's fine to ask,
>>>> but will anyone come up with that sort of funding?
>>>
>>> I don't think Mark is asking for you or I to fund the exercise. He's
>>> asking for the PSF to fund it.
>>
>> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
>> half the money.
> 
> Right. I'd misinterpreted the fact that the PSF was to get half the
> money as meaning they weren't doing the fundraising. My
> misunderstanding, thanks for the clarification.

You are welcome! IMHO you got it right in your initial posting.

I'm irritated that Steven is spreading FUD although Mark's documents
clearly states his intention [1]:

The PSF seems to be the obvious organization to coordinate funding.

Mark is asking the PSF to organize a fund raiser and to act as a
trustee. This is the most logical and reasonable approach. We don't want
to have another requests incident, do we? For stage 2-4 Mark is also
willing to put his reputation and financial security in jeopardy *and*
give the PSF half of the income in exchange for little risk.

Christian

[1] https://github.com/markshannon/faster-cpython/blob/master/funding.md
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5JEPPD4ZFHSPLMBOFF4F444HG72M3E2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 00.14, Steven D'Aprano wrote:
> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
> 
>> What I don't see is where the money's coming from. It's fine to ask,
>> but will anyone come up with that sort of funding?
> 
> I don't think Mark is asking for you or I to fund the exercise. He's 
> asking for the PSF to fund it.

No, he is not. Mark is asking the PSF to organize a fund raiser and keep
half the money.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TOKFADEIOASTU3KMAIMYA226TIJ6IM3T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Resignation from Stefan Krah

[Christian Heimes]

On 09/10/2020 15.48, Antoine Pitrou wrote:
> On Fri, 9 Oct 2020 05:04:55 +0300
> Ivan Pozdeev via Python-Dev  wrote:
>> I don't see the point of requiring to "write an apology", especially *before 
>> a 12-month ban*. If they understand that their behavior is 
>> wrong, there's no need for a ban, at least not such a long one; if they 
>> don't, they clearly aren't going to write it, at least not now (they 
>> might later, after a few weeks or months, having cooled down and thought it 
>> over). So all it would achieve is public shaming AFAICS. Same 
>> issue with the threat of "zero tolerance policy" -- it's completely 
>> unnecessary and only serves to humiliate and alienate the recipient.
> 
> That's my impression as well.  Also, we now have an unmaintained module
> (_decimal) requiring specific competence.

Please elaborate. I feel like you are insinuating something with your
"unmaintained module" remark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5KTGHFOGDYBUBD6JAC3A7QELDSIZYDLP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Resignation from Stefan Krah

[Christian Heimes]

On 09/10/2020 04.04, Ivan Pozdeev via Python-Dev wrote:
> I don't see the point of requiring to "write an apology", especially
> *before a 12-month ban*. If they understand that their behavior is
> wrong, there's no need for a ban, at least not such a long one; if they
> don't, they clearly aren't going to write it, at least not now (they
> might later, after a few weeks or months, having cooled down and thought
> it over). So all it would achieve is public shaming AFAICS. Same issue
> with the threat of "zero tolerance policy" -- it's completely
> unnecessary and only serves to humiliate and alienate the recipient.


I have been the victim of Stefan's CoC violations on more than one
occasion. He added me to nosy list of a ticket just to offend and
humiliate me. For this reason I personally asked the SC to make a
sincere apology a mandatory requirement for Stefan's reinstatement as a
core dev.

I would have been fine with a private apology. However Stefan has also
verbally attacked non-core contributors. In one case another core dev
and I contacted the contribute in private to apologize and ensure that
the contributor was not alienated by Stefan's attitude. Therefore it
makes sense that the SC has requested a public, general apology.

Why are you more concerned with the reputation of a repeated offender
and not with the feelings of multiple victims of harassment? As a victim
of Stefan's behavior I feel that an apology is the first step to
reconcile and rebuild trust.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LGYATAHYU2EL5BH4NPKKZQDIIGEVNXQL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [OT] I'm unsubscribing from this tire fire (formerly known as python-dev)

[Christian Heimes]

Y'all,

trigger warning: strong opinion

The Urban Dictionary defines the term "tire fire":

  A horrifying mess, either literally or figuratively
  foul-smelling, that seems to last forever.

The term describes my current view of python-dev perfectly. It has
always been a problematic and mentally draining place for, sometimes
even toxic. But the recent PEP-8 discussion trumps every past incident
(reference to US politics intended).


To every person still replying on the PEP-8 thread:

  You are making us sick and should be ashamed of yourself!

And I don't mean 'sick' in the figurative sense. You are literally
hurting people who are spending their free and personal time to develop
open source software for you. I know of at least three cases among
Python core developers with symptoms like sleep disorder, tremor,
anxiety, and panic attacks. One core dev wrote publicly that they were
forced to take psychotropic medicine to counter a panic attack after
they have read just a few messages.


At one point I have even considered to retire from Python core
development completely. I'm profoundly disgusted and appalled by the
racist attitudes and self-importance of some people as well as an
unrelated incident on BPO last week. The two reasons I'm not leaving are
 several core developers that I'm happy to call friends and Python
communities beyond predominantly male and Western participants on the
PEP-8 thread. Communities like PyLadies, PyCon Africa, PyLATAM, and
PyCon APAC make me proud and happy to be a member of the Python
community. I have met fantastic people at Python and OSS events in the
Caribbean, India, and East Europe. I don't want to abandon people I
cherish and grew fond of.


At least one other core developer has abandoned python-dev last week.
Others have stopped participating and posting on python-dev years ago. I
will follow their example now.

Goodbye
Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LR3RWME7NYAVAWGD2ZD5NPZAGL7VVI7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: test_ssl.py hangs with SSL 1.1 built with no threads

[Christian Heimes]

On 27/04/2020 19.27, mig28sua...@hotmail.com wrote:
> Hello!
> 
> This is my first time posting to the group. 
> 
> I've been running builds of Python 3.7.x on CentOS Linux release 7.7 
> (64bit/Intel Core 2 Duo) 
> and I ran into hangs with test_ssl.py when using latest SSL 1.1.1d sources. 
> 
> I've done a full compilation from source for Python 3.7.7 and SSL 1.1 in my 
> workspaces. 
> 
> From what I can tell the problem is when SSL 1.1 is built with no threading
> there is no locking enabled by python. 
> 
> This one line change will make the hangs in test_ssl.py  go away:
> 
> Index: Modules/_ssl.c
> ===
> --- Modules/_ssl.c  (revision 70)
> +++ Modules/_ssl.c  (working copy)
> @@ -5875,7 +5875,7 @@
>  if (!_setup_ssl_threads()) {
>  return NULL;
>  }
> -#elif OPENSSL_VERSION_1_1 && defined(OPENSSL_THREADS)
> +#elif OPENSSL_VERSION_1_1
>  /* OpenSSL 1.1.0 builtin thread support is enabled */
>  _ssl_locks_count++;
>  #endif
> 
> 
> There appears to be an assumption in _ssl.c and test_ssl.y that SSL 1.1 will 
> be threaded
> but this may not be true (as in my case). 

Python requires a thread-safe OpenSSL build. I have pushed PR
https://github.com/python/cpython/pull/19953 to 3.7, 3.8, and master.
The hashlib and ssl module will now fail when OpenSSL is not thread-safe.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZEKUSVZVCGPXEZJCLRQ2FPNYQ5O7KHCM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: sqlite3 module and thread-safe SQLite library

[Christian Heimes]

On 18/12/2019 00.56, Gregory P. Smith wrote:
> 
> On Tue, Dec 17, 2019 at 12:30 PM Kacvinsky, Tom
> mailto:tom.kacvin...@vector.com>> wrote:
> 
> We ran into an issue where having the SQLite library built with
> -DSQLITE_THREADSAFE=0,
> but then the sqlite3 module (really, the _sqlite3.so0 crashing in
> threading code.  So I  have
> to ask if it is intended that the sqlite3 Python module always  be
> built with a thread safe
> SQLite library.
> 
> Thanks,
> 
> Tom
> 
> 
> Given your experience I think you've answered the question: Yes.  But it
> would be good if we could detect this at build time (is there a way to
> do that?) and prevent the module from being built against the sqlite3
> library compiled in this unusual mode..  Threading support is required
> in order to be a valid CPython platform, so all libraries used should be
> aware of that.

We have to detect the problem at run-time. Compile time is not
sufficient. It's possible to compile Python with a thread-safe version
of libsqlite3, then run Python with a unsafe version of libsqlite3.

sqlite3 has two APIs to query thread safety and three different settings
for threading:

   sqlite3_threadsafe()
   sqlite3_config()
   SQLITE_CONFIG_SINGLETHREAD
   SQLITE_CONFIG_MULTITHREAD
   SQLITE_CONFIG_SERIALIZED

Only serialized is fully thread safe.

I would start by checking that (sqlite3_threadsafe() ==
SQLITE_CONFIG_SERIALIZED).

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4T7E24TYDDH5ASVGLYYY4POJVUBJEYT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Documenting sorted/min/max prerequisites

[Christian Heimes]

On 14/12/2019 15.20, Serhiy Storchaka wrote:
> 14.12.19 15:29, Steven D'Aprano пише:
>> I might be misinterpreting the evidence, but sorting works on objects
>> that define `__gt__` without `__lt__`.
>>
>> py> class A:
>> ... def __init__(self, x): self.x = x
>> ... def __gt__(self, other): return self.x > other.x
>> ...
>> py> L = [A(9), A(1), A(8)]
>> py> L.sort()
>> py> [obj.x for obj in L]
>> [1, 8, 9]
> 
> This class supports '<':
> 
 A(1) < A(2)
> True
> 
> The `<` operator try to use `__lt__`, but if it is not defined falls
> back to `__gt__`.

The operation "a < b" also fallback back to B.__gt__, when A.__lt__
returns NotImplemented for instances of B.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PAMPK6TS7TVBDAB54OEJEBJCNT75VW3X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Christian Heimes]

On 03/12/2019 21.04, Ethan Furman wrote:
> On 12/03/2019 09:31 AM, Guido van Rossum wrote:
> 
>> I think it’s too soon to worry about this. I don’t see a reason to
>> harass people who maintain code based that were just recently migrated.
> 
> I'm happy to go with this, since my libraries still do the 2/3 straddle.
> 
> Do we want to set a date/version where we will drop code that only
> exists to make 2/3 projects easier?

How about when the last major Linux distro with Python 2 reaches end of
general support? According to Wikipedia that would be 2023 for Ubuntu
18.04 LTS, 2024 for RHEL 8.0, and 2028 for SUSE SLES 15.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3XZZ7FSE3W57VSFBHCNAXIEKTLQWPJBD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Christian Heimes]

On 03/12/2019 19.09, Barry Warsaw wrote:
> On Dec 3, 2019, at 09:16, Serhiy Storchaka 
> wrote:
>>
>> The 'u" string literal prefix was removed in 3.0 and reintroduced
>> in 3.3 to help writing the code compatible with Python 2 and 3
>> [1]. After the dead of Python 2.7 we will remove some deprecated
>> features kept for compatibility with 2.7. When we are going to
>> deprecate and remove the "u" prefix?
>
> I think removing or even deprecating it will cause unnecessary code
> churn in third party libraries, for very little value.  Let’s just
> call it a wart until Python 4000.

I'm 100% with Barry. We can certainly document the "u" string prefix
as deprecated. I'm strongly against removing it from Python 3 or even
raising a deprecation warning. Let's leave it to linters and maybe
consider deprecation for Python 4. Or maybe not.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4GGYZ6XNEH5OQDYC6VLH2MADUVD5LGF3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Restricted Entry Point from PEP-551/578

[Christian Heimes]

On 21/11/2019 21.19, Jason Killen wrote:
> I knew the audit hooks were new but didn't realize they were quite that
> new.  I didn't mean to come across as pejorative asking if people cared
> about this.  The fact that I had trouble finding more information made
> me think this good stuff had been left by the wayside.  It's new, I'll
> to pump the brakes a little and let things take their course.
> 
> I wasn't aware of the configuration system for 3.8.  I'll look into
> that.  When I was in there poking around I kept wondering why we didn't
> have a fancy configuration system.  The good news is we do I just didn't
> know about it.
> 
> You're completely right about my implementation logging to a file.  I
> fully admit my code wasn't camera ready.  Once I learned that the hooks
> existed I wanted to give them a try.  Once I got it working I wanted to
> get some feedback.
> 
> Thank you for the feedback.  I'm glad somebody is thinking about these
> types of things and I'd love to help if I can.

I'm sorry, I wasn't trying to stop you or discourage you in any way. It
was my intent to provide an explanation why there is not much adoption
of the new hooks yet. Please go ahead, put the paddle to the metal and
play around with the new features!

For example you could look into the new config system and figure out
what is missing to build an spython interpreter.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5KKLTIKNY37GK4ZK4DJGGOHAPZFQHGN5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Restricted Entry Point from PEP-551/578

[Christian Heimes]

On 21/11/2019 18.27, Jason Killen wrote:
> I sent in a couple of PRs, accepted and merged (Thanks!), lately that
> switch to using io.open_code when appropriate.  In the process of making
> those PRs I spent a bit of time reading the two related PEPs.  In
> PEP-551 there's a suggestion that people use a restricted entry point in
> production environments.  I googled around a bit and couldn't find any
> evidence that there was an existing implementation, at least not made
> public, that people were using in a general sense.  So I created a
> branch from my fork and over the last few days have implemented part of
> what's suggested in PEP-551.  Specifically my changes remove most of the
> command line options, ignore envvars (except for a possible logging
> filename for the audit hooks), and registers an audit hook that logs
> everything to the defined envvar when provided or stderr if not.

As Steve already pointed out, https://github.com/zooba/spython contains
examples examples for Windows and Linux. The linux_xattr is a PoC that I
developed for our joined talk at EuroPython 2019.

> Now the questions:
> 1) Does anybody care?  Is anyone currently doing this or planning on
> doing this?
> 2) Do we want to provide an "official" version of a restricted entry
> point that could be used as-is or easily modified per specific needs? 
> Seems kinda silly to make everyone roll their own version but I'm happy
> to yield to the will of the people.
> 3) What's the chance we wanna merge something like this into the
> official master branch?  I accomplished what I wanted to do using a few
> #ifdef's and some funky makefile magic.  I think it would merge easily. 
> Maintaining a fork sounds like a lot of work to me.

Yes, we still care. Steve has been more active on the Windows side than
me on the Linux side. Please keep in mind that the feature is new to
Python 3.8 and 3.8.0 just came out a little more than a month ago. It's
going to take a while until major vendors will make use of the feature.

PEP 578 features require distribution and vendor-specific adjustments.
For example it makes sense to disable byte code compilation and usage on
Windows. On the other hand RPM based platforms like RHEL always want to
use precompiled byte code because PYC files are shipped with RPM packages.

> And here's the code:
> I'm very open to suggestions.  I basically have no idea what I'm doing.
> I haven't touched C in about 7 years so don't expect the Mona Lisa.
> https://github.com/python/cpython/compare/master...jsnklln:PEP551_restricted_entry_point

Thanks for your work! Prototypes like this are useful to figure out how
Python's configuration should be improved. Eventually it should not be
necessary to modify any code in CPython to create an spython
interpreter. I like to reach a state in which it is possible to
configure all these flags by providing a custom configuration. Victor's
new config system in 3.8 can accomplish almost everything. It is
currently not possible to modify argument parsing, though.

One comment about your implementation:
You are logging to a file. It's not very useful to log to a target in
the same security context except for debugging. If an attacker is able
to compromise the interpreter, then the attacker most likely gains
enough privileges to wipe the file. That's why Steve uses the Windows
event log in his examples and I'm going for syslog and journald. These
logging system run as a different user and log files are protected by
additional means, e.g. SELinux. I have been working on structures
journald logging recently.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HCSME57BS2AEAATLFMPLCKMWUYZX4SCM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: urllib request HTTPS memory leak?

[Christian Heimes]

On 16/09/2019 23.37, V-ron P wrote:
> Hello Python Dev,
> 
> One of my HTTPS servers leaked over 8 GB of memory over the past week.
> After troubleshooting the issue, I managed to narrow it down to the
> minimal test case below:
> 
> https://gist.github.com/hydrogen-mvm/d588521c2138e0f98aa94b18e7dedfdb  
> 
> (Note that the 100 threads are not necessary, but they help exaggerate
> and accelerate the memory leaking.)
> 
> The memory leak happens if I send a HTTPS request, regardless if it's
> GET or POST.  The most bizarre part is the destination address does
> -not- need to exist and it still leaks! (In the script above it points
> to a non-existent HTTPS server on your localhost.)
> 
> This leak does not happen for HTTP (plaintext) requests, which suggests
> that the bug might be SSL-related.
> 
> My OS is Windows 7 and my Python version is:
> 
> Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916
> 64 bit (AMD64)] on win32
> 
> A friend of mine tested the script on his Windows 10 machine and also
> experienced the same memory leaking effect.
> 
> Please confirm if this is in fact a bug in Python or this is a bug in my
> environment. Thanks.

Yes, we are aware of a memory leak on Windows. A recent change has
caused a memory leak and performance regression in the Windows-only part
of the ssl module. The bug is fixed in 3.7 branch but not yet released.

See https://bugs.python.org/issue37702 and
https://bugs.python.org/issue35941 for more details.

As a workaround I suggest that you create a single SSLContext with
ssl.create_default_context() and reuse the context in all HTTP queries.
You can share the context across threads w/o locking and use it for all
HTTPS connections. A single SSLContext is also more efficient.

Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CYKQCXVPJBFGUDMEE7QFQ5EWQF3LUJDG/

[Python-Dev] Re: Docs: audit event table empty

[Christian Heimes]

On 09/07/2019 14.02, Julien Palard wrote:
> Hi Christian,
> 
>> the table with auditing events does not render on docs.python.org,
>> https://docs.python.org/3.9/library/audit_events.html. Steve and I are
>> going to present the auditing feature tomorrow at EuroPython. It would
>> be helpful to have the table available.
> 
> This was not an easy one... and it may be a Sphinx issue, yet I'm still not 
> sure, maybe Steve can shed some light on it:
> 
> It's the "-j" option of sphinx-build (to parallelize) that causes the issue. 
> I double checked it (full commands at the end of the message in case someone 
> want to reproduce it):
> 
> - Run with -j4 → No table
> - Run without -j → Table is here
> - Run again with -j4 → No table!
> - Run again without -j → Table is back!
> 
> I'm patching docsbuild-scripts to stop using -j4 with is not really helpfull 
> anyway as docsbuild script is parallelizing by starting multiple sphinx-build 
> (for multiple languages / versions).
> 
> I also copied the file and invalidated the cache, so 
> https://docs.python.org/3.9/library/audit_events.html is good again.
> 
> If I'm too slow testing locally and releasing a new docsbuild_script.py, the 
> cron MAY break the file again, don't hesitate to poke me if it happen without 
> me noticing first.

Fantastic! It'll do for now. Thank you!

I'll follow up with you after EuroPython.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6IW5QY7VRSILMJ5R475VSM25V7OAVQDB/

[Python-Dev] Docs: audit event table empty

[Christian Heimes]

Hi,

the table with auditing events does not render on docs.python.org,
https://docs.python.org/3.9/library/audit_events.html. Steve and I are
going to present the auditing feature tomorrow at EuroPython. It would
be helpful to have the table available.

It works on Steve's and my local machine without any issues. I suspect
it's either an outdated sphinx version or caching issue. Could somebody
from the docs team or shell access to the docs machine please look into
the matter?

Thanks!
Christian

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/POWT35ULU2CPELWQ6BRTLTU5H3YKHQZW/

[Python-Dev] Re: 3.7.4 final delayed [Was: Python 3.7.4rc1 (and 3.6.9rc1) cutoffs ahead, now set for 2019-06-17]

[Christian Heimes]

On 29/06/2019 21.53, Ned Deily wrote:
> On Jun 6, 2019, at 01:43, Ned Deily  wrote:
>>
>> https://discuss.python.org/t/python-3-7-4rc1-and-3-6-9rc1-cutoffs-ahead-now-set-for-2019-06-17/1824
>> [...]
>> Following the rc1 cutoff, changes merged to the
>> 3.7 branch will be released in 3.7.5 three months from now unless you
>> mark the issue as a release blocker prior to **3.7.4 final**, planned for
>> release on **2019-06-28**, and explain why the change should be
>> cherry-picked into the final release.
> 
> Update:  3.7.4 final is delayed at least a few days
> 
> A few last minute release blocker issues were identified shortly before 3.7.4 
> final was about to tagged as planned on 2019-06-28, in particular, a couple 
> of TLS 1.3 issues which are of particular importance since we are migrating 
> Windows and macOS installers to OpenSSL 1.1.1 with this release.  We are now 
> on hold awaiting resolutions for the remaining items and then we will need to 
> decide whether another release candidate is needed.  I am hopeful we will be 
> able to proceed by Monday 2019-07-01; I will keep you updated.  And thanks 
> for your help!

Thanks Ned!

the issues are related to TLS 1.3 and client cert authentication. TLS
1.3 dropped renegotiation in favor of rekeying and post-handshake
authentication. PHA is required for conditional authentication in HTTP,
e.g. when a server only requires authentication for some paths or
request methods.

PR https://github.com/python/cpython/pull/14421 fixes a regression in my
PHA code for Python's ssl module. OpenSSL behaves slightly differently
than documented. SSLContext.post_handshake_auth = True also enabled
server cert validation. Alex has reviewed the PR but likes to have a
second opinion.

PR https://github.com/python/cpython/pull/14448 enables PHA for http.client.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46U6OW6UQ7AGVKF6XIJVL7SPT3MKTDI3/

[Python-Dev] Re: EuroPython 2019 sprints

[Christian Heimes]

On 13/06/2019 18.28, Steve Dower wrote:
> Hi all
> 
> Just letting everyone know that I signed up any core devs who are going
> to be at EuroPython to run sprints :) (including myself, since I'll be
> there)
> 
> Like PyCon US, the sprints at EuroPython will attract many first-time
> contributors, so it will be helpful to have as many people who
> understand our workflow around to help out. We might also be able to get
> some of our own contributions done.
> 
> For those who didn't hear, the EuroPython Society is offering a grant
> for core developers to get free attendance at EuroPython (see
> https://www.europython-society.org/core-grant). If you take this, it
> would be a nice gesture of thanks to help support the sprints!
> 
> And in the meantime, if you see an easy bug on the tracker, please mark
> it as "Easy" or "Easy (C)" and post an explanation of how to fix it.
> That way we will have a good supply of tasks for new contributors.

I'm at EP but I'm leaving on Saturday already. There is a small chance
that I can attend the sprints for a couple of hours.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RA2IGTNPMTMW5GWB52IYMFXCSNTY3CNN/

[Python-Dev] Re: python3 -bb and hash collisions

[Christian Heimes]

On 18/06/2019 18.32, Daniel Holth wrote:
> set([u"foo", b"foo]) will error because the two kinds of string have the
> same hash, and this causes a comparison. Is that correct?

Yes, it will fail with -bb, because it turns comparison between str and
bytes into an error. This can also happen with other strings when
hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a
set starts with PySet_MINSIZE - 1 == 8 and increases over team.

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/

Re: [Python-Dev] PEP 590: vectorcall without tp_call

[Christian Heimes]

On 29/05/2019 15.29, Petr Viktorin wrote:
> On 5/29/19 2:25 PM, Jeroen Demeyer wrote:
>> Hello,
>>
>> I have one implementation question about vectorcall which is not
>> specified in PEP 590: what should happen if a type implements
>> vectorcall (i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set
>> tp_call (i.e. tp_call == NULL)? I see the following possibilities:
>>
>> 1. Ignore this problem/assume that it won't happen. This would be bad,
>> since callable(obj) would be False even though obj() would succeed.
>>
>> 2. Raise SystemError.
>>
>> 3. Automatically set tp_call to PyVectorcall_Call.
>>
>> I would vote for 3 since it's the most user-friendly option. There is
>> also no way how it could be wrong: it ensures that tp_call and
>> vectorcall are consistent.
> 
> That sounds like a good idea for PyType_FromSpec.
> 
> For static types I either wouldn't bother at all, or only check in debug
>  builds and fail with Py_FatalError.

You could add a check to PyType_Ready() and have it either return an
error or fix tp_call.

Christian


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 23/05/2019 02.58, Steven D'Aprano wrote:
> On Wed, May 22, 2019 at 01:31:18PM +0200, Christian Heimes wrote:
>> On 22/05/2019 12.19, Steven D'Aprano wrote:
>>> I don't think this PEP should become a document about "Why you should 
>>> use PAM". I appreciate that from your perspective as a Red Hat security 
>>> guy, you want everyone to use best practices as you see them, but it 
>>> isn't Python's position to convince Linux distros or users to use PAM.
>>
> 
>> I think the PEP should make clear why spwd is bad and pining for The 
>> Fjords. The document should point users to correct alternatives. There 
>> is no correct and secure way to use the spwd module to verify user 
>> accounts. Any use of spwd for logins introduces critical security 
>> bugs.
> 
> When you use absolute language about security without considering 
> threat models, like "there is no ... way" and "Any use", you lose 
> credibility in my eyes.
> 
> I have a Linux desktop where I am the only user but not the only user 
> account. If I use spwd, what vulnerabilty am I introducing? That's not a 
> rhetorical question. If spwd does introduce a threat that isn't already 
> there, then please educate me, I genuinely want to know.

I can give you more details once I have resolved some CVEs. The problem
can result into full system compromise by a local or remote attacker
without any trace in the system audit and security logs. Depending on
other circumstances, the issue is CVSS HIGH to CRITICAL, perhaps up to
CVSS score 9.9.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 22/05/2019 02.44, Brett Cannon wrote:
> It also doesn't help that no one is listed in the experts index for the 
> module either..

Excellent point! The PEP now lists the presence / absence of experts.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 594: discussion-to discuss.python.org

[Christian Heimes]

Please use 
https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704
 for feedback and discussion.

Thank you,
Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 22/05/2019 06.59, Stephen J. Turnbull wrote:
> Christian Heimes writes:
> 
>  > It's all open source. It's up to the Python community to adopt
>  > packages and provide them on PyPI.
>  > 
>  > Python core will not maintain and distribute the packages. I'll
>  > merely provide a repository with packages to help kick-starting the
>  > process.
> 
> This looks to me like an opening to a special class of supply chain
> attacks.  I realize that PyPI is not yet particularly robust to such
> attacks, and we have seen "similar name" attacks (malware uploaded
> under a name similar to a popular package).  ISTM that this approach
> to implementing the PEP will enable "identical name" attacks.  (By
> download count, stdlib packages are as popular as Python. :-)

I don't consider this an argument against my proposal, but an argument in favor 
of improving PyPI.


I propose a deal: If you get PEP 453 (ensurepip) revoked, ensurepip removed 
from the standard library, and the recommendation for the requests package on 
urllib.request replaced with a big, fat security warning, then I'll reconsider 
my proposal to recommend PyPI.


:)

My PEP acts in good faith. As long as CPython's stdlib ships pip and embraces 
PyPI, I don't see any reason to distrust PyPI. Yes, PyPI is not Fort Knox. In 
my humble opinion it's more than secure enough for my proposal.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 22/05/2019 12.19, Steven D'Aprano wrote:
> I don't think this PEP should become a document about "Why you should 
> use PAM". I appreciate that from your perspective as a Red Hat security 
> guy, you want everyone to use best practices as you see them, but it 
> isn't Python's position to convince Linux distros or users to use PAM.

I think the PEP should make clear why spwd is bad and pining for The Fjords. 
The document should point users to correct alternatives. There is no correct 
and secure way to use the spwd module to verify user accounts. Any use of spwd 
for logins introduces critical security bugs.

By the way, all relevant BSD, Linux, and Darwin (macOS) distributions come with 
PAM support. Almost all use PAM by default. AFAIK only the minimal Alpine 
container does not have PAM installed by default. This is not Red Hat trying to 
evangelize the world. PAM is *the* industry standards on Unix-like OS.

> To put it another way... I think that if you want to make the case for 
> PAM, put it on the web (a blog?) and link to it.
> 
> As far as the spwd module is concerned, on the one hand you're saying 
> "we should remove it because nobody should ever read from /etc/shadow", 
> and then on the other hand you're all "but go ahead and read /etc/shadow 
> if you like, it is perfectly trivial to do":
> 
>> By the way, the /etc/shadow shadow(5) format is trivial and can be 
>> parsed with a few lines of code. There is no need to use spwd.
> 
> so I think you're undercutting your own argument. If reading from 
> /etc/shadow is such a bad thing that we must remove it, why tell people 
> that they can parse it themselves?
> 
> Not that we could stop them, even if we wanted to.

Steven, I feel like you are turning my own words and arguments against me.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 22/05/2019 01.11, Glenn Linderman wrote:
> On 5/21/2019 2:00 PM, Nathaniel Smith wrote:
>> On Tue, May 21, 2019 at 10:43 AM Glenn Linderman  
>> wrote:
>>> After maintaining my own version of http.server to fix or workaround some 
>>> of its deficiencies for some years, I discovered bottle.py. It has far more 
>>> capability, is far better documented, and is just as quick to deploy. While 
>>> I haven't yet converted all past projects to use bottle.py, it will likely 
>>> happen in time, unless something even simpler to use is discovered, 
>>> although I can hardly imagine that happening.
>> bottle.py uses http.server for its local development mode (the one you
>> see in their quickstart example at the top of their README). Same with
>> flask, django, and probably a bunch of other frameworks. It's *very*
>> widely used.
>>
>> -n
>>
> The source for bottle.py version 0.13-dev has an import for http.client, but 
> not http.server. I hadn't tracked down every indirect dependency in the 
> bottle.py source code, but it seems that if one uses the "default server" for 
> bottle, that it is "wsgiref", imported from wsgiref.simple_server, and that 
> in turn does import BaseHTTPRequestHandler and HTTPServer from http.server.
> 
> It is the higher-level code in http.server that has significant deficiencies 
> that have caused me problems over the years... a "SimpleHTTPRequestHandler" 
> that is so simple it doesn't do POST, PUT or PASTE, a "CGIHTTPRequestHandler" 
> that only implements part of the CGI protocol, only CGI support in POST, no 
> support for PUT or PASTE, and no support for https, and not much bug fix 
> activity in those areas.
> 
> Maybe http.server should be split into the "basic parts" (used by bottle.py, 
> and other frameworks), and the "higher-level parts", which could then be 
> discarded by this PEP! At this point, though, I'd have to agree that the 
> whole should not be discarded. Thanks for making me dig deeper.

The idea has merrit. However I feel its out of scope for the PEP. The 
http.server module and socketserver module are still widely used for debug and 
toy examples.

Could you please open a bug to track your proposal? We may pursue it in a 
couple of years from now.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 22/05/2019 08.30, Giampaolo Rodola' wrote:
> 
> 
> On Tue, 21 May 2019 at 04:30, Antoine Pitrou  > wrote:
> 
> 
> NNTP is still quite used (often through GMane, but probably not only) so
> I'd question the removal of nntplib.
> 
> 
> I concur nntplib should be left alone. There are possibly even less used 
> network protocols such as telnet (tenetlib) which are not targeted by this 
> PEP but could have been by following the same logic. FTP is another one that 
> despite no longer popular, still has a constant user base (you’d be surprised 
> by the amount of traffic we got over FTP in the last company I worked for). 
> Overall, I think the bar for a module removal should be set very high, 
> especially for “standard” things such as these network protocols, that 
> despite being old are not likely to change. That means that also the 
> maintenance burden for python-dev will be low or close to none after all. 

* To be honest, I missed telnetlib. https://github.com/python/peps/pull/1073
* The maintenance burden for nntplib is actually high, because tests are flaky.
* nntplib has no maintainer and is missing features like COMPRESS extension.

> It seems to me also spwd/crypt modules fall into this category (all UNIX 
> platforms have the shadow password db, which is nice to have in python out of 
> the box). In that case the removal appears to be more justified by the 
> security implications than them being not widely used though, so I would use 
> more caution and treat them differently (eg. opt for doc warning + 
> investigate a possible replacement). Also note that spwd could be used to 
> parse /etc/passwd, despite I admit its primary use case is password checking. 
> Certain users may even not care about the additional security provided by PAM 
> (eg. internal devop scripts).

spwd and crypt are dead batteries, because their usefulness has been surpassed 
about two decades (!) ago. They are also very dangerous batteries because they 
leak hydrofluoric acid at scale. It's as least as bad as the acid + bathtub 
scene from the first season of Breaking Bad [1]. HF is nasty [2].

I can reveal more details in a week or two.

Christian

[1] https://breakingbad.fandom.com/wiki/Hydrofluoric_acid
[2] https://en.wikipedia.org/wiki/Hydrofluoric_acid#Health_and_safety

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 22/05/2019 06.20, Arfrever Frehtes Taifersar Arahesis wrote:
> 2019-05-21 00:06 UTC+02:00, Christian Heimes  wrote:
>> On 20/05/2019 23.27, Antoine Pitrou wrote:
>>> Removing the crypt module would remove support for system-standard
>>> password files.  I don't understand the rationale.
>>
>> Applications *must* not access system-standard password files directly. On
>> any sanely and securely configured systems, application cannot even access
>> system password files like /etc/shadow. Access restrictions and system
>> security policies will prevent read access. Also applications cannot assume
>> that users are present in any user file. They may come from LDAP, SSSD,
>> ActiveDirectory, or other sources.
>>
>> The correct way to interact with system users is to use the proper APIs,
>> that are NSS (name service switch) and PAM (pluggable authentication
>> modules). NSS looks up and enumerate users and groups. PAM performs password
>> validation and much, much, much more. The pwd and grp modules use the
>> correct APIs to interact with NSS. If you need to check or change passwords,
>> you must go through PAM.
> 
> It is possible to have a modern Linux desktop system with PAM not
> installed at all, and therefore not used.
> 
> Examples of packages in Gentoo Linux which have OPTIONAL dependency on PAM:
> shadow, sudo, openssh, libcap, systemd, util-linux, screen, cronie,
> polkit, cups, sddm, kscreenlocker, xscreensaver
> (So a KDE Plasma desktop environment and its direct and indirect
> dependencies can be installed without PAM.)
> 
> The suggested substitutes for spwd module, i.e. python-pam and
> simpleplam, look like they would not work on a PAM-free system.
 
Thanks for bringing this up. I don't think we need to care about this care.

A PAM-free Linux system is an IMHO very special and exotic case. It's certainly 
not a setup anybody should run on a server. There are a lot of good reasons to 
use PAM. I'll update the BPO with reasons soonish.

By the way, the /etc/shadow shadow(5) format is trivial and can be parsed with 
a few lines of code. There is no need to use spwd.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python in next Windows 10 update

[Christian Heimes]

On 21/05/2019 22.30, Steve Dower wrote:
> Hi all
> 
> Just sharing this here because I think it's important for us to be aware of 
> it - I'm not trying to promote or sell anything here :) (Those who were at 
> the language summit have seen this already.)
> 
> In the next Windows 10 update that starts rolling out today, we (Microsoft) 
> have added "python.exe" and "python3.exe" commands that are installed on PATH 
> *by default* and will open the Microsoft Store at the page where we (Python 
> core team) publish our build.
> 
> This makes it a 1-2 click process to get from a clean machine to having a 
> usable Python install ("python.exe" -> opens Store -> "Get it Free" -> 
> "python.exe" now works!)
> 
> The associated blog post:
> 
> https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/

Thanks for doing this and lowering the entree barrier for every Windows user. :)

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 20.35, Guido van Rossum wrote:
> On Tue, May 21, 2019 at 11:17 AM Christian Heimes  <mailto:christ...@python.org>> wrote:
> 
> I'm already facing opposition for modules that are less controversial and 
> useful than http.server, too.
> 
> 
> There's another argument here. This is an "omnibus" PEP, meaning it proposes 
> many unrelated changes. In order to get a consensus to pass the PEP, it may 
> be necessary to compromise. IOW I would recommend removing modules from the 
> PEP that bring up strong opposition, *even* if you yourself feel strongly 
> that those modules should be removed.
> 
> The vast majority of modules on the list hasn't elicited any kind of feedback 
> at all -- those are clearly safe to remove (many people are probably, like 
> myself, hard-pressed to remember what they do). I'm not saying drop anything 
> from the list that elicits any pushback, but once the debate has gone back 
> and forth twice, it may be a hint that a module still has fans. Threatening 
> to open a CVE is more likely to reduce support for the PEP than it is to 
> convince anyone.

It was not a threat, but an illustration how critical the flaw with spwd + 
crypt is. The approach performs only authentication and completely bypasses any 
authorization. It does not take any login restrictions into account like 
account enabled flag, host/service based access control, IP restriction, 
credential strength, and so on. I would give the issue a CVSS rating between 
8.3 (high) to 9.6 (critical), perhaps 
CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:L

By the way, Giampaolo and I have known each other for many years. I know that 
he'll address the issue and file a CVE himself.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 21/05/2019 20.18, Giampaolo Rodola' wrote:
> No, the statement is correct. I may have to explain this even further.
> 
> The approach in pyftpdlib is the wrong and IMO deserves a CVE. The 
> crypt() + spwd() approach is flawed on multiple levels. For example it 
> bypasses account restriction, access control, and login session. It also 
> requires you to run the service as root and with SELinux disabled or an 
> unconfined context -- a bad combination. There is only one correct way to 
> perform a credential check: use PAM.
> 
> spwd can't be fixed. It could only be replaced with a completely 
> different API that wraps PAM and Windows's authentication API.
> 
> Christian
> 
> PS: Authentication, authorization, and identity management are part of my 
> day job at Red Hat.
> 
> 
> Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt 
> doc deserve a warning. This is probably out of the scope of the PEP, but I 
> wonder if the 3 third-party alternatives mentioned in the PEP are mature 
> enough and could be evaluated for stdlib inclusion (the part re. PAM / 
> password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8 
> and the alternative added in 3.9 before the 3.10 removal.

Sorry, I didn't even occur to me that anybody was still using spwd. I would 
have added a warning much earlier. There is now 
https://bugs.python.org/issue36997 to track the problem.

Could you do me a favor and open a feature request?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 16.46, Guido van Rossum wrote:
> +1. Let's keep colorsys then.

I let colorsys off the hock, https://github.com/python/peps/pull/1070

Thanks for your feedback, Walter and Petr!

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 18.29, Glenn Linderman wrote:
> On 5/20/2019 2:20 PM, Christian Heimes wrote:
>> On 20/05/2019 23.12, Andrew Svetlov wrote:
>>> socketserver.py is also questionable
>> I briefly though about the module, but didn't consider it for removal. The 
>> http.server, xmlrpc.server, and logging configuration server are implemented 
>> on top of the socketserver. I don't want to remove the socketserver module 
>> without a suitable replacement for http.server in the standard library.
> 
> But http.server could be on the remove list too... it gets mighty little 
> support, has very little functionality, and implements a CGI interface 
> (although that also has very little functionality), and you have the CGI 
> tools on the remove list, rendering the CGI interface implemented by 
> http.server less easily usable.
> 
> Further, it doesn't directly support https:, and browsers are 
> removing/reducing support for http:.
> 
> I can't speak to xmlrpc or logging configuration.

Hi,

thanks for bringing this topic up. Initially I considered http.server, too. But 
as Guido put it, it's both used and useful for local testing and quick hacks. 
I'm already facing opposition for modules that are less controversial and 
useful than http.server, too. 

I have two remarks:

1) The http.server does not act as a CGI server by default. In CGI server mode, 
it does not depend on the cgi module.
2) The lack of HTTPS support is not a major problem for connections on 
localhost. There is a RFC draft to consider any connection to "localhost" as 
secure, https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-06

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 21/05/2019 18.08, Giampaolo Rodola' wrote:
> 
> 
> On Tue, 21 May 2019 at 21:13, Christian Heimes  <mailto:christ...@python.org>> wrote:
> 
> crypt
> ~
> 
> The `crypt <https://docs.python.org/3/library/crypt.html>`_ module 
> implements
> password hashing based on ``crypt(3)`` function from ``libcrypt`` or
> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of 
> poor
> quality and insecure. Users are discouraged to use them.
> 
> * The module is not available on Windows. Cross-platform application need
>   an alternative implementation any way.
> * Only DES encryption is guarenteed to be available. DES has an extremely
>   limited key space of 2**56.
> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension.
>   SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
>   algorithm that is still secure. However it's in glibc and therefore not
>   commonly available on Linux.
> * Depending on the platform, the ``crypt`` module is not thread safe. Only
>   implementations with ``crypt_r(3)`` are thread safe.
> * The module was never useful to interact with system user and password
>   databases.
> 
> 
> This is actually not true. Their main use case is to compare passwords 
> against the shadowed password db:
> https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413
> A quick search on searchcode.com <http://searchcode.com> shows both spwd and 
> crypt modules are used. I am no security expert (and I wasn’t aware they are 
> insecure until now, since the doc doesn’t mention it) but I would prefer 
> seeing these 2 fixed or improved rather than bluntly removed. 

PS: "pw1 != pw2" on line 418 is also vulnerable to side channel attack. You 
must use a constant timing comparison operator like hmac.compare_digest() to 
compare password digests.

https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L418
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 21/05/2019 18.08, Giampaolo Rodola' wrote:
> 
> 
> On Tue, 21 May 2019 at 21:13, Christian Heimes  <mailto:christ...@python.org>> wrote:
> 
> crypt
> ~
> 
> The `crypt <https://docs.python.org/3/library/crypt.html>`_ module 
> implements
> password hashing based on ``crypt(3)`` function from ``libcrypt`` or
> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of 
> poor
> quality and insecure. Users are discouraged to use them.
> 
> * The module is not available on Windows. Cross-platform application need
>   an alternative implementation any way.
> * Only DES encryption is guarenteed to be available. DES has an extremely
>   limited key space of 2**56.
> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension.
>   SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
>   algorithm that is still secure. However it's in glibc and therefore not
>   commonly available on Linux.
> * Depending on the platform, the ``crypt`` module is not thread safe. Only
>   implementations with ``crypt_r(3)`` are thread safe.
> * The module was never useful to interact with system user and password
>   databases.
> 
> 
> This is actually not true. Their main use case is to compare passwords 
> against the shadowed password db:
> https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413
> A quick search on searchcode.com <http://searchcode.com> shows both spwd and 
> crypt modules are used. I am no security expert (and I wasn’t aware they are 
> insecure until now, since the doc doesn’t mention it) but I would prefer 
> seeing these 2 fixed or improved rather than bluntly removed. 

No, the statement is correct. I may have to explain this even further.

The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() + 
spwd() approach is flawed on multiple levels. For example it bypasses account 
restriction, access control, and login session. It also requires you to run the 
service as root and with SELinux disabled or an unconfined context -- a bad 
combination. There is only one correct way to perform a credential check: use 
PAM.

spwd can't be fixed. It could only be replaced with a completely different API 
that wraps PAM and Windows's authentication API.

Christian

PS: Authentication, authorization, and identity management are part of my day 
job at Red Hat.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 21/05/2019 17.31, Antoine Pitrou wrote:
> 
> As I said, if the main annoyance with nntplib is the sporadic test
> failures, then the relevant tests can be disabled on CI.
> 
> NNTP itself is still used, even if less and less.

I don't like the idea to drop a third of the test cases for nntplib -- and the 
30% that actually test that Python's NNTP library can talk to an actual NNTP 
server. IMHO it's more beneficial for core development and for nntplib to have 
it maintained by somebody that cares about the library.

Even a deprecation in 3.8 means that the module has to be maintained until EOL 
of 3.9, which is around 2026.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Christian Heimes]

On 21/05/2019 17.35, Guido van Rossum wrote:
> OK, sounds like nntplib can stay — this time. 
> 
> On Tue, May 21, 2019 at 08:33 Antoine Pitrou  > wrote:
> 
> 
> As I said, if the main annoyance with nntplib is the sporadic test
> failures, then the relevant tests can be disabled on CI.
> 
> NNTP itself is still used, even if less and less.

I'd rather have it moved outside the core and maintained by people that 
actually care about NNTP support.

Greg, you proposed the deprecation of nntplib. Any comments?

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 15.54, Victor Stinner wrote:
> IMHO we should simply acknowledge this fact by mentioning it in the
> PEP. We are aware that "pip install" is not always a valid option, but
> we decided anyway to deprecate/remove modules because <...>.

I like the idea. Could you create an issue or PR, please?

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 594: update 1

[Christian Heimes]

Hi,

I have updated the PEP with feedback from discussions. The highlights are:

* Deprecate parser module
* Keep fileinput module
* Elaborate why crypt and spwd are dangerous and bad
* Improve sections for cgitb, colorsys, nntplib, and smtpd modules
* The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable 
substitutions.
* Mention that socketserver is going to stay for http.server and xmlrpc.server
* The future maintenance section now states that the deprecated modules may be 
adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574


I'll be traveling the next couple of days and will only have limited 
opportunities to respond on feedback.

Christian

---
PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-May-2019
Post-History: 21-May-2019


Abstract


This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats and APIs that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.


Rationale
=

Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often refrained to as "batteries included"
philosophy and was one of the corner stones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. The introduction of the cheese shop (PyPI), setuptools,
and later pip, it became simple and straight forward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.

On the other hand, Python's standard library is piling up cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.

* Any additional module increases the maintenance cost for the Python core
  development team. The team has limited resources, reduced maintenance cost
  frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
  de-facto solution for a problem. A majority of users only pick 3rd party
  modules to replace a stdlib module, when they have a compelling reason, e.g.
  lxml instead of `xml`. The removal of an unmaintained stdlib module
  increases the chances of a community contributed module to become widely
  used.
* A lean and mean standard library benefits platforms with limited resources
  like devices with just a few hundred kilobyte of storage (e.g. BBC
  Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
  (e.g. pyodide) also benefit from reduced download size.

The modules in the PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example
least controversial are 30 years old multimedia formats like ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980ties. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be cooperate environments or class rooms where external
code is not permitted without legal approval.

* The usage of FTP is declining, but some files are still provided over
  the FTP protocol or hosters offer FTP to upload content. Therefore
  ``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` module are widely used. They are mature
  modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
  kids and can make crazy sounds. Making a computer generate crazy sounds is
  powerful and highly motivating exercise for a 9yo aspiring developer. It's
  a fun battery to keep.


Deprecation schedule


3.8
---

This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released
a few months before Python 2.7 will reach its end of lifetime. We expect that
Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and
2020. To reduce churn and to allow a smooth transition from Python 2,
Python 3.8 will neither raise `DeprecationWarning` nor remove any
modules that have been scheduled for removal. Instead deprecated modules will
just be *documented* as deprecated. Optionally modules may emit a
`PendingDeprecationWarning`.

All 

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 15.01, Steven D'Aprano wrote:
> Christian, I'm glad that you are privileged enough to find it simple and 
> straight forward to download and install, but for many Python users, it 
> is not so simple or straight forward.
> 
> Many Python users don't have the privilege of being able to install 
> arbitrary, unvetted packages from PyPI. They get to use only packages 
> from approved vendors, including the stdlib, what they write themselves, 
> and nothing else. Please don't dismiss this part of the Python community 
> just because they don't typically hang around in the same forums we do.
> 
> I've worked with organisations where downloading and installing software 
> from the internet was grounds for instant dismissal. I've worked with 
> organisations with regulatory requirements to do due-dilegance on their 
> entire software stack, and getting permission to install an unapproved 
> library could take 3-6 months elapsed time and dozens of person-hours, 
> including a full review of the licencing and copyright by lawyers.
> 
> I've also worked with kids using school computers who don't have either 
> the legal permission or the knowledge to use pip install.
> 
> Sometimes their school administrators are ... how shall I put this 
> kindly? ... over zealous in their efforts to protect the students from 
> malware and spyware (apart from the school's own spyware, of course...) 
> and rather lacking in their understanding of the difference between 
> piracy and Open Source software. Getting Python installed by the school 
> admiinistrator is one thing, but allowing the kids to run pip and 
> install software themselves is unthinkable.
> 
> And remember, in some juristictions, installing software from the 
> internet can put you in breach of some draconian laws. At the very 
> least, kids may face expulsion.

This argument has bring brought up several times. For that reason the PEP lists 
mostly modules that deal with historic and irrelevant data formats. The wave 
module stays because it is useful in education contexts. The use case is 
explicitly mentioned in the latest version of the PEP.

Do you have a concrete and tangible case example, in which a deprecated module 
is absolutely necessary, a user gets permission to update Python, but does not 
get permission to get a PyPI package installed?

You could argue that cgi, spwd, and crypt are not history data formats. My 
counter argument: *If* you have to work in a severely restricted environment, 
*then* you are surely not allowed to run your own CGI web server or access the 
system password database.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 14.06, Anders Munch wrote:
> Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan...@python.org] På 
> vegne af Christian Heimes
>> * The removed modules will be available through PyPI.
> 
> Will they?  That's not the impression I got from the PEP. 

It's all open source. It's up to the Python community to adopt packages and 
provide them on PyPI.

Python core will not maintain and distribute the packages. I'll merely provide 
a repository with packages to help kick-starting the process.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 13.50, Victor Stinner wrote:
> Hi Christian,
> 
> I dislike the PEP 594 title: "Removing dead batteries from the
> standard library". A module is never "dead", there are always users,
> even if there are less than 5 of them.

I'm open for suggestions for a better title.

> Extract of the Rationale: "The modules are mostly historic data
> formats and APIs that have been superseded a long time ago"
> 
> If we take the example of NNTP: even if it's old and "superseded",
> people still uses NTTP. Python doesn't have to be opinionated about
> formats.

NNTP was added late in the writing of the PEP. I can rephrase the rational

> Wait, I like the overall PEP. I'm just talking about how it's explained.
> 
> IMHO the question here is if the core developers want to continue to
> pay the price of the maintenance burden for modules which have "few"
> users (define few...). The other question is if it would be acceptable
> to move these modules to PyPI. "import wave" would continue to work as
> previously, but the maintenance would be moved *outside* Python.
> 
> Who will maintain these modules on PyPI? Do we have to define this right now?

Whoever steps up and is interested to maintain the module. But we don't have to 
define this now. We have more than a year to come up with a plan. Except for 
the parser module, all modules will be maintained by Python core until 2026.

> If a module is pure Python, well, the easy solution is to embed it
> into your code: cp /path/to/cpython/Lib/.py
> /path/to/yourapp/.py. If it's a C extension, it's more
> complicated.

This only affects nis, ossaudiodev, spwd, and msi. For crypt I already have a 
ctypes wrapper. ossaudiodev is dead and has been replaced by ALSA about two 
decades ago. spwd is bad and users should use PAM instead.

> The PEP is full of "Substitute: none". I'm not comfortable with that.
> I would prefer to require that there is always a solution, like
> putting the code on PyPI and let it die there. The old mojo "the
> stdlib is where modules die" would become "PyPI is where old stdlib
> modules die" :-)

The latest version of the PEP lists many additional substitutes. I'll post a 
new version soonish.

 
> Python itself is bad at fixing DeprecationWarning: imp and asyncore
> are deprecated for years, but they are still widely used inside
> Python...

We are keeping the old packages to make transition from 2.7 to 3.8 easier. This 
is going to change with 3.9.

> I dislike DeprecationWarning *but* ... well, does it really hurt so
> much to keep these modules around? asyncore has know bugs, *but* many
> tests of our test suite are based on that and we managed to make these
> tests very reliable only with minor work.

I'm struggeling with asyncore in SSL tests, mostly because I don't grasp how 
asyncore works and I have no intention to dig into a deprecated technology.

> Even if it's not said this way, in fact this PEP reopens the question
> of splitting the stdlib. It would be helpful to have some references
> into the PEP about previous discussions on this topic. Recent
> discussion:
> 
> "Amber Brown: Batteries Included, But They're Leaking "
> http://pyfound.blogspot.com/2019/05/amber-brown-batteries-included-but.html
> 
> By the way, even if it's not directly related, the PEP doesn't address
> one of the common question: be able to override a stdlib module with a
> more recent version. For example, if the Python provided by your OS
> contains a bug in asyncio, you are stuck and cannot easily upgrade it
> to get a new version without the fix.

As you said, it's not directly related. These topics are not within scope for 
the PEP. Please open a new thread. I like to keep this thread on topic.


> For me, "getopt" is a dead battery. The PEP says "Although users are
> encouraged to use argparse instead, the getopt module is still widely
> used."
> 
> Well, that makes sense. But so, what is the metric to decide if a
> module is "widely used" or not?
> 
> Should we keep a module at soon as at least one user says "I'm using
> it!". In other replies, users asked to keep spwd, fileinput, nntplib,
> etc.

getopt is very short, has no maintainance overhead, and useful for quick hacks. 
fileinput stays as well for the same reason.

> Sorry, I mostly opened questions. I don't have answers. I just wanted
> to share that this PEP scares me a little bit. At least, the current
> version of the PEP.

The PEP will be implemented over a course of two releases and three years. It 
gives us enough time to address concern or change our minds.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 13.08, André Malo wrote:
> On Montag, 20. Mai 2019 23:27:49 CEST Antoine Pitrou wrote:
>> NNTP is still quite used (often through GMane, but probably not only) so
>> I'd question the removal of nntplib.
>>
>> cgitb used to be used by some Web frameworks in order to format
>> exceptions.  Perhaps one should check if that's still the case.
> 
> I concur with both of those.
> There's software in production using both. (It doesn't mean it's on pypi or 
> even free software).

There is always somebody who uses a feature. This argument blocks any 
innovation or cleanup. Victor just reminded me of https://xkcd.com/1172/ .

* The removed modules will be available through PyPI.
* You don't have to start to worry until Python 3.10 is released in over 3 
years from now.
* The modules are fully supported in Python 3.8 and 3.9. Python 3.9 will reach 
EOL late 2026 or early 2027.

That's plenty of time.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 12.19, Giampaolo Rodola' wrote:
> I find this one useful and would be a bit sad to see it go. FWIW I use it in 
> pyftpdlib and I suppose there are other apps out there relying on UNIX 
> password db for authentication. The fact that it’s a C module is also an 
> incentive to leave it in the stdlib IMO (pure python modules can easily be 
> copied in the project instead of retrieving them from PYPI as a third party 
> dep - e.g. this is how I am likely going to replace asyncore/asynchat).

If you use the spwd module for authentication, then you have a major security 
problem in your application. You must use the PAM stack to authenticate access 
to a service.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com