[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-10-21 Thread Victor Stinner
On Wed, Oct 20, 2021 at 10:58 AM Petr Viktorin  wrote:
> I think this info should be in the PEP.

Ok, we added (and completed) the list to the PEP:
https://www.python.org/dev/peps/pep-0670/#macros-converted-to-functions-since-python-3-8

> If the PEP is rejected, would all these previous changes need to be
> reverted? Or just the ones done in 3.11?

I don't know. I guess that it can be decided once the PEP will be rejected :-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QHL2ACN7LU6XKUZRW44A43LHHXLUUE3M/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-10-19 Thread Victor Stinner
One of my motivation to write this PEP was decide how to solve the
issue: "[C API] Disallow using PyFloat_AS_DOUBLE() as l-value"
https://bugs.python.org/issue45476

I proposed two fixes:

* Convert macros to static inline functions:
https://github.com/python/cpython/pull/28961
* Fix the macro, add _Py_RVALUE(): https://github.com/python/cpython/pull/28976

I would prefer to static inline functions ;-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HSLZAOEJLSBS4F2YRHJEXYQHCTDNKLU6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-10-19 Thread Victor Stinner
Extra info that I didn't put in the PEP to keep the PEP short.

Since Python 3.8, multiple macros have already been converted,
including Py_INCREF() and Py_TYPE() which are very commonly used and
so matter for Python performance.

Macros converted to static inline functions:

* Py_INCREF(), Py_DECREF(), Py_XINCREF(), Py_XDECREF(): Python 3.8
* PyObject_INIT(), PyObject_INIT_VAR(): Python 3.8
* Private functions: _PyObject_GC_TRACK(), _PyObject_GC_UNTRACK(),
_Py_Dealloc(): Python 3.8
* Py_REFCNT(): Python 3.10
* Py_TYPE(), Py_SIZE(): Python 3.11

Macros converted to regular functions in Python 3.9:

* PyIndex_Check()
* PyObject_CheckBuffer()
* PyObject_GET_WEAKREFS_LISTPTR()
* PyObject_IS_GC()
* PyObject_NEW(): alias to PyObject_New()
* PyObject_NEW_VAR(): alias to PyObjectVar_New()

To keep best performances on Python built without LTO, fast private
variants were added as static inline functions to the internal C API:

* _PyIndex_Check()
* _PyObject_IS_GC()
* _PyType_HasFeature()
* _PyType_IS_GC()

--

Many of these changes have been made to prepare the C API to make
these structure opaque:

* PyObject: https://bugs.python.org/issue39573
* PyTypeObject: https://bugs.python.org/issue40170

Don't access structure members at the ABI level, but abstract them
through a function call.

Some functions are still static inline functions (and so still access
structure members at the ABI level), since the performance impact of
converting them to regular functions was not measured yet.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U7TKPCJ5FQJHEIUXZIZRYZL4ZWU5WSE6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] PEP 670: Convert macros to functions in the Python C API

2021-10-19 Thread Victor Stinner
Hi,

Erlend and me wrote a PEP to move away from macros in the Python C
API. We are now waiting for feedback :-) Read the PEP online:
https://www.python.org/dev/peps/pep-0670/

There is a copy of the PEP below for inline replies.

Victor

---

PEP: 670
Title: Convert macros to functions in the Python C API
Author: Erlend Egeberg Aasland ,
Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 19-Oct-2021
Python-Version: 3.11


Abstract


Convert macros to static inline functions or regular functions.

Remove the return value of macros having a return value, whereas they
should not, to aid detecting bugs in C extensions when the C API is
misused.

Some function arguments are still cast to ``PyObject*`` to prevent
emitting new compiler warnings.


Rationale
=

The use of macros may have unintended adverse effects that are hard to
avoid, even for experienced C developers. Some issues have been known
for years, while others have been discovered recently in Python.
Working around macro pitfalls makes the macro coder harder to read and
to maintain.

Converting macros to functions has multiple advantages:

* By design, functions don't have macro pitfalls.
* Arguments type and return type are well defined.
* Debuggers and profilers can retrieve the name of inlined functions.
* Debuggers can put breakpoints on inlined functions.
* Variables have a well defined scope.
* Code is usually easier to read and to maintain than similar macro
  code.  Functions don't need the following workarounds for macro
  pitfalls:

  * Add parentheses around arguments.
  * Use line continuation characters if the function is written on
multiple lines.
  * Add commas to execute multiple expressions.
  * Use ``do { ... } while (0)`` to write multiple statements.

Converting macros and static inline functions to regular functions makes
these regular functions accessible to projects which use Python but
cannot use macros and static inline functions.


Macro Pitfalls
==

The `GCC documentation
<https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_ lists several
common macro pitfalls:

- Misnesting
- Operator precedence problems
- Swallowing the semicolon
- Duplication of side effects
- Self-referential macros
- Argument prescan
- Newlines in arguments


Performance and inlining


Static inline functions is a feature added to the C99 standard. Modern C
compilers have efficient heuristics to decide if a function should be
inlined or not.

When a C compiler decides to not inline, there is likely a good reason.
For example, inlining would reuse a register which require to
save/restore the register value on the stack and so increase the stack
memory usage or be less efficient.


Debug build
---

When Python is built in debug mode, most compiler optimizations are
disabled.  For example, Visual Studio disables inlining. Benchmarks must
not be run on a Python debug build, only on release build: using LTO and
PGO is recommended for reliable benchmarks. PGO helps the compiler to
decide if function should be inlined or not.


Force inlining
--

The ``Py_ALWAYS_INLINE`` macro can be used to force inlining. This macro
uses ``__attribute__((always_inline))`` with GCC and Clang, and
``__forceinline`` with MSC.

So far, previous attempts to use ``Py_ALWAYS_INLINE`` didn't show any
benefit and were abandoned. See for example: `bpo-45094
<https://bugs.python.org/issue45094>`_: "Consider using
``__forceinline`` and ``__attribute__((always_inline))`` on static
inline functions (``Py_INCREF``, ``Py_TYPE``) for debug build".

When the ``Py_INCREF()`` macro was converted to a static inline
functions in 2018 (`commit
<https://github.com/python/cpython/commit/2aaf0c12041bcaadd7f2cc5a54450eefd7a6ff12>`__),
it was decided not to force inlining. The machine code was analyzed with
multiple C compilers and compiler options: ``Py_INCREF()`` was always
inlined without having to force inlining. The only case where it was not
inlined was the debug build. See discussion in the `bpo-35059
<https://bugs.python.org/issue35059>`_: "Convert ``Py_INCREF()`` and
``PyObject_INIT()`` to inlined functions".


Disable inlining


On the other side, the ``Py_NO_INLINE`` macro can be used to disable
inlining.  It is useful to reduce the stack memory usage. It is
especially useful on a LTO+PGO build which is more aggressive to inline
code: see `bpo-33720 <https://bugs.python.org/issue33720>`_. The
``Py_NO_INLINE`` macro uses ``__attribute__ ((noinline))`` with GCC and
Clang, and ``__declspec(noinline)`` with MSC.


Specification
=

Convert macros to static inline functions
-

Most macros should be converted to static inline functions to prevent
`macro pitfalls`_.

The following macros should not be converted:

* Empty macros. Example: ``#define Py_HAV

[Python-Dev] Re: What is __int__ still useful for?

2021-10-13 Thread Victor Stinner
Hi Antoine,

I have a lot of troubles to reminder how Python converts numbers, I
collected notes about the Python "number tower" and the C
implementation:
https://pythondev.readthedocs.io/numbers.html

Honestly, I don't understand well the difference between __int__() and
__index__().

* https://docs.python.org/dev/reference/datamodel.html#object.__int__
* https://docs.python.org/dev/reference/datamodel.html#object.__index__

Victor

On Wed, Oct 13, 2021 at 7:11 PM Antoine Pitrou  wrote:
>
>
> Hello,
>
> It used to be that defining __int__ allowed an object to be accepted as
> an integer from various functions, internal and third-party, thanks to
> being implicitly called by e.g. PyLong_AsLong.
>
> Today, and since bpo-37999, this is no longer the case.  It seems that
> __int__ has now become a strict equivalent to __trunc__.  Of course,
> user code can still look up and call the __int__ method explicitly (or
> look up the nb_int slot, in C), but that's a bit of anti-pattern.
>
> Is there a point in having three special methods __index__, __int__ and
> __trunc__, if two are them are practically interchangeable?
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/K6TEYMDY5NEDV4MHH6EGIOQWDOAKSPJV/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QFMFPMCSDWV7S5QGPYS7K2VN4CHF5KDL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] svn.python.org require an HTTP authentication: r77062 links at bugs.python.org

2021-10-08 Thread Victor Stinner
Hi,

I'm digging into the Python bug tracker history and I found links to
Subversion commits:
"Fixed in r77062 (trunk), r77063 (py3k)."
https://bugs.python.org/issue1811#msg96910

Roundup adds links which are redirected:

* https://hg.python.org/lookup/r77062 ->
https://svn.python.org/view?view=revision=77062
* https://hg.python.org/lookup/r77063 ->
https://svn.python.org/view?view=revision=77063

Problem: the svn.python.org links require me to log in with an
username and password, and I don't know how to log in. If I cancel the
prompt, it fails with: an HTTP Error 401 "Unauthorized".

Who manages this Subversion server? Would it be possible to remove
this authentication?

I know that the Misc/svnmap.txt file in Python contains a mapping from
Subversion to Mercurial commits, but Python now uses Git, so it's not
convenient. Maybe a new file mapping Subversion to Git commits should
be created? Or the https://hg.python.org/lookup/ service should
redirect to Mercurial commits at least.

Using this map, I was able to manually retrieve the r77062 commit:
https://hg.python.org/cpython/rev/c4e60988834df0fd437ae3422da0cab4acd6fce0

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UR5SKKQ3XJTP7Q2IBMMZTNICZ7BDOKVZ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 654 except* formatting

2021-10-04 Thread Victor Stinner
To stay consistent with PEP 8, exception groups should use 4 spaces.

Victor

On Sun, Oct 3, 2021 at 5:54 PM Irit Katriel via Python-Dev
 wrote:
>
>
> We wonder if people have a view on which of the following is clearer/better:
>
> 1. except *E as e:  //  except *(E1, E2) as e:
> 2. except* E as e:  //  except* (E1, E2) as e:
>
> (The difference is in the whitespace around the *).
>
> At the moment * is a separate token so both are allowed, but we could change 
> that (e.g., make except* a token), and in any case we need to settle on a 
> convention that we use in documentation, etc.
>
> It is also not too late to opt for a completely different syntax if a better 
> one is suggested.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4B256YKUPW5P2M44GG5H6FBL3PSV6ODP/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A2REM55FHTETDZUPRVDWTVSXC273GHZW/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Should I care what -3.14 // inf produces?

2021-09-30 Thread Victor Stinner
Hi Jeff,

The decimal module docstring starts with:

"""
This is an implementation of decimal floating point arithmetic based on
the General Decimal Arithmetic Specification:

http://speleotrove.com/decimal/decarith.html

and IEEE standard 854-1987:

http://en.wikipedia.org/wiki/IEEE_854-1987

Decimal floating point has finite precision with arbitrarily large bounds.
"""

I suggest you to look into these standards.

Victor

On Thu, Sep 30, 2021 at 9:13 AM Jeff Allen  wrote:
>
> Is an implementation of Python free to make up its own answers to division 
> and modulus operatons in the case of inf and nan arguments?
>
> The standard library documentation says only that // is "rounded towards 
> minus infinity". The language reference says that :
>
> x == (x//y)*y + (x%y),
> the modulus has the same sign as y, and
> division by (either kind of) zero raises ZeroDivisionError .
>
> It's consistent, but it doesn't define the operator over the full range of 
> potential arguments. In Python 3.8 and 3.9:
>
> >>> from decimal import Decimal
> >>> Decimal('-3.14') // Decimal('Infinity')
> Decimal('-0')
> >>> -3.14 // float("inf")
> -1.0
> >>> import math
> >>> math.floor(-3.14 /  float("inf"))
> 0
>
> I can see sense in all three answers, as possible interpretations of "rounded 
> towards minus infinity", but I quite like decimal's. There seem to be no 
> regression tests for floor division of floats, and for modulus only with 
> finite arguments, perhaps intentionally.
>
> --
>
> Jeff Allen
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/BXYBSUMNSP6AAAS6OL23ANSML4IOARVB/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GLFGZ6D2YTFL3HYO4JL4OR6UAANR7JP6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] PEP: Taking the Python C API to the Next Level

2021-09-28 Thread Victor Stinner
Hi,

I would like to change the Python C API. I failed to write a single
document listing all constraints and proposing all changes that I
would to do. For example, my previous PEP 620 contains too many
changes and is too long.

Here is my attempt to focus on the bare minimum and (what I consider
as) the least controversial part: list current usages of C API and
constraints of these usages. This *informal* PEP should be the base of
future PEP changing the C API.

The current draft lives at:
https://github.com/vstinner/misc/blob/main/cpython/pep-c-api-next-level.rst

My PEP is based on HPy Next Level Manifesto written by Simon Cross:
https://github.com/hpyproject/hpy/wiki/c-api-next-level-manifesto

To reach most users of the C API, I cross-posted this email to
python-dev, capi-sig and hpy-dev.


+
Taking the Python C API to the Next Level
+

::

PEP: xxx
Title: Taking the Python C API to the Next Level
Author: Victor Stinner 
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 28-Sep-2021
Python-Version: 3.11

While the C API is a key of the Python popularity, it causes multiple
subtle and complex issues. There are different ways to use the C API,
each usage has its own constraints, and some constraints are exclusive.
This document lists constraints but doesn't propose changes, it only
gives vague ideas how to solve some issues. More concrete C API changes
will require writing separated PEPs.

C extensions are a key component of the Python popularity
=

The Python popularity comes from its great programming language and from
its wide collection of modules freely available on PyPI. Many of the
most popular Python modules rely directly or indirectly on C extensions
written with the C API. The Python C API is a key component of the
Python popularity.

For example, the numpy project is now a common dependency on many
scientific projects and a large part of the project is written by hand
with the C API.

**Abandoning or removing the C API** is out of question. Years ago, the
incomplete C API support was the main drawback of PyPy, since PyPy only
supported a minority of C extensions.

Today, CPython still have a similar issue. **When Cython or numpy don't
support a new Python version** (because of incompatible C API changes),
many Python projects depending on them are cannot be installed,
especially during the development phase of the next Python version.


Backward compatibility and unmaintained C extensions


One important property of the C API is the backward compatibility.
Developers expect that if their C extension works on Python 3.10, it
will work unmodified in Python 3.11: building the C extension with
Python 3.11 should be enough.

This property is even more important for unmaintained C extensions.
Sometimes, unmaintained just means that the only maintainer is busy or
overwhelmed for a few months. Sometimes, the project has no activity for
longer than 5 years.

When an incompatible change is introduced in the C API, like removing a
function or changing a function behavior, there is a **risk of breaking
an unknown number of C extensions**.

One option can be to update old C extensions when they are built on
recent Python versions, to adapt them to incompatible changes. This
conversion is non trivial and cannot handle all kinds of incompatible
changes.


Migration plan for incompatible changes
===

There should be a **sensible migration path** for large C extensions
(e.g.  numpy) when incompatible changes are introduced. Whenever
possible, it should be possible to write a **single code base** compatible
with old and new Python versions.

A **compatibility layer** can be maintained externally.  Cython and
numpy have their own internal compatibility layer.

There should be a way to easily pick up common errors introduced by
migrating.

One practical way to **minimize the number of broken projects** is to
attempt to check in advance if an incompatible change is going to break
popular C extensions. For broken C extensions, propose a fix and wait
until a new release includes the fix, before introducing the change in
Python. Obviously, it doesn't solve the problem of less popular C
extensions and private C extensions.


Obtain the best possible performance


There are two main reasons for writing a C extension: implement a
function which cannot be written in pure Python, or write a **C
accelerator**: rewrite the 10% of an application in C where 90% of the
CPU time is spent. About the former use case, the intent is to obtain
the best possible performance. Tradeoffs are made with portability: it
is acceptable to only support a limited number of Python versions and to
only support a limited number of Python

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
The Python Debug Build document lists changes compared to a release build:
https://docs.python.org/dev/using/configure.html#python-debug-build

Sometimes, I'm confused that "./python" (Python built locally in debug
mode) displays warnings, whereas "python" (Fedora package) doesn't.


See also the Python Development Mode:
https://docs.python.org/dev/library/devmode.html#devmode

The documentation starts with: "The Python Development Mode introduces
additional runtime checks that are too expensive to be enabled by
default."

Victor

On Tue, Sep 28, 2021 at 3:33 PM Eric V. Smith  wrote:
>
> On 9/28/2021 9:17 AM, Antoine Pitrou wrote:
> > On Tue, 28 Sep 2021 09:14:38 -0400
> > "Eric V. Smith"  wrote:
> >> On 9/28/2021 9:10 AM, Antoine Pitrou wrote:
> >>> On Tue, 28 Sep 2021 08:55:05 -0400
> >>> "Eric V. Smith"  wrote:
> > So I prefer to teach everybody how to use "-X frozen_modules=off" if
> > they want to hack the stdlib for their greatest pleasure. I prefer
> > that such special use case requires an opt-in option, the special use
> > case is not special enough to be the default.
>  I agree with Victor here: I'd rather have #1.
> 
>  As a compromise, how about go with #1, but print a warning if python
>  detects that it's not built with optimizations or is run from a source
>  tree (the conditions in #2 and #3)? The warning could suggest running
>  with "-X frozen_modules=off". I realize that it will probably be ignored
>  over time, but maybe it will provide enough of a reminder if someone is
>  debugging and sees the warning.
> >>> What would be the point of printing a warning instead of doing just
> >>> what the user is expecting?
> >> To me, the point would be to get the same behavior no matter which
> >> python executable I run, and without regard to where I run it from.
> > But why do you care about this?  What does it change *concretely*?
>
> It reduces the number of things I have to remember which are different
> based on where I'm running python (or which executable I'm running).
>
> Eric
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/2FN2OCYGMLU5XST44N7SFDUAI426VRB3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F6SHNS7EU2JCBV4FMYMFPC2JZZJV2L6Y/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
Would it be possible to add a __file__ attribute?

Victor

On Tue, Sep 28, 2021 at 2:47 PM Pablo Galindo Salgado
 wrote:
>
> > What is the annoyance? What is different between frozen and not frozen?
>
> One interesting consequence of what Eric mentioned (They have a different 
> loader and repr.  Also, frozen modules do not
> have __file__ set (and __path__ is always []).) is that frozen modules don't 
> have a `__file__` attribute IIRC and therefore
>  tracebacks won't include the source.
>
>
> On Mon, 27 Sept 2021 at 22:31, Victor Stinner  wrote:
>>
>> Hi Eric,
>>
>> Which stdlib modules are currently frozen? If I really want to hack
>> site.py or os.py for whatever reason, I just have to use "python3 -X
>> frozen_modules=off"?
>>
>> > 1. always default to "on" (the annoyance for contributors isn't big 
>> > enough?)
>>
>> What is the annoyance? What is different between frozen and not frozen?
>>
>> Victor
>>
>> On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  
>> wrote:
>> >
>> > We've frozen most of the stdlib modules imported during "python -c
>> > pass" [1][2], to make startup a bit faster.  Import of those modules
>> > is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>> > to "off" but we'd like to default to "on".  The blocker is the impact
>> > on contributors.  I expect many will make changes to a stdlib module
>> > and then puzzle over why those changes aren't getting used.  That's an
>> > annoyance we can avoid, which is the point of this thread.
>> >
>> > Possible solutions:
>> >
>> > 1. always default to "on" (the annoyance for contributors isn't big 
>> > enough?)
>> > 2. default to "on" if it's a PGO build (and "off" otherwise)
>> > 3. default to "on" unless running from the source tree
>> >
>> > Thoughts?
>> >
>> > -eric
>> >
>> >
>> > [1] https://bugs.python.org/issue45020
>> > [2] FWIW, we may end up also freezing the modules imported for "python
>> > -m ...", along with some other commonly used modules (like argparse).
>> > That is a separate discussion.
>> > ___
>> > Python-Dev mailing list -- python-dev@python.org
>> > To unsubscribe send an email to python-dev-le...@python.org
>> > https://mail.python.org/mailman3/lists/python-dev.python.org/
>> > Message archived at 
>> > https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/CLODS7B5Z3UEZTQ7QIALG2DWB4H37EWP/
>> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CRL7C7I2VHGI56TFJMK6U53LCKRRHDA3/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  wrote:
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?

Honestly, for me, #1: always on, is the most reasonable choice.

I dislike when Python behaves differently depending on subtle things
like "was it built with optimizations" or "is Python started from its
source tree"?

When I built Python without optimization and/or from its source tree,
I do that to debug an issue. If the bug goes away in this case, it can
waste my time.

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.

--

It means that the site module module can no longer be "customized" by
modifying directly the site.py file (inject a path in PYTHONPATH env
var where the customized site.py lives). But there is already a
supported way to customize the site module: create a module named
"sitecustomize" or "usercustomizer". I recall that virtualenv likes to
override stdlib site.py with its own code. tox uses virtualenv by
default. Someone should check if freezing site doesn't break
virtualenv and tox, since they seem to be popular in Python. The venv
doesn't need to override site.py and tox can use venv if I recall
correctly.

If site.py customization is too popular, I would suggest to not freeze
this one, until the community stops doing that.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FWAYUPG5DRRHFULNPXQOZF3NR6QZ37K2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-27 Thread Victor Stinner
On Tue, Sep 28, 2021 at 12:52 AM Eric Snow  wrote:
> On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner  wrote:
> > Which stdlib modules are currently frozen? If I really want to hack
> > site.py or os.py for whatever reason, I just have to use "python3 -X
> > frozen_modules=off"?
>
> The single-source-of-truth is Tools/scripts/freeze_modules.py.  After
> running "make regen-frozen" you'll find a cleaner list in
> Python/frozen_modules/MANIFEST.  You can also look at the generated
> code in Makefile.pre.in or Python/frozen.c.  Finally, you can run
> "./python -X frozen_modules=on -c 'import _imp;
> print(_imp._frozen_module_names())'"

Ok, so compared to Python 3.10, the following 13 stdlib modules can
now be frozen:

* _collections_abc
* _sitebuiltins
* abc
* codecs
* genericpath
* io
* ntpath
* os
* os.path
* posixpath
* site
* stat
* zipimport

(I tested on Linux. I guess that the list is the same on other
operating systems.)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RYRZ7GLC6ULGXZJ4KHRIGH4XCIMW5M4H/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-27 Thread Victor Stinner
Hi Eric,

Which stdlib modules are currently frozen? If I really want to hack
site.py or os.py for whatever reason, I just have to use "python3 -X
frozen_modules=off"?

> 1. always default to "on" (the annoyance for contributors isn't big enough?)

What is the annoyance? What is different between frozen and not frozen?

Victor

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  wrote:
>
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?
>
> -eric
>
>
> [1] https://bugs.python.org/issue45020
> [2] FWIW, we may end up also freezing the modules imported for "python
> -m ...", along with some other commonly used modules (like argparse).
> That is a separate discussion.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CLODS7B5Z3UEZTQ7QIALG2DWB4H37EWP/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Regressions caused the recent work on ceval.c and frame objects

2021-09-19 Thread Victor Stinner
Hi,

The recent optimization work on ceval.c and frame objects introduced
regressions and the situation is stuck for 4 months (bpo-43760).

Right now, maybe the best would be to revert the 2 commits in the 3.10
branch, to get more time in Python 3.11 development cycle to solve
these issues.


(1) "The DISPATCH() macro is not as efficient as it could be (move
PyThreadState.use_tracing)"
https://bugs.python.org/issue43760

This change introduced an incompatible C API change. It's not
documented in What's New in Python 3.10 and there is no solution for
the 4 broken projects (including Cython).

I proposed a C API but so far, nobody implemented it.

Another option is to fix each project since the list is short. Right
now, only 4 projects are known to be broken. Fixing Cython is not
enough, you need to get a new release of broken projects (using
Cython) to regenerate the C code with the updated Cython.


(2) "Performance regression 3.10b1 and later on Windows: Py_DECREF()
not inlined in PGO build"
https://bugs.python.org/issue45116

Changes made in bpo-43760 caused a performance regression on the PGO
build of Windows.

It's a tricky issue about thresholds in compiler PGO optimization,
inlining or not static inline functions, number of statements per
function, etc.

It was proposed to workaround the specific case of the huge
_PyEval_EvalFrameDefault() function (3500 lines of C code) by
converting again some static inline functions to macros. They were
macros in Python 3.8 and were fine in Python 3.9. The performance
regression with the Windows PGO build was introduced by the recent
ceval.c work.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XLEMW7PWYGIK2RHHICO3CNITZ4ETO3OZ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: How to decipher Windows errors on builds?

2021-09-16 Thread Victor Stinner
Hi Mark,

https://bugs.python.org/issue45220 now tracks these Windows build errors.

Victor

On Thu, Sep 16, 2021 at 3:25 PM Mark Shannon  wrote:
>
> Hi,
>
> I'm getting build errors on Windows for
> https://github.com/python/cpython/pull/28386
>
> The error message is:
>
> C:\Program Files (x86)\Windows
> Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188:
> D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa01688(47)
> : fatal error RC1116: RC terminating after preprocessor errors
> [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
>
> Can anyone translate that into English for me?
>
> It suggests that there are errors in the preprocessor. But how do I tell
> what the error is?
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/SOCIYBNX5VZYPZFQYKWRCCPPN7TNVUMJ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ZTTG53Z6GDXBQZSPNYK2BVG7MPX2BGN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: python3.10rc2 compilation on android/termux/clang12.0.1 fails

2021-09-16 Thread Victor Stinner
Hi,

On Wed, Sep 15, 2021 at 7:16 PM Sandeep Gupta  wrote:
> I get following warnings and errors:
> Python/pytime.c:398:10: warning: implicit conversion from 'long' to 'double' 
> changes value from 9223372036854775807 to 9223372036854775808 
> [-Wimplicit-const-int-float-conver
> sion]
>if (!_Py_InIntegralTypeRange(_PyTime_t, d)) {

This warning is tracked at: https://bugs.python.org/issue39277

I proposed a fix but i'm not sure that it's correct and so I never did
anything with this PR: https://github.com/python/cpython/pull/17933

> Python/bootstrap_hash.c:141:17: error: implicit declaration of function 
> 'getrandom' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
>n = getrandom(dest, n, flags);
>^
> Python/bootstrap_hash.c:145:17: error: implicit declaration of function 
> 'getrandom' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
>n = getrandom(dest, n, flags);

This code path should only be taken if HAVE_GETRANDOM macro is
defined. Is it defined in your pyconfig.h file?

Is the getrandom() function available in your libc?

configure.ac tries to build this code:
---
#include 

int main() {
char buffer[1];
const size_t buflen = sizeof(buffer);
const int flags = 0;
/* ignore the result, Python checks for ENOSYS at runtime */
(void)getrandom(buffer, buflen, flags);
return 0;
}
---

Maybe Android needs a different #include? Or a different test?

> Not sure if this is limitation of the platform or Python codebase needs fixes.

Enhancements are welcomed :-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H35G7QNDUFNQS6AXMN76K3FRE63BXOXF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Bug fixed in the bugs.python.org OAuth-based authentication: user logged as the wrong account

2021-09-13 Thread Victor Stinner
Hi,

A bug has been identified and *fixed* in the OAuth-based
authentication code used on the Python bug tracker bugs.python.org
(BPO) to log in with GitHub, Launchpad or Google. Under some
conditions, it was possible to be logged as another person account. We
are only aware of a single user affected by the issue. We are not
aware of any account takeover.

All bugs at bugs.python.org are public: being logged as the wrong
account cannot give access to private bugs. The main risk is if an
attacker could be logged as an administrator (the "Coordinator" role)
which allows to change the bug tracker configuration and to change
accounts (add/remove roles, see/change the email address, etc.). We
are not aware of any abuse.

All OAuth accounts have been removed in the database to fully fix the
issue. Users using OAuth-based authentication must associate again
(once) their GitHub, Launchpad or Google account with their BPO
account.

A BPO account contains the following information: Name, Login Name,
GitHub Name, Organisation, Timezone, Homepage, Contributor Form
Received, Is Committer, E-mail address, Alternate E-mail addresses.
All fields but Name and Timezone are hidden to other accounts, only
coordinators can see all fields of other accounts. You can check in
the "Your Details" page for the your account change log.

Thanks Ammar Askar, Berker Peksağ and Ee Durbin who fixed the bug!

Source code of bugs.python.org (Roundup fork):
https://github.com/psf/bpo-tracker-cpython

The OAuth-based authentication is an extension written for
bugs.python.org. The bug report and its fix:

* https://github.com/python/bugs.python.org/issues/64
* 
https://github.com/psf/bpo-tracker-cpython/commit/0a32e072aafca20c0bf51cf16fb6a7328cdd720a

Report issues with bugs.python.org:
https://github.com/python/bugs.python.org/issues

To report sensitive issues, write to: secur...@python.org

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CIXIB6EMN7HOPMXFJI7EBK7V7OPK4E7H/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Spam from Daniel Scott

2021-09-10 Thread Victor Stinner
Hi,

For 1 or 2 years, I'm getting spam messages from "Daniel Scott". It
started with empty replies to multiple Python mailing lists. Mailing
list owners: please block his email address. I never ever saw any
useless message from this name (sorry if a real Daniel Scott plans to
contribute to Python and will be blocked by mistake).

Today, I got an email to my @python.org email account.

And "Daniel Scott" also commented a closed issue:
"I left you to see conduct on phython assigned."
https://github.com/python/cpython/pull/23734#issuecomment-916568696

SPAM SPAM SPAM SPAM SPAM SPAM SPAM

What can we do to reduce the spam?

For GitHub, I reported the comment as a spam:

(o) I want to report abusive content or behavior
=> I want to report SPAM, a user that is disrupting me or my
organization's experience on GitHub, or a user who is using my
personal information without my permission
==> A user's profile or repository is SPAM.

In the comment, I added a link to this email :-)

"Daniel": Please stop what you are doing and try to find a better way
to use your free time. Thanks!

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MOOL5KWUKG2ZODH2MFTQOFNULMN5M227/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-09 Thread Victor Stinner
I proposed bytes.byte earlier in this thread:
https://mail.python.org/archives/list/python-dev@python.org/message/KBVVBJL2PHI55Y26Z4FMSCJPER242LFA/

Gregory dislikes the name: "I don't *like* to argue over names (the
last stage of anything) but I do need to point out how that sounds to
read".
https://mail.python.org/archives/list/python-dev@python.org/message/DGJWM3VMNMDBUTGYG72H5WLKDWBYFSUV/

That's why I proposed: bytes.fromchar(). I still like bytes.byte() :-)

Victor

On Thu, Sep 9, 2021 at 11:07 AM Antoine Pitrou  wrote:
>
> On Thu, 9 Sep 2021 18:55:04 +1000
> Nick Coghlan  wrote:
> >
> > P.S. The fact that it *didn't* look like the inverse operation for
> > `int.from_bytes` was one advantage of calling the method
> > `bytes.fromord` instead of `bytes.fromint`, but I'm still happy the SC
> > is right that `bytes.fromint` is a more comprehensible method name
> > overall.
>
> Perhaps we can call it `bytes.byte` to make it unambiguous?
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/WZUPBP4UASRCJLAKP6FMQJLLMYJY22CL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6W4G32NOBXAQ73VESVE4UL7AZIWUAD6A/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-09 Thread Victor Stinner
Hum, it seems like this is a confusion between converting a whole
bytes *string* to/from an integer, and converting a single *character*
to/from an integer.

I propose to rename PEP 467 method bytes.fromint(n) to =>
bytes.fromchar(n) <= to convert an integer to a single *character*: it
fails if n is not in the [0; 255] range. "char" comes from
"character", as "bchr()" means "bytes character".

For C programmers, the usage of the "char" type is common for a single
*character*. The char type is not treated as an integer, but part of a
character string. All string functions take "char*" type (strcpy,
printf, etc.). Converting an integer to a "char" in C: "int x = 1;
char ch = (char)x;".

I suggest to *not* add a builtin function bchr(), it's not common
enough to justify to add it: it's trivial to create you own bchr()
function:

bchr = bytes.fromchar

By the way, it's a little unfortunate that int methods have an
underscore in their name (int.to_bytes, int.bit_length,
int.as_integer_ratio), whereas bytes methods have no undersore in
their name (bytes.removeprefix, bytes.islower). I guess that we should
follow the trend of existing methods: so no underscore for
bytes/bytearray methods.

Victor

On Wed, Sep 8, 2021 at 7:06 PM Brandt Bucher  wrote:
>
> Steven D'Aprano wrote:
> > To me, it sounds like should be the opposite of int.from_bytes.
> > >>> int.from_bytes(b'Hello world', 'little')
> > 121404708502361365413651784
> > >>> bytes.from_int(121404708502361365413651784, 'little')
> > # should return b'Hello world'
> > If that's not the API being suggested, that's going to be confusing.
>
> I'm a bit lost here... why are we convinced at all that we need a new way to 
> do this? Hasn't this functionality already existed for years?
>
> >>> x = int.from_bytes(b"*", "little")
> >>> x
> 42
> >>> x.to_bytes(1, "little")
> b'*'
>
> Brandt
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/FMG5K4BOX5GSUR2KU3G5ZLBBUIC3EQKD/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGUWZ7FV7CA5LOCPIFOP3WUP6Z5NQDTD/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-08 Thread Victor Stinner
On Wed, Sep 8, 2021 at 7:46 AM Steven D'Aprano  wrote:
> >>> bytes.from_int(121404708502361365413651784, 'little')
> # should return b'Hello world'

Really? I don't know anyone serializing strings as a "bigint" number.
Did you already see such code pattern in the wild? Usually, bytes are
serialized as... bytes, no? Sometimes, bytes are serialized as base64
or hexadecimal to go through into an ASCII ("7-bit") bytestream. But I
don' recall any file format serializing bytes as a single large
decimal number.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/65FBFRCWV4V5SAP44EQG3XKHW6Q7C3QL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: deprecated-removed

2021-09-08 Thread Victor Stinner
Hi Serhiy,

For me, "deprecated" should be used if the removal is *not* planned:
the feature is too popular, and removing it would break too much 3rd
party code.

"deprecated-removal" should be used if the feature removal *is*
planned. IMO it's ok if we forget to remove the feature and keep it
longer than expected. But it should help users to raise their
awareness that the removal *will* happen. Having a release version
gives a deadline.

Victor

On Wed, Sep 8, 2021 at 12:37 PM Serhiy Storchaka  wrote:
>
> I always thought that the "deprecated" directive is used if the term of
> removing the deprecated feature is not set yet, and the
> "deprecated-removed" directive is used if it is set. After removing the
> feature both directives are replaced with the "versionchanged" directive
> which only specifies the version of removing, not version of deprecation.
>
> But after reading the devguide [1] I am no longer so confident. The
> devguide clearly describes it as specifying version at which the feature
> "is" removed, not "will be" removed. Maybe I always used this directive
> incorrectly? Or its meaning was changed with time? Or I incorrectly
> understand the devguide wording? Or devguide is incorrect?
>
> Before 3.8 I only seen "deprecated-removed" with future removing
> version. The first use "deprecated-removed" for feature which is already
> deleted happened in 3.8 (maybe the directive was just not updated), and
> there are now many uses for features removed in asyncio in 3.10.
>
> What was the initial meaning of "deprecated-removed", how it should be
> used now, and is the devguide correct?
>
> [1]
> https://devguide.python.org/documenting/?highlight=deprecated-removed#paragraph-level-markup
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/SWBM2N4EFPVQFP4TX6Q33L5OK2WPBFRU/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/STQ6JSRYODHMNIEYTYSCM3AZH3MSMRWC/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making code object APIs unstable

2021-09-06 Thread Victor Stinner
Oh, I didn't know this *existing* C API function:

PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno)

So Cython could be modified to use it, no?

Victor

On Tue, Sep 7, 2021 at 12:44 AM Guido van Rossum  wrote:
>
> On Fri, Sep 3, 2021 at 4:12 PM Victor Stinner  wrote:
>>
>> On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum  wrote:
>> > FWIW I've applied for an exception from the two-release deprecation policy 
>> > from the SC:
>> > https://github.com/python/steering-council/issues/75
>>
>> On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I
>> didn't check how many are using Cython.
>
>
> Most of them. :-)
>
> I wrote a script that to do a similar search on the 4000 most popular 
> packages, disregarding Cython-generated files (these have "/* Generated by 
> Cython  */" in their first line). Now the list collapsed to this:
>
> Cython-3.0a7.tar.gz: 11 hits in 3 files
> frozendict-2.0.6.tar.gz: 14 hits in 8 files
> gevent-21.8.0.tar.gz: 1 hits in 1 files
> JPype1-1.3.0.tar.gz: 1 hits in 1 files
> mypy-0.910.tar.gz: 2 hits in 1 files
> reportlab-3.6.1.tar.gz: 1 hits in 1 files
> setuptools-9.1.tar.gz: 1 hits in 1 files
>
> Of these:
>
> Cython: obviously :-)
> frozendict: calls PyCode_NewEmpty; seems to include modified CPython headers
> gevent: Uses Cython's __Pyx_PyCode_New in a generated .h file
> JPype: calls PyCode_NewEmpty
> mypy: PyCode_NewEmpty mentioned in a comment
> reportlab: calls PyCode_NewEmpty
> setuptools: in a file generated by Pyrex (Cython's predecessor)
>
> There wasn't a single call to PyCode_NewWithPosOnlyArgs in any of these apart 
> from Cython.
>
> In addition, I just heard from the SC that they've approved the exception. So 
> we will remove these two APIs from 3.11 without deprecation. I've filed 
> https://bugs.python.org/issue45122 to get this done (looking for volunteers).
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQOKM6V7UJA2M6LBZ4JPHLVRK3OCHNRU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making code object APIs unstable

2021-09-03 Thread Victor Stinner
On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum  wrote:
> FWIW I've applied for an exception from the two-release deprecation policy 
> from the SC:
> https://github.com/python/steering-council/issues/75

On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I
didn't check how many are using Cython.

yarl-1.6.3.tar.gz
wsaccel-0.6.3.tar.gz
wordcloud-1.8.1.tar.gz
uvloop-0.16.0.tar.gz
tslearn-0.5.2.tar.gz
uamqp-1.4.1.tar.gz
tweedledum-1.1.0.tar.gz
tinycss-0.4.tar.gz
thriftpy-0.3.9.tar.gz
thriftpy2-0.4.14.tar.gz
Theano-PyMC-1.1.2.tar.gz
Theano-1.0.5.tar.gz
TA-Lib-0.4.21.tar.gz
tables-3.6.1.tar.gz
ssh2-python-0.26.0.tar.gz
srsly-2.4.1.tar.gz
ssh-python-0.9.0.tar.gz
statsmodels-0.12.2.tar.gz
sphinx-gallery-0.9.0.tar.gz
sktime-0.7.0.tar.gz
Shapely-1.7.1.tar.gz
wxPython-4.1.1.tar.gz
scikit-image-0.18.2.tar.gz
sasl-0.3.1.tar.gz
scikit-surprise-1.1.1.tar.gz
s2sphere-0.2.5.tar.gz
ruptures-1.1.4.tar.gz
runstats-2.0.0.tar.gz
ruamel.yaml.clib-0.2.6.tar.gz
reedsolo-1.5.4.tar.gz
recordclass-0.15.1.tar.gz
reportlab-3.6.1.tar.gz
rasterio-1.2.6.tar.gz
rapidfuzz-1.4.1.tar.gz
qiskit-terra-0.18.1.tar.gz
pyzmq-22.2.1.tar.gz
pyxDamerauLevenshtein-1.7.0.tar.gz
PyWavelets-1.1.1.tar.gz
python-crfsuite-0.9.7.tar.gz
py_spy-0.3.8.tar.gz
pysimdjson-4.0.2.tar.gz
pysam-0.16.0.1.tar.gz
pypcap-1.2.3.tar.gz
pyngrok-5.0.6.tar.gz
PyLBFGS-0.2.0.13.tar.gz
pyjq-2.5.2.tar.gz
pyhacrf-datamade-0.2.5.tar.gz
pyflux-0.4.15.tar.gz
pygame-2.0.1.tar.gz
pyemd-0.5.1.tar.gz
pydevd-2.4.1.tar.gz
pydevd-pycharm-212.5080.18.tar.gz
plyvel-1.3.0.tar.gz
pmdarima-1.8.2.tar.gz
peewee-3.14.4.tar.gz
paramiko-2.7.2.tar.gz
osmium-3.2.0.tar.gz
orderedset-2.0.3.tar.gz
numpydoc-1.1.0.tar.gz
numdifftools-0.9.40.tar.gz
numba-0.53.1.tar.gz
numcodecs-0.8.1.tar.gz
NetfilterQueue-0.8.1.tar.gz
neobolt-1.7.17.tar.gz
Naked-0.1.31.tar.gz
mypy-0.910.tar.gz
msgpack-python-0.5.6.tar.gz
msgpack-1.0.2.tar.gz
mojimoji-0.0.11.tar.gz
mpi4py-3.1.1.tar.gz
matrixprofile-1.1.10.tar.gz
marisa-trie-0.7.7.tar.gz
lupa-1.9.tar.gz
lxml-4.6.3.tar.gz
lsm-db-0.6.4.tar.gz
linearmodels-4.24.tar.gz
lightfm-1.16.tar.gz
Levenshtein-0.13.0.tar.gz
leven-1.0.4.tar.gz
lda-2.0.0.tar.gz
jsonobject-0.9.10.tar.gz
jq-1.2.1.tar.gz
JPype1-1.3.0.tar.gz
jenkspy-0.2.0.tar.gz
implicit-0.4.4.tar.gz
imgui-1.3.0.tar.gz
imbalanced-learn-0.8.0.tar.gz
imagecodecs-2021.7.30.tar.gz
httptools-0.3.0.tar.gz
httpretty-1.1.4.tar.gz
hmmlearn-0.2.6.tar.gz
hdbscan-0.8.27.tar.gz
gssapi-1.6.14.tar.gz
grpcio-tools-1.39.0.tar.gz
grpcio-1.39.0.tar.gz
graphene-federation-0.1.0.tar.gz
GPy-1.10.0.tar.gz
gluonnlp-0.10.0.tar.gz
gevent-21.8.0.tar.gz
gensim-4.0.1.tar.gz
fuzzyset-0.0.19.tar.gz
fuzzysearch-0.7.3.tar.gz
frozendict-2.0.6.tar.gz
flower-1.0.0.tar.gz
Fiona-1.8.20.tar.gz
fastrlock-0.6.tar.gz
fastparquet-0.7.1.tar.gz
fastdtw-0.3.4.tar.gz
fastavro-1.4.4.tar.gz
edlib-1.3.8.post2.tar.gz
editdistance-0.5.3.tar.gz
econml-0.12.0.tar.gz
dtaidistance-2.3.2.tar.gz
DoubleMetaphone-0.1.tar.gz
django-localflavor-3.1.tar.gz
dependency-injector-4.35.2.tar.gz
dedupe-hcluster-0.3.8.tar.gz
dedupe-2.0.8.tar.gz
ddtrace-0.51.2.tar.gz
cytoolz-0.11.0.tar.gz
Cython-0.29.24.tar.gz
correctionlib-2.0.0.tar.gz
clickhouse-driver-0.2.1.tar.gz
cityhash-0.2.3.post9.tar.gz
cchardet-2.1.7.tar.gz
causalml-0.11.1.tar.gz
Cartopy-0.19.0.post1.tar.gz
av-8.0.3.tar.gz
asyncpg-0.24.0.tar.gz
astral-2.2.tar.gz
arch-5.0.1.tar.gz
arcgis-1.9.0.tar.gz
altgraph-0.17.tar.gz
aiokafka-0.7.1.tar.gz
aiohttp-3.7.4.post0.tar.gz
affinegap-1.11.tar.gz

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6RO2WFU5Q7UQDVL72IRMT4T6L4GEAKB6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
I saw Python projects injecting fake frames for XML and JSON parsers,
maybe also configuration file (.ini?) parsers. So text files which
have line numbers ;-)

On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
 wrote:
> I don't think we should think on those terms. We certainly don't want to be 
> on a case
> where yet again we cannot change the internals because we have an official 
> C-API exposed.

PyCode_New() is annoying since it requires to provide *all* arguments.
I'm thinking of an Frame API which only allows to set 2 values:
filename and line number. Nothing else.

Something like: frame = PyFrame_New(filename, lineno).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AII6O6ZXEENM7FJH3JRJIPTG4BRKJ3G5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
If creating a fake frame is a common use case, we can maybe write a
public C API for that. For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.

Victor

On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel  wrote:
>
> Guido van Rossum schrieb am 13.08.21 um 19:24:
> > In 3.11 we're changing a lot of details about code objects. Part of this is
> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine
> > Grained Error Locations in Tracebacks).
> >
> > As a result, the set of fields of the code object is changing. This is
> > fine, the structure is part of the internal API anyway.
> >
> > But there's a problem with two public API functions, PyCode_New() and
> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
> > signatures are incompatible with previous versions, and they have to be
> > since the set of values needed to create a code object is different. (The
> > types.CodeType constructor signature is also changed, and so is its
> > replace() method, but these aren't part of any stable API).
> >
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP
> > 387 stable ABI. What should we do?
> >
> > A. We could deprecate them, keep (restore) their old signatures, and create
> > crippled code objects (no exception table, no endline/column tables,
> > qualname defaults to name).
> >
> > B. We could deprecate them, restore the old signatures, and always raise an
> > error when they are called.
> >
> > C. We could just delete them.
> >
> > D. We could keep them, with modified signatures, and to heck with ABI
> > compatibility for these two.
> >
> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the
> > posonlyargcount (which is the only difference between the two), and d*mn
> > the torpedoes.
> >
> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New()
> > (and deprecate it).
> >
> > If these weren't part of the stable ABI, I'd choose (E). [...]
>
> I also vote for (E). The creation of a code object is tied to interpreter
> internals and thus shouldn't be (or have been) declared stable.
>
> I think the only problem with that argument is that code objects are
> required for frames. You could argue the same way about frames, but then it
> becomes really tricky to, you know, create frames for non-Python code.
>
> Since we're discussing this in the context of PEP 657, I wonder if there's
> a better way to create tracebacks from C code, other than creating fake
> frames with fake code objects.
>
> Cython uses code objects and frames for the following use cases:
>
> - tracing generated C code at the Python syntax level
> - profiling C-implemented functions
> - tracebacks for C code
>
> Having a way to do these three efficiently (i.e. with close to zero runtime
> overhead) without having to reach into internals of the interpreter state,
> code objects and frames, would be nice.
>
> Failing that, I'm ok with declaring the relevant structs and C-API
> functions non-stable and letting Cython use them as such, as we always did.
>
> Stefan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QYDIQ5SS3FQCTDBBZ3JTI643KV4F6TNY/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

2021-08-30 Thread Victor Stinner
Hi Ammar,

On Tue, Aug 24, 2021 at 12:17 AM Ammar Askar  wrote:
> 1. Weekly summary emails with bug counts and issues from the week,
> example: 
> https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/

These emails annoy me. I learnt to mark them as read, as soon as I receive them.

I prefer to pick my favourite bugs directly at bugs.python.org and
then follow email notifications. Well, I'm already getting too many
notifications in general :-)

> 2. Emails sent to the new-bugs-announce and python-bugs-list for new
> issues and comments to existing issues.

I don't know these lists.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OWZMAX3MZPJDCJ4L6KSWQK2V4EUHYLCR/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: A ban from Core Developer spaces.

2021-08-27 Thread Victor Stinner
Hi,

I collected past events about bans and code of conduct incidents in
the Python community:
https://pythondev.readthedocs.io/diversity.html#python-code-of-conduct-bans

It may help some people to see how bans and incidents are handled in Python.

See also the https://pythondev.readthedocs.io/community.html page.

Contact me in private (no need to spam this public list) if you would
like helping me to complete this list ;-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MNBQGCVZXPJMZPULMTEXUNWOQGQQ3C7M/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: My apologies to the list

2021-08-27 Thread Victor Stinner
Hi Antoine,

I have an alternative to GMane. I'm using a different email address
for anything related to Python (and OSS projects in general):
python-dev list, bug tracker, buildbots, GitHub notifications, etc.
Some days, I don't read this email account at all, but I can still
read my personal and profesional emails, without seeing
"notifications" on the Python topics that I'm following (and not be
tempted to reply!).

For me, this level of granularity works pretty well ;-)

Victor

On Thu, Aug 26, 2021 at 11:10 AM Antoine Pitrou  wrote:
>
> On Wed, 25 Aug 2021 12:22:29 -0700
> Brett Cannon  wrote:
> > I just wanted to apologize for any angst or noise my replies to Marco
> > caused folks. I should have known that correcting Marco on how to address
> > me would have triggered an outsized reply (the real intent of that overall
> > email was to communicate the process of banning someone to the rest of the
> > list so people didn't leave faster than we addressed it; obviously didn't
> > work). It appears my judgement was more impaired than I realized while I
> > have been sick.
> >
> > As such, I have amended my rules for when I *don't* read OSS emails (this
> > excludes SC business):
> >
> >1. At least one day every weekend (to avoid having my whole week ruined
> >by mean people)
> >2. Outside of work hours one month out of the year (to prevent burn-out)
> >3. On vacation (to unplug appropriately)
> >4. While I am ill (to prevent impaired judgement)
> >
> > I have actually gone as far as to separate my personal email and OSS/
> > python.org email accounts into separate email providers so that I can
> > enforce this separation cleanly.
>
> For the record, my personal arrangement for years has been to read most
> open source mailing-lists using GMane, on a NNTP reader separate from my
> main mail client.  This works fine when I don't want to read open
> source-related e-mails :-)
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/VLQUDDVMJKL7VDNFZLMSZGTJL2PWCU5Q/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OFDDNTIQODMMXVUYH4W2H2LF2KGZEDVH/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Victor Stinner
Since Cython is a common consumer of this C API, can somone please dig
into Cython to see exactly what it needs in terms of API? How does
Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
copy an existing function to only override some fields, something like
CodeType.replace(field=new_value)?

If possible, I would prefer that Cython only uses the *public* C API.
Otherwise, it will be very likely that Cython will break at every
single Python release. Cython has a small team to maintain the code
base, whereas CPython evolves much faster with a larger team.

Victor

On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
>
> Doing a search of a huge codebase (work), the predominant user of PyCode_New* 
> APIs appears to be checked in Cython generated code (in all sorts of 
> third_party OSS projects). It's in the boilerplate that Cython extensions 
> make use of via it's __Pyx_PyCode_New macro.  
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
>
> I saw very few non-Cython uses.  There are some, but at a very quick first 
> glance they appear simple - easy enough to reach out to the projects with a 
> PR to update their code.
>
> The Cython use will require people to upgrade Cython and regenerate their 
> code before they can use the Python version that changes these. That is not 
> an uncommon thing for Cython. It's unfortunate that many projects on ship 
> generated sources rather than use Cython at build time, but that isn't _our_ 
> problem to solve. The more often we change internal APIs that things depend 
> on, the more people will move their projects towards doing the right thing 
> with regards to either not using said APIs or rerunning an up to date code 
> generator as part of their build instead of checking in generated unstable 
> API using sources.
>
> -gps
>
>
> On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum  wrote:
>>
>> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:
>>>
>>> [...]
>>> A cloning-with-replacement API that accepted the base code object and the 
>>> "safe to modify" fields could be a good complement to the API deprecation 
>>> proposal.
>>
>>
>> Yes (I forgot to mention that).
>>
>>>
>>> Moving actual "from scratch" code object creation behind the Py_BUILD_CORE 
>>> guard with an underscore prefix on the name would also make sense, since it 
>>> defines a key piece of the compiler/interpreter boundary.
>>
>>
>> Yeah, we have _PyCode_New() for that.
>>
>>>
>>> Cheers,
>>> Nick.
>>>
>>> P.S. Noting an idea that won't work, in case anyone else reading the thread 
>>> was thinking the same thing: a "PyType_FromSpec" style API won't help here, 
>>> as the issue is that the compiler is now doing more work up front and 
>>> recording that extra info in the code object for the interpreter to use. 
>>> There is no way to synthesise that info if it isn't passed to the 
>>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>>
>>
>> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC pushed 
>> for this and implemented it). You gave me an idea now: the C equivalent to 
>> .replace() could use the same input structure; one can leave fields NULL 
>> that should be copied from the original unmodified.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> Pronouns: he/him (why is my pronoun here?)
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/67DMIW7NQE6M6LEPLANXKZQEFOFVPBBL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FFE6ULML5CWOBMFCZKTRCSSHKTPBFRFG/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

2021-08-17 Thread Victor Stinner
See also:

* https://bugs.python.org/issue44881 "Consider integration of
PyObject_GC_UnTrack() with the trashcan C API".
* https://bugs.python.org/issue44897 "Integrate trashcan mechanism
into _Py_Dealloc"

The issue44897 would make all these macros useless ;-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZTTQP5YYAKVBQWHZPOVQ4FREZJPPV3PQ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

2021-08-17 Thread Victor Stinner
If the macro is deprecated, please provide an detailed explanation how
to port existing C code to the new C API, in What's New In Python 3.X
(version where the macro is deprecated, Python 3.11 if I understood
correctly).

Also, is there a way to write a single code base working on Python
3.6-Python 3.11? It seems like mypy uses macros to support Python <
3.8 and Python >= 3.8:

#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)


A search in PyPI top 1000 project gives me 8 projects (I was too lazy
to check the full top 5000). It would be great to help these projects
(propose pull requests) to migrate to the new C API.

(*) pyrsistent-0.18.0.tar.gz

> ./pvectorcmodule.c:  Py_TRASHCAN_SAFE_BEGIN(self);
> ./pvectorcmodule.c:  Py_TRASHCAN_SAFE_BEGIN(self);

(*) PyGObject-3.40.1.tar.gz

> ./gi/pygi-resulttuple.c:Py_TRASHCAN_SAFE_BEGIN (self)

(*) pycurl-7.44.1.tar.gz

> ./src/easy.c:Py_TRASHCAN_SAFE_BEGIN(self);
> ./src/multi.c:Py_TRASHCAN_SAFE_BEGIN(self);
> ./src/share.c:Py_TRASHCAN_SAFE_BEGIN(self);

(*) mypy-0.910.tar.gz

> ./mypyc/lib-rt/CPy.h:
---
#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
#define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
#define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)
#endif
---

(*) pypi-top-1000_2021-08-17/multidict-5.1.0.tar.gz

> ./multidict/_multidict.c:Py_TRASHCAN_SAFE_BEGIN(self);

(*) pypi-top-1000_2021-08-17/immutables-0.16.tar.gz

> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)
> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)
> ./immutables/_map.c:Py_TRASHCAN_SAFE_BEGIN(self)

(*) guppy3-3.1.1.tar.gz

> ./src/sets/nodeset.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/sets/immnodeset.c:Py_TRASHCAN_SAFE_BEGIN(it)
> ./src/sets/immnodeset.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/hv.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/classifier.c:Py_TRASHCAN_SAFE_BEGIN(op)
> ./src/heapy/nodegraph.c:Py_TRASHCAN_SAFE_BEGIN(v)
> ./src/heapy/hv_cli_rel.c:Py_TRASHCAN_SAFE_BEGIN(op)

(*) frozendict-2.0.6.tar.gz

> 

Victor

On Tue, Aug 17, 2021 at 12:02 PM Łukasz Langa  wrote:
>
> Hi everybody,
> I'd like to revive this thread as I feel like we have to do something here 
> but some consensus is needed first.
>
> To recap, the current state of things is as follows:
> - in March 2000 (d724b23420f) Christian Tismer contributed the "trashcan" 
> patch that added Py_TRASHCAN_SAFE_BEGIN and Py_TRASHCAN_SAFE_END macros which 
> allow destroying nested objects non-recursively.
> - in May 2019 (GH-11841 of BPO-35983) Antoine Pitrou merged a change by 
> Jeroen Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END (unintentionally?) 
> backwards incompatible; this was released in Python 3.8.0.
> - by the way, GH-11841 introduced a new pair of macros (because they have 
> different signatures) called simply  Py_TRASHCAN_BEGIN and Py_TRASHCAN_END.
> - by that time there was already a follow-up PR open (GH-12607) to improve 
> backwards compatibility of the macros, as well as introduce tests for them; 
> this was never merged.
> - in Feb 2020 (0fa4f43db08) Victor Stinner removed the trashcan mechanism 
> from the limited C API (note: not ABI, those are macros) since it accesses 
> fields of structs not exposed in the limited C API; this was released in 
> Python 3.9.0.
> - in May 2020 Irit noticed that the backwards incompatibility (BPO-40608) 
> causes segfaults for C API code that worked fine with Python 3.7. Using the 
> new macros requires code changes but doesn't crash.
>
> Now, there are a couple of things we can do here:
> Option 1: Finish GH-12607 to fix the old macros, keeping in mind this will 
> restore compatibility lost with Python 3.8 - 3.10 only for users of 3.11+
> Option 2: Review and merge GH-20104 that reverts the macro changes that make 
> old client code segfault -- unclear what else this needs and again, that 
> would only fix it for users of 3.11+
> Option 3: Abandon GH-12607 and GH-20104, instead declaring the old macros 
> deprecated for 3.11 and remove them in 3.13
>
> I personally agree with Irit, voting +1 for Option 3 since the old macros 
> were soft-deprecated already by introducing new macros in 3.8, and more 
> importantly made incompatible with pre-3.8 usage.
>
> Let's talk on how to proceed.
>
> - Ł
>
>
>
> On 26 Apr 2021, at 23:55, Irit Katriel via Python-Dev  
> wrote:
>
>
> Re https://bugs

[Python-Dev] Re: [python-committers] Roundup to GitHub Issues migration

2021-08-05 Thread Victor Stinner
Hi Ezio,

What is the status of the migration of Python issues from
bugs.python.org (Roundup) to GitHub? Is it still a work-in-progress or
is it stalled?

Victor

On Mon, Jun 21, 2021 at 4:20 AM Ezio Melotti  wrote:
>
> As you might know, PEP 581 (Using GitHub Issues for CPython) has been
> approved.  I've been working with Ewa, Ee, the Working Group, the
> Steering Council, and the GitHub folks to make this happen, and the SC
> encouraged me to give you all a quick update.
>
> This effort is being tracked at
> : this board reflects
> the current status of the project.  The PEPs (including PEP 588 --
> GitHub Issues Migration Plan) haven't been updated yet and might
> contain outdated information, so please refer to the psf/gh-migration
> repo for the latest updates.
>
> During the next phase I will work with the WG to sort out all the
> major issues that we might encounter, and then I will once again reach
> out to you to gather feedback from the wider audience that follows
> these mailing lists.
>
> Best Regards,
> Ezio Melotti
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/QQT7DOKTBKZRFLT6GUJLNQOC3YDLBSLU/
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MLP7A3SSX5G6GMILAKEWWK5TGRA7AK54/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Python3 OSF/1 support?

2021-08-04 Thread Victor Stinner
Hi,

I suggest you to start by forking the python/cpython repository and
keep your changes in a branch. You can share it on a website, maybe
with a tarball including your patches. If it gets enough popularity,
maybe we can consider later to include these changes.

Since Alpha hardware is not longer produced and OSF/1 doesn't seem to
be maintained anymore, I'm not excited by maintaining new changes in
Python for this outdated platform. Even if the newly added code is
seen as "dead code" (unused), it has a cost. Future readers will have
to dig into the mailing list archive and the Internet to understand
why it's there, if it's still correct, if it should be removed or not,
etc.

As Brett wrote, I would prefer to first clarify the rules (PEP 11) to
support or not a new platform in Python.

> Admitted, I haven't figured out what is happening with the posixsubprocess 
> module.

Spawning processes is not stricly required, but it's highly
recommended to use Python. Many stdlib modules use subprocess directly
or indirectly. It's ok if your platform doesn't support it or now.

Anyway, it's cool to support Python on old platforms, happy hacking!

Victor

On Wed, Jul 14, 2021 at 6:34 AM Jay K  wrote:
>
> Hi. I have an Alpha/OSF machine up and running.
> It is little slow, but it works ok.
>
> I would like to run Python3 on it.
>
> I have it "compiling and working". It was easy enough so far.
>   https://github.com/python/cpython/pull/27063
>
> Admitted, I haven't figured out what is happening with the posixsubprocess 
> module.
>
> Yes? No? Maybe?
>
> I can possibly provide ongoing "support" (I don't expect it will be much) and 
> a buildbot (if really required),
> if I can get it working a bit more, if this is approved, etc. I haven't 
> looked yet at what buildbot involves.
>
> Thank you,
>  - Jay
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/NK6ER7SIFQDEZ2W7OANZNGNFPO3FV7HW/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TEJGMQLHULIYYJKXU2BYRMSCH5ZWCPT5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Repealing PEP 509 (Add a private version to dict)

2021-08-04 Thread Victor Stinner
Hi,

I wrote the PEP 509 as part of my abandonned "FAT Python" project
which was a ahead-of-time optimizer using runtime guards to deoptimize
code. I planed to abandon this PEP as well, but the dictionary version
was used by LOAD_GLOBAL opcode cache which made the version useful and
so the PEP was accepted in Python 3.6.

As soon as LOAD_GLOBAL and LOAD_ATTR optimizations are not lost, I'm
perfectly fine with removing the dictionary version. As it was said in
other messages, there is now a per-key version which can be used
instead, if I understand correctly.

Victor

On Thu, Jul 29, 2021 at 12:41 PM Mark Shannon  wrote:
>
> Hi everyone,
>
> I would like to repeal PEP 509. We don't really have a process for
> repealing a PEP. Presumably I would just write another PEP.
>
> Before I do so, I would like to know if anyone thinks we should keep
> PEP 509.
>
> The dictionary version number is currently unused in CPython and just
> wastes memory. I am not claiming that we will never need it, just that
> we shouldn't be required to have it. It should be an internal
> implementation detail that we can add or remove depending on requirements.
>
> Thoughts?
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/GV4CW3T7SUTJOYSCP6IJMV4AHDNNZIPV/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J6GMNVLSLBBZB766ZNTJO3FU3ORJ75NN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Windows buildbots may be broken

2021-08-04 Thread Victor Stinner
Hi Jason,

One month ago, I changed the Buildbot configuration:
---
Use Git "fresh" method

Git "clean" method keeps most files created by a previous build. Use
Git "fresh" method instead to ignores .gitignore rules and so remove
all generated files (like ".o" files): run a fresh build rather than
an incremental build.

https://github.com/python/buildmaster-config/commit/f83c0a321dac8bb7d0df2e3098e7d36862f0b18a
https://github.com/python/buildmaster-config/pull/255
---

See https://docs.buildbot.net/latest/manual/configuration/steps/source_git.html
documentation.

I made this change to fix the following issue:
https://mail.python.org/archives/list/buildbot-sta...@python.org/thread/Z27NUT4OY7IGUNQXCTMO4GGSR7RWVB4J/

setup.py failed with: "RuntimeError: subprocess not supported for
isolated subinterpreters". It seems like aarch64 Fedora Rawhide 3.x
doesn't use a fresh build, but incremental build. I get such error
when I use incremental build.

Victor

On Fri, Jul 30, 2021 at 5:53 PM Jason R. Coombs  wrote:
>
> Jeremy informed me that due to a race condition on a test file and the 
> CPython repo configuration for newline conversion, build bots on Windows may 
> now be failing.
>
> If you run such a buildbot, please consider running this command on your repo 
> to bypass the issue:
>
> git rm -r :/ ; git checkout HEAD -- :/
>
>
> You may want to consider adding this command after every update to the repo 
> to avoid the stale state.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/EZPAFN3BSVDBZC7CRAJEFIQVM6JOSGU5/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7UDB2PHSBI66WDTVRA6D5NMJLO734ZNZ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-08-04 Thread Victor Stinner
On Tue, Aug 3, 2021 at 7:54 PM Ethan Furman  wrote:
> I would rather keep `bchr` and lose the `.fromint()` methods.

I would prefer to only have a bytes.byte(65) method, no bchr()
built-in function. I would prefer to keep builtins namespace as small
as possible.

bytes.byte() name is similar to bytes.getbyte(). I cannot find "int"
in the name of other bytes methods.

>some_var = bytearray(bchr(65))
> vs
>some_var = bytearray.from_int(65)

bytearray(bchr(65)) sounds less efficient.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KBVVBJL2PHI55Y26Z4FMSCJPER242LFA/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?

2021-07-08 Thread Victor Stinner
Ok, thanks.

I created https://bugs.python.org/issue44584 to deprecate the feature
and schedule its removal in Python 3.12.

Victor

On Wed, Jul 7, 2021 at 8:09 PM Guido van Rossum  wrote:
>
> I agree, that flag was probably last useful when we were debugging the 
> threading machinery itself. (Evidenced by it only working on --with-pydebug, 
> which is for when you're debugging CPython itself, not any user apps.)
>
> On Wed, Jul 7, 2021 at 9:33 AM Gregory P. Smith  wrote:
>>
>>
>>
>> On Wed, Jul 7, 2021 at 2:28 AM Victor Stinner  wrote:
>>>
>>> Hi,
>>>
>>> Does anyone use threading debug PYTHONTHREADDEBUG=1 env var on a
>>> Python debug build? If not, can I just remove it?
>>>
>>> --
>>>
>>> To fix a race condition at thread exit on Linux using the glibc, I
>>> removed calls to pthread_exit() (PyThread_exit_thread()) in the
>>> _thread module:
>>>
>>>https://bugs.python.org/issue44434
>>>
>>> A side effect of this change is the removal of the
>>> "PyThread_exit_thread called" threading debug log when using
>>> PYTHONTHREADDEBUG=1 environment variable.
>>>
>>> I never used PYTHONTHREADDEBUG. I just tried it and it produces tons
>>> of output in stdout about locks. It looks basically useless because it
>>> produces way too many logs, and it pollutes stdout (ex: most Python
>>> tests fail when it's enabled).
>>>
>>> This debug mode requires to build Python in debug mode (./configure
>>> --with-pydebug):
>>>
>>>https://docs.python.org/dev/using/configure.html#python-debug-build
>>>
>>> IMO there are enough external debugging tools to debug threading
>>> issues. Python no longer has to come with its built-in logs.
>>>
>>> I propose to deprecate the feature in Python 3.11 and remove it in 2
>>> releases (Python 3.13).
>>
>>
>> I agree with its removal.
>>
>>>
>>> Victor
>>> --
>>> Night gathers, and now my watch begins. It shall not end until my death.
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at 
>>> https://mail.python.org/archives/list/python-dev@python.org/message/NMLGCDRUKLZSTK4UICJTKR54WRXU2ZGJ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/F3MFU5YD6H6NIPW5ULIRNGAMSPSWXYPN/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2JIZV4ZWRWHBAMYJYL2ELX5SOHFBSTCN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?

2021-07-07 Thread Victor Stinner
Hi,

Does anyone use threading debug PYTHONTHREADDEBUG=1 env var on a
Python debug build? If not, can I just remove it?

--

To fix a race condition at thread exit on Linux using the glibc, I
removed calls to pthread_exit() (PyThread_exit_thread()) in the
_thread module:

   https://bugs.python.org/issue44434

A side effect of this change is the removal of the
"PyThread_exit_thread called" threading debug log when using
PYTHONTHREADDEBUG=1 environment variable.

I never used PYTHONTHREADDEBUG. I just tried it and it produces tons
of output in stdout about locks. It looks basically useless because it
produces way too many logs, and it pollutes stdout (ex: most Python
tests fail when it's enabled).

This debug mode requires to build Python in debug mode (./configure
--with-pydebug):

   https://docs.python.org/dev/using/configure.html#python-debug-build

IMO there are enough external debugging tools to debug threading
issues. Python no longer has to come with its built-in logs.

I propose to deprecate the feature in Python 3.11 and remove it in 2
releases (Python 3.13).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NMLGCDRUKLZSTK4UICJTKR54WRXU2ZGJ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Is the Python review process flawed?

2021-07-01 Thread Victor Stinner
Thanks Brian for your nice summary! Let me complete it.

Warning: I didn't look at https://github.com/python/cpython/pull/4819
at all. My email is a really general remark on the Python workflow.
You in this email is not the OP, but "any contributor" ;-)

What contributors don't see is that regressions are very common in
Python: bugs introduced by recent changes (feature or bugfix).
Usually, the author of the change is gone when the regression pops up,
and the core developers who merged the PR "must" handle the
regression.

And it's not about a single regression and you're done ever, you can
get more. In the worst case, the fix for a regression introduced a
second regression. And then you have to handle new backward
compatibility issue when fixing a stable Python version :-(

What happens usually is that some modules have no maintainer. Once you
merge a single change in a module, boom! You are now the new
maintainer for your entire life time! More and more people will ask
you to look at their "very important" bug blocking their production
and their very specific use case. You will get more and more reviews
request: "since you merged a single change 12 months ago, I'm sure
that you will love to review my precius bugfix! Come on, it's
trivial!". Now you realize that you don't know the design of the
module. You don't know its history. You know nothing, and the entire
world now have very high expectation, since you merged an obvious and
trivial change.

It happens to me all the time. Most stdlib modules have no maintainer
and past maintainers are gone for a long time.

Merging a PR is very "expensive" for a core developer. There are very
good reasons to not merge a PR. The status quo is much more
comfortable for the core dev who decided to *not* be the one who
merged your PR.

I prefer to pretend that I know nothing about Python and only focus on
the code area that I know the best, to reduce the risk of regression
when merging a change. If you don't know well a module, the risk of
missing a bug (breaking a specific code path) is way higher.

The other problem is that it's very rare that a PR is reviewed by more
than a single core developer. The core developer will be the only
responsible person to handle the future incoming mess introduced by
the "obvious and short bugfix".

I exaggerate the severity of regressions to explain my point, but
trust me, they are more likely than what you are thinking. Also, more
contributors are around for 1 to 6 months, whereas core developers are
usually around for 1 to 5 years. It's a different time scale in terms
of responsibilities.

Victor

On Tue, Jun 29, 2021 at 7:21 PM Brian Curtin  wrote:
>
> On Tue, Jun 29, 2021 at 10:38 AM  wrote:
>>
>> I just stumbled upon the following issue and subsequent pull request. It is 
>> a very small bugfix. There is currently a bug in Python and this pull 
>> request fixes it. It's not a new feature or an enhancement, it is a bugfix! 
>> Yet, it doesn't get reviewed, nor merged. And this has been going on since 
>> March 2017. Why not just merge this? It's not like it's huge or obstructing 
>> or obfuscating the current code base? There's always time to write an 
>> improvement or a complete rewrite of this module feature later for an 
>> upcoming minor release. But if there is currently a bug in Python and the 
>> bugfix is available - why doesn't it get merged?
>>
>> https://github.com/python/cpython/pull/4819
>
>
> For starters, the PR is closed in favor of another issue that has reviews and 
> a discussion, but even the smallest change like that requires a lot out of a 
> reviewer. Looking at that change, I don't personally know that it's correct, 
> so I'd have to take the time to figure out that it's correct. It includes no 
> tests, so I certainly don't trust that it's correct, so it looks incomplete 
> to me.
>
> Time is irrelevant here—there's no need to rush things because a change 
> appears small. What if that one line change is even more wrong than before? I 
> have merge access and if I just said "ah it's a small PR and it's been open 
> for a while, I'll just merge it for them," any change to Python has the 
> possibility to affect a huge amount of people.
>
> When I got the shutil.which feature merged, the PR had been open for I 
> believe 11 years and it was mostly complete in the initial patch outside of a 
> few small issues, and the change itself wasn't a lot of code. To just have 
> merged it because it was open for 11 years would have been the wrong thing to 
> do. It needed to cover some things it didn't initially cover, it needed tests 
> and documentation, and it wasn't merged until it was completed and properly 
> reviewed.
>
>> If this doesn't get fixed, doesn't that mean the Python review process is 
>> flawed? Sure, Python is an open source project and many people just work in 
>> their free time. But this doesn't really apply here, does it? The bugfix is 
>> available. Only a review is required. All this is 

[Python-Dev] Re: Roundup to GitHub Issues migration

2021-06-21 Thread Victor Stinner
Hi,

As a bug triager, it's very frustrating when I ask for more
information about a bug, and I get no reply. Usually, such bug is
closed as "out of date" (lack enough information to be debugged).

The requirement for a GitHub account was well known when PEP 581 was
accepted. The PEP was approved. It's now time to move on!

The creation of bugs.python.org account and bugs.python.org
authentication is often broken. It was common that OpenID
authentication was broken for 1 month or longer. Moreover,
bugs.python.org doesn't support 2FA currently, whereas GitHub does.
GitHub looks more reliable *and* more secure.

Also, please don't forget an important point: bugs.python.org is
barely maintained. See the list of open issues:
https://github.com/python/bugs.python.org/issues

For example, "Not receiving email from the bug tracker" was reported
25 days ago and got no answer :-(

It's all about trade-offs. Don't under estimate the cost of operating
our own bug tracker. System administration is not free.

Victor

On Mon, Jun 21, 2021 at 11:46 AM Baptiste Carvello
 wrote:
>
> Hi,
>
> Le 21/06/2021 à 04:20, Ezio Melotti a écrit :
> > This effort is being tracked at
> > : this board reflects
> > the current status of the project.  The PEPs (including PEP 588 --
> > GitHub Issues Migration Plan) haven't been updated yet and might
> > contain outdated information, so please refer to the psf/gh-migration
> > repo for the latest updates.
>
> since 2019, I've been waiting for proposed solutions to PEP 588's first
> open issue ("A GitHub account should not be a requirement"). Now would
> be the time to think about it, but I see no such reflection mentioned in
> the repo.
>
> Cheers,
> Baptiste
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/2GJQKB27XD6LKGKWJXTXLEURNMF5ZJT7/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JOQKA3AUGEYQZWZ2NJATOKK4APESMBF4/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Difficulty of testing beta releases now available

2021-06-11 Thread Victor Stinner
For "pip install numpy", a new numpy release should fix the issue.

The issue was already fixed in packaging 20.8:

* 
https://github.com/pypa/packaging/commit/611982be44d7d91453a97082f58d8ea349f89d00
* https://github.com/pypa/packaging/pull/355

The fix is already included in  wheel 0.36.2 (which includes a
vendored copy of packaging):

* https://github.com/pypa/wheel/commit/f6fd2472c7a4a836a3ebe12eab96a1ed8a31c061

And numpy pyproject.toml was updated to use wheel 0.36.2 which contains the fix:

* https://github.com/numpy/numpy/commit/9569f40917c80b9506f0969b1e89e2fc119d4ae3

The pyproject.toml fix is not part of 1.20.3:
https://github.com/numpy/numpy/blob/v1.20.3/pyproject.toml

--

I didn't even know that it was possible to require a specific wheel
version in pyproject.toml to build a binary wheel package in pip!

Current pyproject.toml:
https://github.com/numpy/numpy/blob/main/pyproject.toml

[build-system]
# Minimum requirements for the build system to execute.
requires = [
"packaging==20.5; platform_machine=='arm64'", # macos M1
"setuptools<49.2.0",
"wheel==0.36.2",
"Cython>=0.29.21,<3.0", # Note: keep in sync with tools/cythonize.py
]

Victor

On Fri, Jun 11, 2021 at 4:30 PM Victor Stinner  wrote:
>
> Hi,
>
> On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer  
> wrote:
> > - Cython doesn't work because of _PyGen_Send change [1]
>
> My team fixed the Python 3.10 compatibility in the Cython 0.29.x
> branch, but the latest 0.29.x release (0.29.23, April 14, 2021)
> doesn't include these fixes yet. A new Cython release is needed. I'm
> not sure how to help Cython to release a new version.
>
> In Fedora, we have downstream patches until a new Cython is released:
> https://src.fedoraproject.org/rpms/Cython/pull-request/28
>
>
> > - numpy cannot be installed because of _Py_HashDouble() change [2]
>
> "pip install numpy" also fails with an error about "incompatible tag".
> It's an issue in packaging which is vendored in wheel which is used
> (as a vendored copy?) by pip to build a local wheel package of numpy.
> numpy doesn't ship wheel packages for Python 3.10 beta releases which
> is a good idea, the ABI is not stable yet.
>
>
> > Can we do any things to improve the situation?  Perhaps using the
> > pre-release functionality of PyPI would help.  We would have to
> > somehow encourage 3rd party packages to upload pre-releases that are
> > compatible with our beta/RC releases.
>
> In Fedora, we use a "COPR build": we partially rebuild the OS with
> Python 3.10, and then we check for package build failures. We are
> working on switching to Python 3.10 by default:
>
> * https://fedoraproject.org/wiki/Changes/Python3.10
> * https://bugzilla.redhat.com/show_bug.cgi?id=1890881
>
> For example, my colleague Lumir submitted the trio bug report ;-)
> https://github.com/python-trio/trio/issues/1899
>
> We are doing our best to report issues to projects. Sometimes we help
> to fix them, but we cannot fix all compatibility issues!
>
> In the past, I tried to work on a small Python script to run the test
> suite of some projects with the main branch of Python. I had practical
> issues:
>
> * There is no simple way to get install test dependencies of a Python
> project, especially non-Python dependencies.
> * When a dependency was not compatible with the next Python, I had to
> maintain a patch until my fix is accepted upstream.
> * Sometimes, I had to pull the code from Git rather than using a
> tarball/ZIP (to not have to wait for a new release).
>
> At the end, all these practical issues need solutions which look very
> closely to a recipe to build a package for a Linux distribution:
>
> * Recipe to pull the source.
> * Recipe to pull build dependencies.
> * Downstream patches (until a fix is merged upstream).
> * Recipe to run tests: we try to always run tests when we build a
> package in Fedora.
>
> Advantages of using Fedora COPR:
>
> * Reuse Fedora infrastructure (building packages need physical servers
> to build packages and run tests)
> * Reuse the public Fedora bug tracker to coordinate the work.
> * Reuse Fedora "specfiles" (recipe to build a RPM package).
> * Prepare Fedora to be updated to the next Python.
>
> Sadly, Fedora COPR might be a little bit too Fedora specific: require
> to understand specfiles, how to submit a PR on a specfile, use the
> Fedora bug tracker, etc.
>
> Fedora Rawhide was just updated to switch Python 3.10 by default!
> /usr/bin/python3 is now Python 3.10 beta2. In COPR we could skip tests
> to build dependencies and discover more compatibiliy issues. In
> Rawhide, tests cannot be skipped and so some packages cannot be built
> yet

[Python-Dev] Re: Difficulty of testing beta releases now available

2021-06-11 Thread Victor Stinner
Hi,

On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer  wrote:
> - Cython doesn't work because of _PyGen_Send change [1]

My team fixed the Python 3.10 compatibility in the Cython 0.29.x
branch, but the latest 0.29.x release (0.29.23, April 14, 2021)
doesn't include these fixes yet. A new Cython release is needed. I'm
not sure how to help Cython to release a new version.

In Fedora, we have downstream patches until a new Cython is released:
https://src.fedoraproject.org/rpms/Cython/pull-request/28


> - numpy cannot be installed because of _Py_HashDouble() change [2]

"pip install numpy" also fails with an error about "incompatible tag".
It's an issue in packaging which is vendored in wheel which is used
(as a vendored copy?) by pip to build a local wheel package of numpy.
numpy doesn't ship wheel packages for Python 3.10 beta releases which
is a good idea, the ABI is not stable yet.


> Can we do any things to improve the situation?  Perhaps using the
> pre-release functionality of PyPI would help.  We would have to
> somehow encourage 3rd party packages to upload pre-releases that are
> compatible with our beta/RC releases.

In Fedora, we use a "COPR build": we partially rebuild the OS with
Python 3.10, and then we check for package build failures. We are
working on switching to Python 3.10 by default:

* https://fedoraproject.org/wiki/Changes/Python3.10
* https://bugzilla.redhat.com/show_bug.cgi?id=1890881

For example, my colleague Lumir submitted the trio bug report ;-)
https://github.com/python-trio/trio/issues/1899

We are doing our best to report issues to projects. Sometimes we help
to fix them, but we cannot fix all compatibility issues!

In the past, I tried to work on a small Python script to run the test
suite of some projects with the main branch of Python. I had practical
issues:

* There is no simple way to get install test dependencies of a Python
project, especially non-Python dependencies.
* When a dependency was not compatible with the next Python, I had to
maintain a patch until my fix is accepted upstream.
* Sometimes, I had to pull the code from Git rather than using a
tarball/ZIP (to not have to wait for a new release).

At the end, all these practical issues need solutions which look very
closely to a recipe to build a package for a Linux distribution:

* Recipe to pull the source.
* Recipe to pull build dependencies.
* Downstream patches (until a fix is merged upstream).
* Recipe to run tests: we try to always run tests when we build a
package in Fedora.

Advantages of using Fedora COPR:

* Reuse Fedora infrastructure (building packages need physical servers
to build packages and run tests)
* Reuse the public Fedora bug tracker to coordinate the work.
* Reuse Fedora "specfiles" (recipe to build a RPM package).
* Prepare Fedora to be updated to the next Python.

Sadly, Fedora COPR might be a little bit too Fedora specific: require
to understand specfiles, how to submit a PR on a specfile, use the
Fedora bug tracker, etc.

Fedora Rawhide was just updated to switch Python 3.10 by default!
/usr/bin/python3 is now Python 3.10 beta2. In COPR we could skip tests
to build dependencies and discover more compatibiliy issues. In
Rawhide, tests cannot be skipped and so some packages cannot be built
yet. For example, astropy has a few failing tests:
https://github.com/astropy/astropy/issues/11821

Again, the practical problem is that numpy cannot easily be installed
on Python 3.10 (pip compatibiltiy tag issue), and so astropy may wait
until Python 3.10 final is released to look into these tests failures.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6YRNA2F2G3RSXWG72WFEFOEAARIA57E5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.

2021-06-09 Thread Victor Stinner
On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev
 wrote:
> Its a bit late to complain (and I’m not affected by this myself), but those 
> functions are part of the stable ABI. The change in 3.10 will break any 
> extensions that use the stable ABI, use these functions and don’t use 
> PY_SSIZE_T_CLEAN.  I have no idea how many of those exist, especially given 
> that the stable ABI doesn’t seem to be used a lot.

Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API*
change. At the ABI level, what changed if that C extensions built
(with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an
error on Python 3.10 (for a few specific argument formats using "#").
Ah you are right, it's an incompatible ABI change.

It might be possible to keep the backward compatibility at the ABI
level by adding a 3rd flavor of "parse" functions:

* parse with size_t: no change
* parse without size_t: stable ABI
* parse without size_t which raises an exception on "#" formats: new
Python 3.10 functions

It's already painful to have 2 flavors of each functions. Adding a 3rd
flavor would make the maintenance burden even worse, whereas Inada-san
wants to opposite (remove the 2nd flavor to only have one!).

A more general question is: do we still want to keep backward
compatibility with Python 3.2 released 10 years ago, or is it ok to
start with a new stable ABI which drops backward compatibility with
Python 3.5 (for example)?

It's maybe time to replace "abi3" with "abi4" in Python 3.10?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ENFYNPCPTEZQSBVB5BA5XYNVFUDUA4GF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Proposal: declare "unstable APIs"

2021-06-04 Thread Victor Stinner
On Fri, Jun 4, 2021 at 12:29 AM Guido van Rossum  wrote:
>> In the C API, there is the internal C API which fits with your
>> description. To access it, you have to declare the
>> Py_BUILD_CORE_MODULE macro. It's not usable directly on purpose. It's
>> an user agreement: I know what I am doing, and I know that this API is
>> not supported nor stable.
>
> Hm, but aren't for example all the fields of code objects (co_name, 
> co_argcount, etc.) in the "non-internal" API? Those (and the functions that 
> manipulate code objects) are a prime example of what I'd consider "unstable".
>
> On https://docs.python.org/3/c-api/code.html it already says about the fields 
> "The fields of this type are subject to change at any time." But I think 
> everything else on that page should be considered unstable as well. (And why 
> do we even have PyCode_GetNumFree()?)

Hum, the C API is somehow off-topic, but let me reply anyway ;-)

As I explained in my PEP 620, the Python C API never had any design.
Things were only exposed because it was easy and technically possible,
and it was a convenient way to define a function in one file and uses
it from another file. But 30 years later, we identified that exposing
some things are causing troubles and we are trying to make the C API
more "abstract".

Exposing directly all structures is causing a lot of headaches at
every new Python 3.x release. Getter and setter functions can handle
structure changes, retrieve information from another structure, return
an error, etc. This abstractation is needed for users to not have to
update their code, and to CPython developers to be able to change
things.

If possible, I would prefer to make PyThreadState, PyCodeObject and
other structures opaque, and only go through getter and setter
functions ;-) PyCode_New() is another problem :-/ The PEP 570 first
changed it to add a new parameter. It broke Cython and other projects.
The change was reverted, and PyCode_NewWithPosOnlyArgs() was added.
The lesson is that it's possible to change PyCodeObject without
breaking PyCode_New() (which handles the "backward compatibility" for
you).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQI2NKFXQ4LYJ2HVFBCHL3KPMQVD4HI4/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Proposal: declare "unstable APIs"

2021-06-03 Thread Victor Stinner
Hi Guido,

It seems like you are talking about the Python API.

In the C API, there is the internal C API which fits with your
description. To access it, you have to declare the
Py_BUILD_CORE_MODULE macro. It's not usable directly on purpose. It's
an user agreement: I know what I am doing, and I know that this API is
not supported nor stable.

I don't know if there would be a way to advertise that a Python API is
unstable. Some projects use an underscore prefix in their module names
to mark them as "private". For example, sub-modules of a package are
called "_something.py" and they exposed in package/__init__.py (or
another public module).

Victor

On Thu, Jun 3, 2021 at 7:14 PM Guido van Rossum  wrote:
>
> This is not a complete thought yet, but it occurred to me that while we have 
> deprecated APIs (which will eventually go away), and provisional APIs (which 
> must mature a little before they're declared stable), and stable APIs (which 
> everyone can rely on), it might be good to also have something like 
> *unstable* APIs, which will continually change without ever going away or 
> stabilizing. Examples would be the ast module (since the AST structure 
> changes every time the grammar changes) and anything to do with code objects 
> and bytecode (since we sometimes think of better ways to execute Python).
>
> So maybe the docs should grow a standard way of saying "this is an unstable 
> API"?
>
> Would we need a PEP to create an initial list of APIs (modules, classes, 
> etc.) that are considered unstable?
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/JM6SQ2YNMDAKXYD5O54QWMVR2X7QOXVL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KG4Z45UW7Y2EY5WMPZ6Z4763CJWUTXH3/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support

2021-06-02 Thread Victor Stinner
On Thu, Jun 3, 2021 at 2:30 AM Joannah Nanjekye
 wrote:
> Is HPy ready yet given IIRC, there is no even first release yet? I stand to 
> be corrected.
> I reckon it will still go through a period of incompatible changes for some 
> time/months too.

HPy is a great project, but even if 90% of top 4000 PyPI extensions
are converted to it, CPython will likely still want to support the
long tail of old C extensions written directly with the Python C API
("Python.h"). My PEP 620 addresses the specific case of CPython which
wants to maximize its backward compatibility. It's a trade-off between
optimizations/major code rewrite and backward compatibility. Sadly,
you have to break eggs to make an omelet (french expression) :-)

I suggest to use Cython or HPy to write *new* C extensions, since they
don't depend (directly) on the C API. At a new Python version, just
update Cython/HPy, and you're good. I hope that tomorrow, Cython/HPy
will even support the limited C API and the stable ABI to not even
have to rebuild wheel packages! Cython already has an (experimental?)
option to restrict emitted C code to the limited C API.

--

I would say that HPy is not complete nor mature yet. IMO it's already
worth it to start playing with it, try to convert your small C
extensions, and send back your early feedback to HPy developers!

HPy links:

* Website: https://hpyproject.org/
* Blog: https://hpyproject.org/blog/
* Documentation: https://docs.hpyproject.org/
* GitHub: https://github.com/hpyproject/hpy/

Status:

* May 2021 (blog): https://hpyproject.org/blog/posts/2021/05/may-status-update/
* April 2021 (doc):
https://docs.hpyproject.org/en/latest/overview.html#current-status-and-roadmap
* March 2021 (blog): https://hpyproject.org/blog/posts/2021/03/hello-hpy/

There is a 0.0.1 release on GitHub (Git tag created last January):
https://github.com/hpyproject/hpy/releases/tag/0.0.1

On PyPI, the 0.0.1 tarball doesn't contain any source: it was only
created to reserve the name on PyPI.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DM3YRZVFYEB672ZINR7HDXCGH3RLVEQ6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support

2021-06-02 Thread Victor Stinner
Hi,

What do you think of promoting the pythoncapi_compat project that I'm
introducing below in the "C API: Porting to Python 3.10" section of
What's New In Python 3.10?

Should this project be moved under the GitHub psf organization to have
a more "future proof" URL?

I would like to promote this project to prepare C extensions
maintainers for the following incompatible C API change (Py_TYPE) that
I would like to push into Python 3.11:
https://github.com/python/cpython/pull/26493
(Py_REFCNT was already converted to a static inline function in Python 3.10.)

I already made this Py_TYPE change in Python 3.10, but I had to revert
it since it broke too many projects. Since last year, I upgraded most
of these broken projects, I created the pythoncapi_compat project, and
I succeeded to use upgrade_pythoncapi.py script and copy the
pythoncapi_compat.h header file in multiple C extensions.

C extensions written with Cython are not affected. I already fixed
Cython last year to emit code compatible with my incoming incompatible
change. If it's not done yet, you only have to regenerate the C files
using a recent Cython version.

--

I wrote a new script which adds Python 3.10 support to your C
extensions without losing Python 2.7 support:
https://github.com/pythoncapi/pythoncapi_compat

To add Python 3.10 support to your C extension, go to its source
directory and run:

  /path/to/upgrade_pythoncapi.py .

It upgrades all C files (.c) in the current directory and
subdirectories. For example, it replaces "op->ob_type" with
"Py_TYPE(op)". It creates an ".old" copy of patched files.

Use the -o option to select operations:

* -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)".
* -o all,-PyMem_MALLOC: run all operations, but don't replace
PyMem_MALLOC(...) with PyMem_Malloc(...).

--

The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h
header file that I wrote to provide recent Python 3.9-3.11 C functions
on old Python versions. Examples: Py_NewRef() and
PyThreadState_GetFrame(). Functions are implemented as simple static
inline functions to avoid requiring to link your extension to a
dynamic library.

You can already use the new Py_NewRef() and Py_IsNone() Python 3.10
functions in your projects without losing support for Python 2.7-3.9!

--

The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".

The _PyFrame_GetBackBorrow() function doesn't exist in the Python C
API, it's only provided by pythoncapi_compat.h to ease the migration
of C extensions. I advise you to replace _PyFrame_GetBackBorrow()
(borrowed reference) with PyFrame_GetBack() (strong reference).

--

This project is related to my PEP 620 "Hide implementation details
from the C API" which tries to make the C API more abstract to later
allow to implement new optimization in CPython and to make other
Python implementations like PyPy faster when running C extensions.

Article on the creation of the pythoncapi project:
https://vstinner.github.io/pythoncapi_compat.html

The main drawback of this project is that it uses regular expressions
to parse C code. Such "parser" can miss C code which has to be patched
manually. In my experience, additional manual changes are really rare
and take less than 1 minute on a very large C extension like numpy.

--

This project only targets extension modules written in C by using
directly the "Python.h" API. I advise you to use Cython or HPy to no
longer be bothered with incompatible C API changes at every Python
release ;-)

* https://cython.org/
* https://hpy.readthedocs.io/

I hope that my script will facilitate migration of C extensions to HPy.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Need help to debug a ssl crash on Windows which prevents merging PRs

2021-06-01 Thread Victor Stinner
On Fri, May 28, 2021 at 6:40 PM Victor Stinner  wrote:
> In the 3.10 branch, it became really hard to merge PRs because the
> following ssl crashs on Windows:
> https://bugs.python.org/issue44252

Update on this bug which blocked the Python 3.10 beta 2 release. It's
now fully fixed!

It was a simple bug in the _ssl.SSLError exception. The problem was
that the crash only occurred on Windows and only if tests were run in
a very specific way. On CIs, the crash was deterministic. When I
debugged the issue manually, I failed to reproduce it. I tried many
different ways to run the tests, none worked.

I recall an old hack: run "import gc; gc.set_threshold(5)" at startup.
It makes crashes related to GC way more likely (the default threshold
of GC generation 0 is 700). I used this hack 3 years ago to debug
another GC bug really hard to reproduce:
https://mail.python.org/pipermail/python-dev/2018-June/153857.html
https://docs.python.org/dev/library/gc.html#gc.set_threshold

Not only the _ssl.SSLError bug is fixed, but Pablo also fixed the
documentation to explain clearly that a traverse function must be
implemented if Py_TPFLAGS_HAVE_GC is set:
https://github.com/python/cpython/commit/8b55bc3f93a655bc803bff79725d5fe3f124e2f0

Moreover, for people who don't read the documentation ;-), I also made
sure that it's no longer possible to create a type with
Py_TPFLAGS_HAVE_GC but with no traverse function:
https://github.com/python/cpython/commit/ee7637596d8de25f54261bbeabc602d31e74f482

By the way, I had to fix two stdlib types (_testcapi and _decimal
modules) which didn't respect that!

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7FHMUDJIZSO4EPREJ5XX6WTYLR3SQTAI/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Need help to debug a ssl crash on Windows which prevents merging PRs

2021-05-28 Thread Victor Stinner
Hi,

In the 3.10 branch, it became really hard to merge PRs because the
following ssl crashs on Windows:
https://bugs.python.org/issue44252

It has a failure rate 1/2 (on average) on the "Windows x86" and
"Windows x64" jobs of GitHub Action and on the Win32 and Win64 jobs of
the Azure Pipelines. I failed to reproduce it on an up to date Windows
10 with an up to date Visual Studio 2019.

I cannot say if it's a race condition, if it's a bug in Python, if
it's a bug in the tests (it sounds unlikely, it worked well previously
and it's a hard crash, not a Python exception), if it's a bug in the C
compiler or in Windows itself...

Since there are other random test failures like test_asyncio, it now
requires multiple "re-run jobs" on GitHub Actions. Example of
test_asyncio test which fails frequently on Windows:
https://bugs.python.org/issue41682

If someone can reproduce https://bugs.python.org/issue44252 crash on
Windows, can you please provide me a SSH access to your machine so I
can debug it? Here is my public SSH key:
https://github.com/vstinner.keys

Is there a way to get a SSH access to a machine of the GitHub Action
CI job or of an Azure Pipelines CI job?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B7DIDGAZWV3WET227Z3LPXI5LUZX2DFP/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

2021-05-27 Thread Victor Stinner
Hi,

The gilectomy was mentioned earlier in the thread. This project was
created by a core dev who had the permission to push directly his work
into the development branch, but it was done on a work.

Another optimization project example was my experimental "FAT Python"
project. The main idea was to specialize functions with some
assumptions, and check these assumptions at the function entry. I
started in a fork and then wrote 3 PEPs to propose to merge the main
changes into Python development branch:

* PEP 509 -- Add a private version to dict
* PEP 510 -- Specialize functions with guards
* PEP 511 -- API for code transformers

The PEP 509 was accepted since it could be used for other
optimizations: Python now uses the dictionary version to optimize
LOAD_GLOBAL.

The two other PEPs were rejected. Even if I was very disappointed when
they were rejected, it seems like it was a wize choice since I was
never able to make Python significantly faster (like 2x faster). It
was only 10-20% faster on some micro-benchmarks. I also had the
permission to push directly, and I think that it was nice to confront
my design and ideas to the community.

I also understand that optimizing Python is really hard and it
requires a lot of preparation work. It's hard to sell the preparation
work since it only introduces regressions and noise without any
concrete speedup. All of this work is required to implement the real
interesting optimization. Moreover, few people are earger to review a
Python fork with deep and large changes in Python internals. The work
must be re-done step by step with more "atomic" (small) changes to
have a readable Git history.

The Instagram team behind the Cinder project may be interested to
review Mark's work before it's merged into Python development branch.
A design PEP would be more convenient than reviewing an concrete
implementation.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CS7WDQYHJN7QV6TQW3CJZ2XRQRKX2RWT/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: IRC #python-dev channel is now on Libera Chat (bye bye Freenode)

2021-05-26 Thread Victor Stinner
By the way, #python also moved to Libera Chat: https://www.pound-python.org/

The PSF (Group Contact) owns  #python*, #pypa* and #psf* channels on Libera.

I updated the IRC informations in the Python devguide.

I'm still migrating #python-dev-notifs bots to Libera.

Victor

On Wed, May 26, 2021 at 12:46 PM Victor Stinner  wrote:
>
> Hi,
>
> The #python-dev IRC channel moves from Freenode to Libera Chat:
> https://libera.chat/ I'm connected as vstinner, see you there ;-) Join
> the channel if you would like to talk about the development of Python
> itself (the VM, bytecode, stdlib, etc.)!
>
> Moreover, bots notifications (GitHub, buildbots, bugs.python.org) are
> now in a new dedicated channel #python-dev-notifs so humans can talk
> together again! :-) The migration is on-going, right now some
> notifications are still on Freenode.
>
> --
>
> Yesterday evening, Freenode admins decided to take the control of the
> #python-fr channel on Freenode (which now redirects to a new
> ##python-fr channel), only because the channel topic contained
> "libera"! The french Python association ("AFPy") managing #python-fr
> lost control and decided to migrate to Libera Chat (libera.chat).
> Announcement in french:
> https://www.afpy.org/posts/actualites/1622023152
>
> I'm disappointed by how the events are going. At least, it motivated
> me to move bots notfications ;-)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R3YTKHTUPMF3W5UH2YP4ACXHBXTQLQUE/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] IRC #python-dev channel is now on Libera Chat (bye bye Freenode)

2021-05-26 Thread Victor Stinner
Hi,

The #python-dev IRC channel moves from Freenode to Libera Chat:
https://libera.chat/ I'm connected as vstinner, see you there ;-) Join
the channel if you would like to talk about the development of Python
itself (the VM, bytecode, stdlib, etc.)!

Moreover, bots notifications (GitHub, buildbots, bugs.python.org) are
now in a new dedicated channel #python-dev-notifs so humans can talk
together again! :-) The migration is on-going, right now some
notifications are still on Freenode.

--

Yesterday evening, Freenode admins decided to take the control of the
#python-fr channel on Freenode (which now redirects to a new
##python-fr channel), only because the channel topic contained
"libera"! The french Python association ("AFPy") managing #python-fr
lost control and decided to migrate to Libera Chat (libera.chat).
Announcement in french:
https://www.afpy.org/posts/actualites/1622023152

I'm disappointed by how the events are going. At least, it motivated
me to move bots notfications ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FQW44V4DZFX4GAGOAMAZAN5KDQXCASDU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Implementing async enumerate

2021-05-25 Thread Victor Stinner
Hi,

I suggest to put such function in a Python module and publish it on
PyPI. I'm sure that the module will quickly grow with more async
flavor of existing functions.

Victor

On Tue, May 25, 2021 at 4:22 AM Filipe Alves Caixeta
 wrote:
>
> Hello,
>
> Python 3.10 comes with the built-in anext() but I think we may be missing 
> some other equivalent built-in functions like list and enumerate.
> I didn't discuss the idea of creating a PEP for this but I would like to get 
> some help with the implementation.
>
> The equivalent Python code would be
>
> async def aenumerate(sequence, start=0):
> n = start
> async for elem in sequence:
> yield n, elem
> n += 1
>
> The code for enumerate if here 
> https://github.com/python/cpython/blob/2f3a87856c7033227577b9ed0c77ed75311430b7/Objects/enumobject.c#L42
>
> The next function creates a tuple and return with (n, elem)
>
> result = PyTuple_New(2);
> PyTuple_SET_ITEM(result, 0, next_index);
> PyTuple_SET_ITEM(result, 1, next_item);
> return result;
>
> For the async implementation I would have to instead return an 
> async_generator_asend object
> and that would resolve to the tuple. Calling tp_as_async->am_anext and 
> getting the value doesn't seem like the solution because I should not get the 
> result at this stage yet.
>
> I would like to have a hint on how to implement this =D
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/7ZRSZAOM5NPF3DOMPS2XNS64L7XP5SBN/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TZBZXWHMZDQSW7BWXBJTNFKAJDAUCZLL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: GDB not breaking at the right place

2021-05-24 Thread Victor Stinner
--with-pydebug default compiler flags is a trade-off between "runtime
checks for debug" and performance. -O0 makes Python slower. For
example, we want to Python CI to run as fast as possible.

I don't want to fight for https://bugs.python.org/issue38350 I simply
learnt to type: ./configure --with-pydebug CFLAGS="-O0" (I have a
shell alias for that).

Victor

On Mon, May 24, 2021 at 5:54 PM Guido van Rossum  wrote:
>
> I'm confused. I've always assumed that --with-pydebug was intended for the 
> situation where you're modifying the C code, so obviously you might have to 
> debug C code. (I know that was the case when we introduced it, decades ago.) 
> If that's not the goal, then what is --py-debug used for?
>
> On Mon, May 24, 2021 at 3:35 AM Victor Stinner  wrote:
>>
>> "Debugging" means many things. Python is built with -Og because it
>> makes Python faster than -O0, and most developers debug Python code,
>> not C code (in gdb).
>>
>> If you don't need to go up to the gdb/lldb level, -Og is fine.
>>
>> It would even make sense to build Python with -O3 in debug mode if you
>> don't debug C code at all, only pure Python code.
>>
>> My proposition to switch to -00 by default was rejected:
>> https://bugs.python.org/issue38350
>>
>> I also love -O0 when I modify C code because it makes the build faster ;-)
>>
>> Fedora Python debug builds are now built with -O0 which makes gdb a
>> way more pleasant experience, not more strange behavior with inlined
>> code or "" local variables or function arguments.
>>
>> Victor
>>
>> On Sun, May 23, 2021 at 3:57 PM Skip Montanaro  
>> wrote:
>> >
>> > > I strongly suggest to only build Python with -O0 when using gdb. -Og
>> > > enables too many optimizations which makes gdb less usable.
>> >
>> > Thanks, Victor. It never made sense to me that you would want any
>> > optimizations enabled when truly debugging code (as opposed to wanting
>> > debug symbols and a sane traceback in production code).
>> >
>> > I'm getting more convinced that the problem I'm seeing is a GCC/GDB
>> > thing, particularly because I can move the erroneous stopping point by
>> > changing the GCC optimization level. I'll probably open a bugzilla
>> > report just so it's on that team's radar screen. In the meantime, to
>> > get going again I wrote a crude script which maps the
>> > file:function:label form to file:linenumber form. That way I can
>> > save/restore breakpoints across GDB sessions and still avoid problems
>> > when the offsets to specific instructions change.
>> >
>> > Skip
>> >
>> > Skip
>>
>>
>>
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/FPWJHHV55JPF6ZQVI2OGMYSIPYN3AXH2/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G7EFOMH7DXC3TUCR3R6UJ7Q2KTT5UN22/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: GDB not breaking at the right place

2021-05-24 Thread Victor Stinner
"Debugging" means many things. Python is built with -Og because it
makes Python faster than -O0, and most developers debug Python code,
not C code (in gdb).

If you don't need to go up to the gdb/lldb level, -Og is fine.

It would even make sense to build Python with -O3 in debug mode if you
don't debug C code at all, only pure Python code.

My proposition to switch to -00 by default was rejected:
https://bugs.python.org/issue38350

I also love -O0 when I modify C code because it makes the build faster ;-)

Fedora Python debug builds are now built with -O0 which makes gdb a
way more pleasant experience, not more strange behavior with inlined
code or "" local variables or function arguments.

Victor

On Sun, May 23, 2021 at 3:57 PM Skip Montanaro  wrote:
>
> > I strongly suggest to only build Python with -O0 when using gdb. -Og
> > enables too many optimizations which makes gdb less usable.
>
> Thanks, Victor. It never made sense to me that you would want any
> optimizations enabled when truly debugging code (as opposed to wanting
> debug symbols and a sane traceback in production code).
>
> I'm getting more convinced that the problem I'm seeing is a GCC/GDB
> thing, particularly because I can move the erroneous stopping point by
> changing the GCC optimization level. I'll probably open a bugzilla
> report just so it's on that team's radar screen. In the meantime, to
> get going again I wrote a crude script which maps the
> file:function:label form to file:linenumber form. That way I can
> save/restore breakpoints across GDB sessions and still avoid problems
> when the offsets to specific instructions change.
>
> Skip
>
> Skip



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FPWJHHV55JPF6ZQVI2OGMYSIPYN3AXH2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: GDB not breaking at the right place

2021-05-22 Thread Victor Stinner
I strongly suggest to only build Python with -O0 when using gdb. -Og
enables too many optimizations which makes gdb less usable.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YDKJNGWQKGSW633KDHKFIY4PBHLOFLWL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The repr of a sentinel

2021-05-20 Thread Victor Stinner
IMO you should consider writing a PEP to enhance sentinels in Python,
and maybe even provide a public API for sentinels in general.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EY6B6PRQ2B54FVG5JK42GR6ZM2VQ7VL2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: The repr of a sentinel

2021-05-14 Thread Victor Stinner
Hi Tal,

Would it make sense to have an unique singleton for such sentinel, a
built-in singleton like None or Ellipsis? I propose the name
"Sentinel".

Sentinel would be similar to None, but the main property would be that
"Sentinel is None" is false :-)

The stdlib contains tons of sentinels:

* _collections_abc: __marker__
* cgitb.__UNDEF__
* configparser: _UNSET
* dataclasses: _HAS_DEFAULT_FACTORY, MISSING, KW_ONLY
* datetime.timezone._Omitted
* fnmatch.translate() STAR
* functools.lru_cache.sentinel (each @lru_cache creates its own sentinel object)
* functools._NOT_FOUND
* heapq: temporary sentinel in nsmallest() and nlargest()
* inspect._sentinel
* inspect._signature_fromstr() invalid
* plistlib._undefined
* runpy._ModifiedArgv0._sentinel
* sched: _sentinel
* traceback: _sentinel

There are different but similar use cases:

* Optional parameter: distinguish between func() and func(arg=value),
a sentinel is useful to distinguish func() from func(arg=None)
* Look into a data structure for a value and store the result in a
value, distinguish if 'result' variable was set ("result is not None"
doesn't work since None is a value). Quick example: "missing =
object(); tmsg = self._catalog.get(message, missing); if tmsg is
missing: ..."

Special cases:

* dataclases._EMPTY_METADATA = types.MappingProxyType({})
* string._sentinel_dict = {}
* enum: _auto_null = object()

Victor

On Thu, May 13, 2021 at 7:40 PM Tal Einat  wrote:
>
> On Thu, May 13, 2021 at 7:44 PM Ethan Furman  wrote:
> >
> > Consider me complaining.  ;-)
>
> +1
>
> > An actual Sentinel class would be helpful:
> >
> >  >>> class Sentinel:
> >  ... def __init__(self, repr):
> >  ... self.repr = repr
> >  ... def __repr__(self):
> >  ... return self.repr
> >  ...
> >
> >  >>> MISSING = Sentinel('MISSING')
> >  >>> MISSING
> >  MISSING
> >
> >  >>> implicit = Sentinel('')
> >  >>> implicit
> >  
>
> Here is my suggestion (also posted on the related bpo-44123), which is
> also simple, ensures a single instance is used, even considering
> multi-threading and pickling, and has a better repr:
>
> class Sentinel:
> def __new__(cls, *args, **kwargs):
> raise TypeError(f'{cls.__qualname__} cannot be instantiated')
>
> class MISSING(Sentinel):
> pass
>
> - Tal
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/URFRF634732GRICGLRPGJEJON2BYQZM4/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBYXQH3NV3YBF7P2HLHB5CD6V3GVTY55/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: [python-committers] [Release manager team communication] master blocked until 3.10 beta 1 is released

2021-05-03 Thread Victor Stinner
Hi,

Please don't attempt to merge PRs until the GitHub issue described
below is solved ;-)

Pablo renamed the default "master" branch to "main" (in a live Twitch
stream ;-)) but got a GitHub internal error! Maybe it's because the
dialog announced that 1.4k+ pull requests and 700+ repositositories
will be impacted which is not a common case. Right now, there are
"main" and "master" branches.

I reported the issue to GitHub Support forum:
https://github.community/t/renaming-python-master-branch-to-main-1-4k-prs-700-repositories-triggered-server-http-error-500/178090

Victor

On Mon, May 3, 2021 at 5:59 PM Pablo Galindo Salgado
 wrote:
>
> Hi everyone,
>
> Unfortunately, the CI for master is currently quite unstable and he had some 
> problems that we
> have been fixing all weekend and today. We are doing our best to stabilize 
> the test suite and making
> sure that everything is ready to go for the release with no issues, but this 
> will take some time (also
> bear in mind that today is a bank holiday in the UK, so I am doing my best to 
> get the release ready
> in a holiday). Given this and the fact that we need to do a branch rename,
> **we have decided to block the master branch** until we can stabilize the 
> test suite.
>
> In the meantime, if you have some urgent fix or PR that you absolutely need 
> to get merge before
> feature freeze, you can add me as a reviewer to said PR so we can evaluate 
> the merge.
>
> Thanks for your understanding.
>
> I will update once master is unblocked again.
>
> Regards from cloudy London, Pablo Galindo Salgado
>
>
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/AKSCARRZJKKU427Z7DIMYO7MK5RSXOFR/
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LCCT76L7ILONS7VKLC72FBJAOYJIBZMU/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: In what tense should the changelog be written?

2021-04-30 Thread Victor Stinner
By the way, https://docs.python.org/dev/whatsnew/3.10.html#improved-modules
is inconsistent:

"asyncio: Added missing ..."

versus

"base64: Add base64.b32hexencode() ..."

You can also find such inconsistencies in "versionchanged" markups.

Victor

On Fri, Apr 30, 2021 at 5:01 AM Larry Hastings  wrote:
>
>
> When one writes one's "blurb" for the changelog, in what tense should it be?  
> I mostly see entries in present tense:
>
> bpo-43660: Fix crash that happens when replacing sys.stderr with a callable 
> that can remove the object while an exception is being printed. Patch by 
> Pablo Galindo.
>
> bpo-41561: Add workaround for Ubuntu’s custom OpenSSL security level policy.
>
> But occasionally I see entries in past tense:
>
> bpo-26053: Fixed bug where the pdb interactive run command echoed the args 
> from the shell command line, even if those have been overridden at the pdb 
> prompt.
>
> bpo-40630: Added tracemalloc.reset_peak() to set the peak size of traced 
> memory blocks to the current size, to measure the peak of specific pieces of 
> code.
>
> I couldn't find any guidance in the Python Dev Guide after sixty seconds of 
> poking around.
>
>
> Obviously this isn't a big deal.  But it might be nice to try and nudge 
> everybody in the same direction.  It'd be pleasant if the changelog read in a 
> more unified voice, and using the same tense and sentence structure would 
> help towards that goal.
>
> If we arrived at a firm decision, maybe "blurb" et al could add a little text 
> suggesting the proper style.
>
>
> Cheers,
>
>
> /arry
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/SUEPLPRN2IY7EGC7XBLWWN2BOLM3GYMQ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BG7AI3YKZ6S2NR5556ZJ6XY6N5NHZWFL/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: In what tense should the changelog be written?

2021-04-30 Thread Victor Stinner
On Fri, Apr 30, 2021 at 6:57 AM Larry Hastings  wrote:
> Your NEWS entry should be written in the present tense, and should start with 
> a verb:
>
> Add foo [...]
> Change bar [...]
> Remove bat [...]
> Fix buffalo.spam [...]

You might suggest using "now" when describing a behavior change: "foo
now does this new thing [...]"

> Function and class names should not be followed by parentheses, unless 
> demonstrating an example call.

Oh, I love putting parentheses when mentionning a function: "foo() now
does thigs new thing". Also, I like to use :func:`foo` Sphinx markup
to add a link in the changelog, and Sphinx adds parentheses for me
*but* I don't add parentheses :-)

By the way, I recently discovered that :data:, :func:, :option:, etc.
accept references written as . Examples:

* :option:`-X utf8 <-X>`
* :data:`sys.flags.dev_mode `
* :c:func:`Py_TYPE(self) `
* :func:`map(f, iterA, iterB, ...) `

I like it! It's convenient to document a function call with
arguements, so the arguments can be named in the sentence (like "self"
in "Py_TYPE(self)").

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QO6IDVU2ZHPHSPJA7GLIFNEKDTGDPWMQ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: In what tense should the changelog be written?

2021-04-30 Thread Victor Stinner
On Fri, Apr 30, 2021 at 6:22 AM Guido van Rossum  wrote:
> There’s something in the dev guide, but not about tense. Worth adding. (My 
> pet peeve is not to write “The foo module resets the bar state when spam 
> happens,” because that isn’t clear about whether that’s the before or after 
> behavior.)

In the past, different people told me that my phrasing was confusing
and suggested me to add "now". It kept this habbit:

"The foo module *now* resets the bar state when spam happens"

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AWH3BEAHTNGLGYFSPB4EQRAYC7DTAPO5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Existing asyncio features scheduled for removal in Python 3.9 and 3.10

2021-04-27 Thread Victor Stinner
Hi,

On Mon, Apr 26, 2021 at 10:51 PM Illia Volochii
 wrote:
> There are a couple of uncompleted asyncio feature removals scheduled
> for 3.9 and 3.10 releases.
> It will be great if we either complete them or reschedule before the
> 3.10 feature freeze. There are two stale pull requests related to
> this.

I would prefer to wait for 3.11 to remove more deprecated features,
these ones can survive a little bit longer.

If possible, I suggest to remove functions at the beginning of a new
Python development cycle, rather than just before the feature freeze.
So projects tested with the development branch of Python can detect
issues early and we can more time to either fix these projects, or to
consider reverting the removals to give more time for updating these
projects.

It's common that we forget to remove deprecated functions and so that
documentation is not accurate. IMO it's not a big deal ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/37M7AYCYUUIUUSN7MQ25UORJH5K2TWHJ/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Should Python typing leverage PEP 263 as a pre-processor step?

2021-04-20 Thread Victor Stinner
I proposed PEP 511 "API for code transformers" for Python 3.6 (in
2016) and it was rejected:
https://www.python.org/dev/peps/pep-0511/#rejection-notice

"""
This PEP was rejected by its author.

This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.

This PEP was also seen as a nice tool to experiment new Python
features, but it is already possible to experiment them without the
PEP, only with importlib hooks. If a feature becomes useful, it should
be directly part of Python, instead of depending on an third party
Python module.

Finally, this PEP was driven was the FAT Python optimization project
which was abandoned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.
"""

IMO the most important part of the PEP 511 was to change the pyc
filename depending on the code transformer used to modify the code.

Victor

On Mon, Apr 19, 2021 at 8:21 PM Luciano Ramalho  wrote:
>
> How about leveraging the `# coding=` hook that exists
> since 2001 to enable the alternative syntax some are advocating for
> type hints?
>
> PEP 263—Defining Python Source Code Encodings
> https://www.python.org/dev/peps/pep-0263/
>
> I've seen experiments in the wild using that to support syntax
> extensions to Python. Just for fun, years ago I wrote a POC for
> Sucuri—a Python dialect with the keywords in Portuguese. The
> pre-processor simply replaced the Portuguese keywords with the English
> ones.
>
> Cheers,
>
> Luciano
>
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> |  Technical Principal at ThoughtWorks
> |  Twitter: @ramalhoorg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/XRLAOOIJZRPDT2AW6LW4UVBEJD5NKLV5/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7HTSTMQK53APVR2OLUTDSYBZLTN2OQSM/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Question regarding the value of PyThreadState.thread_id

2021-04-17 Thread Victor Stinner
Hi,

There are two reasons:

* PyThread_get_thread_native_id() was only added recently (Python 3.8)
* PyThread_get_thread_native_id() is not portable and not available on
all platforms: the function is only declared if the
PY_HAVE_THREAD_NATIVE_ID macro is defined.

Maybe PyThreadState.thread_id could use
PyThread_get_thread_native_id() if avaialble, or
PyThread_get_thread_ident() otherwise. But that sounds like an
incompatible change.

Another approach would be to add a *new* structure member, like
thread_native_id. It would not exist if the PY_HAVE_THREAD_NATIVE_ID
macro is not defined.


Include/pythread.h:

#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
|| defined(__OpenBSD__) || defined(__NetBSD__) || defined(_WIN32) ||
defined(_AIX)
#define PY_HAVE_THREAD_NATIVE_ID
PyAPI_FUNC(unsigned long) PyThread_get_thread_native_id(void);
#endif


thread_pthread.h implementation:

unsigned long
PyThread_get_thread_native_id(void)
{
if (!initialized)
PyThread_init_thread();
#ifdef __APPLE__
uint64_t native_id;
(void) pthread_threadid_np(NULL, _id);
#elif defined(__linux__)
pid_t native_id;
native_id = syscall(SYS_gettid);
#elif defined(__FreeBSD__)
int native_id;
native_id = pthread_getthreadid_np();
#elif defined(__OpenBSD__)
pid_t native_id;
native_id = getthrid();
#elif defined(_AIX)
tid_t native_id;
native_id = thread_self();
#elif defined(__NetBSD__)
lwpid_t native_id;
native_id = _lwp_self();
#endif
return (unsigned long) native_id;
}

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.


On Fri, Apr 16, 2021 at 10:47 PM Gabriele  wrote:
>
> Hi all.
>
> My apologies if this is a topic that's been discussed already, but I
> wasn't able to locate anything in the archive on the subject. I was
> wondering if there's a fundamental reason for using
> PyThread_get_thread_ident instead of PyThread_get_thread_native_id for
> the value of the thread_id field of the PyThreadState object.
>
> The reason why I'm asking is that I would like to have easy access to
> the native TID on Linux to identify the /proc/stat file associated
> with the thread from an external process. At the moment I have to
> resort to calling process_vm_readv to copy the struct pthread over to
> local VM and then guess where the tid field might be. So, if there's
> no fundamental reason for thread_id to be PyThread_get_thread_ident, I
> would like to propose to change it to PyThread_get_thread_native_id
> instead.
>
> Thanks,
> Gabriele
>
> --
> "Egli è scritto in lingua matematica, e i caratteri son triangoli,
> cerchi, ed altre figure
> geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola;
> senza questi è un aggirarsi vanamente per un oscuro laberinto."
>
> -- G. Galilei, Il saggiatore.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6U47ITBNRZWNWELWXBZNAOAELGBAP3T6/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: How about using modern C++ in development of CPython ?

2021-04-16 Thread Victor Stinner
Hi,

I used C++ in the past to write a small game. My experience was that
the compilation was quite slow and many compiler errors were hard to
understand.

I love the fact the CPython is written in C because its build time is
under one minute for a fresh build, and way faster for an incremental
build (modifying a single C file).

I'm part of the "trial-and-error" church. I modify random parts of the
C code, build, test and repeat until it works as I want ;-)

A *fresh* build (after make clean) of CPython on my laptop (8 threads,
4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh
build in release mode (make -j10 and gcc -O3) takes 44 seconds on the
same laptop.

CPython is made of around 500K lines of C code.

Victor

On Fri, Apr 16, 2021 at 7:19 PM  wrote:
>
> The benefits:
>
> 1. You will link with high quality libstdc++ with lots of reusable containers 
> without writing your own "buggy" one.
> 2. C++ is much much more maintainable than pure C. It drastically increase 
> number of contributors that what like writing high quality and maintainable 
> code without reinventing the wheel.
>
> My personal stop of contributing in CPython is that it is written in pure C !!
> I wrote code in both: pure C and C++, but I like writing code in C++, because 
> it simplifies things without losing perfomance
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/YNVO3NM7J77AHVOPGCOTWZELTTHHA7YB/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PAIR6UTGNHYKGCTU3GRZSUVECJ2L2W3X/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

2021-04-16 Thread Victor Stinner
The warnings.filterwarnings() function uses a regular expression for
message and module arguments. My request is only about the Python
command line interface.

By the way, an user requested to use a regex for the module field of
-W option and PYTHONWARNINGS env var:
https://bugs.python.org/issue34624

Victor

On Fri, Apr 16, 2021 at 3:22 PM Ivan Pozdeev via Python-Dev
 wrote:
>
> It'll probably be easier to change the warnings filter semantic 
> (https://docs.python.org/3/library/warnings.html#the-warnings-filter)
> to which those values are directly passed.
>
> On 16.04.2021 16:07, Victor Stinner wrote:
> > Hi,
> >
> > I propose to change the -W command line option and the PYTHONWARNINGS
> > environment variable to use the message as a regular expression in
> > Python 3.10. Or does anyone have a reason to keep the current behavior
> > as it is?
> >
> > I created https://bugs.python.org/issue43862 for this change.
> >
> > --
> >
> > Python provides two ways to specify warnings filters:
> >
> > * -W command line option: can be used multiple times
> > * PYTHONWARNINGS environment variable: can contain multiple options
> > separated by commas
> >
> > While the Python API warnings.filterwarnings(action, message="", ...)
> > uses the message as a regular expression, -W and PYTHONWARNINGS
> > require to match *exactly* the *whole* message.
> >
> > For example, if you only want to ignore the new distutils deprecation
> > warning, you must write exactly:
> >
> > $ ./python -X dev -W 'ignore:The distutils package is deprecated and
> > slated for removal in Python 3.12. Use setuptools or check PEP 632 for
> > potential alternatives:DeprecationWarning' -c 'import distutils'
> >
> > I use -X dev to show DeprecationWarning, or you can also use -Wdefault
> > if you prefer.
> >
> > If the deprecation warning changes in Python or if you have a single
> > typo, the warning is not ignored. Example with a typo ("3.13" rather
> > than "3.12"):
> >
> > $ ./python -X dev -W 'ignore:The distutils package is deprecated and
> > slated for removal in Python 3.13. Use setuptools or check PEP 632 for
> > potential alternatives:DeprecationWarning' -c 'import distutils'
> > :1: DeprecationWarning: The distutils package is deprecated
> > and slated for removal in Python 3.12. Use setuptools or check PEP 632
> > for potential alternatives
> >
> > The PYTHONWARNINGS has another limitation: you cannot specify a
> > message if it contains a comma (","). Hopefully, Python doesn't raise
> > warnings containing comma, right? Well... Just one example:
> >
> > Lib/distutils/sysconfig.py:554:warnings.warn('SO is
> > deprecated, use EXT_SUFFIX', DeprecationWarning, 2)
> >
> > You cannot only ignore the message:
> >
> > $ PYTHONWARNINGS='ignore:SO is deprecated, use
> > EXT_SUFFIX:DeprecationWarning' ./python -c 'import sys;
> > print(sys.warnoptions); print(len(sys.warnoptions))'
> > Invalid -W option ignored: invalid action: 'use EXT_SUFFIX'
> > ['ignore:SO is deprecated', ' use EXT_SUFFIX:DeprecationWarning']
> > 2
> >
> > You can only try to use "module" and "lineno" parameters of a warning
> > filter, which are more fragile and hard to use to use.
> >
> > Victor
>
> --
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ADUDHMNJIYERRA5MHF4GGB2OXV2XJC37/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IIWWB5IZ32H3O4QHYUGRA2D7DYMVSUIY/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Can we change python -W option and PYTHONWARNINGS to use a regex for the message?

2021-04-16 Thread Victor Stinner
Hi,

I propose to change the -W command line option and the PYTHONWARNINGS
environment variable to use the message as a regular expression in
Python 3.10. Or does anyone have a reason to keep the current behavior
as it is?

I created https://bugs.python.org/issue43862 for this change.

--

Python provides two ways to specify warnings filters:

* -W command line option: can be used multiple times
* PYTHONWARNINGS environment variable: can contain multiple options
separated by commas

While the Python API warnings.filterwarnings(action, message="", ...)
uses the message as a regular expression, -W and PYTHONWARNINGS
require to match *exactly* the *whole* message.

For example, if you only want to ignore the new distutils deprecation
warning, you must write exactly:

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.12. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'

I use -X dev to show DeprecationWarning, or you can also use -Wdefault
if you prefer.

If the deprecation warning changes in Python or if you have a single
typo, the warning is not ignored. Example with a typo ("3.13" rather
than "3.12"):

$ ./python -X dev -W 'ignore:The distutils package is deprecated and
slated for removal in Python 3.13. Use setuptools or check PEP 632 for
potential alternatives:DeprecationWarning' -c 'import distutils'
:1: DeprecationWarning: The distutils package is deprecated
and slated for removal in Python 3.12. Use setuptools or check PEP 632
for potential alternatives

The PYTHONWARNINGS has another limitation: you cannot specify a
message if it contains a comma (","). Hopefully, Python doesn't raise
warnings containing comma, right? Well... Just one example:

Lib/distutils/sysconfig.py:554:warnings.warn('SO is
deprecated, use EXT_SUFFIX', DeprecationWarning, 2)

You cannot only ignore the message:

$ PYTHONWARNINGS='ignore:SO is deprecated, use
EXT_SUFFIX:DeprecationWarning' ./python -c 'import sys;
print(sys.warnoptions); print(len(sys.warnoptions))'
Invalid -W option ignored: invalid action: 'use EXT_SUFFIX'
['ignore:SO is deprecated', ' use EXT_SUFFIX:DeprecationWarning']
2

You can only try to use "module" and "lineno" parameters of a warning
filter, which are more fragile and hard to use to use.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JMKLA4RUJLTORBPPTM4BWL76VNNMKYVG/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

2021-04-15 Thread Victor Stinner
SOURCE_DATE_EPOCH is not a random variable, but is a *standardised*
environment variable:
https://reproducible-builds.org/docs/source-date-epoch/

This page explains the rationale. See the “Lying about the time” /
“violates language spec” section ;-)

More and more projects adopted it. As I wrote, the Python stdlib
already uses it in compileall and py_compile modules.

Victor

On Thu, Apr 15, 2021 at 12:34 PM Antoine Pitrou  wrote:
>
> On Thu, 15 Apr 2021 11:28:03 +0200
> Victor Stinner  wrote:
> > If gzip is modified to use SOURCE_DATE_EPOCH timestamp, you get a
> > reproducible binary and you don't need to add a new constant ;-)
> > SOURCE_DATE_EPOCH is a timestamp: number of seconds since Unix Epoch
> > (January 1, 1970 at 00:00).
>
> Changing the behaviour of a stdlib module based on an environment
> variable sounds a bit undesirable.  That behaviour can be implemented
> at a higher-level in application code (for example the tarfile or
> zipfile command line).
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/HPX62SVAMT6ELIKCDWE2JDTY4ATX2NKU/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MHDIQZZXQJRBSXDMQKV4JYR6J5UU2OFH/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

2021-04-15 Thread Victor Stinner
Paul Bryan:
> Seems like this is something that should make its way into stdlib?

In the last 10 years, the trend is more to remove anything related to
packaging *outside* the stdlib :-) Since it's evolving way faster than
Python releases and the stdlib cannot be updated once installed, I
don't think that it's a good idea.

On Thu, Apr 15, 2021 at 12:22 AM Antoine Pitrou  wrote:
> Tangentially, until now projects could use distutils's LooseVersion if
> they wanted to compare version numbers reliably.  With distutils being
> deprecated, they'll have to either depending on packaging (which is a
> large dependency just for comparison version numbers) or vendor
> packaging's Version class (which is doable but still some bothersome
> additional work).

If packaging is too big and if packaging maintainters like the idea,
maybe packaging.version could be moved into a dedicated package? I
didn't look if it makes sense from a technical point of view.

$ wc -l packaging/version.py  packaging/_structures.py
  556 packaging/version.py
   86 packaging/_structures.py
  642 total

version.py uses _structures.py (InfinityType, NegativeInfinityType).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GT6NZDIKA222ADKKSJN7W4WZROMAIRDF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

2021-04-15 Thread Victor Stinner
If gzip is modified to use SOURCE_DATE_EPOCH timestamp, you get a
reproducible binary and you don't need to add a new constant ;-)
SOURCE_DATE_EPOCH is a timestamp: number of seconds since Unix Epoch
(January 1, 1970 at 00:00).

Victor


On Wed, Apr 14, 2021 at 8:15 PM  wrote:
>
> The gzip specification [1] makes clear that the mtime field is always present.
> The time is in Unix format, i.e., seconds since 00:00:00 GMT, Jan.  1, 1970.
> MTIME = 0 means no time stamp is available. Hence no need for a
> new constant NO_TIMESTAMP.
>
> So this is primarily a documentation problem [2]. For this, I will create a
> pull request to gzip.py.
>
> Joachim
>
> [1] https://www.ietf.org/rfc/rfc1952.txt
> [2] 
> https://discuss.python.org/t/gzip-py-allow-deterministic-compression-without-time-stamp
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/LCPWERWIFG4AJS6DPHNEMOGBYJ2APDJ3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FZTBVELC53IX6CGRCG53IGECJC3SANLE/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

2021-04-14 Thread Victor Stinner
Hi,

gzip.NO_TIMESTAMP sounds like a good idea. But I'm not sure about
changing the default behavior. I would prefer to leave it unchanged.

I guess that your problem is that you don't access gzip directly, but
uses a higher level API which doesn't give access to the timestamp
parameter, like the tarfile module?

If your usecase is reproducible build, you may follow py_compile
behavior: the default behavior depends if the SOURCE_DATE_EPOCH
environment variable is set or not:

def _get_default_invalidation_mode():
if os.environ.get('SOURCE_DATE_EPOCH'):
return PycInvalidationMode.CHECKED_HASH
else:
return PycInvalidationMode.TIMESTAMP

Victor

On Wed, Apr 14, 2021 at 6:34 PM Joachim Wuttke  wrote:
>
> gzip compression, using class GzipFile from gzip.py, by default
> inserts a timestamp to the compressed stream. If the optional
> argument `mtime` is absent or None, then the current time is used [1].
>
> This makes outputs non-deterministic, which can badly confuse
> unsuspecting users: If you run "diff" over two outputs to see
> whether they are unaffected by changes in your application,
> then you would not expect that the *.gz binaries differ just
> because they were created at different times.
>
> I'd propose to introduce a new constant `NO_TIMESTAMP` as
> possible value of `mtime`.
>
> Furthermore, if policy about API changes allows, I'd suggest
> that `NO_TIMESTAMP` become the new default value for `mtime`.
>
> How to proceed from here? Is this the kind of proposals that
> has to go through a PEP?
>
> - Joachim
>
> [1]
> https://github.com/python/cpython/blob/6f1e8ccffa5b1272a36a35405d3c4e4bbba0c082/Lib/gzip.py#L163
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/OTUGLATLYB736SAPPRWSSXWAKM5JHWZN/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5FLBLVY3DJFGIBMED57SASLS5ASZ65KF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

2021-04-14 Thread Victor Stinner
On Wed, Apr 14, 2021 at 7:48 AM Christopher Barker  wrote:
> So what do you'all think? After thirteen years, it would be nice to put this 
> to bed.

There are two main use cases for versions:

* Display them to the user
* Compare versions to check if one is newer, older or the same

I dislike using strings for comparison. You need to use
packaging.version for that:
https://packaging.pypa.io/en/latest/version.html

Many C libraries provide the version as a number of as 3 numbers
(major, minor, micro). In its C API, Python provides all of them:

* PY_VERSION_HEX: single number
* (PY_MAJOR_VERSION, PY_MINOR_VERSION, PY_MICRO_VERSION,
PY_RELEASE_LEVEL, PY_RELEASE_SERIAL): as 5 numbers
* PY_VERSION: string

In my Python projects, I like to provide the version as a tuple which
can be used directly for comparison: version_a <= version_b. Example:

VERSION = (2, 2, 1)
__version__ = '.'.join(map(str, VERSION))

The tuple might contain strings like "beta" or "rc", as soon as
comparison makes sense ;-) Sadly, such tuple is no standardized. Which
part is the major version? How to format it as a string?

Good luck with trying to standardize that ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBBYB5AWX76O3TOUFATRKSU2QND2TPKS/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Boundaries between numbers and identifiers

2021-04-14 Thread Victor Stinner
Also, would it be possible to enhance to tokenizer to report a
SyntaxWarning, rather than a SyntaxError?

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CH7SLXKIKX47KVCWEJEMOB35BCIM7Y5U/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Boundaries between numbers and identifiers

2021-04-13 Thread Victor Stinner
It would be useful to first estimate how many projects would be broken
by such incompatible change (stricter syntax).

Inada-san wrote
https://github.com/methane/notes/blob/master/2020/wchar-cache/download_sdist.py
to download source files using
https://hugovk.github.io/top-pypi-packages/ API (top 4000 PyPI
projects).

Victor

On Tue, Apr 13, 2021 at 10:59 PM Guido van Rossum  wrote:
>
> On Tue, Apr 13, 2021 at 12:55 PM Serhiy Storchaka  wrote:
>>
>> 26.04.18 21:37, Serhiy Storchaka пише:
>> > In Python 2.5 `0or[]` was accepted by the Python parser. It became an
>> > error in 2.6 because "0o" became recognizing as an incomplete octal
>> > number. `1or[]` still is accepted.
>> >
>> > On other hand, `1if 2else 3` is accepted despites the fact that "2e" can
>> > be recognized as an incomplete floating point number. In this case the
>> > tokenizer pushes "e" back and returns "2".
>> >
>> > Shouldn't it do the same with "0o"? It is possible to make `0or[]` be
>> > parseable again. Python implementation is able to tokenize this example:
>> >
>> > $ echo '0or[]' | ./python -m tokenize
>> > 1,0-1,1:NUMBER '0'
>> > 1,1-1,3:NAME   'or'
>> > 1,3-1,4:OP '['
>> > 1,4-1,5:OP ']'
>> > 1,5-1,6:NEWLINE'\n'
>> > 2,0-2,0:ENDMARKER  ''
>> >
>> > On other hand, all these examples look weird. There is an assymmetry:
>> > `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize
>> > visually the boundary between a number and the following identifier or
>> > keyword, especially if numbers can contain letters ("b", "e", "j", "o",
>> > "x") and underscores, and identifiers can contain digits. On both sides
>> > of the boundary can be letters, digits, and underscores.
>> >
>> > I propose to change the Python syntax by adding a requirement that there
>> > should be a whitespace or delimiter between a numeric literal and the
>> > following keyword.
>> >
>>
>> New example was found recently (see https://bugs.python.org/issue43833).
>>
>>  >>> [0x1for x in (1,2)]
>>  [31]
>>
>> It is parsed as [0x1f or x in (1,2)] instead of [0x1 for x in (1,2)].
>>
>> Since this code is clearly ambiguous, it makes more sense to emit a
>> SyntaxWarning if there is no space between number and identifier.
>
>
> I would totally make that a SyntaxError, and backwards compatibility be 
> damned.
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/OU3USHVMXZJD4SA3FJGQQVQYAORHY5BM/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5VYOQRW4DOVDNSIB3G7GBHSUL5ZC3QZO/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Please review the new Python Build System documentation

2021-04-12 Thread Victor Stinner
Hi,

Please review this documentation:

https://docs.python.org/dev/using/configure.html

If you have comment, you can directly write a PR (please notify me by
mentioning @vstinner in a comment). Or if you prefer, you can reply
directly by email and I will try to address your remarks.

--

Over the last years, the configure script got more and more options. I
was always disappointed by the lack of documentation. It is also
really hard to understand how Python pass compiler and linker files
when building C code. So I wrote a documentation about configure,
preprocessor, compiler and linker flags, and a very brief description
of the Python Build System.

I chose to not document *all* configure/compiler options. I skipped
generic configure options like --prefix. I copied the documentation
that I could find, but sadly many options are still not documented
yet.

For the first time, I wrote a description of in the infamous "Python
Debug Build", especially ("all") effects of this build:

https://docs.python.org/dev/using/configure.html#debug-build

I also took the opportuniy to add links from configure --with-xxx
options to this documentation, and also add links from "debug mode" or
"debug build" terms to the Debug Build section.

Note: the online documentation is only updated once per day. The job
building the documentation of all Python versions in all supported
languages takes a whole day! If you want to help, visit:
https://github.com/python/docsbuild-scripts/

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G3JJPRWMU2W6ARNVAUALDJOUNZSP343M/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: When we remove 'U' mode of open()?

2021-04-08 Thread Victor Stinner
That sounds reasonable ;-)

Victor

On Thu, Apr 8, 2021 at 3:02 AM Inada Naoki  wrote:
>
> We are close to 3.10 beta and it is not ideal timing for removing.
> So my proposal is:
>
> * Remove 'U' in fileinput, because it makes my task little simpler.
> * Remove 'U' in other places in Python 3.11, after 3.10 branch is
> created (and master branch is renamed to main).
>
> On Thu, Apr 8, 2021 at 5:45 AM Brett Cannon  wrote:
> >
> >
> >
> > On Wed, Apr 7, 2021 at 10:01 AM Serhiy Storchaka  
> > wrote:
> >>
> >> 07.04.21 19:13, Victor Stinner пише:
> >> > Hi Inada-san,
> >> >
> >> > I'm +0 on removing again the flag, but I would prefer to not endorse
> >> > the responsibility. I am already responsible for enough incompatible
> >> > changes in Python 3.10 :-D
> >> >
> >> > Some context on this "U" open mode. The flag is accepted by many
> >> > functions opening files. It is deprecated (emit DeprecationWarning)
> >> > for 9 years (Python 3.3, 2012).
> >>
> >> It was silently deprecated before 3.3 (perhaps it was no-op since 3.0).
> >>
> >> I added DeprecationWarning with intention to remove this option in all
> >> functions accepting it. The only non-trivial support of the "U" mode was
> >> left in ZipFile.open(), and it was broken since beginning.
> >
> >
> > I think at this point the DeprecationWarning has definitely been on long 
> > enough, there was an explicit warning about it in Python 3.9, and 3.10 will 
> > be nearly 2 years removed from 2.7 reaching EOL which is the only place 
> > where "U" may still be used. So I think it's fine to drop "U" in 3.10.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/VTROKN5UOU3EN6F3OLX5RUK7TVETAXKB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/NYVORKRSH562UMAXXLSJOOW5ECBA3HC5/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/633HYC4O3TCOZARWXI6JCDFO2SBZHXRT/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: When we remove 'U' mode of open()?

2021-04-07 Thread Victor Stinner
Hi Inada-san,

I'm +0 on removing again the flag, but I would prefer to not endorse
the responsibility. I am already responsible for enough incompatible
changes in Python 3.10 :-D

Some context on this "U" open mode. The flag is accepted by many
functions opening files. It is deprecated (emit DeprecationWarning)
for 9 years (Python 3.3, 2012).

The flag is currently ignored and is basically only kept for backward
compatibility with Python 2, whereas Python 2.7 support ended in
January 2020 (1 year ago).

I removed the flag in Python 3.9:

* https://bugs.python.org/issue37330
* 
https://github.com/python/cpython/commit/e471e72977c83664f13d041c78549140c86c92de

The removal broke the Waf build system used by the Samba project. Waf
code is copied into Samba source code (as Python copies
autoconf/automake files). So even if Waf and then the Waf copy in
Samba are updated to not pass the "U" flag, trying to build an old
Samba version will fail if the flag is removed. Andrew Bartlett
explained that it's a practical issue when bisecting a bug in Samba.
It's annoying, but likely easy to workaround (fix the local waf copy
temporarily).

Moreover, the removal broke 11 packages in Fedora:

* aubio
* openvswitch
* python-SALib
* python-altgraph
* python-apsw
* python-magic-wormhole-mailbox-server
* python-munch
* python-parameterized
* python-pylibmc
* python-sphinx-testing
* veusz

Miro Hrončok and me proposed to revert the removal in Python 3.9:
https://mail.python.org/archives/list/python-dev@python.org/thread/EYLXCGGJOUMZSE5X35ILW3UNTJM3MCRE/

The removal was reverted (accept again the flag) in Python 3.9:

* https://bugs.python.org/issue39674#msg363195
* 
https://github.com/python/cpython/commit/942f7a2dea2e95a0fa848329565c0d0288d92e47

LWN article summarizing the issue: https://lwn.net/Articles/811369/

I added a section at the very beginning of What's New in Python 3.9 to
request developers to check for DeprecationWarning in their project:
https://docs.python.org/dev/whatsnew/3.9.html#you-should-check-for-deprecationwarning-in-your-code

The idea was that Python 3.9 is the last version supporting the flag,
developers are now warned, and so the flag should be removed from
Python 3.10.

That being said, I'm kind of cautious. Each time I introduce a minor
incompatible change breaking a few projects (say 5 projects or less),
many people get angry and complain without trying to understand the
rationale. Moreover, they are silent when I say that there was a
DeprecationWarning for 9 years.

I didn't check if the 11 projects + waf + samba have been updated to
no longer pass the deprecated "U" flag.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AAPTZUY46IG2H75TQKMWHOQBVTOCXN2K/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: [python-committers] NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems

2021-04-06 Thread Victor Stinner
Hi,

About this very specific ABI issue, one long term solution would be to
exclude the PyThreadState structure from the C API, to not rely on it
the ABI level.

I started to add getter functions in Python 3.9:
PyThreadState_GetInterpreter(), PyThreadState_GetFrame() and
PyThreadState_GetID(). I'm working on updating C extensions to use
these getter functions, rather than accessing directly PyThreadState
members. I wrote a new pythoncapi_compat.h header file (in an exteral
project, pythoncapi_compat) to provide getter functions to Python
2.7-3.8. Cython gives me most of the work, since it gets and sets many
PyThreadState members.

You can follow the progress at: https://bugs.python.org/issue39947

Victor

On Sat, Apr 3, 2021 at 9:45 PM Łukasz Langa  wrote:
>
> The memory layout of PyThreadState was unintentionally changed in the recent 
> 3.9.3 bugfix release. This leads to crashes on 32-bit systems when importing 
> binary extensions compiled for Python 3.9.0 - 3.9.2. This is a regression.
>
> We will be releasing a hotfix 3.9.4 around 24 hours from now to address this 
> issue and restore ABI compatibility with C extensions built for Python 3.9.0 
> - 3.9.2.
>
> Details:
> https://bugs.python.org/issue43710
>
>
> - Ł
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/OUCNKMFBNGFK2AI4B7S7MF5O6UVLBSMB/
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BOEUCEILH5TWMZIY3DIHTLX3S7O2XKIF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Weird io.OpenWrapper hack to use a function as method

2021-03-31 Thread Victor Stinner
I created https://bugs.python.org/issue43682 "Make static methods
created by @staticmethod callable".

Oh, I didn't know this old bpo-20309 issue closed as "not a bug". But
it proposed to modify many other wrappers like @classmethod. I only
propose to make static methods created @staticmethod callable.

Victor

On Wed, Mar 31, 2021 at 4:19 PM Ethan Furman  wrote:
>
> On 3/31/21 6:49 AM, Victor Stinner wrote:
>
> > tl; dr *Maybe* staticmethod could be modified to become callable?
>
> There have been other requests to make staticmethod callable, one of them 
> being
>
>https://bugs.python.org/issue20309
>
> +1 for having it done.
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/KITFURNWOLEMETD2CLM6LKXCFUNZUNUP/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D6CI33WWUMQTA4VKVKIPZGGJ4XCQTTI2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Weird io.OpenWrapper hack to use a function as method

2021-03-31 Thread Victor Stinner
A friend, Antoine Rozo, wrote a variant of staticmethod which is
callable. With this decorator, it works in A, B and C cases:
---
class simplefunction:
def __init__(self, func):
self.func = func
def __get__(self, owner, instance):
return self.func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)

@simplefunction
def func():
print("my func")

class MyClass:
method = func

func() # A
MyClass.method() # B
MyClass().method() # C
---

It works without the __get__() method, but in this case, we go through
the __call__() indirection for A, B and C cases, rather than only
going through __call__() indirection in A case.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OQUM4RZBIGHF6UJYQDFAMTHOZOU5DLKS/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Weird io.OpenWrapper hack to use a function as method

2021-03-31 Thread Victor Stinner
tl; dr *Maybe* staticmethod could be modified to become callable?


io.open is a built-in function (type "builtin_function_or_method"). It
behaves differently than _pyio.open which is a Python function (type
"function").

The difference is in the LOAD_METHOD bytecode which uses __get__()
descriptor if available. Built-in function has no __get__() method and
so the function is used directly as a method. Python function has a
__get__() descriptor which returns the function unchanged when a class
method is requested, and create a bound method if an instance method
is requested:
---
def func():
...

FunctionType = type(func)

class MyClass:
method = func

class_method = MyClass.method
assert class_method is func
assert class_method is FunctionType.__get__(func, None, MyClass)

obj = MyClass()
instance_method = obj.method
assert instance_method.__self__ is obj
assert instance_method.__func__ is func
# each __get__() call creates a new bound method
assert instance_method == FunctionType.__get__(func, obj, type(obj))
---

@staticmethod decorator avoids the creation of the bound method:
---
def func(): ...

class MyClass:
method = staticmethod(func)

# method = MyClass.method
attr = MyClass.__dict__['method']
method = type(attr).__get__(attr, None, MyClass)
assert method is func
---

The drawback is that the object created by staticmethod cannot be
called :-( The following code raises a TypeError:
---
wrapped = staticmethod(func)
wrapped()
---

*Maybe* staticmethod could be modified to become callable?

Victor


On Wed, Mar 31, 2021 at 2:34 PM Victor Stinner  wrote:
>
> Hi,
>
> The io module provides an open() function. It also provides an
> OpenWrapper which only exists to be able to store open as a method
> (class or instance method). In the _pyio module, pure Python
> implementation of the io module, OpenWrapper is implemented as:
>
> class OpenWrapper:
> """Wrapper for builtins.open
>
> Trick so that open won't become a bound method when stored
> as a class variable (as dbm.dumb does).
>
> See initstdio() in Python/pylifecycle.c.
> """
> def __new__(cls, *args, **kwargs):
> return open(*args, **kwargs)
>
> I would like to remove this class which is causing troubles in the PEP
> 597 implementation, but I don't know how. Simplified problem:
> ---
> def func():
> print("my func")
>
> class MyClass:
> method = func
>
> func() # A
> MyClass.method() # B
> obj = MyClass()
> obj.method() # C
> ---
>
> With this syntax, A and B work, but C fails with TypeError: func()
> takes 0 positional arguments but 1 was given.
>
> If I decorate func() with @staticmethod, B and C work, but A fails
> with TypeError: 'staticmethod' object is not callable.
>
> Is OpenWrapper the only way to have a callable object which works in
> the 3 variants A, B and C?
>
> A, B and C work if MyClass is modified to use staticmethod:
>
> class MyClass:
> method = staticmethod(func)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KAZ5HR2ZLPBZ76FZZOZI5RU35FYDBDI7/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Weird io.OpenWrapper hack to use a function as method

2021-03-31 Thread Victor Stinner
Hi,

The io module provides an open() function. It also provides an
OpenWrapper which only exists to be able to store open as a method
(class or instance method). In the _pyio module, pure Python
implementation of the io module, OpenWrapper is implemented as:

class OpenWrapper:
"""Wrapper for builtins.open

Trick so that open won't become a bound method when stored
as a class variable (as dbm.dumb does).

See initstdio() in Python/pylifecycle.c.
"""
def __new__(cls, *args, **kwargs):
return open(*args, **kwargs)

I would like to remove this class which is causing troubles in the PEP
597 implementation, but I don't know how. Simplified problem:
---
def func():
print("my func")

class MyClass:
method = func

func() # A
MyClass.method() # B
obj = MyClass()
obj.method() # C
---

With this syntax, A and B work, but C fails with TypeError: func()
takes 0 positional arguments but 1 was given.

If I decorate func() with @staticmethod, B and C work, but A fails
with TypeError: 'staticmethod' object is not callable.

Is OpenWrapper the only way to have a callable object which works in
the 3 variants A, B and C?

A, B and C work if MyClass is modified to use staticmethod:

class MyClass:
method = staticmethod(func)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QZ7SFW3IW3S2C5RMRJZOOUFSHHUINNME/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] New public C API functions must not steal references or return borrowed references

2021-03-25 Thread Victor Stinner
Hi,

A new Include/README.rst file was just added to document the 3 C API
provided by CPython:

* Include/: Limited C API
* Include/cpython/: CPython implementation details
* Include/internal/: The internal API

I would like to note that *new* public C API functions must no longer
steal references or return borrowed references.

Don't worry, there is no plan to deprecate or remove existing
functions which do that, like PyModule_AddObject() (streal a
reference) or PyDict_GetItem() (return a borrowed reference). The
policy is only to *add* new functions.

IMO for the *internal* C API, it's fine to continue doing that for
best performances.

Moreover, the limited C API must not expose "implementation details".
For example, structure members must not be accessed directly, because
most structures are excluded from the limited C API. A function call
hiding implementation details is usually better.

Here is a copy of the current Include/README.rst file:

The Python C API


The C API is divided into three sections:

1. ``Include/``
2. ``Include/cpython/``
3. ``Include/internal/``


Include: Limited API


``Include/``, excluding the ``cpython`` and ``internal`` subdirectories,
contains the public Limited API (Application Programming Interface).
The Limited API is a subset of the C API, designed to guarantee ABI
stability across Python 3 versions, and is defined in :pep:`384`.

Guidelines for expanding the Limited API:

- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference
- Macros should not expose implementation details
- Please start a public discussion before expanding the API
- Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.

It is possible to add a function or macro to the Limited API from a
given Python version.  For example, to add a function to the Limited API
from Python 3.10 and onwards, wrap it with
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A``.


Include/cpython: CPython implementation details
===

``Include/cpython/`` contains the public API that is excluded from the
Limited API and the Stable ABI.

Guidelines for expanding the public API:

- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference


Include/internal: The internal API
==


With PyAPI_FUNC or PyAPI_DATA
-

Functions or structures in ``Include/internal/`` defined with
``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
exposed only for specific use cases like debuggers and profilers.


With the extern keyword
---

Functions in ``Include/internal/`` defined with the ``extern`` keyword
*must not and can not* be used outside the CPython code base.  Only
built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN``
macro defined) can use such functions.

When in doubt, new internal C functions should be defined in
``Include/internal`` using the ``extern`` keyword.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QKFDGIAFPZJZE22GJ5OOMXOWQGFEDSU5/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Tests now start with only 131 imported modules, instead of 233

2021-03-23 Thread Victor Stinner
On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev
 wrote:
> Also, how is the now-split-off funcionality to be invoked? Does it require 
> two or more imports now, or it's imported on demand when one
> invokes an appropriate test.support entry?

By the way, splitting test.support into sub-modules is also a way to
work around the lack of lazy import in Python. It would be nice to
have this feature, but so far, no consensus on the lazy import
specification was reached.

One issue is that some use cases need the import side effects to be
applied as soon as the import is done ... It goes against the
principle of lazy import :-( I don't recall the details.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KZERLO3VIUG37AS4DIUITFSJHFU4DDSF/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Tests now start with only 131 imported modules, instead of 233

2021-03-23 Thread Victor Stinner
Hi Ivan,

On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev
 wrote:
> I didn't quite get what the big effect is. Saving 30 milliseconds?

I started to dig into this issue while debugging a random crash on AIX
(bpo-40091). A test_threading test using fork randomly crashed on AIX.
I discovered that the crash was triggered by the logging module. It
surprised me since threading and test_threading don't use the logging
module.

"import test.support" not only loads code but also has side effects
because it runs code. Some examples:

* "import logging" calls atexit.register() and os.register_at_fork().
* "import random" calls os.register_at_fork().
* "import readline" sets SIGWINCH signal handler.

In general, it's perfectly fine in tests to import modules which have
side effects. But I'm talking about the very specific case of the
Python test suite which tests low-level Python internals, any side
effect causes multiple annoying issues and makes debugging way harder.

I'm working for years on making the Python test suite more reliable.
These side effects indirectly make tests less reliable and less
reproducible.

IMO test_threading must not test the logging module. If someone wants
to test the relationship between logging, threading and fork, a new
dedicated test should be written in test_logging. In this case, there
are already fork tests in test_logging ;-)

By the way, I'm also working on fixing these random crashes ;-) For
example, I partially fixed the AIX crash at fork in a generic way by
adding new _at_fork_reinit() methods to locks: see also the underlying
_PyThread_at_fork_reinit() function.


> Also, how is the now-split-off funcionality to be invoked? Does it require 
> two or more imports now, or it's imported on demand when one
> invokes an appropriate test.support entry?

For example, TESTFN should now be get from test.support.os_helper.

Some tests use "from test.support.os_helper import TESTFN", other
tests prefer "from test.support import os_helper" and then use
"os_helper.TESTFN".

All tests in Lib/test/ have been updated last year, you don't have to
do anything.

By the way, the test module should not be used outside Python, it's
clearly specified in its documentatioin ;-) On Fedora, the "test"
module is not installed by default, but is in a separated python3-test
package. (Another reason is to reduce the disk space.)


I hope that it gives you a better idea why these changes were made ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OVGPFLZ762MKCGXFCXI7Y6QNR5PKJF7L/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Tests now start with only 131 imported modules, instead of 233

2021-03-22 Thread Victor Stinner
Hi,

tl;dr Tests of the Python Test Suite now start with 131 imported
modules, instead of 233. For example, asyncio, logging,
multiprocessing and warnings are no longer imported by default.

--

The Python test suite is run by a custom test runner called
"libregrtest" (test.libregrtest). It can detect reference leaks, write
the output as JUnit XML, run tests in parallel with multiple
processes, etc. Tests are written with test.support which is a
collection of helper functions.

Over the years, test.support got more and more functions, and
libregrtest got more and more features. The problem is that they
import more and more modules. For example, "import test.support"
imports 173 modules in Python 3.8!

$ python3.8
>>> import sys, sys
>>> before=set(sys.modules); import test.support; after=set(sys.modules)
>>> len(after - before)
173

In these imports, you can find some "heavy" modules like asyncio and
multiprocessing. Moreover, some modules have "side effects" on import.
For example, "import logging" registers an "at fork" callback.

In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to
reduce the number of test.support imports from 171 to 25! test.support
was splitted into multiple sub-modules:

- bytecode_helper
- hashlib_helper
- import_helper
- logging_helper
- os_helper
- script_helper
- socket_helper
- threading_helper
- warnings_helper

I continued the work in bpo-41718 to reduce the number of libregrtest
imports, since libregrtest also imported many modules.

A dummy test which does nothing in the master branch now only has 131
modules in sys.modules, whereas in Python 3.9 it has 233 (+102)
modules! For example, asyncio, logging, multiprocessing and warnings
are no longer imported by default.

"import test.support" is now faster. master branch compared to Python
3.8 using ./python -c 'import test.support' command and Python built
in release mode:

[py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster

More realistic command running a single test method: ./python -m test
test_os -m test_access

[py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster

The only annoying point is that splitting test.support into
sub-modules was not backported to Python 3.8 and 3.9, and so it will
make backports a little bit more annoying. Sorry about that!

--

If you know that two modules should be tested together, please write a
dedicated test for that ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I3OQTA3F66NQUN7CH2NHC5XZTO24QCIK/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Non-monotonically increasing line numbers in dis.findlinestarts() output

2021-03-17 Thread Victor Stinner
Hi Skip,

I modified co_lnotab format when I worked on the FAT Python
optimization project which modified the AST to emit more efficient
bytecode. My optimizer moved instructions, the line number could
become non-monotonic (..., line 2, line 3, line 2, line 4, ...), and
so lnotab (..., +1, -1, +2, ...) could be negative.

Example with loop unrolling:

def func1():
for i in ("a", "b"):
print(i)

is optimized:

def func2():
i = "a" # line 1
print(i) # line 2 (+1)

i = "b" # line 1 (-1)
print(i) # line 2 (+1)

I'm happy to see that Python 3.10 now also implements faster bytecode
which rely on this change ;-)

It seems like loop unrolling is not implemented in Python 3.10 yet. I
guess that Python/ast_opt.c has no strategy yet to decide if an
optimization should be applied or not depending on the code size. Loop
unrolling can make code way bigger.

Victor

On Thu, Mar 18, 2021 at 12:12 AM Ned Batchelder  wrote:
>
> On 3/17/21 6:41 PM, MRAB wrote:
> > On 2021-03-17 22:10, Skip Montanaro wrote:
> >
> >> I stumbled on this while trying to generate a line number table in my
> >> side project register VM. As I understand it, the line number delta
> >> in the output table is supposed to always be >= 0. In my code I'm
> >> using dis.findlinestarts() to determine the line numbers for each
> >> block. Perhaps I should be modifying its results. OTOH, maybe it's a
> >> bug. (If that's the consensus, I will create an issue.)
>
> co_lnotab has had negative deltas since 3.6.
>
> --Ned.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/EK2O6SE6J6LROGNHKPQYTQNJH5VFE2M4/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQAT4BHRJN4DKBLJ27MG6PWWVQJOUTFY/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Reviving PEP 473

2021-03-16 Thread Victor Stinner
For best performance (reduce time to build an exception object), I
would be interested to only format the error message on demand. For
example, when str(exception) is called.

The problem is the Exception.args attribute. Example:

$ ./python
Python 3.10.0a6+
>>> exc=AttributeError("%s object has no attribute %s" % ("MyObject", "name"))
>>> str(exc)
'MyObject object has no attribute name'
>>> exc.args
('MyObject object has no attribute name',)

Currently, args is the raw positional arguments passed to the
Exception constructor, and they are very likely many applications
relying on Exception.args.

For backward compatibility, we could store "%s object has no attribute
%s" and ("MyObject", "name"), and build the args tuple on demand and
format the string at the first str() call.

At the C level, args is exposed directly as PyBaseExceptionObject.args
and the PyBaseExceptionObject structure is part of the public C API.
Changing it would be a C API incompatible change.

By the way, the PEP 473 doesn't say anything about it and has no
"Backward compatibility" section :-(

Victor

On Sat, Mar 13, 2021 at 7:49 PM Sebastian Kreft  wrote:
>
> Hi dev-team, I'm reopening the discussion of PEP 473, which was closed due to:
>
> The steering council felt the PEP was too broad and not focused enough. 
> Discussions about adding more attributes to built-in exceptions can continue 
> on the issue tracker on a per-exception basis (and obviously here for any 
> broader points, e.g. performance implications as I know that has come up 
> before when the idea of storing relevant objects on exceptions).
>
> Before the PEP was rejected by the SC, I had submitted 
> https://github.com/python/cpython/pull/6271 and the last response said:
>
> You may propose a newly revised PEP which overcomes the last SC feedback.
> But without the accepted PEP, this PR can not be merged.
>
> Please clarify how can I proceed with adding some of the specified fields to 
> the mentioned exceptions. Should I create a new PEP as suggested in the PR or 
> should we create individual BPO issues for each exception modification.
>
> --
> Sebastian Kreft
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4XVFFNBHXN5KBTHJGMX6ZTN6KOW4Z7UG/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LVI7S6O5QVVTZN5A6DLYHEKGHW6EIZUH/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: [python-committers] Accepting PEP 597 (Add optional EncodingWarning)

2021-03-15 Thread Victor Stinner
Congratulation INADA-san! I'm impressed by your tenacity :-)

Last months, I followed your different propositions on
discuss.python.org to use UTF-8 by default in Python. It's good to see
the first non-controversial part being accepted! I hope that this PEP
will help to move towards a world where we don't guess encodings
anymore, but make them very explicit!

Once the whole stdlib and most of top PyPI projects will be fixed to
no longer emit EncodingWarning, I will become safer to opt-in for
UTF-8 by default by enabling the Python UTF-8 Mode!
https://docs.python.org/dev/library/os.html#python-utf-8-mode

One day, we will silently switch Python to UTF-8 by default, and
nobody will notice! ;-)

Victor

On Mon, Mar 15, 2021 at 7:46 PM Thomas Wouters  wrote:
>
> Hi Inada,
>
> Thank you for submitting PEP 597 (Add optional EncodingWarning). The Steering 
> Council is happy with the PEP, and hereby accepts it. The SC is of the 
> opinion that we should move towards making UTF-8 the default encoding, and 
> this PEP will make it easier to do so, and mitigate some of the confusion 
> around the default encoding in the meantime.
>
> That being said, the SC would like to invite you and others with interest to 
> work on a PEP for a long term plan to, in fact, change the default encoding 
> to UTF-8. We don't want to make irrevocable decisions until we have a clear, 
> viable plan with a good upgrade path and with community support, but we do 
> believe we need to make that decision sooner rather than later.
>
> With thanks from the whole Python Steering Council,
> Thomas.
> --
> Thomas Wouters 
>
> Hi! I'm an email virus! Think twice before sending your email to help me 
> spread!
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/YIZGJBMDQAOZHD6MXQCLEXIAKGUQFMQM/
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UHS2MSJR4LOITOJEFS3BCLEOONKHX36C/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Reviving PEP 473

2021-03-13 Thread Victor Stinner
Context: PEP 473 "Adding structured data to built-in exceptions"
proposed in 2014.
https://www.python.org/dev/peps/pep-0473/

Abstract: "Exceptions like AttributeError, IndexError, KeyError,
LookupError, NameError, TypeError, and ValueError do not provide all
information required by programmers to debug and better understand
what caused them. Furthermore, in some cases the messages even have
slightly different formats, which makes it really difficult for tools
to automatically provide additional information to diagnose the
problem. To tackle the former and to lay ground for the latter, it is
proposed to expand these exceptions so to hold both the offending and
affected entities."

For example the PEP proposes to add "target" and "attribute"
attributes to AttributeError.

PS: Please give at least the PEP title for me when you mention a PEP
only by its number, I'm unable to remember what are the 500+ PEP :-(

Victor

On Sat, Mar 13, 2021 at 7:49 PM Sebastian Kreft  wrote:
>
> Hi dev-team, I'm reopening the discussion of PEP 473, which was closed due to:
>
> The steering council felt the PEP was too broad and not focused enough. 
> Discussions about adding more attributes to built-in exceptions can continue 
> on the issue tracker on a per-exception basis (and obviously here for any 
> broader points, e.g. performance implications as I know that has come up 
> before when the idea of storing relevant objects on exceptions).
>
> Before the PEP was rejected by the SC, I had submitted 
> https://github.com/python/cpython/pull/6271 and the last response said:
>
> You may propose a newly revised PEP which overcomes the last SC feedback.
> But without the accepted PEP, this PR can not be merged.
>
> Please clarify how can I proceed with adding some of the specified fields to 
> the mentioned exceptions. Should I create a new PEP as suggested in the PR or 
> should we create individual BPO issues for each exception modification.
>
> --
> Sebastian Kreft
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4XVFFNBHXN5KBTHJGMX6ZTN6KOW4Z7UG/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PLPUG6MFVHHHVONJK6ZHPORCQYJS5QTH/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: [Release management] schedule for renaming the default branch

2021-03-11 Thread Victor Stinner
Hi Serhiy,

On Thu, Mar 11, 2021 at 8:33 AM Serhiy Storchaka  wrote:
> I have above 200 feature branches in my local repository. Will renaming
> the master branch cause any problems?

I don't think that you need to do anything on your machine nor on your open PRs.

When I use "git switch -c new_branch" command to create a new branch,
the created branch doesn't "track" its parent branch by default.
Example:

$ git branch -vv
(...)
* gilstate_init a6959b8971 bpo-43311: Create GIL autoTSSkey ealier
  master9a9c11ad41 [upstream/master] bpo-43287: Use PEP 590
vectorcall to speed up filter() (GH-24611)

My "gilstate_init" local branch doesn't track any branch, whereas my
local "master" branch tracks upstream/master (my upstream remote is
g...@github.com:python/cpython.git).

Usually, when I want to easily see the differences between a local
branch and my local master branch (to use my "git out" alias), I type
"git branch --set-upstream-to=master":

$ git switch gilstate_init
$ git branch --set-upstream-to=master
$ git out
a6959b8971 bpo-43311: Create GIL autoTSSkey ealier

where my "git out" alias is the command:

$ git log '@{upstream}..' --pretty='format:%Cred%h%Creset %s' --color --reverse

You can check that gilstate_init now tracks my local master branch:

$ git branch -vv
(...)
  gilstate_init a6959b8971 [master: ahead 1] bpo-43311: Create GIL
autoTSSkey ealier
  master9a9c11ad41 [upstream/master] bpo-43287: Use PEP 590
vectorcall to speed up filter() (GH-24611)

Use "git switch -c new_branch --track master" to create a new branch
based on master which tracks the master branch.

Maybe I should track upstream/master rather than my local master
branch, but when I fetch the upstream remote, I always update my local
master branch, so in my case, it's the same in pratice :-) And
"master" is shorter to type than "upstream/master".

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T7IS4WR5TTTI7ULDMH6JRQ7YTTJN7MID/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Steering Council update for February

2021-03-10 Thread Victor Stinner
Hi,

Could we please stick to the point of renaming a Git branch? Today,
renaming a branch is easy. The rationale has been given. I don't think
any argument is going to make the Steering Concil changing their mind
("the consensus was that we should do that"). If you want to help,
please remain at the technical level to help making this migration as
smooth as possible for everybody.

On Wed, Mar 10, 2021 at 2:09 PM Evpok Padding  wrote:
> If the discussion here can stay civil for changes with far more repercussions 
> for Python (e.g. controversial PEPs), surely it can stay civil for this too.

Thanks. For people able to remain constructive and civil, please join
the discussion on the devguide on "listing terms which should be
avoided":
https://github.com/python/devguide/issues/605

While it's not easy to give an exhaustive list of terms which should
be avoided, IMO it would prevent future discussion like this thread.
I'm tired of such discussion which is not constructive at all and goes
nowhere. People love to take it as an opportonity to troll and quickly
messages become more and more offensive.

IMO 
https://chromium.googlesource.com/chromium/src/+/master/styleguide/inclusive_code.md
is a good example to follow: it's short and gives concrete advices for
better terms.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5QI7XFD6EBKZOZ4OSERCOZW5PBORQNWG/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: CFLAGS_NODIST and -qalias=noansi

2021-03-08 Thread Victor Stinner
On Fri, Mar 5, 2021 at 5:47 PM Michael Felt  wrote:
> Fyi: using `git log` I have tried to get any clue re: CFLAGS_NODIST
> and/or -qalias=noansi - but I do not seem to be skilled enough to find
> that information.

Try "git blame configure.ac" and search for "noansi".

It was introduced in bpo-41721 with this commit:

commit 84a7917b4c9afec07575065cffa143b91fe98c14
Author: Stefan Krah 
Date:   Fri Sep 4 22:33:17 2020 +0200

bpo-41721: Add xlc options (GH-22096)

* https://bugs.python.org/issue41721
* https://github.com/python/cpython/pull/22096

The rationale seems to be: https://bugs.python.org/issue41721#msg376396

Note: it would be nice to add a comment in configure.ac with a link to
the bpo when a new C flag is introduced.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FM2QBWEVY7GX53P4VSGZJZWZJ4JLNJNW/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Need help to fix known Python security vulnerabilities

2021-03-08 Thread Victor Stinner
Hi,

The Python bug tracker currently has 78 open issues of the type
Security. If you are looking for something to do to help the Python
project, please go through the list (search for open issues with
Type=security at bugs.python.org), discuss the different solutions how
to address these vulnerabilities, and maybe even propose a fix.

Here are some examples.

== tarfile ==

For example, the tarfile module has a known directory traversal
vulnerability (unsafe by default), whereas the GNU tar command is
fixed: the -P/--absolute-names option must be used explicitly to get
the unsafe behavior.

I suggest to make the Python tarfile module safe by default, and add
an option to allow absolute paths. Issue reported 7 years ago:
https://bugs.python.org/issue21109

== webbrowser ==

Another example, on Windows, the webbrowser seems be has a shell
command injection vulnerability, reported 2 years ago:
https://bugs.python.org/issue36021

== XML ==

Python XML parsers have at least two known vulnerabilities: "billion
laughs" and "quadratic blowup" which are documented:
https://docs.python.org/dev/library/xml.html#xml-vulnerabilities

The third party defusedxml module address these vulnerabilities:
https://pypi.org/project/defusedxml/

But Python remains unsafe by default, issue reported 8 years ago:
https://bugs.python.org/issue17239


== tempfile ==

The tempfile library does not check the prefix argument, which can be
exploited to create files outside tmpdir by using directory traversal.

Issue reported 3 years ago:
https://bugs.python.org/issue35278

The same issue was found and treated as a vulnerability in PHP
(CVE-2006-1494) and Ruby (CVE-2018-6914).


== Issues involving URLs ==

There are multiple issues involving URLs:

"ReDoS in urllib.request"
https://bugs.python.org/issue43075

"http.server: Open Redirection if the URL path starts with //"
https://bugs.python.org/issue43223

"urlparse of urllib returns wrong hostname"
https://bugs.python.org/issue36338

"[CVE-2015-2104] Urlparse insufficient validation leads to open redirect"
https://bugs.python.org/issue23505

"urlparse library detecting wrong hostname leads to open redirect vulnerability"
https://bugs.python.org/issue35748

"http.server can be abused to redirect to (almost) arbitrary URL"
https://bugs.python.org/issue32084

"urllib may leak sensitive HTTP headers to a third-party web site"
https://bugs.python.org/issue33661

"Unnecessary URL scheme exists to allow 'URL: reading file in urllib"
https://bugs.python.org/issue37820

"A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! "
https://bugs.python.org/issue32085


Happy hacking!

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PHHQXJYDFWBIKBAHTVATHBL5DO3ER3BE/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Which thing is "Development Mode"

2021-02-26 Thread Victor Stinner
I suggest to assign a color to each development mode to distinguish them.

Victor

On Fri, Feb 26, 2021 at 7:50 PM Coyot Linden  wrote:
>
> As of 3.7, there is now a python feature called Development Mode:
>
> https://docs.python.org/3/library/devmode.html#devmode
>
> Which has a confusingly similar and nearly identical name to a setuptools 
> feature:
>
> https://packaging.python.org/guides/distributing-packages-using-setuptools/#working-in-development-mode
>
> It seems like a bit of bikeshedding could create a name for one of them that 
> disambiguates them.
>
> Best,
>
> coyot
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/42NQQHQEQ4FTRP5CKI5XSKD3GA6L5IIT/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3VNZCIENVRB3OXMP3GDVSUNNJADYWBY2/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Moving threadstate to thread-local storage.

2021-02-26 Thread Victor Stinner
Sure. The plan is to use __thread when possible ;-)

Victor

On Thu, Feb 25, 2021 at 4:58 AM 谢俊逸 via Python-Dev
 wrote:
>
> On MacOS & iOS, __thread variable is 35% faster than using 
> pthread_getspecific.
>
> getSpecific cost: 0.000649
> getTLS cost: 0.000423
>
>
> - (void)test { double t1 = CFAbsoluteTimeGetCurrent(); for (int i = 0; i < 
> 10; i++) { [self getSpecific]; } double t2 = CFAbsoluteTimeGetCurrent(); 
> printf("getSpecific cost: %f\n", t2 - t1); double t3 = 
> CFAbsoluteTimeGetCurrent(); for (int i = 0; i < 10; i++) { [self getTLS]; 
> } double t4 = CFAbsoluteTimeGetCurrent(); printf("getTLS cost: %f\n", t4 - 
> t3); } - (int)getSpecific { int * value = pthread_getspecific(tlsKey); return 
> *value; } - (int)getTLS { return *tlv_v5; }
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/KTHAKNENSBKURE7I2SRVXEPJ6NDNCACI/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/23MHKR44CZTGW7K3NCZADBRWDRA2BHIN/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

2021-02-22 Thread Victor Stinner
On Mon, Feb 22, 2021 at 12:51 PM Ivan Pozdeev via Python-Dev
 wrote:
> IIRC I suggested earlier that buildsbots should be integrated into the PR 
> workflow in order to make it the contributor's rather than a core
> dev's burden to fix any breakages that result from their changes.

Some buildbot worker take 2 to 3 hours per build. Also, it would not
scale. Buildbots are not fast enough to handle the high number of PR
and PR updates.

When there is clear relationship between a buildbot failure and a
merged PR, a comment is added automatically showing the failed test
and explanation how to investigate the issue. It's there for 2 years
thanks to Pablo, and so far, I rarely saw developers paying attention
to these failures. They just ignore it.

I'm not trying to blame anyone. Contributing to Python requires a lot
of free time. I'm only trying to explain in length what does the
"maintenance burden" mean in practice, since some people are
pretending that supporting a platform is free.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KIVBXMBFUF73PKNDOBGRMWJORS2DNXSM/
Code of Conduct: http://python.org/psf/codeofconduct/


  1   2   3   4   5   6   7   8   9   10   >