[Python-Dev] RFC on PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library

[Shantanu Jain]

Hello all,

Link to the PEP: https://www.python.org/dev/peps/pep-0680/
Please redirect discussion to: https://discuss.python.org/t/13040

PEP 680 proposes adding the tomllib module to the standard library for
parsing TOML . The main motivation is to address
bootstrapping problems with pyproject.toml based builds.

Thank you to everyone who has already left comments, suggestions and reacts
on the Discuss thread! We welcome any further suggestions before submitting
to the Steering Council :-)

Thanks,
Shantanu Jain
Taneli Hukkinen
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BVWOZUY4M7B5WNOJCAAMS6VXWAQ6NA6R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Steven D'Aprano]

On Wed, Jan 26, 2022 at 02:40:32PM -0800, Neil Schemenauer wrote:
> On 2022-01-18 23:14, Gregory P. Smith wrote:
> >
> >Our stdlib unittest already enables warnings by default per 
> >https://bugs.python.org/issue10535.
> >
> >Getting the right people to pay attention to them is always the hard part.
> 
> I wonder if we can do a bit better in that regard.  When I install 3rd 
> party packages, I create a usercustomize.py file that uses 
> filterwarnings() to turn off all the warnings I don't care about.  I 
> don't know how but maybe we could make that easier to do.  That way, you 
> don't get buried in warnings coming from code you don't maintain.

Or maybe, as a developer (not an end-user of an app), you could be more 
proactive in reporting those warnings to the third party, and 
encouraging them to fix them. Maybe even submitting a patch?

If we use a library, then we surely care about that library working 
correctly, which means that if the library generates warnings, we 
*should* care about them. They are advanced notice that the library is 
going to break in the future.

Of course I understand that folks are busy maintaining their own 
project, and have neither the time nor the inclination to take over the 
maintenance of every one of their dependencies. But we shouldn't just 
dismiss warnings in those dependencies as "warnings I don't care about" 
and ignore them as Not My Problem.

Like it or not, it is My Problem and we should care about them.

Especially in the case of open source software, the lines of 
responsibility are blurred. Open source libraries are not proprietary 
black boxes with a strict division between the vender that supplies the 
library and the people who use the library. They are fully transparent, 
we can see the warnings and, at least potentially, see how to fix them. 
And we have the legal right to.

This is a hard problem, but it is not solely a technical problem. It is 
partly a social problem, and you cannot fix social problems with 
technology. People are ignoring the warnings, and not just the immediate 
developers of the software, but their downstream users.

The open source mantra about many eyes making bugs shallow doesn't work 
when everyone is intentionally closing their eyes to the warnings of 
pending bugs.



-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BCEKHDT76GGG2E65DCCIX4DC6OL5CIW4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 678: Enriching Exceptions with Notes

[Zac Hatfield Dodds]

Hi all,

I've written PEP 678, proposing the addition of a __note__ attribute
which can be used to enrich displayed exceptions.  This is particularly
useful with ExceptionGroup, or where exception chaining is unsuitable, and
ensures that 'orphan' output is not produced if the annotated exception is
caught.

Link to the PEP: https://www.python.org/dev/peps/pep-0678/

*Please, redirect all discussions to:*
https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374

Thanks for your time, and to those who have already made (very helpful!)
comments!

Best wishes from just-moved-to San Francisco,
Zac Hatfield-Dodds
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IMNZE4HN7VNWS5SROMFOLHQQVYHUYT77/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python no longer leaks memory at exit

[Eric Snow]

On Thu, Jan 27, 2022 at 8:40 AM Victor Stinner  wrote:
> tl; dr Python no longer leaks memory at exit on the "python -c pass" command 
> ;-)

Thanks to all for the effort on this!

Would it be worth adding a test to make sure we don't start leaking
memory again?

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ELLXKXMQAZ3WMLDDNKU7QLR6AGE36JJR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python no longer leaks memory at exit

[Dong-hee Na]

Hey Victor

First of all, I would like to thank Victor for all his efforts to resolve
this issue.
And I want to thank the other contributors to this issue as well!
I am very happy to hear that we can close bpo-1635741.
This has been a long mission requiring patience and
I'm proud to finally be able to put an end to it.

Let's move forward to the next step!

Warm regards,
Dong-hee

2022년 1월 28일 (금) 오전 12:37, Victor Stinner 님이 작성:

> Hi,
>
> tl; dr Python no longer leaks memory at exit on the "python -c pass"
> command ;-)
>
>
> == Bug report ==
>
> In 2007, the bpo-1635741 issue was reported on SourceForge:
> "Interpreter seems to leak references after finalization".
>
> This bug is 15 years old. It saw the bugs migration from SourceForge
> to Roundup (bugs.python.org) in 2007, the code migration from
> Subversion to Mercurial in 2009, and the second code migration from
> Mercurial to Git  in 2016!
>
> Link to the main issue: https://bugs.python.org/issue1635741
>
> Some people reported similar issues seen by WinCRT debug (bpo-6741),
> MSVC Debug C Runtime (bpo-32026), GCC AddressSanitzer (bpo-26888), or
> LeakSanitizer and Valgrind (bpo-21387).
>
> In general, "leaking memory" at Python exit doesn't matter, since the
> operating system releases all memory when a process completes. It
> matters when Python is embedded in an application. For example, Python
> can be used by a plugin which is unloaded and reloaded multiple times.
> It also matters for sub-interpreters.
>
>
> == Tedious task ==
>
> In the last 3 years, many people helped fixing this old issue by
> converting static types to heap types, adding a module state to C
> extension modules, converting extensions to the multi-phase
> initialization API (PEP 489), fix many memory leaks, fix bugs, etc.
>
> When it became possible to cleanly unload an extension module (thanks
> for the multi-phase init), tests on sub-intepreters (which load and
> unload extension modules) showed many *old* reference leaks: all of
> them have been fixed! When a test is run in sub-interpreters, Python
> is able to detect leaks, whereas currently it doesn't check for memory
> leaks at Python exit.
>
>
> == Python 3.10 regressions ==
>
> During the Python 3.10 development, we identified and fixed 3 major
> regressions caused by this work:
>
> * Converting static types to heap types make them mutable: the
> Py_TPFLAGS_IMMUTABLETYPE flag was added; 68 types use it in Python
> 3.11 (bpo-43908).
>
> * It became possible to create uninitialized objects by instanciating
> types (which should not be instanciated directly): the
> Py_TPFLAGS_DISALLOW_INSTANTIATION was added; 41 types use it in Python
> 3.11 (bpo-43916). For example, you cannot create an instance of
> type(sys.flags).
>
> * Heap types must implement the full GC protocol: Py_TPFLAGS_HAVE_GC
> flag, traverse and clear functions. Otherwise, the GC is unable to
> break reference cycles, whereas a type contains (multiple) strong
> references to itself (in the MRO tuple and in methods). All heap types
> have been fixed to fully implement the GC protocol (bpo-40217).
>
>
> == PEPs ==
>
> The work relies on multiple PEPs:
>
> * PEP 489: Multi-phase extension module initialization
> * PEP 573: Module State Access from C Extension Methods
> * PEP 630: Isolating Extension Modules
>
>
> == Persons who helped fixing the issue ==
>
> Incomplete list of people who helped to fix this issue:
>
> * Christian Heimes (modules: symtable, _hashlib, ,_random, grp, pwd,
> _queue, spwd, _struct, gc, _posixshmem, _posixsubprocess, select,
> resource)
> * Dong-hee Na (modules: _statistics, itertools, _heapq, _collections,
> _uuid, math, _stat, syslog, errno, fcntl, mmap, _dbm, _gdbm, _lzma,
> faulthandler, _bisect)
> * Eric Snow (PEP 573)
> * Erlend Egeberg Aasland (modules: _sqlite3, _sre, _multibytecodec)
> * Hai Shi (modules: _json, _codecs, _crypt, _contextvars, _abc, _bz2,
> _locale, audioop, _ctypes_test, _weakref, _locale)
> * Marcel Plch (PEP 573)
> * Martin von Löwis (PEP 3121)
> * Mohamed Koubaa (modules: sha256, multiprocessing, _winapi, _blake2,
> _sha3, _signal, _sha1, _sha512, _md5, zlib, _opcode, _overlapped,
> _curses_panel, termios, _sha256, scproxy, cmath, _lsprof, unicodedata)
> * Nick Coghlan (PEP 489, PEP 573)
> * Paulo Henrique Silva (modules: time, operator, _functools)
> * Petr Viktorin (PEP 489, PEP 573, PEP 630)
> * Stefan Behnel (PEP 489)
> * Victor Stinner (modules _string, mashal,_imp, _warnings, _thread)
>
> The work was scatted into many sub-issues, it was hard for me to track
> all persons who contributed, sorry about that! I only searched for
> "New changeset" in bpo-1635741.
>
> For me, it was a very pleasant collaborative work :-) Contributors
> wrote pull requests, I reviewed and merged them. Slowly, contributors
> started to review each others and shared some recipes for these tasks.
>
>
> == Close the very old bpo-1635741 issue ==
>
> Today, in the main development branch, Python no longer leaks memory
> at 

[Python-Dev] Python no longer leaks memory at exit

[Victor Stinner]

Hi,

tl; dr Python no longer leaks memory at exit on the "python -c pass" command ;-)


== Bug report ==

In 2007, the bpo-1635741 issue was reported on SourceForge:
"Interpreter seems to leak references after finalization".

This bug is 15 years old. It saw the bugs migration from SourceForge
to Roundup (bugs.python.org) in 2007, the code migration from
Subversion to Mercurial in 2009, and the second code migration from
Mercurial to Git  in 2016!

Link to the main issue: https://bugs.python.org/issue1635741

Some people reported similar issues seen by WinCRT debug (bpo-6741),
MSVC Debug C Runtime (bpo-32026), GCC AddressSanitzer (bpo-26888), or
LeakSanitizer and Valgrind (bpo-21387).

In general, "leaking memory" at Python exit doesn't matter, since the
operating system releases all memory when a process completes. It
matters when Python is embedded in an application. For example, Python
can be used by a plugin which is unloaded and reloaded multiple times.
It also matters for sub-interpreters.


== Tedious task ==

In the last 3 years, many people helped fixing this old issue by
converting static types to heap types, adding a module state to C
extension modules, converting extensions to the multi-phase
initialization API (PEP 489), fix many memory leaks, fix bugs, etc.

When it became possible to cleanly unload an extension module (thanks
for the multi-phase init), tests on sub-intepreters (which load and
unload extension modules) showed many *old* reference leaks: all of
them have been fixed! When a test is run in sub-interpreters, Python
is able to detect leaks, whereas currently it doesn't check for memory
leaks at Python exit.


== Python 3.10 regressions ==

During the Python 3.10 development, we identified and fixed 3 major
regressions caused by this work:

* Converting static types to heap types make them mutable: the
Py_TPFLAGS_IMMUTABLETYPE flag was added; 68 types use it in Python
3.11 (bpo-43908).

* It became possible to create uninitialized objects by instanciating
types (which should not be instanciated directly): the
Py_TPFLAGS_DISALLOW_INSTANTIATION was added; 41 types use it in Python
3.11 (bpo-43916). For example, you cannot create an instance of
type(sys.flags).

* Heap types must implement the full GC protocol: Py_TPFLAGS_HAVE_GC
flag, traverse and clear functions. Otherwise, the GC is unable to
break reference cycles, whereas a type contains (multiple) strong
references to itself (in the MRO tuple and in methods). All heap types
have been fixed to fully implement the GC protocol (bpo-40217).


== PEPs ==

The work relies on multiple PEPs:

* PEP 489: Multi-phase extension module initialization
* PEP 573: Module State Access from C Extension Methods
* PEP 630: Isolating Extension Modules


== Persons who helped fixing the issue ==

Incomplete list of people who helped to fix this issue:

* Christian Heimes (modules: symtable, _hashlib, ,_random, grp, pwd,
_queue, spwd, _struct, gc, _posixshmem, _posixsubprocess, select,
resource)
* Dong-hee Na (modules: _statistics, itertools, _heapq, _collections,
_uuid, math, _stat, syslog, errno, fcntl, mmap, _dbm, _gdbm, _lzma,
faulthandler, _bisect)
* Eric Snow (PEP 573)
* Erlend Egeberg Aasland (modules: _sqlite3, _sre, _multibytecodec)
* Hai Shi (modules: _json, _codecs, _crypt, _contextvars, _abc, _bz2,
_locale, audioop, _ctypes_test, _weakref, _locale)
* Marcel Plch (PEP 573)
* Martin von Löwis (PEP 3121)
* Mohamed Koubaa (modules: sha256, multiprocessing, _winapi, _blake2,
_sha3, _signal, _sha1, _sha512, _md5, zlib, _opcode, _overlapped,
_curses_panel, termios, _sha256, scproxy, cmath, _lsprof, unicodedata)
* Nick Coghlan (PEP 489, PEP 573)
* Paulo Henrique Silva (modules: time, operator, _functools)
* Petr Viktorin (PEP 489, PEP 573, PEP 630)
* Stefan Behnel (PEP 489)
* Victor Stinner (modules _string, mashal,_imp, _warnings, _thread)

The work was scatted into many sub-issues, it was hard for me to track
all persons who contributed, sorry about that! I only searched for
"New changeset" in bpo-1635741.

For me, it was a very pleasant collaborative work :-) Contributors
wrote pull requests, I reviewed and merged them. Slowly, contributors
started to review each others and shared some recipes for these tasks.


== Close the very old bpo-1635741 issue ==

Today, in the main development branch, Python no longer leaks memory
at exit for the simplest command: "python3 -c pass"!

Using a Python debug build, you can check the "python -I -X
showrefcount -c pass" command (if you get a negative reference count,
see bpo-46449), or you can use a memory debugger like Valgrind.

While the work is not 100% done, it's a great milestone (at least for
me ;-)! 15 years after bpo-1635741 was reported, finally I can close
it! Sadly, this bug will no see the bugs migration from Roundup
(bugs.python.org) to GitHub ;-)

There are still some static types which should be converted to heap
types and some extensions which should be be ported to the multi-phase

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Christopher Barker]

On Wed, Jan 26, 2022 at 12:50 PM Neil Schemenauer 
wrote:

> maybe we should be more aggressive about showing PendingDeprecationWarning
> if it comes from code that seems to be written by the user, e.g. outside
> site-packages
>
> That would be pretty straightforward.

> or not from a package installed by pip.  The exact logic of that is
> complicated though.
>
> or impossible, I always use pip install -e on teh code I'm developing
(though that could be detected, I presume), and if I have more than one of
my own packages, then they are piip installed.

Ido think we could work harder on documenting how to selectively disable
warnings though.

-CHB




___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HB4ZSI5UWSI3R3ASTIOCAYNU7YPBB57F/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HDAOJ5A6GERC33NOJNINITQFL6U52XDP/
Code of Conduct: http://python.org/psf/codeofconduct/