[Python-Dev] Re: Debugging of native extensions on windows

[Steve Dower]

On 3/14/2023 7:13 AM, Rokas Kupstys wrote:
> Still, i think there can be an improvement in this area, and it would
> likely be quite cheap. The biggest problem is people being unaware what
> is going on. IsDebuggerPresent()/CheckRemoteDebuggerPresent() could be
> used for checking debugger presence and when debugging state of main
> process and child process do not match, launcher could print some link
> to documentation describing what is going on and how situation could be
> solved. I am just not sure about any possible race conditions (no idea
> how fast debuggers attack to child processes).

Detecting when a debugger is attached and printing a debug message on 
exit from the launcher might be a neat idea. Care to submit a bug at 
https://github.com/python/cpython?


The change would likely have to go into PC\launcher.c (for venvs) as 
well as PC\launcher2.c right now (for py.exe).


Native debuggers usually hook process creation, so they should attach 
immediately with no risk of missing anything.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2NH3T2KZTXIIWSPPUTYA3S66XEDG56MK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Debugging of native extensions on windows

[Steve Dower]

Hi Rokas

The typical solution (which I myself use frequently) is to enable your 
debugger to attach to child processes automatically. It can make things 
a bit noisier, but it's generally manageable, especially if you've got 
breakpoints set in your own code.


Another option is to not use the virtual environment, but set PYTHONPATH 
to your environment's Lib\site-packages directory and then run the base 
interpreter directly. Most of the time, this will be identical, but it 
avoids the extra process.


Unfortunately, for a variety of reasons, we can't get away from the 
redirector process as long as virtual environments keep their current 
design. As changing the design would be just as disruptive, we've not 
done anything yet, nor do we have any plans to change anything.


Finally, most discussion about Python occurs at 
https://discuss.python.org/ these days. You'll likely find more help and 
discussion over there.


Cheers,
Steve

On 3/13/2023 3:25 PM, Rokas Kupstys wrote:

Hello!

I am dropping this mail to bring up an issue which cost me three good 
evenings of time. Now that i figured it out i believe it is quite a 
serious usability problem.


Gist of the problem: i have some C++ code wrapped with SWIG, so a native 
extension. As with all software - there was a bug. However, no matter 
what i did - i could not debug it in a native debugger. I ran 
".venv/Scripts/python.exe script.py" in a native debugger and 
breakpoints would not be hit, application would crash eventually. This 
was especially confusing, because exact same setup worked just fine on 
linux. I eventually stumbled on to process list showing 
".venv/Scripts/python.exe" having spawned a subprocess... Which led me 
to "PC/launcher.c" which is what ".venv/Scripts/python.exe" really is. I 
cant find much information about this behavior in documentation even 
after the fact. All in all, this was quite confusing. So now every time 
i want to debug a native extension i have to start a program and then 
attach a debugger to it, instead of just hitting "Debug" button in IDE. 
It gets worse if crash happens immediately, which means i have to resort 
to things like adding a message box somewhere to block execution and 
give me enough time to attach the debugger. It works in the end, but 
user experience is really not great. But whats worse - this is such a 
non-obvious behavior that many more people may trip on it and waste 
their time. Documenting this behavior would be of little help too, as 
there is no clear path from the issue to the documentation on the matter...


So there it is. I am sure it is the way it is for a good reason. 
However, this is a very error-prone process which is likely to waste 
people's time. So maybe this behavior could be reconsidered? Or maybe 
there is a solution already, which escaped me?




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/424LIHYHRQWM5VLQF2PAMLKGCNCKSJDF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C- Python on Windows

[Steve Dower]

On 15Jan2023 0922, Guenther Sohler wrote:

Now when i want to get my project compiled in windows, whats the easiest
development chain ?
Is there something like a python.dll which i can link to my project and
having an embedded python interpreter ?

Maybe the question is too simple, but i could not yet find the right place
to read docs.
Thank you for your hints!


There's an overview of our embeddable package and some discussion of how 
to use it at https://docs.python.org/3/using/windows.html#windows-embeddable


You'll likely also need a full install to get all the developer kit 
pieces (such as the headers and libs files), but the embeddable package 
is one you can include straight into your project and redistribute. The 
binaries are the same, it's just laid out differently.


And yes, discuss.python.org is the best place to ask these days.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DHEON2LAXFWRH5NUYAOJSYVTDRH4EL4N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC: expose ssl internals for use by ctypes/cffi

[Steve Dower]

On 11/30/2022 4:52 PM, chris...@weinigel.se wrote:

Does this seem like a good idea?  As I said, I feel that it is a bit ugly, but 
it does mean that if someone wants to use some SSL_really_obscure_function in 
libcrypto or libssl they can do that without having to rebuild all of CPython 
themselves.


Broadly, no, I don't think it's a good idea. We don't like encouraging 
users to do things that make it hard to support them in the future.


Nonetheless, it's one that I've had to do, and so realistically I think 
it's okay to *enable* the hack without endorsing it. This is one of the 
reasons I switched the Windows builds to dynamically linked OpenSSL 
builds (they used to be statically linked, which meant there was no way 
to get at the unused exports). So now you can use `import _ssl; 
ctypes.CDLL("libssl-1_1")` to get at other exports from the module if 
you need them, and there's a similar trick to get the raw handle that I 
don't recall off the top of my head.


But the only reason I'd ever want to document this is to tell people not 
to rely on it. If you control your environment well enough that you can 
guarantee it'll work for you, that's great. Nobody else should ever 
think they're doing the "right thing".


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AJ3PQTOMD4GL27LH5XZR7XUDWU3XKUUE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: problem with Distributed File System Replication and Namespacing and different versions of Python 3

[Steve Dower]

On 10/20/2022 1:07 PM, rainonthescarecrowhumanwhe...@gmail.com wrote:

What happens is, when injecting into the sys.path the domain names it doesnt import but 
when injecting into the sys.path the "real" file server path it works, 
generally speaking. We have been facing this issue in such different python 3 versions 
and i was wondering what makes this work in Python 3.10.2.


We added some path fixes into importlib in 3.10 that will result in 
better path resolution during import on Windows, particularly when 
resolving partial paths. You'll find them in 
Lib/importlib/_bootstrap_external.py and Modules/posixmodule.c (they're 
not real obvious - you'll be best to diff those files).


You're unlikely to easily backport them to earlier versions, so your 
current workaround is probably best. If the problems are due to 
*writing* bytecode files, you might also be able to use 
PYTHONPYCACHEPREFIX to reference a local directory and avoid issues that 
way.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BYPWOPJV3SCYGX4ZMZWEY44WD5VP25QI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 681 – Data Class Transforms

[Steve Dower]

+1. Glad it's not just me

On 6/6/2022 2:36 PM, Victor Stinner wrote:

First, I understood that "Arbitrary Literal String Type" was adding a
new built-in types for "literal strings" :-) Nope. It's just about
type annotations ;-)


I was also excited about the new built-in type :-)

Pablo is already separating the PEPs in new release announcements (or 
has at least agreed to, not sure any releases have happened since I asked).


But with the amount of work going on in separate venues these days, 
separating "CPython implementation" from "Language specification" from 
"Typing specification" would be helpful. (Packaging is already separate, 
but doesn't appear in release announcements anyway.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3VJEOP2VYQ66G4PS665QV52P5TGRYP5O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Steve Dower]

I prefer separate header files, provided people outside of core always 
have one (presumably "Python.h") that should be included first and 
includes enough info to check which headers will be available (i.e. the 
version defs).


Modifying preprocessor definitions for different Python versions, or 
having to set them before knowing what version is being used, seems more 
complicated.


Cheers,
Steve

On 5/30/2022 8:26 PM, Brett Cannon wrote:

We discussed having leading underscores for this API tier, and it was decided 
that a leading underscore was preferred.

This did start a discussion, though, about whether we should control API 
access/opt-in via `#include` by having `.h` files that convey what API the user 
is opting into, or use `#define` to control what gets exposed via `Python.h`. 
The general feeling was that the header file idea is ideal, but it is a little 
extra work to transition to if you want to be compatible with older versions of 
Python that wouldn't have the header files (Victor's compatibility project 
could help here). The question for the team is whether separate header files 
makes sense to others, or would people prefer using `#define` and `Python.h` to 
control API access/opt-in?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NMIULT7IJA77KYNFQGNQYV5LOVLV3FSV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Issue: 92359 - Python 3.10 IDLE 64-bit doesn't open any files names code (code.py, code.pyw) - found a partial fix but looking for input

[Steve Dower]

On 5/14/2022 8:37 PM, Terry Reedy wrote:

On 5/14/2022 12:40 AM, Guido van Rossum wrote:
Probably "Edit with IDLE" should be changed. I have no idea where that 
is defined.


I presume somewhere in PCBuild.  Steve Dower knows and is in charge of 
the Windows installer.


FTR, the behaviour for the traditional installer lives in 
Tools/msi/tcltk/tcltk_reg.wxs, and the behaviour for the Store install 
is in PC/python_uwp.cpp.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DUTVAITI3PKENGS5HQVQB2JDXQNC6WRB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Steve Dower]

On 5/10/2022 9:38 PM, Christopher Barker wrote:
So if you can't entirely drop distutils in the stdlib, I don't think we 
can count on the rest of the community being able to drop it as well.


The only reason distutils is used to build Python is for 
self-bootstrapping. setuptools would have to avoid any module that may 
potentially be built as an extension module if it were to be used at 
that stage of build, which is unfair (and this use is also likely to 
become Makefile for Linux and macOS, just as it uses static build 
scripts on Windows already).


The other uses are tests or tools. There's nothing wrong with 
setuptools's functionality, merely the fact that it isn't part of the 
CPython repository and we don't like relying on third-party repositories 
for a CPython build/test.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y2LVG23ZMH3JOBBZFQ2D2P554DZWRI36/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Steve Dower]

If we're not shipping it, we should just move it out of the way.

Otherwise, there are a few places in the various Windows packages that 
will need it to be explicitly excluded. We should already have similar 
exclusions for the test suite and a few other modules, but they're a bit 
scattered around PC, PCbuild and Tools/msi. Easier to just remove it 
from Lib.


Cheers,
Steve

On 5/10/2022 9:24 PM, Victor Stinner wrote:

On Tue, May 10, 2022 at 6:16 PM Steve Dower  wrote:

I agree. The internal tools that use it are all in our Tools directory,
so we can move distutils there and explicitly add an import path to
locate it. No need to keep it in the stdlib (Lib/) at all.

Migrating to Makefile builds is probably better long-term, but not as
important as moving distutils out from the stdlib so that setuptools can
rely on their copy being the "main" one.


With my current PR, Lib/_distutils/ is not installed by "make
install". Moving it to Tools/ would also work.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JG2BVL54C3RYXFFIX2TL7C53YPIWQHBQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12

[Steve Dower]

On 5/10/2022 4:33 PM, Christian Heimes wrote:

On 10/05/2022 13.18, Victor Stinner wrote:

test_peg_generator and test_cppext build C extensions with distutils.
Until these tests are modified to use something else, we still need to
keep distutils. So I propose to rename it to _distutils to remove it
from the stdlib. Maybe these tests can use ensurepip to install
setuptools which provides distutils.

There is also the c-analyzer tool which uses distutils to run a C 
preprocessor.


We can easily take care of test_cppext and add the build step to 
Makefile. How does test_peg_generator depend on distutils? I don't see 
an import of distutils in the directory.


I would prefer to fix and remove all imports of distutils before we 
resort to renaming the package. Please give us time until alpha 1. There 
is no need to rush it *now*.


I agree. The internal tools that use it are all in our Tools directory, 
so we can move distutils there and explicitly add an import path to 
locate it. No need to keep it in the stdlib (Lib/) at all.


Migrating to Makefile builds is probably better long-term, but not as 
important as moving distutils out from the stdlib so that setuptools can 
rely on their copy being the "main" one.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LGF4BJMN3H7L6QFTZTDBMOA2GPZQFHC6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Test Python 3.11 beta1 with PYTHONSAFEPATH=1

[Steve Dower]

On 5/9/2022 5:24 PM, Victor Stinner wrote:

If PYTHONSAFEPATH=1 only breaks a minority of use cases, maybe we can
consider to make it the default, as Perl did in Perl 5.26 (2017) to
increase its security:
https://perldoc.perl.org/perl5260delta#Removal-of-the-current-directory-(%22.%22)-from-@INC

Perl has an environment variable to get the old (Perl 5.24) behavior:
PERL_USE_UNSAFE_INC=1.

If enough people consider that it would be a good idea to change the
default, I can maybe write a full PEP (I already have some notes).
Even if the PEP is rejected, it might be a good thing to write down
everything about this topic since it's a common issue hit by users
learning Python and a common question of people auditing the Python
security. I was asked a few months ago about changing the default to
increase Python security.


It's possibly worth writing it down, but I'm pretty sure it would impact 
more people than it's worth. I quite often see people who are relying on 
both empty sys.path[0] and implicit namespace packages for all of their 
imports, often without realising it.


If we are able to add an warning on import via an empty sys.path entry, 
that might be useful enough, but it could also serve as a deprecation 
warning without necessarily putting a timeline on it (and also as 
advertising for the new option).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FBDN23BFF24OZPZAXAFXQE3KMFFXHTTC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Using the Python C API in C++

[Steve Dower]

On 5/2/2022 2:21 PM, Victor Stinner wrote:
Maybe it 
should be made optional, but so far I failed to test if distutils has an 
available C++ compiler.


Considering the current status of distutils, you should probably put 
that check in configure or the Makefile and build the extension from 
there. Then the test suite can check whether it was built or not.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46C4ZJZZ5KSPHS4WRICHEOEQZWHXHWWT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Add -P command line option to not add sys.path[0]

[Steve Dower]

On 4/26/2022 10:46 AM, Victor Stinner wrote:

I propose adding a -P option to Python command line interface to "not
add sys.path[0]":
https://github.com/python/cpython/pull/31542

See the documentation in the PR for the exact behavior of this option.
I prefer to add an environment variable, only pass the option
explicitly on the command line.


Another viable option might be to add an option to imply "import site", 
which would work together with -I to:

* ignore environment variables (-E/-I)
* omit implicit CWD imports (-I)
* still process .pth files (-?)
* still include site-packages and user site-packages in sys.path (-?)

It seems likely that the proposed -P would almost always be used with 
-E, since if you can't control CWD then you presumably can't control 
environment variables either.


The existing ._pth functionality starts by implying -I, and allows 
"import site" in the file to explicitly include site. A command-line 
option matching this behaviour would be consistent. There's also already 
configuration in our structures for import site, so there'd be no need 
to add new fields to public APIs for the option.


The biggest issue I see is that the obvious command line options for 
"import site" are already used to imply "do not import site". But then, 
-P isn't obvious either. Maybe an -X option would suffice?


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7XKFGCQJCWXU37SFZQA6RVAPOGOQLWDG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Declarative imports

[Steve Dower]

On 4/8/2022 12:51 PM, Daniel Pope wrote:

On Fri, 8 Apr 2022 at 12:23, Steve Dower  wrote:


I've read the rest of the thread so far, and agree strongly that we
can't do this at the language/runtime level.


You mean the hoisting, right?

I don't see any reason why an import expression without hoisting would
be impractical. But I'd like to hear your thoughts if you think it is.


Sure, __import__() or better yet, importlib.util.import_module() are 
perfectly good ways to do an import in an expression. Both have exactly 
the right amount of code-smell as well, and only a few surprising ways 
to interfere with the behaviour of apparently completely unrelated code ;)


As others pointed out, implicitly defining variables that outlive the 
evaluation of an expression is incredibly messy. It was bad enough 
getting assignment expressions defined well enough to (a) not be a 
terrible footgun and (b) be possible for the average developer to 
predict what will happen when they're used (arguably this isn't true 
either, but the "average" developer probably isn't doing things that 
will hit the edge cases).


We don't want to have to figure something like that out again. Certainly 
not for the sake of typing - the SC has already ruled that language 
development won't be driven by the convenience of static type users.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SRAIPD7DQH36DNBZVEAA6I56IZHLGA66/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Declarative imports

[Steve Dower]

I've read the rest of the thread so far, and agree strongly that we 
can't do this at the language/runtime level.


However ...

On 4/8/2022 9:24 AM, Malthe wrote:

This is an idea which has been brought up before, sometimes introduced
as "heresy". But an interesting twist has surfaced now which is
typing.

[SNIP]

We want typing to pick up these imports. And this twist has a second
leg which is that we often need to import symbols simply in order to
type some argument type or return type. This leads to a great many
more imports to type.


Type checkers can do whatever they like. The syntax has to remain a 
valid Python expression, but the semantics of it can include implicit 
imports if that's what you want it to do.


I'd certainly make use of it on those occasions I actually put type 
annotations in code (rather than .pyi files).


But it doesn't need the runtime to do anything differently if the module 
isn't really going to be used at that point.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/57NDZA7B3Z4PLGS5JON5YXOP44KI4LVU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 4/5/2022 11:52 PM, Victor Stinner wrote:

On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:

I don't see any additional discussion on the bug, and the prevailing
opinion from actual users of this API is that it probably shouldn't
change,


 From what I understood my change basically only impacts "pydevd"
users. Fabio who works on that project said that he was fine with
that change ("I'm ok with changes"):
https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55ANVKHGG74D62HDBOFLC4EXWJ26/


"I'm okay with" isn't really a sign of support, particularly in this 
kind of relationship where *we* have all the power. Fabio doesn't have a 
choice but to be okay with whatever we decide, and he obviously accepts 
that.


I'm still going to strongly advocate for not going out of our way to 
break users like Fabio. Just because we have a reputation for doing it 
doesn't mean we should keep doing it.



For me, all impacted users were made aware and joined the discussion.
It's very rare to be able to reach *all* users impacted by an
incompatible C API change!


There's no way we reached all the users :) But we did reach more than usual.


On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:

and certainly shouldn't become internal without additional
guarantees about stability.


The discussed API is not stable, Brett repeated that. The internal API
is not stable on purpose. I missed the discussion proposing to design
a stable API for PEP 523.


I can't tell whether you missed the discussion or if you're deliberately 
misrepresenting it as a persuasive technique.


The API may not be "stable" according to our typical rules around public 
C APIs, but it has explicit rules about its own stability (well, had, 
though once the revert goes in it will have them again).


Nobody suggested making it *more* stable than it already was. We just 
want to treat its existing stability rules with respect and either 
transition out of them slowly, or leave them as they are.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/226B54YXCYX4JNYTVGNSOVN6ONJZX7PC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Steve Dower]

On 02Apr2022 0925, Paul Moore wrote:

On Sat, 2 Apr 2022 at 02:30, Christopher Barker  wrote:


On Fri, Apr 1, 2022 at 4:06 AM Steve Dower  wrote:


The main difference is that 'immutables' offers you a stable/versioned
interface to use it, while the one that's in CPython is an internal
implementation detail. If one day we find a better design, we can just
switch to it, while 'immutables' probably can't. If we've exposed as a
public interface in the core runtime, it's much more complicated.



I don't understand the issue here:

If we expose a "frozendict" as built in python object then only the API of that 
object needs to remain stable, not the implementation.

And it seems that's an API that is already clearly defined.

+ 1 from me -- just the other day I was wishing it was there.


There would presumably need to be be a C API as well, and that would
probably expose more of the implementation unless handled carefully.
Without seeing an actual implementation, it's hard to know for sure.


Yes, and the request in this thread was "expose the HAMT", not "create a 
new data type that uses it under the covers".


My opposition to the new data type was that it *wasn't* frozendict, 
primarily because we decided that dicts had to preserve insertion order. 
So it was proposed as "frozenmap", but was more like a dict than map(). 
It's also slightly slower for lookups (as well as technically not O(1), 
though close enough for moderate N), and the primary benefit is reduced 
memory usage when you mutate it (but it's frozen...? ;) ).


All of that kind of adds up to make it a distraction that's likely worse 
than a dict subclass with __setitem__ overridden (if that's the 
behaviour you want). It's a fantastic data structure for when you need 
it, but it's one that deserves to be researched and understood before 
simply dropping it into your code. An optional package is a perfectly 
good place for it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VFSXMPGG7Q3YOAZKKNHLLW5TPHJPU3OS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Steve Dower]

On 4/1/2022 11:48 AM, zhang kai wrote:
Thanks Victor and Pablo. I will check the discussion of PEP 603. It's a 
little weird to use the immutables library when it's code in already in 
CPython but I'm glad it's an option.


The main difference is that 'immutables' offers you a stable/versioned 
interface to use it, while the one that's in CPython is an internal 
implementation detail. If one day we find a better design, we can just 
switch to it, while 'immutables' probably can't. If we've exposed as a 
public interface in the core runtime, it's much more complicated.


(For what it's worth, the major thing that held up contextvars in the 
first place was justifying why it needed a new data structure that 
wasn't already in the core runtime. So we didn't adopt it lightly, and 
making sure we kept the freedom to change it was an important compromise.)


There are plenty of other things in this same category, and because we 
want to keep things as stable as possible while also improving 
performance and reliability, we have to keep pretty tight limits on what 
we promise will remain stable. Most of our discussions are about finding 
this balance ;)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KDJLDBZ5FIMUCVBKGEWCA2T4NGTZB4CW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 4/1/2022 10:01 AM, Victor Stinner wrote:

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850


So what was the point of this discussion then?

I don't see any additional discussion on the bug, and the prevailing 
opinion from actual users of this API is that it probably shouldn't 
change, and certainly shouldn't become internal without additional 
guarantees about stability.


Did we all just waste a whole lot of electrons discussing a foregone 
conclusion?


(My apologies to the people I invited into this thread. I genuinely 
thought it would help to have outside perspective on this potential change.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZGWG3KKGGWI7HLYSJUDQVTXFUNGZKD3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/30/2022 4:42 PM, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, 
"Unstable APIs". These are public but are not guaranteed to be backwards 
compatible in feature releases (though I feel they should remain so in 
bugfix releases).


Agreed. This is definitely a new category, and it seems the only thing 
we're debating now is whether or not to add/omit the underscore.


I'm not sure whether those should have a leading underscore or not. 
Perhaps (like some other languages do and like maybe we've used in a few 
places) the name could just include the word "Unstable"?


I don't think we have "Unstable" anywhere, though we do have "Unsafe" 
(which is more about threading than stability, so not right for this). 
But I'd be okay with that as a compromise.


I'd prefer to not have public-unstable APIs hidden behind the same 
preprocessor directive as internal APIs. That's a big switch to throw 
that may also activate other settings - for example, on Windows we will 
set the minimum Windows version in our headers if you enable internal 
APIs, and disable automatic linking of the Python DLL. Easy enough 
things to work around, but they probably need to be explicitly 
documented as well if we're going to document public APIs as requiring 
Py_BUILD_CORE (and I don't want to have to document that kind of stuff).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZEPJSIJNXQKXTOE2MRNS6GJRP52WLDEF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Steve Dower]

On 30Mar2022 1132, Baptiste Carvello wrote:

Le 28/03/2022 à 18:44, Steve Dower a écrit :

I think to most people "batteries included" doesn't necessarily mean
"standard library," with all that implies. It just means "it came with
the first thing that I installed" (or alternatively, "at no additional
charge").


A point I have not seen made, is that some uses really need *standard*
batteries.


I've certainly not forgotten it, it's just every time I try to make the 
point it doesn't seem to help. So until I find a clear way to argue 
that, yes, a standard "datetime" value makes sense, but no, a standard 
"HTTP headers dictionary" is unnecessary, I'm avoiding bringing it up 
again ;)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OCY7VGDIXJAQ55XAHOCYFZOBF6O4P4I4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Steve Dower]

On 30Mar2022 1124, Barney Gale wrote:
I'd like to become a maintainer for the pathlib module, if possible. I 
know the code and tests inside-out, and I've been wrestling the 
internals for past few Python releases. I check the bugs/PRs at least 
every week and help wherever I can.


Antoine is still active, so if he's fine with that, I'm also supportive. 
You're certainly familiar enough with that module.


I can also look after urllib if it would help. I have much less 
experience with that module vs pathlib, so I'll need to spend some time 
familiarizing myself with the code and the outstanding bug reports. But 
I do have plenty of experience with networking, HTTP, URLs, etc :-). 
I'll spend some time reading the bug tracker to start.


Please! I skim the open issues occasionally, but definitely feel out of 
my depth to decide on issues (e.g. should 
urlparse("http://[::1]spam:80;) raise? - I have literally no idea :) ), 
but feel free to nosy me on issues you think you've figured out and I 
can help confirm your logic and do merges. Or if another core dev has 
been helping you, keep nosying them.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DRHN22EUHGYCEUHJ3GSMEYMM2BQCS3QR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/28/2022 10:44 PM, Jason Ansel via Python-Dev wrote:

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
proposal may break the next major release of PyTorch.

The related project is TorchDynamo, which can be found here:
https://github.com/facebookresearch/torchdynamo

We will likely move this into the core of PyTorch closer to release.

If the changed happens, would PyTorch still be able to use the eval frame API?  
Or would it prevent from being used entirely?


You'd be able to use it, but if we don't nail down the compatibility 
guarantees then you might find PyTorch doesn't work properly against 
3.11.(n+1) while 3.11.n was fine.


Right now, the API is allowed/expected to change between 3.x releases 
(which normally we don't allow without a deprecation period) but it 
still has to remain compatible within a single 3.x release. Making it 
fully internal *without adding a stability guarantee* means it could 
change more frequently, which you wouldn't be able to handle as an 
installable package.


It's *unlikely* that it'll change that often, because there are still 
other public interfaces that cannot. But, the plan behind this is to 
make more stuff internal so that it can be modified more freely, so we 
may see that rate of change increase.


In this world, your best bet is for TorchDynamo to become a full CPython 
fork. And when that's your best bet, it should set off alarms everywhere 
;) But that is what relying on internal APIs implies, and is certainly 
what people would have to do if we were to remove the existing PEP's 
interface. (It's what Pyjion was doing before the PEP was written, and 
it's why Pyjion doesn't have to be a full fork these days.)


So you probably want to state explicit support for either keeping the 
APIs public and slightly-flexible, or making them internal but stable. 
(Public and stable won't work at all for us, and normal internal won't 
work at all for you.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MU56YBMPWSVMK3HAUJOBRW4KEN2ZLVCG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Steve Dower]

On 3/28/2022 7:26 PM, Paul Moore wrote:

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to. Is
there actually any new development here, or is it just a repeat of the
same positions people have expressed in the past, based on the same
underlying realities? (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).


Yep, same. Nothing much has changed, but if those of us who know that 
nothing has changed fail to reiterate those points, everyone new to the 
discussion will assume that everyone agrees with them.


Just deleted a long rant to the discussion in general (not directed at 
Paul at all ;) ) that would have achieved nothing, but I'll summarise to 
this: email doesn't fix bugs; maintainers fix bugs. Please let us know 
*publicly* if you want to become the maintainer for a stdlib module and 
then we can support them, but if nobody is willing/able/ready to care 
for them it's irresponsible for us to keep shipping them to users.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I37FCHGNGCKACI4C4S3OA6MO4MZLRABY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Steve Dower]

I think to most people "batteries included" doesn't necessarily mean 
"standard library," with all that implies. It just means "it came with 
the first thing that I installed" (or alternatively, "at no additional 
charge").


Given there are *plenty* of existing distros out there that install more 
than just the standard library for the benefit of their users, and yet 
the people asking for a fatter stdlib don't seem to want them, perhaps 
we just need to officially endorse one?


I've got no issue preinstalling a curated sumo.txt into the Windows 
installers, provided we're clear about support (i.e. none from us) and 
someone besides me is tracking everything in that curated list. That 
probably requires new volunteers who are somehow already trustworthy, 
but that should be easier than people to actually write and maintain new 
functionality on stdlib timelines.


We would still have to have the "light" distribution, as many of our 
users wouldn't be able to use a "this is full of unsupported stuff" 
package, but at least most users could be pointed to the bigger install 
by default and not even care that the batteries just came in the same 
package and aren't actually an integral part of the core product.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TC5EEDFNSZJBR22GQDJ32UWVTQNILU2P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/22/2022 11:28 PM, Victor Stinner wrote:

On Tue, Mar 22, 2022 at 7:33 PM Steve Dower  wrote:

After a normal deprecation period, yes?


There is no backward compatibility warranty and no deprecation process
for private APIs.


And yet you're asking the question, which means you know these are 
special ;)


The PEP says: "The API is purposefully listed as private to communicate 
the fact that there are no semantic guarantees of the API between Python 
releases."


Absence/presence isn't a semantic guarantee, it's an availability 
guarantee. Code using them should be able to rely on their presence, and 
ideally their prototype (though it seems we've messed that up in the 
past), but shouldn't expect code that worked against 3.8 to also work 
against 3.9 or 3.10.


Perhaps in hindsight, we could have not used the underscore and just 
explicitly described them as being behaviorally unstable between major 
versions. I guess that would have raised exactly the same question though.


The point is, it's a documented API that we've told people they can use. 
We can't simply revoke that without telling people that it's going to 
happen, even if we covered ourselves for there being version changes 
that affect how they need to be used.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF75IDAHS66BKJZDAAQDGOW2IGU2KME5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/22/2022 6:07 PM, Victor Stinner wrote:

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):


After a normal deprecation period, yes?


While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.


Pyjion (https://github.com/tonybaloney/Pyjion) uses it, and ptvsd never 
did. I also understand that pydevd - and implicitly debugpy - disabled 
its use of the API by default because of reliability, but that was a few 
years back so maybe they fixed it. So it's likely only being used by the 
project that requested it, though I don't think that's enough to justify 
skipping normal deprecation.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X6EC7KW4B5CB4R5CS4WSQYWSXIYFV4J3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Restrict the type of __slots__

[Steve Dower]

On 3/18/2022 2:38 PM, Eric V. Smith wrote:
There a practical need for these changes. See 
https://bugs.python.org/issue46382 for a case where dataclasses needs 
__slots__ to be iterated over after the class has been created. And it 
would be good if __slots__ accurately reflected the slots that were 
actually created.


It seems totally sufficient for dataclasses to require that base class 
__slots__ are a string or iterables of strings. Which is apparently what 
would happen today, since anything else would cause incorrect behaviour.


A single special case doesn't have to become a language-wide backwards 
compatibility break, even if it perhaps would've been better to have had 
the restriction from the start.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R5PBNN36BMUM72QPRJG4DAHSD6UKUY3O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python 3.9.11

[Steve Dower]

We're still working on it. Hopefully it will be today, but we have had 
other issues arise that have delayed things by a few days (including new 
critical OpenSSL fixes that were just released yesterday).


Cheers,
Steve

On 15Mar2022 1743, Prasad, PCRaghavendra wrote:

Hi Team,

Can someone please let us know the release date of Python 3.9.11 ( with 
libexpat 2.4.8 security issues fixed )


In the python.org releases it was mentioned as 14-march-2022, but still, 
I couldn’t see the bin/source code.


Can someone help with this

Thanks,

Raghavendra

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZPC2ZLVUSNMKUYENL22OGEGXZCTMEPGN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

[Steve Dower]

On 23Feb2022 2240, h.vetin...@gmx.com wrote:

On top of that, I think the formulation you chose does not map correctly to
actual compiler capabilities. C99 (minus C11 optionals) only arrived in VS2019,
there were still gaps in VS2017.


Additionally, Visual Studio version doesn't directly map to the compiler 
version. For example, VS2022 contains multiple older versions of the 
compiler.


Currently we use the v142 toolset for releases (selected by default in 
desktop development workloads in VS 2019), and can't migrate to v143 
(selected by default in VS 2022) until some compiler bugs are fixed. If 
we're going to specify anything about MSVC, I'd prefer it was the v142 
label rather than the Visual Studio version (though we can include the 
latter as a convenience for the reader, and hopefully clearly enough 
that people stop filing bug reports about it...)


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KJ5XCX7IOV6GR2MWPXV6TNJ3P5G5NLSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Steve Dower]

On 2/8/2022 9:54 PM, Steven D'Aprano wrote:

Regarding your second point, about domain-specific tasks, I don't
understand. If your domain-specific task doesn't need to use floats,
just don't use floats. There's surely no need to invent a whole new
language for some task because your app only needs to do string
processing, say. Have I misunderstood what you are trying to say?


Only that some people get very concerned about following "standards", 
and we can help them by making our standard easier to follow ("more 
permissive," as you suggest, is fine wording by me).


Some people see things that are written down as rules to be followed, 
while others see them as guidelines. When we write something down wrong 
or loosely, we can cause a surprising amount of angst within the first 
group (for example, I fairly frequently get worried questions about the 
VS version for building CPython due to how the wiki and devguide mention 
it).


I'm just trying to make sure we don't add more of these, by ensuring 
that we properly scope things. In this case *CPython requires IEEE 754 
floats*, but the Python language does not.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KANDB5WROSPKE7TANO4A75IOD3IDBQZI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

[Steve Dower]

On 2/8/2022 9:52 AM, Petr Viktorin wrote:

On 07. 02. 22 18:09, Victor Stinner wrote:

In 2022, C99 is now well supported by C compilers supported by Python:
GCC, clang, MSVC.



Does MSVC support all of C99? I haven't found any documentation claiming 
that... but I'm also not familiar with MSVC.


Do we need to support a subset like "C99 except the features that were 
removed in C11"?


I couldn't find any information either (from public sources, at least, 
haven't asked the team directly yet).


All the C99 library is supposedly supported, but there are (big?) gaps 
in the compiler support. Possibly these are features that were removed 
in C11? I don't know what is on that list. It seems C11 and C17 are 
supported [1], though I know for sure we have contributors who aren't 
using a recent enough compiler version to have it :)


Personally, I see no reason to "require" C99 as a whole. We have a style 
guide already, and can list the additional compiler features that we 
allow along with guidance on updating existing code and ensuring 
compatibility.


I don't see much risk requiring the C99 standard library, though. It's 
the compiler features that seem to have less coverage.


Cheers,
Steve

[1]: 
https://devblogs.microsoft.com/cppblog/c11-and-c17-standard-support-arriving-in-msvc/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UZACKMZGN2PUGNQ74FQX2V4UTQAA6DWB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Steve Dower]

On 2/8/2022 1:35 AM, Gregory P. Smith wrote:

While we're at it are  64 bit floats required for either cPython or
Python the language?


CPython: yes.  we use a double.
Python the language: no.  (float is single precision on many micropython 
platforms as it saves precious ram and performance, plus microcontroller 
fpu hardware like an M4 is usually single precision 32bit)


Agreed. CPython should be specific, Python should be as vague as 
possible. Otherwise, we would prevent _by specification_ using Python as 
a scripting language for things where floats may not even be relevant.


It's not even about hardware capabilities (unless you believe Python is 
only for running in a terminal, in which case we'll just have to 
disagree). It could be about an app using short Python snippets as a 
customisation language for some domain-specific task rather than 
inventing a brand new language for it. The more 
"it's-only-Python-if-it-has-X" restrictions we have, the less appealing 
we become.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4GVXJZJ6EKNIEQVFIKRGU7PNN3OTNUP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Steve Dower]

On 2/6/2022 4:44 PM, Christian Heimes wrote:
If I had the power and time, then I would replace urllib with a simpler, 
reduced HTTP client that uses platform's HTTP library under the hood 
(WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten, 
maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or 
aiohttp are much better suited than urllib.


I'm +1 on this, though I think it would have to be in place before the 
"two releases until removal" kicked in for urllib.request.


The stdlib can't get by without at least the basic functionality of curl 
built in natively. But we can do this on most platforms without 
vendoring OpenSSL, which is a HUGE win. Then our default behaviour could 
correctly use proxies (including auto-config), CA certificate bundles, 
integrated authentication, and other OS features that are currently 
ignored by our core.


Chances are we could keep simple urlopen() calls in place, and use the 
deprecation as a "potential change of behaviour" without necessarily 
having to break the API. I'm yet to come across a case where making a 
trivial urlopen() call _better_ would break things (the cases I've seen 
that would break are things like "using an OpenSSL environment variable 
to configure something that I wish had been automatic").


The nature of network/internet access is that we have to break things 
periodically anyway, because all the code that was written over the last 
30+ years is eventually going to be found to be exploitable. I'd be 
quite happy to say "Python gives you what your OS gives you; update the 
OS for fixes".


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2P3RL7PAOZZFZ7PRGO6FJRMKR6MM2VXH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

[Steve Dower]

On 2/5/2022 4:09 PM, Guido van Rossum wrote:
On Sat, Feb 5, 2022 at 5:18 AM Steve Dower <mailto:steve.do...@python.org>> wrote:


The gap this has with what I'd like to see is that it will only work
for
compile-time strings. If instead it could work for an arbitrary uint8_t
pointer, rather than an embedded array, that allows even runtime
strings
to be very cheaply passed into the interpreter (and yes, the caller has
to manage the lifetime, but that's pretty easy).


What's the use case for that, that isn't covered by PyUnicode_FromString()?


No dynamic memory allocation (and hence, no deallocation) for any case 
where the original string is already managed, which is really quite 
common. Access to the memory manager is what requires thread 
affinity/the GIL for primitive object construction, and string copies - 
especially with transcoding - are the expensive part.


For cases where strings are just passed around but never manipulated 
(e.g. a lot of filesystem operations, or runtime/interpreter 
configuration), the string may never have to be decoded at all. It's 
almost as good as a tagged pointer, but without breaking our existing 
object model (assuming all the PyUnicode_* functions learn how to handle 
them, which is necessary).


But it's purely a transparent optimisation for most users, with an added 
opportunity for those using native APIs and probably decent complexity 
for us as maintainers. There are a lot of edge cases to handle that I'm 
sure people will use to shoot down the idea, which is why rather than 
debate details here I'd rather build it and define its boundaries of 
usefulness, though for now there's plenty of stuff I'd rather do than 
both of these, so it remains an idea for now :)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6SBJZ7X6XNSU76K53IC4HPNWNJDT3IX2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

[Steve Dower]

On 04Feb2022 2303, Guido van Rossum wrote:
You *can* allocate unicode objects statically. We do it in deepfreeze, 
and Eric's PR under discussion here 
(https://github.com/python/cpython/pull/30928 
) does it. I wonder if a 
better solution than that PR wouldn't be to somehow change the 
implementation of _Py_IDENTIFIER() to do that, and make the special 'Id' 
APIs just aliases for the corresponding unicode-object-based APIs? It 
wouldn't be ABI compatible, but none of those APIs are in the stable ABI.


Indeed. I'm sure you can appreciate how I skimmed over that bit :) (plus 
I've mostly kept up with this by chatting with Eric, rather than 
reviewing all the code)


The guts of the code in question for those who don't want to find it:
#define STRUCT_FOR_ASCII_STR(LITERAL) \
struct { \
PyASCIIObject _ascii; \
uint8_t _data[sizeof(LITERAL)]; \
}

Which is then used inside another struct to statically allocate all the 
objects.


The gap this has with what I'd like to see is that it will only work for 
compile-time strings. If instead it could work for an arbitrary uint8_t 
pointer, rather than an embedded array, that allows even runtime strings 
to be very cheaply passed into the interpreter (and yes, the caller has 
to manage the lifetime, but that's pretty easy).


So this is perfect for the needs we have for our internal API, but I 
wouldn't want it to become _the_ public API. (I also wouldn't want to 
have two public APIs that are subtly different, because in a few years 
time everyone will forget why and be very confused about it all.) Not 
sure I'm motivated enough to build it myself, but I'm getting there ;) 
so perhaps I'll put something together that does what I'd like and give 
us concrete things to discuss.


(Is there a requirement that an Id only contain ASCII characters (i.e., 
7-bit)?)


I don't think it's unreasonable to require that for these internal 
identifier strings, but it would be a shame to do so for arbitrary strings.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LAKPW7A4NP3O6FCWTX3PBOWWZWNM75F3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

[Steve Dower]

On 2/4/2022 5:37 PM, Eric Snow wrote:

On Thu, Feb 3, 2022 at 3:49 PM Eric Snow  wrote:

I suppose I'd like to know what the value of _Py_IDENTIFIER() is for
3rd party modules.


Between Guido, Victor, Stefan, and Sebastian, I'm getting the sense
that a public replacement for _Py_IDENTIFER() would be worth pursuing.
Considering that it would probably help numpy move toward
subinterpreter support, I may work on this after all. :)

(For core CPython we'll still benefit from the statically initialized
strings, AKA gh-30928.)


For me, I'd love to see a statically allocated string type (for a real 
example that's used in Windows, check out [1], specifically when he gets 
to the fast-pass strings).


Essentially, a bare minimum struct around a char* (and/or wchar_t*) that 
acts as a PyUnicodeObject but doesn't ever allocate anything on the 
heap. This would also be helpful for config strings, which are often 
static but need to be copied around a lot, and good for passthrough 
strings that a native extension or host app might insert and receive 
back unmodified.


Because there's nothing to deallocate, it can be "created" and stored 
however the caller likes. As soon as Python code does anything with it 
other than passing it around, a regular PyUnicodeObject is allocated 
(just like the HSTRING example).


I'd expect usage to look very much like the intern examples earlier in 
the thread, but if we actually return a whole struct then we aren't on 
the hook for the allocations.


Cheers,
Steve

[1]: https://devblogs.microsoft.com/oldnewthing/20160615-00/?p=93675
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DGX4GBMDJYYFAE7OSVMBGKYAO2HPP3PT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

[Steve Dower]

On 1/28/2022 6:17 PM, Antonio Cuni wrote:

Of course, in order to be fully usable, the HPy universal ABI will need special 
support by PyPI/pip/etc, because at the moment it is impossible to package it 
inside a wheel, AFAIK.


It's totally possible, it's just that none of the existing tools will 
automatically generate the tags you need. (These are most critical in 
the filename itself, and also appear in 1-2 bits of metadata that 
currently are unused AFAIK.)


Basically, instead of just "cp310" (or "abi3", etc.), you'll want to use 
dots to separate each supported version ("cp38.cp39.cp310"). That will 
match the wheel to any of those versions.


You can even do the same with OS platforms if you prefer fewer/bigger 
wheels over more platform-specific ones.


Python on all platforms since IIRC 3.6 (maybe 3.5?) also have version 
and platform-specific tags in extension modules. These do not support 
combining tags as in wheels (and unfortunately do not match wheel tags 
at all), but do allow you to have version/platform-specific 
.pyd/.dylib/.so files in a single wheel. Again, it's just that none of 
the current build backends will help you do it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MLJXBKETEY7YJIPBRKKOVWKS6HKN2PMG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

[Steve Dower]

On 1/28/2022 5:15 PM, Barry Warsaw wrote:

On Jan 28, 2022, at 09:00, Steve Dower  wrote:


Does HPy have any clear guidance or assistance for their users to keep it up to 
date?

I'm concerned that if we simply substitute "support the C API for everyone" with 
"support the C API for every version of HPy" we're no better off.


Will it ever make sense to pull HPy into the CPython repo so that they evolve 
together?  I can see advantages and disadvantages.  If there’s a point in the 
future where we can just start promoting HPy as an official alternative C API, 
then it will likely get more traction over time.  The disadvantage is that HPy 
would evolve at the same annual pace as CPython.


Possibly, but we'd have to be really careful to not actually *evolve* 
HPy. It would essentially be a new stable API, but ideally one that uses 
all the preprocessor tricks we can (and perhaps runtime tricks) to 
compile against any CPython version rather than just the one that it 
comes with.


PSF "ownership" is probably enough to make it official (for those people 
who need everything to be "official"). I don't think that's necessary, 
but it does smooth the path for some people to be willing to use it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XCJH743KWSZ6436PU7RLG4NCX62ELTPR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

[Steve Dower]

Does HPy have any clear guidance or assistance for their users to keep 
it up to date?


I'm concerned that if we simply substitute "support the C API for 
everyone" with "support the C API for every version of HPy" we're no 
better off.


I think it can be done with clear communication from the HPy project 
(and us when we endorse it) that they will *never* break compatibility 
and it's *always* safe (and indeed, essential) for their users to use 
the latest version. But that's a big commitment that I can't sign them 
up for.


Cython seems to manage it okay. I can't remember the last compat issue I 
had there that wasn't on our (C-API) side.


Thoughts?

Cheers,
Steve

On 1/28/2022 4:50 PM, Victor Stinner wrote:

Wait, where is the HPy project in that plan? :-) The HPy project
(brand new C API) is a good solution for the long term!

My concerns about HPy right now is that, in short, CPython has to
continue supporting the C API for a few more years, and we cannot
evolve CPython before it will become reasonable to consider removing
the "legacy" C API.

I explained that in details in the PEP 674 (Disallow using Py_TYPE()
and Py_SIZE() macros as l-values):
https://www.python.org/dev/peps/pep-0674/#relationship-with-the-hpy-project

In parallel, we should continue promoting the usage of Cython, cffi,
pybind11 and HPy, rather than using directly the C API.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KHJ3IBDAM7OJNECT33FIZBDN3N5HMWYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.11 enhanced error location - can it be smarter?

[Steve Dower]

On 1/18/2022 7:59 PM, Pablo Galindo Salgado wrote:
We considered using colours and other markers such as bold text, but 
that opens a considerable can of worms with detecting in all systems and 
configurations if that can be done. I have been told that some of these 
situations are quite tricky and is not as easy as checking for tty support.


If someone wants to contribute a way to detect reliably (in C) if the 
terminal supports ANSI colours, I'm happy to consider switching to that 
instead of the underlying.


I'll save some time - no such mechanism exists on Windows. There's no 
reliable way to detect what's displaying console output, and while many 
now support ANSI colours, we can't be assured of it.


Omitting the line of ^ where over x% (75%? 90%?) of characters on the 
line would be marked would be fine by me.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ICDRA5LFN4YZ45GBQM5E5GYDEWCSWT4I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Steve Dower]

On 1/18/2022 2:44 PM, Antoine Pitrou wrote:

We propose to revert the following 2 changes in Python 3.11 and
postpone them in a later Python version, once most projects will be
compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
* Removals from configparser module (bpo-45173) - broke 28 Fedora packages


Doesn't this show, once again, that making DeprecationWarning silent by
default was a mistake?


That's a hypothesis, but we don't have enough information to prove it. 
If we'd gone the other way, perhaps we'd be looking at massive 
complaints from "regular" end users about all the noisy warnings that 
they can't fix and saying that making it noisy was the mistake.


Discovering during alpha that some packages haven't updated for the 
release that hasn't happened yet isn't the end of the world. Reverting 
the changes now is probably a bit premature - realistically we can undo 
these anytime during beta if we discover that packages are unable to be 
fixed over the next 9 months.


That said, I'm fine with reverting the changes on the basis that they 
cause churn for no real benefit. If someone wants to argue that the 
benefit is worthwhile, that's fine, as long as they also argue in favour 
of the churn.


We shouldn't pretend to be surprised that something we changed causes 
others to have to change. We *know* that will happen. Either we push 
forward with the changes, or we admit we don't really need them. With 
this amount of time before the release, we can't blame downstream users 
for reverting it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PGR4WUPMVGF67IWX2O7EKA2NSHFYEX5P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Steve Dower]

On 1/17/2022 8:47 PM, Barry Scott wrote:




On 17 Jan 2022, at 06:35, Tim Peters  wrote:

[Guido]

I don't think there's a way to do a PGO build from Visual Studio; but
a command prompt in the repo can do it using `PCbuild\build.bat --pgo`.
Just be patient with it.


Thanks! That worked, and was easy, and gave me an executable that runs
"// 10" at supernatural speed.

Alas, Visual Studio will not show a disassembly window unless the
debugger is running, and there appears to be no way to convince it to
run its debugger without it first recompiling the source file you're
staring at. Which it recomplies without benefit of PGO.


The trick you need is to close the project you use to build python
from source then you can open the python.exe and run that under the
debugger. Because it can find the python.pdb it will source/disasm as
you want.


Or you can Debug/Attach to the process if you start it running outside 
of Visual Studio.


You should also be able to just select the "PGUpdate" configuration 
(either from the dropdown in the toolbar, on Build/Configuration 
Manager). Provided you've done a PGO build from the command line, it 
will have the profile, and should be able to fairly quickly rebuild and 
reapply it - even with code changes - and start debugging.


Or you can try and enable the preprocessed assembler output from the 
compiler. I'm not sure how that handles PGO though, since the assembly 
comes from the compiler but it's the linker that does most of the 
optimisation. Attaching to a running process is what I normally do.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UFGYPAOZ5G7MN2QYYGUDAGJ3X6OCYP6F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Steve Dower]

On 12/7/2021 6:52 PM, Sebastian Berg wrote:

One thing we once did in NumPy (for a runtime problem), was to
intentionally break everyone at pre-release/dev time to point out what
code needed fixing.  Then flip the switch back at release time as to
not break production.
After a long enough time we enabled it for release mode.

Not saying that it was nice, but it was the only alternative would have
been to never fix it.


I like this idea. We'd have to turn it back for RC, and ensure that it's 
possible to have working code both before/after the change. We may be 
getting enough usage during beta for it to be worthwhile, though we 
still have the problem of knock-on effects (where e.g. until NumPy 
works, nothing that depends on it can even begin testing).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQ7KF656QIHPKLT5Z23P3W5OYOGH4J5K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Tool to search in the source code of PyPI top 5000 projects

[Steve Dower]

FTR, I don't consider the top projects on PyPI to be representative of 
our user base, and *especially* not representative of people compiling 
native modules.


This is not a good way to evaluate the impact of breaking changes.

It would be far safer to assume that every change is going to break 
someone and evaluate:

* how will they find out that upgrading Python will cause them to break
* how will they find out where that break occurs
* how will they find out how to fix it
* how will they manage that fix across multiple releases
* how will we explain that upgrading and fixing breaks is better for 
*them* than staying on the older version


This last one is particularly important, as many large organisations 
(anecdotally) seem to have settled on Python 3.7 for a while now. 
Inevitably, this means they're all going to be faced with a painful time 
when it comes to an upgrade, and every little change we add on is going 
to hurt more. Every extra thing that needs fixing is motivation to just 
rewrite in a new language with more hype (and the promise of better 
compatibility... which I won't comment specifically on, but I suspect 
they won't manage it any better than us ;) ).


This is not the case for the top PyPI projects. They incrementally 
update and crowdsource fixes, often from us. The pain is distributed to 
the level of permanent background noise, which sucks in its own way, but 
is ultimately not representative of much of our user base.


So by all means, use this tool for checking stuff. But it's not a 
substitute for justifying every incompatible change in its own right.


/rant

Cheers,
Steve

On 12/2/2021 11:44 PM, Victor Stinner wrote:

Hi,

I wrote two scripts based on the work of INADA-san's work to (1)
download the source code of the PyPI top 5000 projects (2) search for
a regex in these projects (compressed source archives).

You can use these tools if you work on an incompatible Python or C API
change to estimate how many projects are impacted.

The HPy project created a Git repository for a similar need (latest
update in June 2021):
https://github.com/hpyproject/top4000-pypi-packages

There are also online services for code search:

* GitHub: https://github.com/search
* https://grep.app/ (I didn't try it yet)
* Debian: https://codesearch.debian.net/


(1) Dowload

Script:
https://github.com/vstinner/misc/blob/main/cpython/download_pypi_top.py

Usage: download_pypi_top.py PATH

It uses this JSON file:
https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.json

 From this service:
https://hugovk.github.io/top-pypi-packages/

At December 1, on 5000 projects, it only downloads 4760 tarball and
ZIP archives: I guess that 240 projects don't provide a source
archive. It takes around 5,2 GB of disk space.


(2) Code search

First, I used the fast and nice "ripgrep" tool with the command "rg
-zl REGEX path/*.{zip,gz,bz2,tgz}" (-z searchs in ZIP and tarball
archives). But it doesn't show the path inside the archive and it
searchs in files generated by Cython whereas I wanted to ignore these
files.

So I wrote a short Python script which decompress tarball and ZIP
archive in memory and looks for a regex:
https://github.com/vstinner/misc/blob/main/cpython/search_pypi_top.py

Usage: search_pypi_top.py "REGEX" output_filename

The code to parse command line option is hardcoded and pypi_dir =
"PYPI-2021-12-01-TOP-5000" are hardcoded :-D

It ignores files generated by Cython and .so binary files (Linux
dynamic libraries).

While "rg" is very fast, my script is very slow. But I don't care,
once the regex is written, I only need to search for the regex once, I
can wait 10-15 min ;-) I prefer to wait longer and have a more
accurate result. Also, there is room for enhancement, like running
multiple jobs in different processes or threads.

Victor


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HO7PS57UCJPJLON2BJPPEBM7I3Q6AM2U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Steve Dower]

On 12/1/2021 12:47 AM, Oscar Benjamin wrote:

On Tue, 30 Nov 2021 at 23:37, Guido van Rossum  wrote:


We should definitely push back on zealous new converts to typing who insist 
that everything should be annotated. But we should also recognize that even in 
their current, far from perfect state, type annotations can provide a lot of 
value, when used right. (Have you run into VS Code yet? It gets tremendous 
value from typing stubs, in the form of improved auto-complete and hover-doc 
functionality.)


I might be misunderstanding but if "hover-doc" is the same as
"mouse-over" then it apparently needs many gigabytes of memory and a
lot of CPU time when you put the mouse over a sympy function that you
are using in your code. It has been suggested that SymPy should fix
this by adding hints like Any but I don't see the point of that: SymPy
has plenty of bugs but this particular bug is in VS Code.


For the record, this is the *old* support that has since been replaced 
(and was based on full program analysis in the Python of ~2010 that had 
no annotations at all, so always struggled with programs that could not 
be thoroughly inferred and particularly with those that generated 
runtime types - Sympy was a special case in it since about 2012 to avoid 
those issues, but that may have been lost when it was migrated from 
Visual Studio to VS Code).


The latest versions of VS Code have a completely new system, that does 
rely very heavily on programs and libraries being annotated, in exchange 
for being much lighter on memory use and preprocessing. But it should 
stay out of your way on unannotated code if you haven't enabled its more 
strict modes.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LHNW4CDJVJKNWTCJSTLVMBOBFUYMSWGZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Steve Dower]

On 11/30/2021 1:16 AM, Guido van Rossum wrote:
Why would it need to be reiterated? Are there really people who believe 
that such code would become invalid? AFAIK *everybody* here agrees that 
this should stay valid. So who would we be reiterating it for?


(Yes, several static type checkers have options that cause the checker 
to complain about unannotated code. But that's not the default behavior 
and running a static checker is a choice, like running a linter. 
Three-space indents or capitalized function names are never going to be 
disallowed either, even though PEP 8 says that's not how you ought to code.)


I know it sounds like a straw-man argument, but I don't think it is. I 
just think the non-straw-men making the argument aren't "important" 
enough to call out specifically - they are "random individual 
programmers who have gotten the wrong impression from another language 
and applied it to Python" rather than "influencers trying to change how 
we code".[1]


And we can always say that the *language* does not require running the 
type checker, but the reality for many developers is that the projects 
and teams they are part of will require it. And often, they require it 
because it's some kind of global "Best Practice" rather than because it 
makes sense for their particular situation.


For example, I saw a snippet fly by on Twitter the other day (again, not 
the fault of the individual who posted it, so I'm not 
naming-and-shaming) showing how to use Black to format a big directory 
of Jupyter notebooks. Convenient, perhaps, but the vast majority of 
people with "big directories of Jupyter notebooks" are going to be much 
happier if you just leave them alone! And yet, "The Community" is 
suggesting we should format all of them in bulk.


THAT'S the kind of thing that also has been happening with typing, and 
why some of us feel the need to publicly re-state things that are all 
agreed upon within this group, but are struggling to be heard over the 
public discourse on the topics. And this kind of reiteration is easier 
when our official documents (PEPs, etc.) state it explicitly.


Cheers,
Steve


[1]: Okay, I will call out the Python Bytes podcast - which I've been on 
a couple of times - as one that regularly opines on topics based on very 
little information and says things more definitively than they've been 
said anywhere else. They have a much bigger (and more easily influenced) 
audience than python-dev.


I'll also say that what I see from LWN is generally very good in this 
respect, their writers handle things well. Though I don't tend to read 
or listen to either of these that often.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SFLYLQO4YVQGV3BUKQO7KS6N367HNG62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should python-config be an stdlib module?

[Steve Dower]

On 10/25/2021 4:24 PM, Filipe Laíns wrote:

So, I guess the existing python-config script just needs to be updated to handle
cross compilation properly.


Including handling when the cross-compiled Python runtime won't actually 
run on the platform you're trying to get the config on ;)


Over at 
https://discuss.python.org/t/towards-standardizing-cross-compiling/10357/ about 
the *only* thing we could agree on is needing a static, cross-platform 
way to read config values from a Python build (essentially "python -m 
sysconfig" without being able to run Python).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XKY5LFHHEIUXSY72C3LJSRH5FWTM5INY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Steve Dower]

Okay, I'll let myself get sucked into responding ONE TIME, but only 
because you gave me such a nice API to work with :)


On 10/18/2021 9:11 PM, Piotr Waszkiewicz wrote:

 > class User(DBModel):
 >    phone: str | None
 >
 > class Publisher(DBModel):
 >   owner: ForeignKey[User] | None
 >
 > class Book(DBModel)
 >     publisher: ForeignKey[Publisher] | None


Imagine wanting to get the phone number of the person that published a 
certain book from the database.

In this situation, with maybe-dot operator I can write:

 > phone_number = book.publisher?.owner?.phone


Consider today, you wrote this as "book.publisher.owner.phone". You 
would potentially get AttributeError, from any one of the elements - no 
way to tell which, and no way to react.


Generally, AttributeError indicates that you've provided a value to an 
API which doesn't fit its pattern. In other words, it's an error about 
the *type* rather than the value.


But in this case, the (semantic, not implementation) *type* is known and 
correct - it's a publisher! It just happens that the API designed it 
such that when the *value* is unknown, the *type* no longer matches.


This is PRECISELY the kind of (IMHO, bad) API design that None-aware 
operators will encourage.



Consider an alternative:

class ForeignKey:
...
def __bool__(self):
return not self.is_dbnull

def value(self):
if self.is_dbnull:
return self.Type.empty() # that is, DBModel.empty()
return self._value


class DBModel:
@classmethod
def empty(cls):
return cls(__secret_is_empty_flag=True)

def __bool__(self):
return not self._is_empty

def __getattr__(self, key):
if not self:
t = self._get_model_type(key)
return t.empty() if isinstance(t, DBModel) else None
...

class User(DBModel):
phone: str | None

class Publisher(DBModel):
owner: ForeignKey[User]

class Book(DBModel)
publisher: ForeignKey[Publisher]


Okay, so as the API implementer, I've had to do a tonne more work. 
That's fine - *that's my job*. The user hasn't had to stick "| None" 
everywhere (and when we eventually get around to allowing named 
arguments in indexing then they could use "ForeignKey[User, 
non_nullable=True]", but I guess for now that would be some subclass of 
ForeignKey).


But now here's the example again:

> book.publisher.owner.phone

If there is no publisher, it'll return None. If there is no owner, it'll 
return None. If the owner has no phone number, it'll return None.


BUT, if you misspell "owner", it will raise AttributeError, because you 
referenced something that is not part of the *type*. And that error will 
be raised EVERY time, not just in the cases where 'publisher' is 
non-null. It takes away the random value-based errors we've come to love 
from poorly coded web sites and makes them reliably based on the value's 
type (and doesn't even require a type checker ;) ).


Additionally, if you want to explicitly check whether a FK is null, you 
can do everything with regular checks:


if book.publisher.owner:
# we know the owner!
else:
# we don't know

# Get all owner names - including where the name is None - but only if
# Mrs. None actually published a book (and not just because we don't
# know a book's publisher or a publisher's owner)
owners = {book.id: book.publisher.owner.name
  for book in all_books
  if book.publisher.owner}

# Update a null FK with a lazy lookup
book.publisher = book.publisher or publishers.get(...)


You can't do anything useful with a native None here besides test for 
it, and there are better ways to do that test. So None is not a useful 
value compared to a rich DBModel subclass that *knows* it is empty.


---

So to summarise my core concern - allowing an API designer to "just use 
None" is a cop out, and it lets people write lazy/bad APIs rather than 
coming up with good ones.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRTRKGY6RLTHZJQ2US4LO7DYLSGXQ5GM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Steve Dower]

Thanks for allowing me to speak for myself, as I said I would when 
contacted off list :)


But as it happens, you've summarised my position very accurately:


he now believes the implementation of these operators would lead people to a 
style of coding which would lead to the proliferation of None as an 
exception-less error result and also throughout data structures. My 
understanding is that his current preference is to focus on functional 
composition and styles of programming that disallow the use of None.


As additional context, C# and the .NET Framework have just gone through 
the exercise of making (and propagating) non-nullable reference types 
(and their "null" is far more like our None than C's NULL). Our type 
checkers are also pushing in this direction by requiring Optional[]. The 
argument in all cases is that it allows for simpler code that can safely 
ignore non-values because they cannot be passed.


Adding ??, ?. and ?[, particularly in their short-circuiting evaluation 
form, encourages more complex code by adding value-based branching 
within an expression. And it encourages it on the "outside" of the API, 
not within it. Which means API designers can more easily justify 
returning None because their caller can use None-aware operators to 
basically ignore it. (I would argue that the API designer should work 
harder to create an API that doesn't ever have to return None.)


To be clear, I'm thinking very much in terms of "impact on what people 
will consider to be Pythonic API design over the next 10 years", that 
is, what people think they *should* do with this, rather than simply 
what they *could*. So it's all very hypothetical and I have no data for 
it, much like when it was decided that Optional[] would be mandatory on 
types where None is permitted.


And with that, I'm bowing out of the fresh discussion (unless the SC 
wants to discuss it directly with me). I'll continue advocating for 
tools/features that help people create better APIs, but I'm not going to 
spend a lot of time arguing against those that I believe will not.


Cheers,
Steve

On 10/15/2021 3:08 AM, Doug Swarin wrote:

Steven D'Aprano wrote:

Hello Doug,
On Thu, Oct 14, 2021 at 03:45:07PM -, Doug Swarin wrote:

I believe strong and valid arguments can be made about the use of None
being a fundamental flaw in some types of coding
Can you elaborate on that? Obviously it is not always appropriate to use

None, but I've never seen it called a *fundamental* flaw.
I know that the null pointer has been called a billion-dollar mistake,
but Python's None is not a null pointer.


I apologize that I may have spoken too strongly here. When I emailed Mr. Dower, 
he mentioned that he now believes the implementation of these operators would 
lead people to a style of coding which would lead to the proliferation of None 
as an exception-less error result and also throughout data structures. My 
understanding is that his current preference is to focus on functional 
composition and styles of programming that disallow the use of None.

I certainly don't mean to speak for him and I hope he will weigh in with a more 
detailed explanation of his thoughts and objections, but I personally disagree 
as a matter of pure practicality. It's just plain useful to be able to easily 
take non-values and safely deal with them without having to constantly check 
for None or to catch and inspect exceptions to see if it's a case that can be 
ignored.

I did indeed think about connecting None to the 'billion dollar mistake' but 
decided against it since as you say None is not a null pointer, and I should 
have chosen my words a little more carefully when revising my initial post 
(likely by removing the word 'fundamental').

Doug

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UIASDCES7GMQAMBNZGQZ65B2HSCPOEMD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Steve Dower]

On 10/6/2021 5:05 PM, Yury Selivanov wrote:
So I'm -1 on `except group` or any variant that uses soft keywords. If 
the SC considers making `group` a proper keyword I can possibly change 
my mind on this.


For the record (and I'm sure I'm not the only one), I'm -100 on making 
it a proper keyword. That would be disastrous (e.g. re.Match.group() 
becomes unusable).


A soft keyword, punctuation, or magic builtin are the only possibilities 
here.


"except all ..." is viable, since it's already a builtin that isn't 
useful as "except all:". But if that's the case, "except ExceptionGroup" 
is equally viable (with perhaps "except ExceptionGroup[Specific, Type]" 
for filtering?)


I'm not going to argue against "except *", as that's already been 
accepted. But any alternative needs to:

* break the same amount of existing code (i.e. none)
* be equally/more readable and discoverable

Since "except *" breaks *no* existing code, that's a pretty easy thing 
to check for in any alternative. But since "*" here has no precedent (as 
we've seen in this discussion), virtually any alternative is going to be 
more readable.


So enjoy bikeshedding, everyone :) Please don't break any of our code.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YOTXVPFABO4YKQ7TSEA3NMGNF47MBH5T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Steve Dower]

On 10/4/2021 5:47 PM, Antoine Pitrou wrote:

On Mon, 4 Oct 2021 12:18:35 -0400
Calvin Spealman  wrote:

On Mon, Oct 4, 2021 at 12:07 PM Guido van Rossum  wrote:


The question was about which style to *recommend* (a la PEP-8).
  


I think the very fact that it can't (or is difficult) be enforced,


How so?  If style checkers are already able to check whitespace around
operators, they should be to check whitespace in this instance as well.

Do you suggest that PEP 8 violations should be detected by the Python
parser itself?


No, but if it isn't decided by *us*, it'll be decided by whoever 
contributes it to Black first.


To me, the "*name" looks most similar to how we write "*args" in a 
function definition, so I'd go for that.


We don't currently modify[1] keywords with punctuation, and that's what 
"except*" looks like, and "except * E" looks like a binary operator 
and/or grit on the screen.


Cheers,
Steve

[1]: Meaning to "give it a different meaning in particular context", not 
_literally_ modify in any permanent sense.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/52KJZMKMFTFHVMS3NXABNFQJRZNLKLX5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 7:15 PM, Ethan Furman wrote:

On 9/27/21 10:50 AM, Guido van Rossum wrote:

 > So my *actual* proposal (call it #2') is to use a separate 
compile-time flag, which is set by `./configure
 > --enable-optimizations` regardless of whether PGO/LTO are possible, 
and which on Windows can be set by
 > `PCbuild\build.bat --pgo` (we could add another flag to disable it, 
but who'd want to?).


I think a configure-time flag is the way to go, and I'm happy to have it 
included with --enable-optimizations.


Having it be implied by an "--enable-optimizations" option is totally 
fine (and we'd add one to build.bat for this), but I still think it 
needs to be discoverable later whether the frozen modules build option 
was used or not, independent of other build options.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QU6H2GS6S4MD7WTDI2Y6BLP46MXJRVFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 6:40 PM, Patrick Reader wrote:

I accidentally a word.

"... module's hash and/or *timestamp* and comparing it ..."

On 27/09/2021 18:38, Patrick Reader wrote:

How about checking each non-frozen module's hash and/or and comparing it to 
that of the frozen module? Would that defeat the performance improvement of 
freezing? Is it just a terrible idea?


The filesystem access is what hurts the most, so yeah, that check won't 
even be considered here without a manual opt-in (which is the option 
Eric is discussing).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VOX3HTK3RBD7KLJ46XZQGS7G4SHVDDX3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 5:51 PM, Eric Snow wrote:

Possible solutions:

1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)


Just to air my concerns regarding option 2 (which I've already spoken to 
Eric about): I feel this option is completely orthogonal to whether PGO 
is used or not, and ought to be discoverable independently.


Essentially, it should be its own configure-time option, and should be 
included somewhere in sysconfig.get_config_var(...).


If we went with #2, there's no reliable way to detect whether 
profile-guided optimisations were used on all CPython builds, which 
means there'd be no reliable way to detect whether frozen modules are 
going to be enabled by default or not. Adding a separate option handles 
this case.


(My overall preference is for #3, FWIW)

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CDG4LSUXRLKCDPJ4RMVXCAPDJ4PNTN4D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Steve Dower]

On 9/21/2021 7:42 PM, Eric V. Smith wrote:
I don't recall exactly why, but I disallowed backslashes inside 
expressions at the last minute before 3.6 was released. It might have 
been because I was interpreting them in a way that didn't make sense if 
a "real" parser were inspecting f-strings. The idea, even back then, was 
to re-allow them when/if we moved f-string parsing into the parser 
itself. I think it's time.


Yeah, we were still trying to figure out whether escapes like "\\n" 
would be evaluated as "\\n" or "\n" in the expression, and decided to 
decide later. If we can clearly articulate which it is now, then let's 
go ahead and enable it.


* The parser will allow nesting quote characters. This means that we 
**could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"
I'm okay with this, with the caveat that I raised in another email: the 
effect on non-Python tools and alternate Python implementations.


As a fairly regular user, I would be very happy to not have to worry 
about mixing quotes. It's also not going to break any existing code, so 
safe enough to enable if we can.


Agreed with Eric on the rest.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZQDQRUI4JZOQT7IW2MU2IGJCGHWZH7Q4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors

[Steve Dower]

On 9/13/2021 8:12 PM, Ethan Furman wrote:

On 9/13/21 9:03 AM, Steve Dower wrote:

 > You *are* allowed to put some (brief) details in the abstract. No 
need to avoid spoilers ;)

 >
 > As it stands, "it is time" on its own is a really bad reason to 
change anything. So you're

 > preemptively making it sound like the PEP has no solid backing.


Abstract


Update the ``repr()``, ``str()``, and ``format()`` of the various Enum 
types

for consistency and to better match their intended purpose.



Better?


You don't have a one sentence summary of what the changes entail? 
"Originally these were based on the idea that ..., but now will work 
better for ... by making the results more consistent with ..." (where 
each "..." is filled with specific things).


It doesn't have to save me reading the whole thing, but if I'm digging 
through documents going "why am I seeing  instead of just '1'", 
it should confirm that this is the right document to read.


(Alternatively, think about writing the tweet here that you want people 
to include when they share your PEP, assuming that 99% of people won't 
actually click through to the document. What can you tell them so that 
they at least know what is coming?)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQI5IVW5B6NQB246YYOSAW5DP3G2ZWAQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors

[Steve Dower]

On 9/11/2021 6:53 AM, Ethan Furman wrote:

Abstract


Now that we have a few years experience with Enum usage it is time to 
update
the ``repr()``, ``str()``, and ``format()`` of the various enumerations 
by their

intended purpose.


You *are* allowed to put some (brief) details in the abstract. No need 
to avoid spoilers ;)


As it stands, "it is time" on its own is a really bad reason to change 
anything. So you're preemptively making it sound like the PEP has no 
solid backing.


I haven't read the rest yet. The abstract is supposed to make me want to 
read it, and this one does not, so I stopped. (Might come back later 
when I'm not trying to catch up on my weekend's email though.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KN7QUBTJGWWP7TZ4PWJIZNVFPWU7S2ID/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Steve Dower]

On 8/16/2021 12:47 AM, Guido van Rossum wrote:
My current proposal is to issue a DeprecationWarning in PyCode_New() and 
PyCode_NewWithPosArgs(), which can be turned into an error using a 
command-line flag. If it's made an error, we effectively have (B); by 
default, we have (A).


Then in 3.13 we can drop them completely.


We definitely had legitimate use cases come up when adding positional 
arguments (hence the new API, rather than breaking the existing one, 
which was the first attempt at adding the feature).


I don't recall exactly what they are (perhaps Pablo does, or they may be 
in email/issue archives), but since they exist, presumably they are 
useful and viable _despite_ the bytecode varying between releases. This 
suggests there's probably a better API we should add at the same time - 
possibly some kind of unmarshalling or cloning-with-updates function?


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NXSTTN3CVCGAYSCUH24OEZZFIBOEJRBD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Repealing PEP 509 (Add a private version to dict)

[Steve Dower]

On 7/29/2021 6:17 PM, Barry Warsaw wrote:

On Jul 29, 2021, at 05:55, Steve Dower  wrote:


Maybe we should have a "Type" other than Standards Track for PEPs that are 
documenting implementation designs, rather than requirements for standardisation?


Wouldn’t Informational fill that need?


Perhaps, though my understanding was that Informational PEPs aren't tied 
to specific Python versions (in this case, a CPython release) and are 
meant to remain "Active" until completely redundant.


Maybe if the PEPs got a preamble covering "this describes part of the 
implementation of CPython as first released in x.y... may have changed 
since that time... refer to source code" it would be clear enough. We 
might want to move them to "Final" after implementation though, unlike 
other Informational PEPs.


Seems like there's _just enough_ nuance there to justify a different 
type. But then, presumably there's also infrastructure around PEP types 
as well that I'm unaware of (and not volunteering to update!), so those 
with the investment in it can have it however they'd like :)


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F2WEKHD77EAWFTCW7PWDEESSZMBL64BI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Repealing PEP 509 (Add a private version to dict)

[Steve Dower]

On 7/29/2021 11:41 AM, Mark Shannon wrote:
The dictionary version number is currently unused in CPython and just 
wastes memory. I am not claiming that we will never need it, just that
we shouldn't be required to have it. It should be an internal 
implementation detail that we can add or remove depending on requirements.


Sounds reasonable.

Maybe we should have a "Type" other than Standards Track for PEPs that 
are documenting implementation designs, rather than requirements for 
standardisation?


If/when we ever get to a point where other implementations want to claim 
to implement "standard" Python, having an explicit distinction here 
would be helpful.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W7TRTCE3PLLKHPWKFYMRZN4KHFK65RWD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [slightly OT] cryptographically strong random.SystemRandom()

[Steve Dower]

On 7/12/2021 4:11 PM, Dan Stromberg wrote:
It looks like CPython could do better on Windows: SystemRandom (because 
of os.urandom()) is good on Linux and mac, but on Windows they use the 
CryptGenRandom deprecated API


Supporting detail: 
https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptgenrandom 



Should I open an issue?


Please do, but note that the API is only deprecated because it was not 
very extensible and the new API is much more comple... er... extensible.


There's nothing wrong with the randomness from the function. It'll be 
using the new API under the covers. So this is an enhancement, not a 
security issue, and should only target 3.11.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N6K3GLAEOXYFAJY5S4Z6APUCKXVYTIEV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is the Python review process flawed?

[Steve Dower]

On 7/1/2021 7:04 PM, Christopher Barker wrote:
PS: All that being said, we, as a community, could do better. For 
instance, someone like me could do high-level triage on bug reports -- I 
need to set aside some time to do that.


In addition/instead of triage, one thing we (as core maintainers) really 
need, which the more involved members of the wider community can 
provide, is *context*.


I presume that if you're on python-dev, you probably have (or have 
access to) a significant Python codebase. Many core developers do as 
well, but we know we don't have great view across all the different ways 
that Python gets used. Some of the "worst" regressions are due to 
scenarios that we simply didn't know existed until they were broken.


We don't need (or wouldn't benefit from) just seeing all of the code, 
because it's the understanding that's important. How would changes 
impact *your* projects, *your* users. Anywhere you can chip in with "we 
do XYZ and this change would [not] impact us this way" or "we'd have to 
mitigate it by doing ..." is very useful.


It's not just a vote, it's the added context that's helpful. Ultimately, 
"votes" don't count for much in merge/reject decisions, because we're 
trying to take into account all users, not just those who show up to 
click a button. Rational arguments based in real and personal impact 
statements are far more valuable, especially if you have context that we 
don't.


So feel free to drop into bugs or PRs when you have relevant context for 
the impact of a change. Try and be as detailed about the impact on you 
as you're allowed (or can be without distracting from the issue). Don't 
be offended if we still think that the impact is "worth it" or that your 
situation isn't actually as impacted as you think (that probably 
indicates that we need to do a better NEWS/Whats New entry for the change).


And yes, these can be *significant contributions* to the project, so if 
you find yourself in a position where you have to justify it to someone 
outside of the project (e.g. using work time for open source), hopefully 
any core developer you've interacted with a bit will gladly write a 
short endorsement for that.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BB3YX3J5NZHBUKCV67O34TPVQLD2CLWA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.

[Steve Dower]

On 6/9/2021 2:20 PM, Petr Viktorin wrote:

On 09. 06. 21 13:09, Paul Moore wrote:

Also, I often use the stable ABI when embedding, so that
I can replace the Python interpreter without needing to recompile my
application and redeploy new binaries everywhere (my use case is
pretty niche, though, so I wouldn't like to claim I represent a
typical user...).


I hope this use case becomes non-niche. I would love it if embedders 
tell people to just use any Python they have lying around, instead of 
vendoring it (or more realistically, embedding JS or Lua instead).


I also hope it becomes non-niche, but I'd rather you started 
embedding/vendoring CPython rather than using anything that just happens 
to be laying around.


The number one issue that *all* of my customers (and their customers) 
have is installation. For most of them, the best way to solve it is to 
not make them install Python themselves, which in many cases means 
vendoring. The more acceptable and easy we can make this process, the 
more Python will be a viable choice against JS or Lua (though with all 
the other C API, threading and initialization issues, it's unlikely that 
embedding CPython is going to become significantly more attractive than 
those two - even IronPython still lives on for embedding because it 
works so well).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36MZXQC3WGDXO6SFTPGT7RC34EMFPP6E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

[Steve Dower]

On 6/3/2021 7:42 PM, Senthil Kumaran wrote:

On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:

The regression may well be a platform issue. I am by no means an expert at
building python; I followed a recipe from the ARCH PKGBUILD of some time


I meant the change in the diff we were suspecting was supposed to be
"Windows" specific. But interesting that it was observed in ARCH. The
non-windows piece of changes were probably responsible


While we were investigating the reliably broken behaviour, we figured 
that related behaviour was *unreliably* broken on all platforms.


Specifically, if you imported a module through a relative path (resolved 
to CWD), changed the CWD, cleared the module cache, and reimported, you 
would still get the original module, contrary to the intent of getting 
the newly resolved module. ("Correct" code would have also cleared the 
import path caches, not just sys.modules.) This was because the module 
info was cached using the relative path, and so would be used later even 
though its absolute path had changed.


Someone reported this change in 3.8 and we decided to revert it in 
existing released, because the bugfix was for an obscure enough 
situation that it really wasn't worth breaking any existing versions. 
For unreleased versions, it's better to fix it so that imports will be 
more reliable in the future.


Based on Robin's original post, it looks like this is exactly the 
scenario we wanted to fix, and it's probably exposed an underlying issue 
in that test suite.


Unfortunately, import caching has changed a lot since Python 2.7, so 
it's very unlikely that one approach will be able to do reliable 
reloading across all of those versions. Your best bet since 3.3 is to 
use the importlib module's methods (but your *best* bet overall is to 
run separate test suites in their own process so you can avoid the CWD 
changes).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDIMZA2KVWZSYUVLWQUZT4TE5BH6QWOV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: GDB not breaking at the right place

[Steve Dower]

On 5/24/2021 9:38 PM, Guido van Rossum wrote:
To the contrary, I think if you want the CI jobs to be faster you should 
add the CFLAGS to the configure call used to run the CI jobs.


Big +1

We should have the most useful interactive development/debugging options 
set by default (or with an obvious option), and use the complex 
overrides in our own automated systems.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TNHXYXH6A77J5Y43JBLVMXC3KF722H6T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Question for potential python development contributions on Windows

[Steve Dower]

When you're installing Visual Studio the C++ tools version is listed 
under the selected components as "v14x".


However, at this stage, the *only* version in circulation is 14.x - mine 
shows v142. Until the 14 changes to a "15", it will be binary compatible 
and so you can use any version at all to build CPython and/or extension 
modules.


Our official releases are always using relatively up-to-date compilers, 
but provided the compatibility is maintained on Microsoft's side, 
there's no need to worry about the specific versions.


Cheers,
Steve

On 5/24/2021 4:49 PM, Guido van Rossum wrote:

How do you check that the C++ tools are v14.x?

On Mon, May 24, 2021 at 1:43 AM Łukasz Langa > wrote:




On 20 May 2021, at 07:03, pjfarl...@earthlink.net
 wrote:

The Python Developers Guide specifically states to get VS2017 for
developing or enhancing python on a Windows system.
__ __
Is it still correct to specifically use VS2017 , or is VS2019 also
acceptable?


We have to update the devguide. VS 2019 is supported, probably even
recommended, as long as the C++ tools are v14.x.

Cheers,
Łukasz

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MJVPA353LMZM4OP6U63QJ5PWQ73ZCGGW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Steve Dower]

On 5/21/2021 9:36 AM, Petr Viktorin wrote:

On 21. 05. 21 3:23, Eric V. Smith wrote:

On 5/20/2021 3:24 PM, Ronald Oussoren via Python-Dev wrote:

One example of this is the definition of dataclasses.field:

|dataclasses.||field|(/*/, /default=MISSING/, 
/default_factory=MISSING/, /repr=True/, /hash=None/, /init=True/, 
/compare=True/, /metadata=None/)


Completely agree. I'm opposed to Ellipsis as a sentinel for this 
reason, at least for dataclasses. I can easily see wanting to store an 
Ellipsis in a field of a dataclass that's describing a function's 
parameters. And I can even see it being the default= value. Not so 
much default_factory=, but they may as well be the same.


And this argument also works for any other single value.
Including the original None.

(It just might not be obvious at first, before that single value starts 
being used in lots of different contexts.)


I think it's a fairly obvious case, and a legitimate one to acknowledge 
(the array one less so - we're talking about a "missing parameter" 
sentinel, so you ought to be testing for it immediately and replacing 
with your preferred/secret default value, not using it for indexing 
without even checking it).


In the example above, it's fairly easy to pass "lambda: ..." as the 
default_factory to work around it, and besides, the existence of rare 
edge cases doesn't mean you have to force everyone into acting like 
they're a rare edge case.


All the other situations where we want arguments with unspecified 
default values can use ..., and the few cases where ... is a valid value 
(semantically, for the API, not syntactically) can spend the time 
figuring out a different API design.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EJYSP7KBD2XA26QTE2FVQP4SS2FDLPB2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Steve Dower]

On 14May2021 0622, micro codery wrote:


There was a discussion a while back ( a year or so?? ) on
Python-ideas that introduced the idea of having more "sentinel-like"
singletons in Python -- right now, we only have None. 

Not quite true, we also have Ellipsis, which already has a nice repr 
that both reads easily and still follows the convention of eval(repr(x)) 
== x. It also is already safe from instantiation, survives pickle 
round-trip and is multi-thread safe.
So long as you are not dealing with scientific projects, it seems a 
quick (if dirty) solution to having a sentinel that is not None.


I don't think using "..." to indicate "some currently unknown or 
unspecified value" is dirty at all, it seems perfectly consistent with 
how we use it in English (and indexing in scientific projects, for that 
matter, where it tends to imply "figure out the rest for me").


All that's really missing is some kind of endorsement (from python-dev, 
presumably in the docs) that it's okay to use it as a default parameter 
value. I can't think of any reason you'd need to accept Ellipsis as a 
*specified* value that wouldn't also apply to any other kind of shared 
sentinel.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VCKTZF45OAFYNIOL5IVNI5HG34BGJBEA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Steve Dower]

On 13May2021 1248, Petr Viktorin wrote:

On 13. 05. 21 11:45, Antoine Pitrou wrote:


Le 13/05/2021 à 11:40, Irit Katriel a écrit :



On Thu, May 13, 2021 at 10:28 AM Antoine Pitrou > wrote:



  I agree that  is a reasonable spelling.


I initially suggested , but now I'm not sure because it 
doesn't indicate what happens when you don't provide it (as in, what 
is the default value).  So now I'm with  or .


"" makes think of a derived class, and leaves me confused. 
"" is a bit better, but doesn't clearly say what the default 
value is, either.  So in all cases I have to read the docstring in 
addition to the function signature.




Is  the term you're looking for?


Perhaps  or ?

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RSRBWH2UK2MKZN7O3PHSNVZFZEE7JIVJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Steve Dower]

On 3/17/2021 7:34 PM, Ivan Pozdeev via Python-Dev wrote:

On 17.03.2021 20:30, Steve Dower wrote:

On 3/17/2021 8:00 AM, Michał Górny wrote:

How about writing paths as bytestrings in the long term?  I think this
should eliminate the necessity of knowing the correct encoding for
the filesystem.


That's what we're trying to do, the problem is that they start as 
strings, and so we need to convert them to a bytestring.


That conversion is the encoding ;)

And yeah, for reading, I'd use a UTF-8 reader that falls back to 
locale on failure (and restarts reading the file). But for writing, we 
need the tools that create these files (including Notepad!) to use the 
encoding we want.




I don't see a problem with using a file encoding specification like in 
Python source files.

Since site.py is under our control, we can introduce it easily.

We can opt to allow only UTF-8 here -- then we wait out a transitional 
period and disallow anything else than UTF-8 (then the specification can 
be removed, too).


The only thing we can introduce *easily* is an error when the 
(exclusively third-party) tools that create them aren't up to date. 
Getting everyone to specify the encoding we want is a much bigger 
problem with a much slower solution.


This particular file is probably the worst case scenario, but preferring 
UTF-8 and handling existing files with a fallback is the best we can do 
(especially since an assumption of UTF-8 can be invalidated on a 
particular file, whereas most locale encodings cannot). Once we openly 
document that it should be UTF-8, tools will have a chance to catch up, 
and eventually the fallback will become harmless.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5B53GCQNYXFBYAHSJKI6I34XAV6S67HN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Steve Dower]

On 3/17/2021 6:08 PM, Stefan Ring wrote:

A somewhat radical idea carrying this to the extreme would be to use
UTF-16 (LE) on Windows. After all, this _is_ the native file system
encoding, and Notepad will happily read and write it.


I'm not opposed to detecting a BOM by default (when no other encoding is 
specified), but that won't help most UTF-8 files which these days come 
with no marker at all.


I wouldn't change the default file encoding for writing though (except 
to unmarked UTF-8, and only with the compatibility approach Inada is 
working on). Everyone has basically come around to the idea that UTF-8 
is the only needed encoding, and I'm sure if it had existed when Windows 
decided to support a universal character set, it would have been chosen. 
But with what we have now, UTF-16-LE is not a good choice for anything 
apart from compatibility with Windows.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LTEJSNOH6EHESXSMXSW352JFG2SF7ZMX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Steve Dower]

On 3/17/2021 8:00 AM, Michał Górny wrote:

How about writing paths as bytestrings in the long term?  I think this
should eliminate the necessity of knowing the correct encoding for
the filesystem.


That's what we're trying to do, the problem is that they start as 
strings, and so we need to convert them to a bytestring.


That conversion is the encoding ;)

And yeah, for reading, I'd use a UTF-8 reader that falls back to locale 
on failure (and restarts reading the file). But for writing, we need the 
tools that create these files (including Notepad!) to use the encoding 
we want.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MVD67FOAJRCNR2XXLJ4JDVFPYGZWYLDP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Steve Dower]

On 2/24/2021 4:26 PM, Christian Heimes wrote:

On 24/02/2021 15.16, Random832 wrote:

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:

Separate directories don't prevent clashes and system breakage. But they
provide an easy way to *recover* from a broken system.


I think it could be turned into a way to prevent them by A) having 
site-packages always take precedence over dist-packages [i believe this is 
already the case] in normal usage and B) providing an option to the 
interpreter, used by system scripts, to exclude site-packages entirely from the 
path.

Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how 
a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread 
made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things 
that entering a venv does such as "python" [and script names such as pip] being an alias for the correct 
version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" 
can influence the "normal environment" and not vice-versa, in the same way that packages installed in the 
main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the 
main installation.


Yes, you are describing one major aspect of my idea for a system Python
interpreter. I'm happy to read that other users are coming to similar
conclusions. Instead of an option I'd create a new executable to lock
down additional things (e.g. isolated mode, code verification hook). A
separate executable would also allow distros to provide a stripped down
interpreter that does not cause bad user experience.


I mean, this is _precisely_ what PEP 370 defines (including the "-s" 
option and PYTHONNOUSERSITE env variable to provide that one-way isolation).


Is the problem that pip doesn't use it by default? Or that the distros 
went and made patches for the runtime rather than patching pip? (For 
Windows installs from the Store, where even admin rights can't do an 
all-users package install, I added a new config file location for pip to 
change this default, but a patch would also have worked.)


Maybe we need an easier way to patch the location of user site packages? 
I also had to do this for the Store install on Windows, and it's a 
little bit of a hack... but maybe having an official recommendation 
would help encourage distributors to use the mechanism?


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IHEECOXYDPJ6ZQJE2QTGPDOOCOP7J37A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Steve Dower]

On 2/22/2021 5:18 PM, Matthias Klose wrote:

On 2/21/21 1:13 PM, Victor Stinner wrote:
Document what is supported, be inclusive about anything else.  Don't make a
distinction yet between legacy and upcoming new architectures.


I agree with this, and I don't see any reason why we shouldn't just use 
the list of stable buildbot platforms as the "supported" list. That 
makes it really clear what the path is to push support onto upstream 
(join up and bring a buildbot with you), and also means that we've got a 
physically restricted set of machines to prove work before doing a release.


Actively blocking anything at all seems unnecessary at the source/build 
level. That's for pre-built binaries and other conveniences.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BKZTBXDYFIEBMVELBOVQ5KGM2ZEXVT2Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 624: Remove Py_UNICODE encoder APIs

[Steve Dower]

On 2/1/2021 5:16 PM, Christian Heimes wrote:

On 01/02/2021 17.39, M.-A. Lemburg wrote:

Can you explain where wchar_t* type is appropriate and how two
conversions is a performance bottleneck?


If an extension has a wchar_t* string, it should be easy
to convert this in to a Python bytes object for use in Python.


How much software actually uses wchar_t these days and interfaces with
Python? Do you have examples for software that uses wchar_t and would
benefit from wchar_t support in Python?

I did a quick search for wcslen in all shared libraries and binaries on
my system


Yeah, you searched the wrong kind of system ;)

Pick up a Windows machine, cross-platform code that originated on 
Windows, anything that interoperates with Java or .NET as well, or uses 
wxWidgets.


I'm not defending the choice of wchar_t over UTF-8 (but I can: most of 
these systems chose Unicode before UTF-8 was invented and never took the 
backwards-incompatible change because they were so popular), but if we 
want to pragmatically weigh the needs of our users above our desire for 
purity, then we should try and support both equally wherever possible.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GYUWANE7IMPU45A257UYQD4ZGUDE6QUX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why aren't we allowing the use of C11?

[Steve Dower]

On 29Jan2021 1715, Victor Stinner wrote:

It seems like declaring a TLS in libpython and using it from an
extension module (another dynamic library) is causing issues depending
on the linker. It "should" work on macOS, but it doesn't.


I'm pretty sure this is not defined in any calling convention that would 
be able to cross dynamic library boundaries, or at least I've never seen 
it shown how this information would be enecoded.


In general, provided the features don't leak out of our compilation 
process (i.e. cross dynamic library boundaries or show up in 
public/implicit header files), I don't have any issue allowing them to 
be used.


Though I don't want to speak for the people who maintain CPython distros 
on smaller/slower/more focused platforms. I'm not sure how best to reach 
them, but I'd rather do that than find out when they complain on the 
issue tracker. Only thing I'm certain of is that we shouldn't assume 
that we know everyone who ever builds CPython.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NWK3O5DPDA2NIP5EEE6JOLKLENKBCAOL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why aren't we allowing the use of C11?

[Steve Dower]

On 29Jan2021 2227, Random832 wrote:

On Thu, Jan 28, 2021, at 22:57, Emily Bowman wrote:
  
On Thu, Jan 28, 2021 at 1:31 PM MRAB  wrote:

I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free)
and it supports C11 and C17.


While an upgrade for Community is free, for Pro/Enterprise without an
annual license it's not.


Something that's hard for me to find a clear answer for in a few minutes of 
searching:

Does the current version of the *Windows SDK* still come with a compiler, and 
can it be used without the Visual Studio IDE [the situation is confusing 
because it seems to install through the visual studio installer now] or at 
least without a license for Visual Studio Pro/Enterprise [i.e. by users who do 
not qualify for community edition] or with earlier versions of the Visual 
Studio IDE?


The Windows SDK contains system headers/libs, no compilers.

Visual Studio includes MSVC, and is one of only a few bundles that does 
(and if you have access to any of the others, you'll know by the massive 
hole in your bank account, so let's assume Visual Studio). It will also 
install the Windows SDK on your behalf if the box is checked, which it 
is by default when you select C++ support.


*Anyone* is allow to use Visual Studio Community to work on open-source 
software. So the only way to not qualify is to refuse to share whatever 
you're working on. Obviously some people will be in that position, but 
it's really not that onerous an ask (almost like the GPL, if you squint 
;) ). Anyone contributing to CPython is allowed to use VS Community to 
do so. [1]


Additionally, you can install VS Community alongside any other edition 
or version and they will be kept separate (as much as is possible, which 
is on Microsoft's side to deal with and shouldn't impact licensing).


Finally, the VS Build Tools are only licensed to you if you have a 
Visual Studio license, which means you need to have qualified for VS 
Community (or paid for a higher one) to use the build tools. Again, if 
you're building open-source software, you've qualified, so it's 
basically a non-issue for us.


Hopefully that settles some concerns. IANAL, but I am on the Visual 
Studio team and have been involved in discussions around how VS 
Community licensing applies to open source ecosystems ever since it was 
first created. If you're still concerned, go pay a lawyer to give you 
more opinions, because you honestly aren't going to find a more informed 
opinion for free on the internet ;)


Cheers,
Steve

1: See page 8 of 
https://visualstudio.microsoft.com/wp-content/uploads/2020/03/Visual-Studio-Licensing-Whitepaper-Mar-2020.pdf

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WBVMFEGGL4NOGDHUWZWM2C6PDYAIAIHD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

[Steve Dower]

On 1/26/2021 8:32 PM, Steve Holden wrote:
If the length of the name is any kind of issue, since the stdlib 
only contains modules (and packages), why not just sys.stdlib_names?


And since the modules can vary between platforms and builds, why 
wouldn't this be sysconfig.stdlib_names rather than sys.stdlib_names?


"Modules that were built into the stdlib" sounds more like sysconfig, 
and having an accurate list seems better than one that specifies (e.g.) 
distutils, ensurepip, resource or termios when those are absent.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7JBPSATSJMONLAGEU5PKTJHZ72MFRXBK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The SC is accepting PEP 632: deprecating distutils

[Steve Dower]

On 1/21/2021 6:30 PM, Brett Cannon wrote:
On behalf of the SC, I'm happy to announce that we have chosen to accept 
PEP 632. Congrats, Steve, and thanks for the work on the PEP!


I'll let Steve outline what the next steps are for implementing the PEP.


Thanks. Work from this point on will be tracked at 
https://bugs.python.org/issue41282


The next immediate step is finishing off 
https://github.com/python/cpython/pull/23142 to do the biggest task of 
bringing sysconfig up to compatibility with distutils.sysconfig. Once 
merged, we'll add the deprecation warning on import 
(Lib/distutils/__init__.py) and close all the current issues in the 
tracker (and any associated PRs). That should be done before 3.10 Beta 1.


From there, I'll spend time publicising the deprecation as much as 
possible (and everyone is welcome to help out) to find enough cases for 
retaining functionality that we can make sensible decisions on where to 
move things. From discussions so far, most of the functionality that 
people expect is well covered by either setuptools or the standard 
library. The rest, officially, should be copied into your own project as 
needed.


Eventually, my plan is to move distutils into our Tools folder and 
update the setup.py we use for building stdlib extension modules (and 
the PEG parser tests) to point at it there. However, if someone else 
updates our build system to do it all through the Makefile, that won't 
be necessary, and I know a few people were keen to do it. I'm not 
*requiring* that it be done, and we have an easy fallback plan, but I'll 
happily *encourage* anyone who wants to update the Makefile to go ahead.


One thing I do *not* intend to do is go through and thoroughly document 
everything that's being deleted. Some have asked or implied that this 
should be done, but it's just not going to happen. Everything beyond the 
package build support and sysconfig is helper functions, and mostly 
pretty rough ones to be honest. There are almost certainly 1st or 3rd 
party modules with better quality implementations, and if not then I 
expect there will be (and if you're one of the really concerned people, 
the best thing you can do to help now that the PEP is accepted is to 
create a new library with these helpers).


Feel free to raise any questions here or on the bug linked above.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PD2XJQTNSOA4HNCL3ZHOKPFAC3N4P5M2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enhancement request for PyUnicode proxies

[Steve Dower]

On 12/29/2020 5:23 PM, Antoine Pitrou wrote:

The third option is to add a distinct "string view" protocol.  There
are peculiarities (such as the fact that different objects may have
different internal representations - some utf8, some utf16...) that
make the buffer protocol suboptimal for this.

Also, we probably don't want unicode-like objects to start being usable
in contexts where a buffer-like object is required (such as writing to
a binary file, or zlib-compressing a bunch of bytes).


I've had to deal with this problem in the past as well (WinRT HSTRINGs), 
and this is the approach that would seem to make the most sense to me.


Basically, reintroduce PyString_* APIs as an _abstract_ interface to 
str-like objects.


So the first line of every single one can be PyUnicode_Check() followed 
by calling the _concrete_ PyUnicode_* implementation. And then we 
develop additional type slots or whatever is necessary for someone to 
build an equivalent native object.


Most "is this a str" checks can become PyString_Check, provided all the 
APIs used against the object are abstract (PyObject_* or PyString_*). 
Those that are going to mess with internals will have to get special 
treatment.


I don't want to make it all sound too easy, because it probably won't 
be. But it should be possible to add a viable proxy layer as a set of 
abstract C APIs to use instead of the concrete ones.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TC3BZJX4DGC2WV32AHIX7A57HQNJ2EMO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Where is the SQLite module maintainer

[Steve Dower]

On 12/28/2020 11:32 AM, Erlend Aasland wrote:
On 27 Dec 2020, at 22:38, Christian Heimes > wrote:

How about you put your name in the expert index instead of him? :)


Thanks for your confidence, but I'm far from an expert :)


Neither is anyone else :)

I'm not looking to start any mentoring right now, but if someone else is 
and you're interested, it should be easy to get you connected. I am more 
than happy to endorse you as a good candidate for becoming our SQLite 
expert.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AM7AVQH6L4AQVTB5UFOBYXICDPXYZBKQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Advice / RTFM needed for tool setup to participate in python development from a Windows host

[Steve Dower]

On 16Dec2020 2114, pjfarl...@earthlink.net wrote:

If anyone has or knows of step-by-step instructions on how to set that debug 
environment up and start the outer-level script with debug breakpoints in the 
DLL I would greatly appreciate it.  I'm also doing my own searches for 
tutorials on debugging python with VS20xx, but have not read/viewed one of 
those yet.


Hi Peter

Hopefully this document is able to help you get set up: 
https://docs.microsoft.com/en-us/visualstudio/python/debugging-mixed-mode-c-cpp-python-in-visual-studio?view=vs-2019


It gives you a few options, depending on whether you've created a Python 
project, a C++ project, or are attaching to a running process. Hopefully 
one of those will suit you. There are some videos out there as well, but 
they're a little older. (If you don't see Python projects, you'll need 
to go back to the Visual Studio installer, select the Python workload 
and the Python Native Development option.)


If you run into problems, there's a "Report a problem" (under Help/Send 
Feedback) inside VS that will get directly to the team, though most 
people are on vacation for the next few weeks. (And yes, "the official 
docs aren't good enough for me to follow" is a problem ;) )


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5WV7VN3TSIZ5SAPMDLMIWQBMEWRT6OMU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Who is target reader of tutorial?

[Steve Dower]

Hopefully that was a dataclass :)

But yes, point taken. Classes need to be there. And now I've gone and 
re-read the table of contents for the tutorial, I really don't have any 
complaints about the high-level ordering. It does seem to go 
unnecessarily deep in some areas (*very* few people will ever need 
position-only parameters, for example, and I'd say all special 
parameters matter more in the tutorial because of how you _call_ them, 
rather than how you define them).


Cheers,
Steve

On 06Nov2020 1714, Guido van Rossum wrote:
I agree with you that the tutorial should focus at users, not library 
developers. But assuming that users will never write a class seems 
wrong. For example, while ago I went through a PyTorch tutorial, which 
assumes barely any programming knowledge, and yet the first or second 
example has the user write a class, as this is apparently the 
conventional way to store parameters for ML models.


--Guido

On Fri, Nov 6, 2020 at 8:32 AM Steve Dower <mailto:steve.do...@python.org>> wrote:


It would also be nice for the tutorial to separate between "things you
need to know to use Python" vs "things you need to write a Python
library".

For example, the fact that operators can do different things for
different values (e.g. int, str, list, pathlib) would be in the first
category, while the details of how to override operators can wait for
the second.

I see many people suffer from content that goes too deep too quickly,
and I'm more and more convinced over time that this is the right place
to draw a separator for Python. Many devs are just using the language
and never implementing a class (or often, even writing a function).
Having a canonical tutorial to get these users through this stage first
before going deeper would be great.

Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VVJOMFE44ZO6UTA432EQGPSHZI23ULQU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Who is target reader of tutorial?

[Steve Dower]

It would also be nice for the tutorial to separate between "things you 
need to know to use Python" vs "things you need to write a Python library".


For example, the fact that operators can do different things for 
different values (e.g. int, str, list, pathlib) would be in the first 
category, while the details of how to override operators can wait for 
the second.


I see many people suffer from content that goes too deep too quickly, 
and I'm more and more convinced over time that this is the right place 
to draw a separator for Python. Many devs are just using the language 
and never implementing a class (or often, even writing a function). 
Having a canonical tutorial to get these users through this stage first 
before going deeper would be great.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E7Y5MB4JJEB3VW2J24HA7JQZH6JRAOMO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Steve Dower]

On 22Oct2020 1341, Marco Sulla wrote:
On Thu, 22 Oct 2020 at 14:25, Mark Shannon > wrote:


MSVC seems to do better jump fusion than GCC.


Maybe I'm wrong, since I only take a look at dict, tuple and set C code, 
but it does not seems to me that there's more than 1-2 GOTOs in every 
CPython function, and they can't be merged.


There are vastly more jumps generated than what you see in the source 
code. You'll need to compare assembly language to get a proper read on this.


But I don't think that's necessary, since processors do other kinds of 
clever things with jumps anyway, that can variously improve/degrade 
performance from what the compilers generate.


Benchmarks on consistent hardware are what matter, not speculation about 
generated code.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JEEZNIP4TPLIA2ZS3QIRWZGXBKPDOVBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 640: Unused variable syntax.

[Steve Dower]

On 20Oct2020 1309, Thomas Wouters wrote:
The reason for this PEP is that pattern matching will make '_' (but not 
any other names) have the behaviour suggested in this PEP, but *only* in 
pattern matching.


Then why is this PEP proposing a different syntax?

At the very least, wait for pattern matching to get in before proposing 
an expansion, so then you won't be caught out suggesting the wrong thing :)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RXIACSYMIP22ILGYYFVPKHIUH62N7233/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 640: Unused variable syntax.

[Steve Dower]

On 20Oct2020 1021, Steven D'Aprano wrote:

In my opinion, having a convention to treat certain variables as
"unused" is great (I'm partial to `__` myself, to avoid clobbering the
special variable `_` in the REPL). But having that be a pseudo-variable
which is *actually* unused and unuseable strikes me as being an
attractive nuisance.


I agree entirely. The convention is fine, and the workaround for when 
you don't want to overwrite a legitimate `_` variable is also fine.


Making `_` enough of an official convention (but how?) that linting 
tools stop warning about it (e.g. type checkers might warn about 
multiple conflicting assignments) seems like an overall happier path, 
that neither makes existing code forwards-incompatible nor future code 
backwards-incompatible.


I don't think we have to codify every emergent coding pattern in syntax.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6GV3KPPPRNF5ISZK4YSAIUUTCQRMX77H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Steve Dower]

On 20Oct2020 0520, Rob Cliffe wrote:

On 19/10/2020 12:42, Steve Dower wrote:

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
TLDR: In os.scandir directory entries, atime is always a copy of 
mtime rather than the actual access time.


Correction - os.stat() updates the access time to _now_, while 
os.scandir() returns the last access time without updating it.


Eryk replied with a deeper explanation of the cause, but fundamentally 
this is what you are seeing.


Feel free to file a bug, but we'll likely only add a vague note to the 
docs about how Windows works here rather than changing anything. If 
anything, we should probably fix os.stat() to avoid updating the 
access time so that both functions behave the same, but that might be 
too complicated.


Cheers,
Steve

Sorry - what you say does not match the behaviour I observe, which is that


Yes, I posted a correction already (immediately after sending the first 
email).


What you are seeing is what Windows decided was the best approach. If 
you want to avoid that, os.stat() will get the latest available 
information. But I don't want to penalise people who don't need it by 
slowing down their scandir calls unnecessarily.


A documentation patch to make this difference between os.stat() and 
DirEntry even clearer would be fine.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NAR7LTW2XMBKAPKLVBQQFVK6EA4ZWQZP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Steve Dower]

On 19Oct2020 1846, Eryk Sun wrote:

On 10/19/20, Steve Dower  wrote:

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:

TLDR: In os.scandir directory entries, atime is always a copy of mtime
rather than the actual access time.


Correction - os.stat() updates the access time to _now_, while
os.scandir() returns the last access time without updating it.


os.stat() shouldn't affect st_atime because it doesn't access the file
data. That has me curious if it can be reproduced.

With NTFS in Windows 10, I'd expect the os.stat() st_atime to change
immediately when the file data is read or modified. With other
filesystems, it may not be updated until the kernel file object that
was used to access the file's data is closed.


I thought I got my self-correction fired off quickly enough to save you 
from writing this :)



For details, download the [MS-FSA] PDF [1] and look for all references
to the following sections:



[1] 
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/860b1516-c452-47b4-bdbc-625d344e2041


Thanks for the detailed reference.


Going back to my initial message, I can't stress enough that this
problem is at its worst when a file has multiple hardlinks. If a
particular link in a directory wasn't the last link used to access the
file, its duplicated metadata may have the wrong file size, access
time, modify time, and change time (the latter is not reported by
Python). As is, for the current implementation, I'd only rely on the
basic attributes such as whether it's a directory or reparse point
(symlink, mountpoint, etc) when using scandir() to quickly process a
directory. For reliable stat information, call os.stat().

I do think, however, that os.scandir() can be improved in Windows
without significant performance loss if it calls GetFileAttributesExW
to get st_file_attributes, st_size, st_ctime (create time), st_mtime,
and st_atime. This API call is relatively fast because it doesn't
require opening the file via CreateFileW, which is one of the more
expensive operations in os.stat(). But I haven't tried modifying
scandir() to benchmark it.


Resolving the path is the most expensive part, even if the file is not 
opened (I've been working with the NTFS team on this area, and we've 
been benchmarking/analysing all of it). There are a few improvements 
coming across the board, but I'd much rather just emphasise that 
os.scandir() is as fast as we can manage using cached information 
(including as cached by the OS). Otherwise we prevent people from using 
the fastest available option when they can, if they don't need the 
additional information/accuracy.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MMRMLWGEV2ZGIACXQTSEQC6TPWGL3UZ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Steve Dower]

On 19Oct2020 1652, Gregory P. Smith wrote:
I'm sure this is covered in MSDN.  Linking to that if it has it in a 
concise explanation would make sense from a note in our docs.


Probably unlikely :) I'm pretty sure this started "perfect" and was then 
wound back to improve performance. But it's almost certainly an option 
somewhere, which means you can't rely on it being either true nor false. 
You just have to be explicit for certain pieces of information.


If I'm understanding Steve correctly this is due to Windows/NTFS storing 
the access time potentially redundantly in two different places. One 
within the directory entry itself and one with the file's own metadata.  
Those of us with a traditional posix filesystem background may raise 
eyeballs at this duplication, seeing a directory as a place that merely 
maps names to inodes with the inode structure (equiv: file entry 
metadata) being the sole source of truth.  Which ones get updated when 
and by what actions is up to the OS.


So yes, just document the "quirk" as an intended OS behavior.  This is 
one reason scandir() can return additional information on windows vs 
what it can return on posix.  The entire point of scandir() is to return 
as much as possible from the directory without triggering reads of the 
inodes/file-entry-metadata. :)


Yeah, I'd document it as a quirk of scandir. There's also a race where 
if you scandir(), then someone touches the file, then you look at the 
cached stat you get the wrong information too (an any platform). Making 
clearer that it's for non-time sensitive queries is most accurate, 
though we could also give an example of "access times may not be up to 
date depending on OS-level caching" without committing us to being 
responsible for OS decisions.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EBWUDEQEPRWJN36FLUUJQWP5EWLPWRPD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Steve Dower]

On 19Oct2020 1242, Steve Dower wrote:

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
TLDR: In os.scandir directory entries, atime is always a copy of mtime 
rather than the actual access time.


Correction - os.stat() updates the access time to _now_, while 
os.scandir() returns the last access time without updating it.


Let me correct myself first :)

*Windows* has decided not to update file access time metadata *in 
directory entries* on reads. os.stat() always[1] looks at the file entry 
metadata, while os.scandir() always looks at the directory entry metadata.


My suggested approach still applies, other than the bit where we might 
fix os.stat(). The best we can do is regress os.scandir() to have 
similarly poor performance, but the best *you* can do is use os.stat() 
for accurate timings when files might be being modified while your 
program is running, and don't do it when you just need names/kinds (and 
I'm okay adding that note to the docs).


Cheers,
Steve

[1]: With some fallback to directory entries in exceptional cases that 
don't apply here.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QHHJFYEDBANW7EC3JOUFE7BQRT5ILL4O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Steve Dower]

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
TLDR: In os.scandir directory entries, atime is always a copy of mtime 
rather than the actual access time.


Correction - os.stat() updates the access time to _now_, while 
os.scandir() returns the last access time without updating it.


Eryk replied with a deeper explanation of the cause, but fundamentally 
this is what you are seeing.


Feel free to file a bug, but we'll likely only add a vague note to the 
docs about how Windows works here rather than changing anything. If 
anything, we should probably fix os.stat() to avoid updating the access 
time so that both functions behave the same, but that might be too 
complicated.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NGMVB7GWDBCPYHL4IND2LBZ2QPXLWRAX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 632: Deprecate distutils module

[Steve Dower]

On 9/11/2020 12:24 PM, Matthias Klose wrote:

On 9/4/20 1:28 PM, Steve Dower wrote:

Hi all.

setuptools has recently adopted the entire codebase of the distutils module, so
that they will be able to make improvements directly without having to rely on
patching the standard library. As a result, we can now move forward with
official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils
docs already recommend switching to setuptools).


it would be nice to already announce that with the 3.9 release.


I think the announcement will have an immediate effect regardless of 
what versions it's tied to.



At the 2018 Language summit, I had a lightning talk to report about the
experience splitting out distutils into a separate binary package, showing some
unexpected usages:

Unexpected / Creative usages:

  - distutils.version
Used “everywhere” ...

  - distutils.spawn: find_executable
Replace with shutil.which

  - distutils.util: strtobool
Rewrite, no equivalent in the stdlib?

  - distutils.sysconfig:
Mostly replaced by sysconfig


(Aside - I've put out a call on Discourse for people to contribute the 
parts of distutils that are used but wouldn't be suitable for using 
setuptools instead. These are good, but don't seem drastic enough to not 
deprecate the module. Thoughts?)



It really would be nice to have recommended replacements, especially for the
version stuff (packaging?)


Considering that the PEP won't be updated after acceptance, and all of 
these recommendations would be to third-party packages (except possibly 
strtobool), how relevant would you predict these to be in three years time?


Setuptools has already been the official recommendation for the whole 
module for a long time, and their page at 
https://setuptools.readthedocs.io/en/latest/distutils-legacy.html 
already lists a few replacements (and I'm sure they'll accept 
contributions :) ). So I'd prefer to point towards that as the most 
up-to-date source of recommendations. Thoughts?


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HDCK3LFOZWDNWLH3NZ2QFNYUPNJAGNP5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 632: Deprecate distutils module

[Steve Dower]

On 07Sep2020 1602, Stefan Krah wrote:

I'm under the impression that distutils has effectively been frozen for
the last decade, except for the substantial improvements you made for the
MSVC part.

For Unix, no one has addressed e.g. C++ support. The underlying reason
has always been that we cannot change anything lest it breaks someone's
package.


So I have some trouble understanding why we have exercised that kind
of restraint (I might have committed a very small C++ patch otherwise)
if we can now break everything at once.


Others have contributed a range of little fixes over time. Maybe they 
blur into the background noise of regular maintenance, but it does add up.


The main reason we end up needing to fix things is because platforms 
change things. Since we don't control _when_ platforms will change 
things, we either do a single controlled break at a planned Python 
version, or we allow it to break for different users at different times 
that are not detectable from version number alone. It won't take long 
for an autoconf-style approach to be necessary for figuring out how to 
use distutils (though hopefully anyone who sees that as the solution 
will also find https://packaging.python.org/ and find better solutions ;) )



Unless you're offering to take over distutils maintenance? In which case,
I'll happily withdraw the PEP :)


No, thanks. :)


Okay, maybe it is a little bit more than background noise ;)


I've looked at the log, most maintenance patches are from a variety of
active core devs (this includes, of course, the MSVC patches from you).

Will they submit patches to setuptools from now on?


If you look at the setuptools history, a variety of contributors have 
submitted much the same (and often better) fixes there. I expect I 
likely will as well, inasmuch as they are needed to help Windows users 
be successful, though I also have my own backend that will likely be 
where big features that interest me end up ;)


It's easier and more satisfying to submit the patches to setuptools, as 
the release comes out much sooner (and can be installed on earlier 
versions), and it will only get easier when _all_ the code can be fixed 
there. Right now, one of the biggest pains there is having to do loads 
of careful monkeypatching to fix one poor choice in the standard library.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7MESZKL5KFX6YBPOZNYTE5PS4PE5OJZM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 632: Deprecate distutils module

[Steve Dower]

On 07Sep2020 1424, Stefan Krah wrote:

On Mon, Sep 07, 2020 at 11:43:41AM +0100, Steve Dower wrote:

Rest assured, I am very aware of air-gapped and limited network systems, as
well as corporate policies and processes. Having distutils in the standard
library does not help with any of these.


Of course it helps.  You can develop extensions without any third party
packages or install them.

Same situation if you are on mobile Internet or in a cafe without Internet
and you want to try something out.


Or if you moved and you don't have cable Internet yet.  Or if you are in
a country where there is no cable Internet.


Air-gapped systems were just an illustration of the problem. I did not
anticipate that people would take it as the centerpiece of my arguments.


Sorry, I didn't mean to imply that - I used "limited network" to capture 
the rest. I make exactly the same arguments at work from your side for 
many other situations, so if you'd like I can give you an even more 
exhaustive list of scenarios where we can't rely on highly available 
internet :)


And yet despite that, I think in this case you're clutching at straws. 
Sure, you can develop extensions without any third party packages, but 
you can't develop them with *supported* first party packages.


Unless you're offering to take over distutils maintenance? In which 
case, I'll happily withdraw the PEP :)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2HBDAXQA6HQF4QKSKAVUZLXZXSZ5SU75/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 632: Deprecate distutils module

[Steve Dower]

Thanks everyone for your input so far.

Rest assured, I am very aware of air-gapped and limited network systems, 
as well as corporate policies and processes. Having distutils in the 
standard library does not help with any of these.


Do not forget that pip (and presently, though not permanently, 
setuptools) are bundled with a recommended CPython distribution through 
ensurepip. I call out "recommended" because distributors (including 
those who are transferring CPython into an air-gapped system) can choose 
to do whatever they like - including leaving out distutils already!


As for defining standards for installation, the sysconfig module 
specifies the paths to install things (though even those can be taken 
with a grain of salt, as it's really the import system and site module 
that determine where things should be installed), but we already defer 
to third parties to do the actual installation (including, yes, those 
who use distutils to help define their install script). All other 
packaging and distribution specifications are listed at 
https://packaging.python.org/specifications/


Also, distutils has never been a "standard" for how to define a package 
(i.e. write a setup.py), just as argparse is not a "standard" for how to 
define a CLI. It's always been possible to do it in other ways, and the 
current standard definition for this is now PEP 517. Since distutils is 
not compatible with PEP 517, it is explicitly *breaking* the standard.


I also wanted to call out this excellent point from Emily:


If you can update to a breaking Python version, but aren't allowed one single 
point version of an external module, you have a process problem.


This is absolutely the case. CPython, and the core team, cannot take 
responsibility for bad policies, and (arguably) should not help users 
work around those policies. It is better to provide useful advice to 
organisations implementing these policies to help them do things 
properly (this is a large part of what I get paid to do these days), 
because otherwise they very quickly just dictate that Python cannot be 
used at all.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V2JKVPEDQPJJHF5W4MJRER5FBH3I4WLU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 632: Deprecate distutils module

[Steve Dower]

Hi all.

setuptools has recently adopted the entire codebase of the distutils 
module, so that they will be able to make improvements directly without 
having to rely on patching the standard library. As a result, we can now 
move forward with official deprecation (in 3.10) and removal (in 3.12) 
(noting that the distutils docs already recommend switching to setuptools).


Full text and discussion at 
https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134


I'm including the original text below, but won't be responding to all 
discussions here (though I'll periodically check in and skim read it, 
assuming things don't go way off track).


Also be aware that I already have some minor changes lined up that are 
not in this text. Refer to the discussion on Discourse if you need to 
see those.


Cheers,
Steve

---

PEP: 632
Title: Deprecate distutils module
Author: Steve Dower 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Sep-2020
Post-History:


Abstract


The distutils module [1]_ has for a long time recommended using the
setuptools package [2]_ instead. Setuptools has recently integrated a
complete copy of distutils and is no longer dependent on the standard
library [3]_. Pip has silently replaced distutils with setuptools when
building packages for a long time already. It is time to remove it
from the (public part of the) standard library.


Motivation
==

distutils [1]_ is a largely undocumented and unmaintained collection
of utilities for packaging and distributing Python packages, including
compilation of native extension modules. It defines a configuration
format that describes a Python distribution and provides the tools to
convert a directory of source code into a source distribution, and
some forms of binary distribution. Because of its place in the
standard library, many updates can only be released with a major
release, and users cannot rely on particular fixes being available.

setuptools [2]_ is a better documented and well maintained enhancement
based on distutils. While it provides very similar functionality, it
is much better able to support users on earlier Python releases, and
can respond to bug reports more quickly. A number of platform-specific
enhancements already exist in setuptools that have not been added to
distutils, and there is been a long-standing recommendation in the
distutils documentation to prefer setuptools.

Historically, setuptools has extended distutils using subclassing and
monkeypatching, but has now taken a copy of the underlying code. [3]_
As a result, the second last major dependency on distutils is gone and
there is no need to keep it in the standard library.

The final dependency on distutils is CPython itself, which uses it to
build the native extension modules in the standard library (except on
Windows). Because this is a CPython build-time dependency, it is
possible to continue to use distutils for this specific case without
it being part of the standard library.

Deprecation and removal will make it obvious that issues should be
fixed in the setuptools project, and will reduce a source of bug
reports and test maintenance that is unnecessary. It will also help
promote the development of alternative build backends, which can now
be supported more easily thanks to PEP 517.


Specification
=

In Python 3.10 and 3.11, distutils will be formally marked as
deprecated. All known issues will be closed at this time.
``import distutils`` will raise a deprecation warning.

During Python 3.10 and 3.11, uses of distutils within the standard
library may change to use alternative APIs.

In Python 3.12, distutils will no longer be installed by ``make
install`` or any of the first-party distribution. Third-party
redistributors should no longer include distutils in their bundles or
repositories.

This PEP makes no specification on migrating the parts of the CPython
build process that currently use distutils. Depending on
contributions, this migration may occur at any time.

After Python 3.12 is started and when the CPython build process no
longer depends on distutils being in the standard library, the entire
``Lib/distutils`` directory and ``Lib/test/test_distutils.py`` file
will be removed from the repository.

Other references to distutils will be cleaned up. As of Python 3.9's
initial release, the following modules have references in code or
comments:

* Lib/ctypes/util.py
* Lib/site.py
* Lib/sysconfig.py
* Lib/_aix_support.py
* Lib/_bootsubprocess.py
* Lib/_osx_support.py
* Modules/_decimal/tests/formathelper.py

As the distutils code is already included in setuptools, there is no
need to republish it in any other form. Those who require access to
the functionality should use setuptools or an alternative build
backend.

Backwards Compatibility
===

Code that imports distutils will no longer work from Python 3.12.

The suggested migration path is to use the equivalent (though

[Python-Dev] Re: Procedure for trivial PRs

[Steve Dower]

On 8/13/2020 9:56 PM, Mariatta wrote:

  a) is it ok to touch 3.9, as it's in rc1?


Yeah bug fixes are accepted to the maintenance branches. I think your PR 
does count as documentation bug fix, so it should be ok to backport to 3.9


At this stage, changes to the 3.9 branch won't go into the 3.9.0 release 
unless you specifically ask Łukasz to include them (though he does have 
a habit of acting out of his own initiative and including them anyway :) ).


Which means you can freely merge anything you like for 3.9.1, provided 
it's okay for a maintenance branch (in your judgement, which is part of 
being a core dev, but feel free to ask others for advice if you're not 
sure).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5RKZ5CP2SSCD43TUZLHN2F72Z7X4XMYX/
Code of Conduct: http://python.org/psf/codeofconduct/