[Python-Dev] Re: Retiring this mailing list ?

[Eric V. Smith via Python-Dev]

On 11/14/2023 1:21 PM, Steve Holden wrote:

On Mon, 13 Nov 2023 at 10:18, Marc-Andre Lemburg  wrote:

[...]

Question: Should we retire and archive this mailing list ?
(I'm asking as one of the maintainers of the ML)

[...]

Hi Marc-Andre,

Maybe just require senders to be members of the python.org 
 domain, and retain the release announcements?


I think the python-announce list serves that purpose. Any time there's 
an announcement here, I also see a separate copy of it on 
python-announce (where I'm a moderator).


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SHV5KJIBK67IYSCPTFDO5HEQV6YKM76O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Fwd: Python 3.11 performance with frame pointers

[Gregory P. Smith]

I suggest re-posting this on discuss.python.org as more engaged active core
devs will pay attention to it there.

On Wed, Jan 4, 2023 at 11:12 AM Daan De Meyer 
wrote:

> Hi,
>
> As part of the proposal to enable frame pointers by default in Fedora
> (https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer), we
> did some benchmarking to figure out the expected performance impact.
> The performance impact was generally minimal, except for the
> pyperformance benchmark suite where we noticed a more substantial
> difference between a system built with frame pointers and a system
> built without frame pointers. The results can be found here:
> https://github.com/DaanDeMeyer/fpbench (look at the mean difference
> column for the pyperformance results where the percentage is the
> slowdown compared to a system built without frame pointers). One of
> the biggest slowdowns was on the scimark_sparse_mat_mult benchmark
> which slowed down 9.5% when the system (including python) was built
> with frame pointers. Note that these benchmarks were run against
> Python 3.11 on a Fedora 37 x86_64 system (one built with frame
> pointers, another built without frame pointers). The system used to
> run the benchmarks was an Amazon EC2 machine.
>
> We did look a bit into the reasons behind this slowdown. I'll quote
> the investigation by Andrii on the Fesco issue thread here
> (https://pagure.io/fesco/issue/2817):
>
> > So I did look a bit at Python with and without frame pointers trying to
> > understand pyperformance > regressions.
>
> > First, perf data suggests that big chunk of CPU is spent in
> _PyEval_EvalFrameDefault,
> > so I looked specifically  into it (also we had to use DWARF mode for
> perf for apples-to-apples
> > comparison, and a bunch of stack traces weren't symbolized properly,
> which just again
> > reminds why having frame pointers is important).
>
> > perf annotation of _PyEval_EvalFrameDefault didn't show any obvious hot
> spots, the work
> > seemed to be distributed pretty similarly with or without frame
> pointers. Also scrolling through
> > _PyEval_EvalFrameDefault disassembly also showed that instruction
> patterns between fp
> > and no-fp versions are very similar.
>
> > But just a few interesting observations.
>
> > The size of _PyEval_EvalFrameDefault function specifically (and all the
> other functions didn't
> > change much in that regard) increased very significantly from 46104 to
> 53592 bytes, which is a
> > considerable 15% increase. Looking deeper, I believe it's all due to
> more stack spills and
> > reloads due to one less register available to keep local variables in
> registers instead of on the stack.
>
> > Looking at _PyEval_EvalFrameDefault C code, it is a humongous one
> function with gigantic switch
> > statement that implements Python instruction handling logic. So the
> function itself is big and it has
> > a lot of local state in different branches, which to me explained why
> there is so much stack spill/load.
>
> > Grepping for instruction of the form mov -0xf0(%rbp),%rcx or mov
> 0x50(%rsp),%r10 (and their reverse
> > variants), I see that there is a substantial amount of stack spill/load
> in _PyEval_EvalFrameDefault
> > disassembly already in default no frame pointer variant (1870 out of
> 11181 total instructions in that
> > function, 16.7%), and it just increases further in frame pointer version
> (2341 out of 11733 instructions, 20%).
>
> > One more interesting observation. With no frame pointers, GCC generates
> stack accesses using %rsp
> > with small positive offsets, which results in pretty compact binary
> instruction representation, e.g.:
>
> > 0x001cce40 <+44160>: 4c 8b 54 24 50  mov
> 0x50(%rsp),%r10
>
> > This uses 5 bytes. But if frame pointers are enabled, GCC switches to
> using %rbp-relative offsets,
> > which are all negative. And that seems to result in much bigger
> instructions, taking now 7 bytes instead of 5:
>
> > 0x001d3969 <+53065>: 48 8b 8d 10 ff ff ffmov
> -0xf0(%rbp),%rcx
>
> > I found it pretty interesting. I'd imagine GCC should be capable to keep
> using %rsp addressing just fine
> > regardless of %rbp and save on instruction sizes, but apparently it
> doesn't. Not sure why. But this instruction
> > increase, coupled with increase of number of spills/reloads, actually
> explains huge increase in byte size of
> > _PyEval_EvalFrameDefault: (2341 - 1870) * 7 + 1870 * 2 = 7037 (2 extra
> bytes for existing 1870 instructions
> > that were switched from %rsp+positive offset to %rbp + negative offset,
> plus 7 bytes for each of new 471 instructions).
> > I'm no compiler expert, but it would be nice for someone from GCC
> community to check this as well (please CC
> > relevant folks, if you know them).
>
> > In summary, to put it bluntly, there is just more work to do for CPU
> saving/restoring state to/from stack. But I don't
> > think _PyEval_EvalFrameDefault example is typical of how application
> code is written, 

[Python-Dev] Re: Switching to Discourse

[Gregory P. Smith]

On Thu, Dec 1, 2022 at 8:37 AM Victor Stinner  wrote:

>
> Should we *close* the python-dev mailing list?
>

I'd be in favor of this. Or at least setting up an auto-responder
suggesting people post on discuss.python.org instead.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JMVE4S3LGMWDLJW2Z5T73K6Z23IZLHYQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC: expose ssl internals for use by ctypes/cffi

[Gregory P. Smith]

On Wed, Nov 30, 2022 at 12:47 PM Steve Dower  wrote:

> On 11/30/2022 4:52 PM, chris...@weinigel.se wrote:
> > Does this seem like a good idea?  As I said, I feel that it is a bit
> ugly, but it does mean that if someone wants to use some
> SSL_really_obscure_function in libcrypto or libssl they can do that without
> having to rebuild all of CPython themselves.
>
> Broadly, no, I don't think it's a good idea. We don't like encouraging
> users to do things that make it hard to support them in the future.
>
> Nonetheless, it's one that I've had to do, and so realistically I think
> it's okay to *enable* the hack without endorsing it. This is one of the
> reasons I switched the Windows builds to dynamically linked OpenSSL
> builds (they used to be statically linked, which meant there was no way
> to get at the unused exports). So now you can use `import _ssl;
> ctypes.CDLL("libssl-1_1")` to get at other exports from the module if
> you need them, and there's a similar trick to get the raw handle that I
> don't recall off the top of my head.
>
> But the only reason I'd ever want to document this is to tell people not
> to rely on it. If you control your environment well enough that you can
> guarantee it'll work for you, that's great. Nobody else should ever
> think they're doing the "right thing".
>

+1 ... and in general if you want access to other OpenSSL APIs not already
in the ssl module, getting them via non-stdlib packages on PyPI would be a
better idea.

https://pypi.org/project/cryptography/ is very well supported.
https://pypi.org/project/oscrypto/ exists and is quite interesting.
the old https://pypi.org/project/M2Crypto/ package still exists and seems
to be maintained (wow).

More context: We don't like the ssl module in the standard library - it is
already too tightly tied to OpenSSL:
https://discuss.python.org/t/our-future-with-openssl/21486

So if you want specific OpenSSL APIs that are not exposed, seeking to see
them added to the standard library where they would then become features
that need to be supported for a very long time, is going to be the most
difficult approach as there'd need to be a very good reason to have them in
the stdlib. Third party libraries that can provide what you need, or
rolling your own libssl API wrappings however you choose to implement them,
are better bets.

-Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5E62GYXUJLTPQWJAPEOLQDHY3F2IDRER/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [CVE-2022-37454] SHA3 vulnerability and upcoming Python patches for 3.7 - 3.10

[Gregory P. Smith]

The patches to 3.6-3.10 have been merged, which means they will go out in
the next Python patch release for those updates. ie:
https://github.com/python/cpython/issues/98517

You can see the planned schedule of those on
https://peps.python.org/pep-0619/ and related similar peps for older
python versions (i never remember pep numbers, i just google for "python
3.8 release schedule" to get to such a PEP myself). For 3.9 and earlier
which are in "as-needed" security mode and no longer ship binaries I expect
we'll get to it within the next month. The patches are public and anyone
can apply them on their own. We've got other fixes for other issues coming,
so rolling one source only security release for every single issue isn't a
great use of time.

I personally didn't feel this one was urgent enough to ask anyone to spend
time doing an emergency security release as triggering the crash requires
someone sending a multi-gigabyte amount of data into a sha3 hash function
in a single .update() method call. That seems like a rare code pattern. How
many applications ever do that vs doing I/O in smaller chunks with more
frequent .update() calls?

Linux or other OS distros that bundle Python manage their own patches to
their runtimes and their security folks may need to be reminded to pick
these patches up if they have not already done so. Their schedules are
independent, they're likely to rapidly patch anything they agree with as
being a security issue. Distros are also likely the only ones who would
apply a backport to 3.6. Anything shipping Python <=3.8 or earlier *should*
have an interest in patching.

fwiw the forum I would've used for this is the github issue itself as that
is the canonical discussion.

-gps

On Mon, Nov 7, 2022 at 11:52 AM mark_topham--- via Python-Dev <
python-dev@python.org> wrote:

> I’m looking for help understanding how Python will release fixes related
> to the SHA3 critical security vulnerability (CVE-2022-37454).  I’ve tried
> to figure this out myself, but I’m far from a Python expert and I’m not
> sure where else I should look.
>
> Apologies in advance if this is the wrong place to ask - if it is, a
> redirect to the correct place would be most appreciated.
>
> Here’s what I’ve found so far:
>
> * Python versions 3.6 through 3.10 appear to be affected
>* 3.6 is end of life, so no fix is expected
>* A code fix appears to have been applied to 3.7 through 3.10
> https://github.com/python/cpython/issues/98517
>* 3.9 and 3.10 by default use OpenSSL1.1.1+ if it’s available,
> appearing to default to the builtin, vulnerable SHA3 implementation if
> OpenSSL is not found (if there’s an exception)
>   * 3.9 introduced this change via bpo-37630 in release 3.9.0 beta1
>   * 3.10 appears to have had this functionality since it was
> originally released
> * 3.11 uses tiny_sha3 and AFAICT was never affected by the CVE
>
> But what I’m having trouble figuring out is when/how these fixes will
> become generally available and ready for users of Python to download.
>
>
> * When will patched releases for Python 3.7-3.10 be released?
> * If pending releases are not in the release pipeline, what other patching
> opportunities exist?
>
> Ultimately I’m trying to set patching expectations for my company’s
> engineering teams who are still running vulnerable versions of Python.
>
> More notes around what i’ve found, in case it helps clarify my questions:
> From the Python project GitHub I can see gh-98517 to fix the buffer
> overflow in Python’s internal _sha3 module (CVE-2022-37454) has been
> committed to the Python 3.7 - 3.10 branches.  I understand that for Python
> releases 3.9 and 3.10 if one is using the OpenSSL 1.1.1+ sha3 modules
> instead of the internal _sha3 module that is already a mitigation.  I also
> understand that Python 3.11 and later has switched to using tiny_sha3, and
> no longer relies on the vulnerable _sha3 module.
>
> Any information you could point me at would be most helpful.  If there is
> a more ideal forum to raise this question, please redirect me there.
>
> Thank you in advance
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/K7IYZUGOOLCGKZOLCZ27RSWZ7OWIP575/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2XGSIIHFMO4QPQB5ZXIER5DDSRF6VMQB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Embedded Python Memory Leaks

[Gregory P. Smith]

On Thu, Oct 6, 2022 at 12:48 PM  wrote:

> Hi Python team
>
> Is any plan to try and fix memory leaks when embedding Python in a C/C++
> application?
>

Please file issues on github with details if you find problems.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HE27VQ4CKLXSA3EUAQCN57LAABMUFWJ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Annotating pure functions to improve inline caching/optimization

[Eric V. Smith via Python-Dev]

You should bring this up on discuss.python.org. It's not going to see 
much if any discussion here.


Eric

On 9/14/2022 10:05 AM, Philipp Burch wrote:

Hello everyone,

the docs on the upcoming 3.11 release state

> This [specializing adaptive interpreter] also brings in another 
concept called inline caching, where Python caches the results of 
expensive operations directly in the bytecode.


I wonder how this caching works, given that the dynamic nature means 
that virtually every operation could have side effects, causing wrong 
behaviour when cached. The only mitigation for this that I can imagine 
is that caching just occurs for basic operations defined in the 
standard library, where it is known that they are free of side effects 
or "pure".


A web search did reveal some discussions[1,2] and a module related to 
dealing with pure functions, but, as far as I see, not related to 
optimization.


As an example, consider a code like this:

```py
@pure
def rot_factor(angle_deg: float) -> complex:
    # This could also be a much more expensive calculation.
    return cmath.exp(angle_deg / 180 * cmath.pi * 1j)

# ...

res: List[Tuple(complex, complex, complex, float)] = []
for x in many:
    res.append((
    x * rot_factor(90),
    x * rot_factor(45),
    x * rot_factor(-45),
    x * math.sin(math.pi/8),
    ))
```

The problem with this code is obvious, every loop iteration calls 
`rot_factor()` with 90, 45 and -45 and will get exactly the same set 
of results. The last factor might already be inline cached by the 
interpreter, since it probably knows that `math.pi` is a constant and 
`math.sin()` is a pure function. Optimizing this by hand (not 
considering a list comprehension or other more sophisticated 
improvements) is easy, but not very pretty:


```py
f_p90 = rot_factor(90)
f_p45 = rot_factor(45)
f_m45 = rot_factor(-45)
f_sin = math.sin(math.pi / 8)
res: List[Tuple(complex, complex, complex, float)] = []
for x in many:
    res.append((
    x * f_p90,
    x * f_p45,
    x * f_m45,
    x * f_sin,
    ))
```

I actually find myself often factoring such data out of loops in 
Python, whereas in C I would just leave that to the optimizer/compiler.


An available option would be to use `@lru_cache` for `rot_factor()`, 
but this will still cause the same dictionary lookups in every 
iteration and it may not work at all in case the function argument(s) 
is/are not hashable.


Now, if the interpreter understood the `@pure` decorator for 
`rot_factor()` indicated above would give it the same opportunity to 
cache the three results throughout the loop, basically creating the 
manually-optimized code above. For these completely static values, it 
could even precompute the results and integrate them into the bytecode.



Has anything like this been considered already, or is the interpreter 
itself capable to perform such optimizations?



Thanks and best regards,
Philipp



[1] 'pure' type annotation for mypy: 
https://github.com/python/mypy/issues/4468


[2] pure-func module: https://pypi.org/project/pure-func/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XDYZRC6L7LBPE3T6RO6S5IVY3J6IMRSJ/

Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A5CATJEKKNZN4XOWXVJ2ICK2Z4A3WFYU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install

[Gregory P. Smith]

-cc:help (bcc)

FWIW the thing to do to move this forward is to open a new PR. Patches
attached to an email are lost like tears in the rain.

What you describe of cross compilation where host and target triples appear
the same but use different libraries is a valid cross compilation case. But
doesn't surprise me that there are bugs given most people never have such a
setup. Even in an "every build should be a cross build" world (which we
sadly don't have) it being broken may not show up.

-gps

On Wed, Jun 15, 2022 at 8:28 AM Victor Stinner  wrote:

> Hi,
>
> While this problem is causing you a lot of troubles, I never had to
> cross compile Python, and I guess that it's the case for most people.
> Changing the Python build system and distutils is stressful since it
> can break Python for the majority of users, rather than leaving the
> minority of users with an old bug. So far, nobody was brave enough to
> "break" the system for cross compilation.
>
> Moreover, as far as I know, cross compiling Python works for the
> common case: different platform triplet. Only the corner case of the
> same triple is causing troubles.
>
> https://github.com/python/cpython/issues/66913 doesn't explain how to
> reproduce the issue, it only gives some info about what doesn't work.
> I don't know how to reproduce the issue. Please comment the issue.
>
> To cross compile Python, I found these documentations:
>
> * https://docs.python.org/dev/using/configure.html#cross-compiling-options
> * WASM: https://github.com/python/cpython/blob/main/Tools/wasm/README.md
>
> Currently, setup.py checks for:
>
> CROSS_COMPILING = ("_PYTHON_HOST_PLATFORM" in os.environ)
>
> Victor
>
>
> On Tue, Jun 14, 2022 at 1:49 AM Dave Blanchard  wrote:
> > Here's what's happening. This is a very old problem reported ages ago
> which has never been fixed. If I set the target arch to i686 (on an x86_64
> build system) Python will cross-compile and bootstrap just fine. But if the
> host and build triple are the same, then the problem will occur. Python
> seems to be ASSuming that if the build and host triple match (in this case,
> both being 'x86_64-linux-gnu') therefore the host and build libraries are
> binary-compatible--which is NOT actually the case when one is
> cross-compiling to a different libc, for example. Actually it's kind of
> brain dead how this is implemented.
> >
> > Some prior discussions/years-old bug reports:
> >
> > https://bugs.python.org/issue39399
> > https://bugs.python.org/issue22724
> > https://bugs.gentoo.org/705970
> >
> > In those links, various hacks are attempted with various degrees of
> success, but then Xavier de Gaye reworked the build system, apparently
> fixing the problem with this pull request in Dec. 2019:
> >
> > https://github.com/python/cpython/pull/17420
> >
> > Unfortunately he became annoyed in the comments, seemingly mostly due to
> the lack of interest from Python to actually do anything about this
> problem, and cancelled his pull request. His fix was never implemented, and
> Python cross-compiling remains broken to this day.
> >
> > I downloaded his patches, made a minor fix or two, and merged them all
> together into the one patch attached to this email. When applied to both my
> build system and target Python, this problem seems to be resolved, for me
> at least. I'll know more later tonight as I get further into the distro
> build process.
> >
> > It's surprising to hear that cross-compiling Python would be any kind of
> "unusual" thing, considering this is something that *always* has to be done
> any time one is bootstrapping anything on a new or different system. It
> surprises me further to see that Python actually requires the *minor*
> version number to be the same for bootstrapping, also. So not only is
> Python 3 essentially a different language from Python 2, each point release
> is different and incompatible also. Amazing.
> >
> > I guess this shouldn't be surprising, after all of the other
> questionable design decisions this project has made over the years. I
> probably also shouldn't be surprised to see such an incredibly important
> bug going unfixed after all this time. It's Python--by far the #1 biggest
> annoyance and pain in the ass out of the 1,000+ packages on my distro,
> ranking just above CUPS and Texlive.
> >
> > Dave
> >
> >
> > On Mon, 13 Jun 2022 16:12:26 -0500 (CDT)
> > Matthew Dixon Cowles  wrote:
> >
> > > Dear Dave,
> > >
> > > > Hello, I'm trying to cross compile a new version of my Linux system
> > > > with some upgraded packages, including a new Glibc.
> > >
> > > I've never had to do that and I am beginning to suspect, from the
> > > lack of replies here better than this one, that nobody else here has
> > > had to either.
> > >
> > > > I've hit a major snag with Python 3.7 (also tried 3.9 with same
> > > > result) which makes it through the compile OK, but then bombs when
> > > > running ensurepip at the end.
> > >
> > > If it were me, I'd set --with-ensurepip to no, 

[Python-Dev] Re: Is it possible to view tokenizer output?

[Eric V. Smith]

python -m tokenize < file-to-parse.py

See the comment at the top of tokenize.py. IIRC, it re-implements the 
tokenizer, it does not call the one used for python code.


Eric

On 5/29/2022 6:59 PM, Jack wrote:
Hi, I'm just getting into the CPython codebase just for fun, and I've 
just started messing around with the tokenizer and the grammar. I was 
wondering, is there a way to just print out the results of the 
tokenizer (as in just the stream of tokens it generates) in a human 
readable format? It would be really helpful for debugging. Hope the 
question's not too basic.


Cheers.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2ZTZBAN5H2ET2IB7EXTKD27R5T6QVHZB/

Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EZ3E7VMT6WAT23EWWLV3O4CXWXYUGJWB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [OT] Re: Raw strings ending with a backslash

[Gregory P. Smith]

On Sat, May 28, 2022 at 12:55 PM Guido van Rossum  wrote:

>
> On Sat, May 28, 2022 at 12:11 MRAB
>
> Names in Python are case-sensitive, yet the string prefixes are
>> case-/insensitive/.
>>
>> Why?
>
>
> IIRC we copied this from C for numeric suffixes  (0l and 0L are the same;
> also hex digits and presumably 0XA == 0xa) and then copied that for string
> prefixes without thinking about it much. I guess it’s too late to change.
>

Given that 99.99% of code uses lower case string prefixes we *could* change
it, it'd just take a longer deprecation cycle - you'd probably want a few
releases where the upper case prefixes become an error in files without a
`from __future__ import case_sensitive_quote_prefixes` rather than jumping
straight from parse time DeprecationWarning to repurposing the uppercase to
have a new meaning.  The inertia behind doing that over the course of 5+
years is high.  Implying that we'd need a compelling reason to orchestrate
it.  None has sprung up.

-gps


> —Guido
> --
> --Guido (mobile)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/27HLMPDURSAN2YCTFWN6LETWQNY4POX7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CCK3UGMOBQKHP5R25UX777EFZCQQE5CL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Raw strings ending with a backslash

[Eric V. Smith]

On 5/28/2022 7:57 AM, Damian Shaw wrote:

That PR seems to make \' and \" not special in general right?

I think this is a more limited proposal, to only change the behavior 
when \ is at the end of a string, so the only behavior difference 
would never receiving the error "SyntaxError: EOL while scanning 
string literal"


How would you know where the end of a string is? I think this is one of 
those things that's easy to look at for a human and figure out the 
intent, but not so easy for the lexer, without some heuristics and 
backtracking. If the trailing single quote is removed below, it changes 
from "backslash in the middle of a string" to "backslash at the end of a 
string, followed by an arbitrary expression.


r'\' + "foo"'

Eric



In which case there should be no backwards compatibility issue.

Damian

On Sat, May 28, 2022 at 12:20 PM Serhiy Storchaka 
 wrote:


28.05.22 12:22, Steven D'Aprano пише:
> Now that we have a new parser for CPython, can we fix the old gotcha
> that raw strings cannot end in a backslash?
>
> Its an FAQ and has come up again on the bug tracker.
>
> https://docs.python.org/3/faq/design.html#id26
>
> https://github.com/python/cpython/issues/93314

I do not think that we can allow this, and it is not related to
parser.

Few years ago I experimented with such change:
https://github.com/python/cpython/pull/15217

You can see that it breaks even some stdlib code, and it will
definitely
break many third-party packages and examples. Technically we can do
this, but the benefit is small in comparison with the cost.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/WWWRFQK4AG52GD3L6WT6QLRGTY2VQRQ2/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/O3STZ54BRQ3T352PIMNEZWCRBGP6FE2O/
Code of Conduct:http://python.org/psf/codeofconduct/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AMEE5VX5UB3KESBDJ47SUIFLHMEAZAGK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Starting a new thread

[Gregory P. Smith]

On Tue, May 10, 2022 at 10:34 AM Barney Stratford <
barney_stratf...@fastmail.fm> wrote:

> > 1. Does t = target(...) start the thread? I think it does.
> I think it does too. In the commonest use case, immediately after creating
> a thread, you start it. And if you want to delay the thread but still use
> the decorator, then you can do that explicitly with some locks. In fact,
> it’s probably better to explicitly delay execution than have hidden
> requirements concerning the timing of thread creation and startup.
>
> > 2. Is it possible to create daemon threads?
> Not at the moment. I did think about this, but felt that simpler is
> better. Like you say, it’d be easy to add. In fact, I might just go ahead
> and add it to the PR in a bit. The simplest way to do it is probably to
> define a second decorator for daemonic threads.
>

If this is even to be added (i personally lean -1 on it), I suggest
intentionally not supporting daemon threads. We should not encourage them
to be used, they were a misfeature that in hindsight we should never have
created. Daemon threads can lead to very bad surprises upon interpreter
finalization - an unfixable problem given how daemon threads are defined to
behave.

> 3. Can you create multiple threads for the same function? I assume t1,
> > t2, t3 = target(arg1), target(arg2), target(arg3) would work.
> That’s exactly what I had in mind. Make it so that thread creation and
> function call look exactly alike. You can call a function as many times as
> you want with whatever args you want, and you can create threads as often
> as you want with whatever args you want.
>
> There isn’t a single use case where the decorator is particularly
> compelling; rather, it’s syntactic sugar to hide the mechanism of thread
> creation so that code reads better.
>

This is my take as well. I don't like calling code to hide the fact that a
thread is being spawned. Use this decorator and if you fail to give the
callable a name communicating that it spawns and returns a thread, you will
have surprised readers of the calling code.

A nicer design pattern is to explicitly manage threads. Use
concurrent.futures.ThreadPoolExecutor. Or use the async stuff that Joao
mentioned or similar libraries. I think we already provide decent batteries
with the threading APIs.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3I7GBDGMHU4BHHR57EFOL6B4JMURFSGE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

> On Apr 26, 2022, at 3:05 PM, Thomas Kehrenberg  wrote:
> 
> 
> Apr 26, 2022 20:32:55 Eric V. Smith :
> 
>> How would runtime consumers of annotations use this?
>> 
>> --
>> Eric
>> 
>>>> On Apr 26, 2022, at 12:05 PM, Thomas Kehrenberg  wrote:
>>> 
>>> If the problem is mostly type annotations, then another potential
>>> solution would be to make use of .pyi files, which are not hamstrung by
>>> circular definitions.  The idea would be that type checkers would merge
>>> the annotations from .pyi files into the annotations in the
>>> corresponding .py file.
>>> 
>>> So:
>>> 
>>> a.py:
>>> 
>>>from b import B
>>> 
>>>class A:
>>>value: B
>>> 
>>> b.py:
>>> 
>>>class B:
>>>value = None
>>> 
>>> b.pyi:
>>> 
>>>from typing import Optional
>>>from a import A
>>> 
>>>class B:
>>>value: Optional[A] = ...
>>> 
>>> The pyi files would kind of act like header files that are used in other
>>> languages.  It would mean that type checkers need to check the .pyi
>>> files against the code in the .py files to verify that they're
>>> consistent with one another.
>>> 
>>> -thomas
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at 
>>> https://mail.python.org/archives/list/python-dev@python.org/message/VWPWN5KWTRPP6VS4PEHJA4SRVMUDU5WR/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> They wouldn't. But I thought PEP 649 solves the runtime problems,
> and that the remaining problems are with static typing
> of circular definitions.

If the class needs access to its own type annotations at runtime (for example, 
if it’s a dataclass), then the circular reference problem still exists with PEP 
649. That’s among the cases we’re trying to resolve. 

Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SV37AN7HYQNAVIE2BMJ5C5AC77FENTCS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

How would runtime consumers of annotations use this?

--
Eric

> On Apr 26, 2022, at 12:05 PM, Thomas Kehrenberg  wrote:
> 
> If the problem is mostly type annotations, then another potential
> solution would be to make use of .pyi files, which are not hamstrung by
> circular definitions.  The idea would be that type checkers would merge
> the annotations from .pyi files into the annotations in the
> corresponding .py file.
> 
> So:
> 
> a.py:
> 
>from b import B
> 
>class A:
>value: B
> 
> b.py:
> 
>class B:
>value = None
> 
> b.pyi:
> 
>from typing import Optional
>from a import A
> 
>class B:
>value: Optional[A] = ...
> 
> The pyi files would kind of act like header files that are used in other
> languages.  It would mean that type checkers need to check the .pyi
> files against the code in the .py files to verify that they're
> consistent with one another.
> 
> -thomas
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/VWPWN5KWTRPP6VS4PEHJA4SRVMUDU5WR/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2VNYLOHCXWBUYTWRGRJC6C4LXQVF45MD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] New functionality for unittest.mock side_effect

[Roy Smith]

I often want a side_effect of "if called with foo, return bar" functionality.  
This is really useful when the order of calls to your mock is indeterminate, so 
you can't just use an iterable.  What I end up doing is writing a little 
function:

def f(x):
data = {
'foo': 'bar',
'baz': 'blah',
}
return data[x]

my_mock.side_effect = f

sometimes I wrap that up in a lambda, but either way it's fidgety boilerplate 
which obfuscates the intent.  It would be nice if side_effect could accept a 
dict and provide the above functionality for free:

my_mock.side_effect = {
'foo': 'bar',
'baz': 'blah',
}

If could handle DEFAULT just like side_effect does now.  And it could throw 
something more useful than KeyError if the key isn't found.

I could see the argument that "but dicts are themselves iterable, so this could 
break some existing mocks which depend on side_effect iterating over the dicts 
keys.  That would be bizarre, but possible.  The fix for that would be invent a 
new attribute, say, return_value_map, which does this, and say that it's an 
error to set more than one of {return_value, return_value_map, side_effect}.

Anyway, if I were to take a whack at coding this up, is it something that would 
be considered for a future release?




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4A24FFKMN2PEB4GRERXUX2BUNVMZR2MA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

On 4/25/2022 9:13 PM, Mehdi2277 wrote:

We could have forward part include all of method signatures/attributes which is 
what a type checker needs. In theory we could do,

forward class Foo:
   x: int
   y: list[str]
   
   def __init__(self, ...)


   def process(self, data: list[float]) -> float: ...

and then later do continue class. If we're willing to write a full header 
equivalent of c++ then I think static typing side will work. It'll be a lot 
more verbose then normal that I'd probably pick other solutions, but I it 
should make it tractable for a type checker.


How would this address the use case presented in the original post?

class A:
    value: B

class B:
    value: A

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2CPE3IB7TUJKEAY633LS3WM7PRQ7R6L6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

On 4/25/2022 1:10 PM, Jim J. Jewett wrote:

(That said, my personal opinion is that this is pretty heavyweight for very 
little gain; why not just create a placeholder class that static analysis tools 
are supposed to recognize as  likely-to-be-replaced later?  And why not just 
use strings giving the expected eventual class name?  It isn't as though the 
analysis can verify whether something actually meets the full intended contract 
before they've also parsed the continuation.)


The experience with PEP 563 shows that string annotations make things 
much more difficult for dataclasses and other users of runtime type 
annotations. The point of PEP 649 and the forward declaration of classes 
is to make it possible to remove string annotations entirely.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZO7M5NCQKR7EPHRVUALRSH2RCKMT5YX7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: What is Modules/rotatingtree.c for?

[Gregory P. Smith]

On Sun, Apr 24, 2022 at 9:24 AM Patrick Reader <_...@pxeger.com> wrote:

> I've just noticed Modules/rotatingtree.{h,c}, which don't seem to be
> used anywhere. Are they just dead code? If so, is there a reason they
> haven't been removed?
>

grep -R rotatingtree ; grep -R _lsprof

rotatingtree is used by the _lsprof module which is the internal
implementation behind cProfile.

-gps


>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/46D6DD3OEZEOH4CUPFKOSLVUG6MHV7BU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W45DZ2RU75HOKELTMRZY7YL3JH7KS7V5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

On 4/24/2022 5:42 AM, Stephen J. Turnbull wrote:

What's the use case for arbitrary expressions vs. a (possibly
qualified) name?  A class factory that produces forward declarations?
Do you have a use case in mind?


It's:

x.py:

--8<
forward class A()
--8<

x_impl.py

--8<
import X

continue class X.A:
    # class body here
--8<

It needs to be an expression because it's not defining a name, it 
referring to an existing name. You could use "from X import A" here and 
avoid a dotted expression, but it still needs to be an expression 
referring to an existing "forward class". Even if you restrict it to not 
having dots, it's logically an expression, not a name binding.


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VV53MWDTYQCHBCAQGXLZ4SFCKUEHPQ2K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

On 4/23/2022 9:55 AM, Jelle Zijlstra wrote:


However, it doesn't solve the problem for base classes. For example, 
str is conceptually defined as `class str(Sequence["str"]):`. A 
forward reference can't make `str` defined when the bases are 
evaluated, because bases are resolved at the `forward class` stage.


Larry's second email "Proto-PEP part 2: Alternate implementation 
proposal for "forward class" using a proxy object" discusses a 
possibility to move the bases and metaclasses to the "continue class" 
stage. It also has the advantage of not changing the behavior of 
__new__, and I think is in general easier to reason about. He and I have 
discussed this approach, but neither of have looked at in enough detail 
to know if the implementation is possible. Some of the concerns are 
noted in that email.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MODT7FKWYNDOR4D7LZRW2UOQB5OQSSJ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Eric V. Smith]

On 4/23/2022 3:28 AM, Tobias HT wrote:



On the other hand, there's something I've been seeing around. I don't 
know if it was introduced by Mypy or something, but its the use of 
declaration files. I think they are saved as pyi files. They just have 
the declaration of a python object, be it class or variable. What if 
we just found a way of reusing that instead?


As they currently exist, stub files (.pyi files) don't contain enough 
information. In particular, they don't have the metaclass information. 
This could be changed, but at that point you basically have the 
"forward" declaration, but it's hidden away where the interpreter can't 
see it.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZNZS7Q2MYGGDW5DJ4LOAQ2FOOULKQV3S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Updating inspect APIs

[Eric V. Smith]

Things like os.stat_result use PyStructSequence_NewType, which is the C 
equivalent of a namedtuple. But a Struct Sequence has the extension of 
being able to have more named fields than the tuple length. It was 
designed, so I understand, just for cases like this. I don't know if it 
would be possible for you to use it from pure Python code, or maybe 
you'd need a C module to expose it.


There was idle talk many years ago of either extending namedtuple or 
creating some new type factory that could do the same from pure Python 
code, but no one ever got serious about it. I'm not sure it would be 
accepted in any event. Buy maybe if you have a concrete use case 
something like that could be done.


Eric

On 4/17/2022 1:20 PM, Pablo Galindo Salgado wrote:

Hi,

We are currently debating in gh-88116 
(https://github.com/python/cpython/issues/88116)
what's the best way forward to update the APIs in the inspect module 
to include the new position information.


These APIs are inspect.getframeinfo, inspect.getouterframes, 
inspect.getinnerframes, inspect.stack and inspect.trace.


The problem is that these APIs return a named tuple that now needs to 
include several new attributes (or one 4 tuple for
the positions). Being named tuples, if we add a new attribute, 
existing unpackings of the tuple will now fail because there
are more elements or the elements are in different positions. Also, it 
will be quite weird to add the new attributes at the

end but leave the line number at the beginning.

What's the best way to proceed here? The suggested way is to create a 
namedtuple subclass that adds the extra attributes
but doesn't allow indexed access to it (there is a precedent to this 
in how we handled updating os.stat_result). I personally
find this quite confusing but it certainly works. There may be other 
options.


What do you think?

Cheers from sunny London,
Pablo Galindo Salgado

___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/RTGG637WWPOWUHUF6TRJYUSBYYSVUPRA/
Code of Conduct:http://python.org/psf/codeofconduct/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JRVT7YZGXTWPJACWAOKR4EUPO6IZIBJQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proposal to deprecate mailcap

[Gregory P. Smith]

+1 add it to the 3.11 deprecations and proactively reach out to the
mitmproxy owners.

(internal code search: aside from mitmproxy I only see a _single_ use of
this in our codebase and it was simply convenient but has a clear simpler
alternative assuming that ~2008 era code is even still in use)

-gps


On Thu, Apr 14, 2022 at 11:49 AM Brett Cannon  wrote:

> A CVE has been opened against mailcap (see
> https://github.com/python/cpython/issues/68966 for details). I'm not
> aware of anyone trying to maintain the module and Victor did a search
> online and didn't find any use of the module in the top 5000 projects on
> PyPI (see the issue). The module is also under 300 lines of Python code
> that only  (https://github.com/python/cpython/blob/main/Lib/mailcap.py),
> so vendoring wouldn't be burdensome.
>
> As such, I'm proposing we deprecate mailcap in 3.11 and remove it in 3.13.
> Any explicit objections?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EB2BS4DBWSTBIOPQL5QTBSIOBORWSCMJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GG2JGVJPKEGS4P5ASSLJGGWUOO2JBCBJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 678: Enriching Exceptions with Notes

[Gregory P. Smith]

On Thu, Jan 27, 2022 at 10:10 AM Zac Hatfield Dodds <
zac.hatfield.do...@gmail.com> wrote:

> Hi all,
>
> I've written PEP 678, proposing the addition of a __note__ attribute
> which can be used to enrich displayed exceptions.  This is particularly
> useful with ExceptionGroup, or where exception chaining is unsuitable, and
> ensures that 'orphan' output is not produced if the annotated exception is
> caught.
>
> Link to the PEP: https://www.python.org/dev/peps/pep-0678/
>
> *Please, redirect all discussions to:*
> https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374
>

FYI - The Steering Council reviewed your updates and is **Accepting** the
latest iteration of *PEP 678: Enriching Exceptions with Notes*. :grin:

A short and sweet reply, we don't have more to add.

-gps on behalf of the Python Steering Council
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QFOWI4WOUOVPSERRYADQNZWJJTKTOR3A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent

[Eric V. Smith]

On 4/9/2022 4:28 PM, Terry Reedy wrote:

On 4/9/2022 5:09 AM, Arfrever Frehtes Taifersar Arahesis wrote:


Not only deletion, but also random assignments:


Ok.  Change "Manual deletion of entries from sys.modules" to "Direct 
manipulation of sys.modules"


I'm not sure it's worth the hassle to document this. There are no doubt 
hundreds of ways to break things. We can't enumerate all of them.


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WGP7SLEIEIDPJEOU3ANSRPGHWKLMM7QE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: About PEPs being discussed on Discourse

[Gregory P. Smith]

On Thu, Apr 7, 2022 at 4:31 PM Jean Abou Samra  wrote:

>
> I'm only a lurker here, but find the split between this mailing list
> and various Discourse categories a little confusing from the outside.
> As far as I understand, the Discourse server was originally an experiment.
> Today, it has grown far past the size of an experiment though. Are there
> any plans to retire either Discourse or the mailing list and use a
> unified communication channel? This is a curiosity question.
>

We feel it too. We've been finding Discourse more useful from a community
moderation and thread management point of view as well as offering markdown
text and code rendering. Ideal for PEP discussions. Many of us expect
python-dev to wind up obsoleted by Discourse as a result. I encourage
everyone to use https://discuss.python.org/ first for Dev audience
communications. And for lurkers and subscribers here to enable email
notifications for categories of interest over there.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2TPHCHVB7NEMMTEWWYBYFZNIFBEE2S4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Gregory P. Smith]

On Fri, Apr 1, 2022 at 2:06 AM Victor Stinner  wrote:

> Hi,
>
> Update on this issue: I merged my 2 PRs.
> https://bugs.python.org/issue46850
>
> The following APIs have been moved to the internal C API:
>
> - _PyFrameEvalFunction type
> - _PyInterpreterState_GetEvalFrameFunc()
> - _PyInterpreterState_SetEvalFrameFunc()
> - _PyEval_EvalFrameDefault()
>
> If you use any of these API in your debugger/profiler project, you
> have do add something like the code below to your project:
> ---
> #ifndef Py_BUILD_CORE_MODULE
> #  define Py_BUILD_CORE_MODULE
> #endif
> #include 
> #if PY_VERSION_HEX >= 0x030B00A7
> #  include  //
> _PyInterpreterState_SetEvalFrameFunc()
> #  include   // _PyEval_EvalFrameDefault()
> #endif
> ---
>
> Contact me if you need help to update your affected projects.
>
> IMO PEP 523 doesn't have to be updated since it already says that the
> APIs are private.
>

Thanks for bringing this up on python-dev, Victor. That was good. But the
point of the discussion should've been to continue working with people
based on the replies rather than proceeding to merge removals of the APIs
after people said they used them.  (echoing Steve and Petr here...)

We discussed this on the steering council today. These APIs were in a weird
state and despite past decisions at the time of PEP-523
 in 2016 they should be treated as
public-ish rather than entirely private. Because we published a document
saying "here they are, use them!" and multiple projects have done so to
good effect.

For 3.11 we'd like those PRs reverted.  We see the following as the better
way forward for these APIs:

Add a new #define that can be set before the #include  that
exposes non-limited but stable within a bugfix/patch releases APIs (ie:
Petr's earlier suggestion).
These would be the first to fall within. To do so we should give these,
behind that #define, non _-prefixed "public style" names as these are
quasi-public and cannot be changed as readily as other true internals.  We
still, per the PEP, reserve the right to turn these into no-op potentially
warning setting APIs in a future release (likely 3.12?) as they are at
least documented as being unstable/private in the PEP.

So in 3.11 these should continue to exist as in 3.6-3.10:
- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()

AND in 3.11:
 - #define protected versions of those without the leading _ become
available.
 - (i'm intentionally not suggesting a #define name, y'all can pick
something)

In 3.12:
 - the _ prefixed versions can go away.  People using the APIs should've
updated their code to use the new #define and new names when building
against >=3.11 by then.
 - Whether the APIs continue to be as useful and act as originally claimed
in PEP 523 is up to the 3.12 implementors (out of scope for this thread).
They occupy a newly defined middle ground between the "forever" style
limited API and the "can break even on bugfix/patch release" internal API
that wasn't a concept for us in 2016 when PEP 523 was written.

Why? Being conservative with things in active use that weren't *really*
private, and similar to what Mark Shannon and Petr said, we *do not* want
people to #define Py_BUILD_CORE_MODULE and start poking at internals in
arbitrary ways. That exposes a whole pile of other things for (ab)use that
are even more unstable. Avoiding that helps avoid temptation to go wild and
helps us identify users.

-gps (steering council hat on)


> Since these APIs were added by PEP 523, I documented these changes in
> What's New in Python 3.11 > C API > Porting to Python 3.11,even if
> these APIs are private.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/DNJC6U36CDA7S7ATEGAMUPABBSEIYHC4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GFOMU7LP63JUVFMWNJNZJLUMZDRPTUYJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Eric V. Smith]

On 3/29/2022 4:55 PM, Skip Montanaro wrote:

I was trying to think through how a "remote" stdlib might work. In the
process, I got to wondering if there are known "specialists" for
various current modules. Every now and then I still get assigned (or
at least made nosy) about something to do with the csv module. Is
there an official module-by-module list of maintainers?
There's the CODEOWNERS file: 
https://github.com/python/cpython/blob/main/.github/CODEOWNERS


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EKNOIU54LPVM3HZJN6WZCTVQIZHOLSYL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing unittest verbose output.

[Gregory P. Smith]

On Sun, Mar 27, 2022 at 2:50 PM Guido van Rossum  wrote:

> Hopefully it only prints the first line of the docstring?
>

Indeed, it's just the first line (traditionally a one line summary per
style).

https://github.com/python/cpython/blob/f6b3a07b7df60dc04d0260169ffef6e9796a2124/Lib/unittest/runner.py#L46


>
> On Sun, Mar 27, 2022 at 2:42 PM Gregory P. Smith  wrote:
>
>> For many of us, this isn't a nuisance. It is a desirable feature.
>>
>> test_xxx functions and methods typically don't have docstrings at all as
>> the majority of tests tend to be concise and obvious with the function name
>> itself describes what its purpose is.  When we added printing the
>> docstring, it was useful, as you can expand upon the purpose of the test
>> more than you can in a reasonable name within the docstring.  We do this
>> all the many times in our own test suite.  When running the tests, you see
>> the docstring and are given more context as to what the test is about.
>> This can be useful when triaging a failure before you've even loaded the
>> source.
>>
>> I don't doubt that someone writes thesis defenses and stores them in
>> their TestCase.test_method docstrings. I'm just saying that is not the norm.
>>
>> I'd accept a PR adding another command line switch for unittest to
>> disable docstring printing in verbose mode.
>>
>> -gps
>>
>>
>> On Sun, Mar 27, 2022 at 12:59 PM Barry Warsaw  wrote:
>>
>>> On Mar 26, 2022, at 17:48, Itay Yeshaya  wrote:
>>> >
>>> > When running unittest with the -v flag, if a test has errors, and has
>>> a docstring, the test name is shown on one line, and the docstring is shown
>>> on the next line with the `ERROR` word.
>>>
>>> This has been a long-standing nuisance, but I’m like Guido.  I pretty
>>> much use pytest for everything these days, except for maybe
>>> unittest.mock.patch.
>>>
>>> -Barry
>>>
>>>
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/ZIMXSRQMWFOE4U3C3MBK6SG5TH26PDRL/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/RJOWQWMMUMBJAZZTWXWEAJSRDVARA2XL/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YULIAATNZVWHHJEASJY3P5UMC4G7ZIPC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing unittest verbose output.

[Gregory P. Smith]

For many of us, this isn't a nuisance. It is a desirable feature.

test_xxx functions and methods typically don't have docstrings at all as
the majority of tests tend to be concise and obvious with the function name
itself describes what its purpose is.  When we added printing the
docstring, it was useful, as you can expand upon the purpose of the test
more than you can in a reasonable name within the docstring.  We do this
all the many times in our own test suite.  When running the tests, you see
the docstring and are given more context as to what the test is about.
This can be useful when triaging a failure before you've even loaded the
source.

I don't doubt that someone writes thesis defenses and stores them in their
TestCase.test_method docstrings. I'm just saying that is not the norm.

I'd accept a PR adding another command line switch for unittest to disable
docstring printing in verbose mode.

-gps


On Sun, Mar 27, 2022 at 12:59 PM Barry Warsaw  wrote:

> On Mar 26, 2022, at 17:48, Itay Yeshaya  wrote:
> >
> > When running unittest with the -v flag, if a test has errors, and has a
> docstring, the test name is shown on one line, and the docstring is shown
> on the next line with the `ERROR` word.
>
> This has been a long-standing nuisance, but I’m like Guido.  I pretty much
> use pytest for everything these days, except for maybe unittest.mock.patch.
>
> -Barry
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZIMXSRQMWFOE4U3C3MBK6SG5TH26PDRL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RJOWQWMMUMBJAZZTWXWEAJSRDVARA2XL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing unittest verbose output.

[Eric V. Smith]

> On Mar 27, 2022, at 11:20 AM, Eric V. Smith  wrote:
> 
> 
> On 3/27/2022 10:51 AM, Jelle Zijlstra wrote:
>> 
>> 
>> El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya () 
>> escribió:
>>> 
>>> Any thoughts on the mentioned solutions?
>> 
>> This is a bit more out there, but can anyone explain why we show the 
>> docstring at all? As far as I can tell it causes a bunch of confusion and 
>> helps little. If there's a failing test, I want the test name so I can 
>> quickly find it in the code; I don't want the docstring which may not even 
>> be unique.
>>  
> I think this is a good idea. The target audience for a docstring isn't the 
> people trying to track down failing tests.
> 
I’m wrong about the docstrings having a different audience. So I’m going to 
retract my support here. 

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L7HIUA64XS4HCZSCBLEM5FOJU5A7D6OA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing unittest verbose output.

[Eric V. Smith]

On 3/27/2022 10:51 AM, Jelle Zijlstra wrote:



El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya 
() escribió:



Any thoughts on the mentioned solutions?


This is a bit more out there, but can anyone explain why we show the 
docstring at all? As far as I can tell it causes a bunch of confusion 
and helps little. If there's a failing test, I want the test name so I 
can quickly find it in the code; I don't want the docstring which may 
not even be unique.


I think this is a good idea. The target audience for a docstring isn't 
the people trying to track down failing tests.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BTF4LZWVTHNHBJVNHXNWSPB2WS4324TO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Accepting PEP 655 - Marking individual TypedDict items as required or potentially-missing

[Gregory P. Smith]

On behalf of the Python Steering Council, we are pleased to accept PEP 655
- Marking individual TypedDict items as required or potentially-missing
.

Thanks for considering the potential for confusion with Optional during the
design and recommending best practices in the “how to teach” section.

A couple SC members have tried using TypedDict and found it painful, this
PEP helps.

-gps for the steering council
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AJEDNVC3FXM5QXNNW5CR4UCT4KI5XVUE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Accepting PEP 675 - Arbitrary Literal String Type

[Gregory P. Smith]

On behalf of the Python Steering Council, we are accepting PEP 675 -
Arbitrary Literal String Type .

TL;DR - PEP 675 allows type checkers to help prevent bugs allowing
attacker-controlled data to be passed to APIs that declare themselves as
requiring literal, in-code strings.

This is a very thorough PEP with a compelling and highly relevant set of
use cases. If I tried to call out all the things we like about it, it’d
turn into a table of contents. It is long, but everything has a reason to
be there. :)

Once implemented, we expect it to be a challenge to tighten widely used
existing APIs that accept str today to be LiteralString for practical
reasons of what existing code calling unrestricted APIs naturally does. The
community would benefit from anyone who attempts to move a widely used
existing str API to LiteralString sharing their experiences, successful or
not.

-gps for the steering council
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XEOOSSPNYPGZ5NXOJFPLXG2BTN7EVRT5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Restrict the type of __slots__

[Eric V. Smith]

On 3/18/2022 4:58 PM, Guido van Rossum wrote:

On Fri, Mar 18, 2022 at 9:40 AM Paul Bryan  wrote:

On Fri, 2022-03-18 at 09:35 -0700, Guido van Rossum wrote:

The motivation has been explained already.


In this thread?


Yes, Eric's message.


What on earth did your test do that got a speedup by using sets?
Was it repeatedly checking whether a variable was in a slot? The
other thing this does is rearrange the order in which slots
appear from run to run (since set order is affected by hash
randomization) and you might have gotten lucky with a popular
attribute being moved to the front, where it's more likely to be
in the memory cache already (cache lines being 64 bytes and
pointers being 8 bytes nowadays).


I created objects in a tight loop, populating attributes, noting
the elapsed time.


It does not make sense that that is correlated to the type of 
__slots__, since __slots__ is not used for instance creation at all 
(it is only used to create the class). I stick to my likely explanation.


Regarding Serhiy's proposal, I'm +0 on disallowing strings, and +0 on 
disallowing things that can't be reiterated (but it's not a problem in 
practice). Given other responses the status quo is likely best.


The PR for the issue I mentioned (https://bugs.python.org/issue46382) 
allows strings and iterables (but not iterators). And it uses a match 
statement, no less! I think that's good enough for dataclasses.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BS2RRX62SHQDTM5O24WKAANPNHKXMG6F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Restrict the type of __slots__

[Eric V. Smith]

On 3/18/2022 10:01 AM, Ronald Oussoren via Python-Dev wrote:




On 18 Mar 2022, at 14:37, Joao S. O. Bueno  wrote:

IMO this is a purism that have little, if any place in language 
restrictions.
I see that not allowing. "run once" iterables could indeed void 
attempts to write
"deliberatly non cooperative code" - but can it even be reliably 
detected?


The other changes seem just to break backwards compatibility for 
little or no gain at all.
There a practical need for these changes. See 
https://bugs.python.org/issue46382 for a case where dataclasses needs 
__slots__ to be iterated over after the class has been created. And it 
would be good if __slots__ accurately reflected the slots that were 
actually created.


It may not be worth the trouble to fix this, but Serhiy’s proposal 
does try to fix a ward.


It may be better to rely on linter’s here, but one way to do this with 
few backward compatibility concerns:


- if __slots__ is a dict keep it as is
- Otherwise use tuple(__slots__) while constructing the class and 
store that value in the __slots__ attribute of the class


That way the value of the attribute reflects the slots that were 
created while not breaking code that uses __slots__ and doesn’t change 
the value after class creation.


I like this approach, too.

Eric



Ronald





On Fri, Mar 18, 2022 at 6:57 AM Ronald Oussoren via Python-Dev 
 wrote:





On 18 Mar 2022, at 10:29, Serhiy Storchaka 
wrote:

Currently __slots__ can be either string or an iterable of strings.

1. If it is a string, it is a name of a single slot. Third-party
code which iterates __slots__ will be confused.

2. If it is an iterable, it should emit names of slots. Note
that non-reiterable iterators are accepted too, but it causes
weird bugs if __slots__ is iterated more than once. For example
it breaks default pickling and copying.

I propose to restrict the type of __slots__. Require it always
been a tuple of strings. Most __slots__ in real code are tuples.
It is rarely we need only single slot and set __slots__ as a string.

It will break some code (there are 2 occurrences in the stdlib
an 1 in scripts), but that code can be easily fixed.


Pydoc supports __slots__ that is a dict, and will use the values
in the dict als documentation for the slots.   I’ve also seen
code using ``__slots__ =  “field1 field2”.split()``. I don’t
particularly like this code pattern, but your proposal would
break this.

Also note that __slots__ only has a side effect during class
definition, changing it afterwards is possible but has no effect
(“class Foo: pass; Foo.__slots__ = 42”). This surprised my
recently and I have no idea if this feature is ever used.

Ronald



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/E32BRLAWOU5GESMZ5MLAOIYPXSL37HOI/
Code of Conduct: http://python.org/psf/codeofconduct/


—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/YQUWR7CYKNM65HR5FZQ3BANR5SNNK6N6/
Code of Conduct: http://python.org/psf/codeofconduct/



—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/


___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/FZFRSHSJ3HQU37V6RFZNHMFGJXUPJ32X/
Code of Conduct:http://python.org/psf/codeofconduct/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J5LE3OPM2GTT3N7BYJTBZDDYYBXQ34Z7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New PEP website is horrible to read on mobile device

[Eric V. Smith]

On 3/16/2022 11:23 AM, Skip Montanaro wrote:

Dang auto-correct... I meant "anti-tracking," in case it wasn't obvious.


It wasn't obvious! I was wondering what nefarious entity had created an 
anti-teaching VPN, and just what that would involve. Thanks for clarifying!


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HJSNHJBUDUYCQGZPSRQ32WHQEZEZPASW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Need Help

[Gregory P. Smith]

The embedded copy of expat was recently upgraded to 2.4.6 in
https://bugs.python.org/issue46794 including on the 3.9 branch.  That will
wind up in 3.9.11 per https://www.python.org/dev/peps/pep-0596/.

If you are using 3.9.5 you may also have a host of other potential security
issues that updating to a recent 3.9.x will address. If you are using 3.9.5
as provided by a Linux or similar OS distribution, I'd expect the OS distro
packager to be applying relevant patches to it themselves (some distros
link to their own managed libexpat instead of using the embedded version)
even if they don't change the version number.

-gps

On Fri, Feb 25, 2022 at 11:43 AM Prasad, PCRaghavendra <
pcraghavendra.pra...@dell.com> wrote:

> Hi All,
>
> we are using the python 3.9.5 version in our application.
>
>
>
> In 3.9.5 it is using libexpat 2.2.8 version, as part of the Black duck
> scan, it is showing critical vulnerabilities in libexpat 2.2.8.
>
>
>
> (CVE-2022-22824
>
> CVE-2022-23990
>
> CVE-2022-23852
>
> CVE-2022-25236
>
> CVE-2022-22823)
>
>
> when there are any issues ( security issues ) in external modules like
> OpenSSL, bzip2, and zlib we were able to get the latest code and build as
> it is straightforward, but libexpat is an internal module to the python and
> we don't see how we can upgrade libexpat alone in python 3.9.5
>
> So is there a way we can build python (ex 3.9.5) which is already carrying
> libexpat 2.2.8 so that it will link to the latest libexpat version (2.4.6 -
> fixed security issues).
>
> Another solution when we searched over the net and from the mails what we
> came to know is we need to wait for Python 3.9.11 where this will be linked
> to libexpat 2.4.6.
>
> Any inputs on this will be helpful.
>
> Thanks,
>
> Raghu
>
> Internal Use - Confidential
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2JHZTKQVVYR67KQRIFF5XEMXDY3FZLMN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/64FLSLO7KN2Q6UDFXAJEX5LPOUJ32NKL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11

[Gregory P. Smith]

On Thu, Feb 24, 2022 at 3:27 PM Victor Stinner  wrote:

> On Thu, Feb 24, 2022 at 11:10 PM Barry  wrote:
> > > "Python 3.11 and newer versions use C11 without optional features. The
> > > public C API should be compatible with C++."
> > > https://github.com/python/peps/pull/2309/files
> >
> > Should is often read as meaning optional when writing specs.
> >  Can you say “must be compatible with C++”.
>
> I plan to attempt to write an actual test for that, rather than a
> vague sentence in a PEP. For now, "should" is a deliberate choice: I
> don't know exactly which C++ version should be targeted and if it's
> really an issue or not.
>

Agreed.  "should" is good because we're not even clear if we currently
actually comply with C++ standards.  i.e. https://bugs.python.org/issue40120
suggests we technically may not for C++ (it is not strictly a superset of C
as we all like to pretend), though for practical purposes regardless of
standards compilers tend to allow that.

We're likely overspecifying in any document we create about what we require
because the only definition any of us are actually capable of making for
what we require is "does it compile with this compiler on this platform? If
yes, then we appear to support it. can we guarantee that? only with
buildbots or other CI" - We're generally not versed in specific language
standards (aside from compiler folks, who is?), and compilers don't comply
strictly with all the shapes of those anyways for either practical or
hysterical reasons. So no matter what we claim to aspire to, reality is
always murkier.  A document about requirements is primarily useful to give
guidance to what we expect to be aligned with and what is or isn't allowed
to be used in new code.  Our code itself always has the final say.

-gps


> For example, C++20 reserves the "module" keyword, whereas Python uses
> it in its C API. Example:
>
> PyAPI_FUNC(int) PyModule_AddType(PyObject *module, PyTypeObject *type);
>
> See:
>
> * https://bugs.python.org/issue39355
> * https://github.com/pythoncapi/pythoncapi_compat/issues/21
>
> --
>
> I made a change in the datatable project to add Python 3.11 support
> using the pythoncapi_compat.h header file. Problem: this *C* header
> file produced new warnings in datatable extension module which with
> built with a C++ compiler:
> https://github.com/h2oai/datatable/pull/3231#issuecomment-1032864790
>
> Examples:
>
> | src/core/lib/pythoncapi_compat.h:272:52: warning: zero as null
> pointer constant [-Wzero-as-null-pointer-constant]
> ||| tstate->c_profilefunc != NULL);
> |^~~~
> |nullptr
>
> and
>
> | src/core/lib/pythoncapi_compat.h:170:12: warning: use of old-style
> cast [-Wold-style-cast]
> | return (PyCodeObject *)_Py_StealRef(PyFrame_GetCode(frame));
> |^   
>
> I made pythoncapi_compat.h compatible with C++ (fix C++ compiler
> warnings) by using nullptr and reinterpret_cast(EXPR) cast if
> the __cplusplus macro is defined, or NULL and ((TYPE)(EXPR)) cast
> otherwise.
>
> datatable also uses #include "Python.h". I don't know there were only
> C++ compiler warnings on "pythoncapi_compat.h". Maybe because
> datatable only uses static inline functions from
> "pythoncapi_compat.h", but it may also emit the same warnings if
> tomorrow some static inline functions of "Python.h" are used.
>
> For now, I prefer to put a reminder in PEP 7 that the "Python.h" C API
> is consumed by C++ projects.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RGNBM5CSUPBQSTZND4PHEV3WUEKS36TP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XUPAVKB7S2NCOGQY2JUMDBSTJADIOBPY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Rejection notice.

[Gregory P. Smith]

On Thu, Feb 10, 2022 at 3:32 AM Shantanu Jain 
wrote:

> Hello!
>
>
> Thanks for the decision, the points raised mostly make sense to me.
> However, I find myself and a few others are a little confused by point 2. I
> can read it as saying the following perhaps slightly contradictory things:
>
>
> "It's good that PEP 677 is conservative and sticks to things Callable can
> do"
>
> "PEP 677 isn't necessary, since Callable can do everything currently
> proposed"
>
> "PEP 677 could be a slippery slope for further syntax expansions that can
> do things Callable cannot"
>
>
> Would the case for new syntax have been stronger if it was proposing
> something Callable couldn't do? E.g., is the concern something like "The
> cost of new syntax is best paid by expanding the realm of what is
> expressible. While we see how PEP 677 could lead to such expansion in the
> future, the merits of future expansion are currently uncertain and the
> current proposal is too costly discussed on its own merits"?
>
>
> Or is the concern forward compatibility in the eventuality of further
> syntax expansions? PEP 677 did discuss "extended syntax", which the
> proposed syntax would be forward compatible with.
> https://www.python.org/dev/peps/pep-0677/#extended-syntax-supporting-named-and-optional-arguments
>
>
> Or something else entirely! Would appreciate any clarity :-)
>

At least we're as consistent as the zen of python itself? ;)  We
collectively struggled with how to word that one. Sorry if it is confusing.

>From my individual perspective, agreeing with point 2 came down to a belief
that if we accepted the simple syntax, a use would arise in the future
where it'd be desirable to gain something more complicated. Even if only
"3%" of callable annotations use those *today*. So we expected to wind up
in future releases with a proposal to extend the 677 callable syntax to
support parsing a more complete def. Good design in that it appears
technically feasible, but perceived bad in that the decision then would be
to go forward into an even more complicated, even harder to parse, world or
to back off and go another direction entirely which would leave 677
callable shorthand as a partial expressive dead end. Think of it as
*attempting* not to accumulate too many ways to do the same thing in the
long term.

If the 677 proposed syntax had been expanded to include more def features
instead of being conservative I think it would've been an even easier
rejection for many of us as that would've triggered more "point 4" feelings
of visual clutter and cognitive load.

-gps



>
> Thank you!
>
> On Wed, 9 Feb 2022 at 19:45, Gregory P. Smith  wrote:
>
>> Thank you PEP authors for producing a very well written and laid out PEP.
>> That made it easy to understand the proposal, rationale, and alternatives
>> considered. We agree that the existing typing.Callable syntax, while
>> capable, is less than ideal for humans. That said, we deliberated last week
>> and ultimately decided to reject PEP-677 Callable Type Syntax.
>>
>> Why? Several considerations led us here:
>>
>> 1. We feel we need to be cautious when introducing new syntax. Our new
>> parser presents understandably exciting opportunities but we don’t want its
>> existence to mean we add new syntax easily. A feature for use only in a
>> fraction of type annotations, not something every Python user uses, did not
>> feel like a strong enough reason to justify the complexity needed to parse
>> this new syntax and provide meaningful error messages. Not only code
>> complexity, humans are also parsers that must look forwards and backwards.
>>
>> 2. While the current Callable[x, y] syntax is not loved, it does work.
>> This PEP isn’t enabling authors to express anything they cannot already.
>> The PEP explicitly chose be conservative and not allow more syntax to
>> express features going beyond what Callable supports. We applaud that
>> decision, starting simple is good. But we can imagine a future where the
>> syntax would desire to be expanded upon.
>>
>> 3. In line with past SC guidance, we acknowledge challenges when syntax
>> desires do not align between typing and Python itself. Each time we add
>> syntax solely for typing it shifts us further in the direction of typing
>> being its own mini-language so we aim to tread lightly in what gets added
>> here. Adopting PEP 677 would lock us into its syntax forever, potentially
>> preventing other future syntax directions.
>>
>> 4. We did not like the visual and cognitive consequence of multiple `->`
>> tokens in a def. Especially when code is not formatted nicely. Th

[Python-Dev] PEP 677 (Callable Type Syntax): Rejection notice.

[Gregory P. Smith]

Thank you PEP authors for producing a very well written and laid out PEP.
That made it easy to understand the proposal, rationale, and alternatives
considered. We agree that the existing typing.Callable syntax, while
capable, is less than ideal for humans. That said, we deliberated last week
and ultimately decided to reject PEP-677 Callable Type Syntax.

Why? Several considerations led us here:

1. We feel we need to be cautious when introducing new syntax. Our new
parser presents understandably exciting opportunities but we don’t want its
existence to mean we add new syntax easily. A feature for use only in a
fraction of type annotations, not something every Python user uses, did not
feel like a strong enough reason to justify the complexity needed to parse
this new syntax and provide meaningful error messages. Not only code
complexity, humans are also parsers that must look forwards and backwards.

2. While the current Callable[x, y] syntax is not loved, it does work. This
PEP isn’t enabling authors to express anything they cannot already. The PEP
explicitly chose be conservative and not allow more syntax to express
features going beyond what Callable supports. We applaud that decision,
starting simple is good. But we can imagine a future where the syntax would
desire to be expanded upon.

3. In line with past SC guidance, we acknowledge challenges when syntax
desires do not align between typing and Python itself. Each time we add
syntax solely for typing it shifts us further in the direction of typing
being its own mini-language so we aim to tread lightly in what gets added
here. Adopting PEP 677 would lock us into its syntax forever, potentially
preventing other future syntax directions.

4. We did not like the visual and cognitive consequence of multiple `->`
tokens in a def. Especially when code is not formatted nicely. Though we
admit the correlation between Python typing and formatter users is high.

## Recommendation for future syntax enhancements:

When proposing a syntax change, low complexity is better. While not always
possible, it’s ideal if it could still be described using our old <=3.8
parser. It is important to have in mind that adding syntax that only our
modern PEG parser can handle could lead to greater cognitive load and
external tooling implementation costs.

This should not be read as a ban on PEG-only syntax, we just think it
should be used for broadly applicable features or else be relatively
unintrusive.

Thanks,
The 3.11 Python Steering Council
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NHCLHCU2XCWTBGF732WESMN42YYVKOXB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API

[Gregory P. Smith]

On Wed, Feb 9, 2022 at 8:54 AM Victor Stinner  wrote:

> On Wed, Feb 9, 2022 at 1:04 PM Petr Viktorin  wrote:
> > > Right now, a large number of macros cast their argument to a type. A
> > > few examples:
> > >
> > > #define PyObject_TypeCheck(ob, type)
> > > PyObject_TypeCheck(_PyObject_CAST(ob), type)
> > > #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
> > > #define PyDict_GET_SIZE(mp)  (assert(PyDict_Check(mp)),((PyDictObject
> > > *)mp)->ma_used)
> >
> > When I look at the Rationale points, and for the first three of these
> > macros I didn't find any that sound very convincing:
> > - Functions don't have macro pitfalls, but these simple macros don't
> > fall into the pits.
> > - Fully defining the argument types means getting rid of the cast,
> > breaking some code that uses the macro
> > - Debugger support really isn't that useful for these simple macros
> > - There are no new variables
>
> Using a static inline function, profilers like Linux perf can count
> the CPU time spend in static inline functions (on each CPU instruction
> when using annotated assembly code). For example, you can see how much
> time (accumulated time) is spent in Py_INCREF(), to have an idea of
> the cost of reference counting in Python. It's not possible when using
> macros.
>
> For debuggers, you're right that Py_INCREF() and PyTuple_GET_ITEM()
> macros are very simple and it's not so hard to guess that the debugger
> is executing their code in the C code or the assembly code. But the
> purpose of PEP 670 is to convert way more complex macros. I wrote a PR
> to convert unicodeobject.h macros, IMO there are one of the worst
> macros in Python C API:
> https://github.com/python/cpython/pull/31221
>
> I always wanted to convert them, but some core devs were afraid of
> performance regressions. So I wrote a PEP to prove that there is no
> impact on performance.
>
> IMO the new unicodeobject.h code is way more readable. I added two
> "kind" variables which have a well defined scope. In macros, no macro
> is used currently to avoid macro pitfalls: name conflict if there is
> already a "kind" macro where the macro is used.
>
> The conversion to static inline macros also detected a bug with "const
> PyObject*": using PyUnicode_READY() on a const Unicode string is
> wrong, this function modifies the object if it's not ready (WCHAR
> kind). It also detected bugs on "const void *data" used to *write*
> into string characters (PyUnicode_WRITE).
>

Nice example PR.  For now what we're suggesting is to proceed with changes
where it cannot lead to compilation warnings (failures in the widely used
-Werror mode) being introduced.  Yes, that may still leave a lot of
desirable cleanup to be done.  But we don't want to block everything while
discussing the rest. Incremental improvement is good even if not everything
is being done yet.

Two differing examples from that PR 31221:

Hold off as unsafe for now: Macros that do things like
(PyWhateverObject*)(op) such as PyUnicode_CHECK_INTERNED(op) should not
have the casting part of macros replaced yet. Doing so could result in a
warning or failure at -Werror compile time if someone was not using a
PyObject*.  Code is *supposed* to, but the cast means anyone could've used
PyUnicodeObject or whatnot itself.  Perhaps use a hybrid approach when
feasible similar to:
  #define PyUnicode_CHECK_INTERNED(op)
_PyUnicode_CHECK_INTERNED((PyASCIIObject *)(op))

That should make it safe.  And I believe you do mention this technique as
something to be used in the PEP.

Safe: PyUnicode_WRITE() in that PR. At first glance that is full of casts
on its data and value parameters so it raises suspicion. But further
analysis shows that data becomes a void* so there is no casting issue there
unless someone were passing a non-pointer in which isn't rightfully
something code should *ever* have done. And value would only be an issue if
someone were passing a non-integer type in, that also seems exceedingly
unlikely as there'd never be a point in writing code like that. So that
kind of change is fine to proceed with.

> The "massive change to working code" part is important. Such efforts
> > tend to have unexpected issues, which have an unfortunate tendency to
> > surface before release and contribute to release manager burnout.
>
> Aren't you exaggerating a bit? Would you mind to elaborate? Do you
> have examples of issues caused by converting macros to static inline
> functions?
>

Not quite the same but I've got a related example similar to what macros
casting pointer becoming a function accepting PyObject* without a cast
*could* do:

Between 3.6 and 3.7 we added const to a number of our public Python C APIs.

Migration to 3.7 required updating all sorts of C and C++ extension module
code to be pedantically correct, up through its call chain in some cases
with matching const declarations on types. (including conditional
compilation based on the Python version to support compilation under both
during 

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Gregory P. Smith]

On Tue, Feb 8, 2022 at 2:41 PM Steven D'Aprano  wrote:

> On Mon, Feb 07, 2022 at 06:23:52PM +, Mark Dickinson wrote:
>
> > - Should we require IEEE 754 floating-point for
> CPython-the-implementation?
> > - Should we require IEEE 754 floating-point for Python-the-language?
>
> If the answer to those questions are Yes, that rules out using Unums,
> posits, sigmoid numbers etc as the builtin float. (The terminology is a
> bit vague, sorry.) Do we want that?
>

It does not rule anything else out should they become viable.  This is just
a statement that to build cpython we require ieee754 support.  It does not
say anything about how our Python float type is implemented internally.

Should a meaningful large-os platform come along that promotes the use of a
different format available from C we could make use of that and loosen the
policy as needed.

What updating our requirement for CPython would mean is that the likely
unexercised outside our own __set_format__ using test suite
"unknown_format" code in
https://github.com/python/cpython/blob/main/Objects/floatobject.c could go
away until such time as a platform with an actual different format springs
into existence.

Driveby floatobject.c code inspection: It is odd that we do
ieee_be/ieee_le/unknown conditionals as a runtime check rather than
configure time check as that means we compile the code for three
implementations into our float implementation on all platforms despite them
each using only one - I guess that was done for testing purposes presumably
in the 1.x era when viable platforms were weirder as standards traction
grew - today I'd call that dead code bloat.

-gps


>
> https://ieeexplore.ieee.org/document/808
>
> https://en.wikipedia.org/wiki/Unum_%28number_format%29
>
> https://github.com/interplanetary-robot/SigmoidNumbers
>
> Posits are hyped as "better than IEEE-754", I have no idea if it is all
> hype or if they actually are better or just different.
>
>
> --
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7LUAQ32ZAHKWBJHLHUYEB7I5BZNDXGB7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N3Z5PFOALORZ4Z7KGNSHJ7QL47D4SYRJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Gregory P. Smith]

On Tue, Feb 8, 2022 at 2:25 PM Steven D'Aprano  wrote:

> On Mon, Feb 07, 2022 at 05:35:17PM -0800, Gregory P. Smith wrote:
>
> > CPython: yes.  we use a double.
> > Python the language: no.  (float is single precision on many micropython
> > platforms as it saves precious ram and performance, plus microcontroller
> > fpu hardware like an M4 is usually single precision 32bit)
>
> If we are to *officially* support non-double floats, it would be nice if
> sys.float_info were to tell us explicitly how wide the floats are rather
> than having to try to reverse engineer it from the other information
> there.
>
> A floating point expert can probably look at this:
>
> sys.float_info(max=1.7976931348623157e+308, max_exp=1024,
> max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021,
> min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16,
> radix=2, rounds=1)
>
> and immediately recognise that those values imply a 64-bit float, but I
> expect most people will not. If Python the language is going to support
> single, double, quad precision floats, and maybe even minifloats with
> just 16 or even fewer bits, then can we please add a field to float_info
> telling us how many bits the floats have?
>

There is no need to know how many bits it is. The meaningful information
about precision and accuracy from a math point of view is already expressed
in float_info.  the size in bits isn't relevant.  You can derive the size
from that if you'd like and are willing to assume a binary format.
binary_bits = ~mant_dig+log2(max_exp)... But that tells you less than
sys.float_info.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CRWZFO5RV2FLXIBH6GV6QGSDXWUZOZCR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Replace debug runtime checks in release mode with assertions in debug mode

[Gregory P. Smith]

What does pyperformance say about --enable-optimizations builds with all of
those compiled out vs today?

I like the runtime safety checking for purposes of making the lives of
Python C API users easier. But without the use of assertion enabled CPython
builds being the norm during development (we should try to shift that norm
in the community and make it the popular CI system default regardless), I'm
concerned this would hurt the developers who need it most.

Having some idea of what it actually gains us performance wise would be
useful.

-gps

On Mon, Feb 7, 2022 at 3:30 PM Brett Cannon  wrote:

>
>
> On Mon, Feb 7, 2022 at 8:59 AM Victor Stinner  wrote:
>
>> On Mon, Feb 7, 2022 at 5:48 PM Guido van Rossum  wrote:
>> > So you're proposing to completely get rid of those three?
>>
>> I don't propose to remove them, but only call them if Python is built
>> in debug mode. Or remove them from the release build, unless
>> ./configure --with-assertions is used.
>>
>>
>> > And you're sure that each and every single call to any of those is
>> better off being an assert()?
>>
>> For many years, many C extensions raised an exception *and* returned a
>> result: that's a bug. The strange part is that in some cases, the
>> exceptions is somehow ignored and the program continues running fine.
>>
>> That's why I added _Py_CheckFunctionResult() and _Py_CheckSlotResult()
>> which helped to catch such bugs. But before that, these programs were
>> running fine :-)
>>
>> So it's not fully clear to me there was really a bug or it's just that
>> Python became more pedantic :-)
>>
>>
>> About PyErr_BadInternalCall(): in 10 years, I saw a few SystemError
>> raised by this function, but usually when I hacked on Python. It's
>> really rare to hit such bug.
>>
>>
>> > (I still haven't gotten into the habit of building in debug mode by
>> default, in part because it *isn't* the default when you invoke ./configure
>> or PCbuild/build.bat.)
>>
>> If you don't develop C extensions, the release mode is faster and enough
>> ;-)
>>
>> Ah. I don't know if CIs like GitHub Actions and Azure Pipelines
>> provide Python debug builds. If if it's not the case, it would be nice
>> to have the choice :-)
>>
>
> They do not:
> https://github.com/actions/python-versions/blob/797eb71c41e47d194f563c7ef01790d734534788/builders/ubuntu-python-builder.psm1#L35-L38
> .
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VRISRYZVU47PQRWKHN77CV5545SKUI5O/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XNB4ZUTFJFRCDCG2HJ36INGH3PMLFMP6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Gregory P. Smith]

On Mon, Feb 7, 2022 at 4:52 PM Christopher Barker 
wrote:

> From the perspective of some that writes a lot of computational code:
>
> On Mon, Feb 7, 2022 at 10:25 AM Mark Dickinson  wrote:
>
>> - Should we require the presence of NaNs in order for CPython to build?
>> - Should we require IEEE 754 floating-point for
>> CPython-the-implementation?
>>
>
> Yes, and yes, together, as Mark suggests
>
> - Should we require IEEE 754 floating-point for Python-the-language?
>>
>
> I would say no, but it should be recommended -- see Victor's example of
> Micro Python -- though does anyone have an authority over that? IIUC, Micro
> Python already has a few differences -- but is anyone saying they shouldn't
> call it Python? Though, with a quick perusal of teh docs, micro Python does
> seem to support NaN and Inf, at least -- not sure about the rest of 754.
>

PEP 7 and PEP 11 are only about CPython. Other Python VMs full or partial
are free to do what they want with how they implement floats if at all.
(they're build time optional in MicroPython as are many Python language
syntax features)

While we're at it are  64 bit floats required for either cPython or Python
> the language?
>

CPython: yes.  we use a double.
Python the language: no.  (float is single precision on many micropython
platforms as it saves precious ram and performance, plus microcontroller
fpu hardware like an M4 is usually single precision 32bit)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X5LWPTTQHS7BZFARSJ3BO5PDO27CRKWO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?

[Gregory P. Smith]

On Mon, Feb 7, 2022 at 11:06 AM Victor Stinner  wrote:

> Hi Mark,
>
> Aha, good, you posted an email to python-dev, good :-) Last days, I
> was trying to collect more data about this topic, especially find
> platforms which *don't* support IEEE 754, before posting to
> python-dev.
>
> Nowadays, outside museums, it's hard to find computers which don't
> implement IEEE 754.
>
> * Since 1998, IBM S/390 supports IEEE 754.
> * Compaq/DEC VAX didn't use IEEE 754, but are no longer built since
> 2000 (you cant still use Python 3.10 if it can be built on it!)
> * Some Cray computers like Cray SV1 (1998) don't implement IEEE 754,
> but other Cray computers like Cray T90 (1995) implement IEEE 754.
>
> There are embedded devices with no hardware FPU: in this case, the FPU
> is implemented in software. I expect it to implement IEEE 754. Is
> CPython a good target for such small CPUs which have no hardware FPU?
> MicroPython may better fit their needs.
>
> On the other side, all moderns CPU architectures support IEEE 754:
> Intel x86, ARM, IBM Power and PowerPC, Compaq/DEC Alpha, HP PA-RISC,
> Motorola 68xxx and 88xxx, SGI R-, Sun SPARC.
>
> Sources:
>
> *
> https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers
> *
> https://stackoverflow.com/questions/2234468/do-any-real-world-cpus-not-use-ieee-754
>
>
> On Mon, Feb 7, 2022 at 7:25 PM Mark Dickinson  wrote:
> > - Should we require the presence of NaNs in order for CPython to build?
> > - Should we require IEEE 754 floating-point for
> CPython-the-implementation?
>
> In the past, when we deprecated the support for an old platform, we
> didn't suddenly remove the code. We made sure that it's no longer
> possible to build on it. So if anyone notices, it's easy to revert
> (ex: remove the few lines in configure).
>
> Would it make sense to trigger a build error on plaforms which don't
> support IEEE 754 in Python 3.11, and later decide if it's time to
> remove the code in in Python 3.12?
>
> Well. If you ask *me*, I'm in favor of removing the code right now. If
> someone needs to support a platform which doesn't support IEEE 754,
> the support can be maintained *outside* the Python source code, as
> external patches or as a Git fork, no?
>
> Honestly, I never got access to any machine which doesn't support IEEE
> 754 (or nobody told me!).
>
>
> > - Should we require IEEE 754 floating-point for Python-the-language?
>
> Right now, I have no opinion on this question.
>
>
> > Note that on the current main branch there's a Py_NO_NAN macro that
> builders can define to indicate that NaNs aren't supported, but the Python
> build is currently broken if Py_NO_NAN is defined (see
> https://bugs.python.org/issue46656). If the answer to the first question
> is "No", then we need to fix the build under Py_NO_NAN. That's not a big
> deal - perhaps a couple of hours of work.
>
> My comment on bpo-46656: "Python uses Py_NAN without "#ifdef Py_NAN"
> guard since 2008. Building Python with Py_NO_NAN never worked. Nobody
> reported such build failure in the last 14 years..."
>
> If anyone would try building/using Python on a platform without NAN, I
> would expect that we would get a bug report or an email. I'm not aware
> of anything like that, so it seems like nobody uses Python on such
> platform.
>
> Victor
>

It's 2022. We should just require both NaN and IEEE-754.

By the time a system is large enough to build and run CPython, it is
exceedingly unlikely that it will not be able to do that. Even if it means
software emulation of IEEE-754 floating point on something large yet odd
enough not to have a FPU.

The places where non-IEEE-754 and non-NaN floating point are likely to
exist are in specialized parallelized coprocessor hardware. Not the general
purpose CPU running CPython.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UQXG7E6ONOQFCIES4PQIMJFZDXNGQJTM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Gregory P. Smith]

On Sun, Feb 6, 2022 at 9:13 AM Paul Moore  wrote:

> On Sun, 6 Feb 2022 at 16:51, Christian Heimes 
> wrote:
>
> > The urllib package -- and to some degree also the http package -- are
> > constant source of security bugs. The code is old and the parsers for
> > HTTP and URLs don't handle edge cases well. Python core lacks a true
> > maintainer of the code. To be honest, we have to admit defeat and be up
> > front that urllib is not up to the task for this decade. It was designed
> > written during a more friendly, less scary time on the internet.
> >
> > If I had the power and time, then I would replace urllib with a simpler,
> > reduced HTTP client that uses platform's HTTP library under the hood
> > (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> > maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> > aiohttp are much better suited than urllib.
> >
> > The second best option is to reduce the feature set of urllib to core
> > HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter,
> > more standard conform parsers for urls, query strings, and RFC 2822
> > instead of RFC 822 for headers.
>
> I'd likely be fine with either of these two options. I'm not worried
> about supporting "advanced" uses. But having no way of getting a file
> from the internet without relying on 3rd party packages seems like a
> huge gap in functionality for a modern language. And having to use a
> 3rd party library to parse URLs will simply push more people to use
> home-grown regexes rather than something safe and correct. Remember
> that a lot of Python users are not professional software developers,
> but scientists, data analysts, and occasional users, for whom the
> existence of something in the stdlib is the *only* reason they have
> any idea that URLs need specialised parsing in the first place.
>
> And while we all like to say 3rd party modules are great, the reality
> is that they provide a genuine problem for many of these
> non-specialist users - and I say that as a packaging specialist and
> pip maintainer. The packaging ecosystem is *not* newcomer-friendly in
> the way that core Python is, much as we're trying to improve that
> situation.
>
> I've said it previously, but I'll reiterate - IMO this *must* have a
> PEP, and that PEP must be clear that the intention is to *remove*
> urllib, not simply to "deprecate and then think about it". That could
> be making it part of PEP 594, or a separate PEP, but one way or
> another it needs a PEP.
>

This would need to be it's own PEP.  urllib et. al. are used by virtually
everybody.  They're highly used batteries.

I'm -1 on deprecating it for that reason alone.

Christian proposes that having a simpler scope rewrite of it might be nice,
but I think disruption to the world and loss of trust in Python would be
similar either way.

-gps


>
> Paul
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KT6TGUBTLLETHES2OVVGZWSGYC5JCEKC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QMSFZBQJFWKFFE3LFQLQE2AT6WKMLPGL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone

[Eric V. Smith]

On 2/3/2022 12:15 PM, Victor Stinner wrote:

I'm working bottom-to-top: prepare PyObject and PyVarObject to become
opaque, *and* top-to-bottom: prepare subclasses (structures
"inheriting" from PyObject and PyVarObject) to become opaque like
PyFrameObject.

IMO if PyObject* becomes a handle, the migration to the HPy API should
be much easier.


It seems to me that moving PyObject* to be a handle leaves you in a 
place very similar to HPy. So why not just focus on making HPy suitable 
for developing C extensions, leave the existing C API alone, and 
eventually abandon the existing C API?


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BWCNBP26BZU2SYLCBVCXXVBMYUTSHE27/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving away from _Py_IDENTIFIER().

[Gregory P. Smith]

On Wed, Feb 2, 2022 at 2:48 PM Eric Snow 
wrote:

> I'm planning on moving us to a simpler, more efficient alternative to
> _Py_IDENTIFIER(), but want to see if there are any objections first
> before moving ahead.  Also see https://bugs.python.org/issue46541.
>
> _Py_IDENTIFIER() was added in 2011 to replace several internal string
> object caches and to support cleaning up the cached objects during
> finalization.  A number of "private" functions (each with a
> _Py_Identifier param) were added at that time, mostly corresponding to
> existing functions that take PyObject* or char*.  Note that at present
> there are several hundred uses of _Py_IDENTIFIER(), including a number
> of duplicates.
>
> My plan is to replace our use of _Py_IDENTIFIER() with statically
> initialized string objects (as fields under _PyRuntimeState).  That
> involves the following:
>
> * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for
> each string that currently uses _Py_IDENTIFIER() (or
> _Py_static_string())
> * statically initialize each object as part of the initializer for
> _PyRuntimeState
> * add a macro to look up a given global string
> * update each location that currently uses _Py_IDENTIFIER() to use the
> new macro instead
>
> Pros:
>
> * reduces indirection (and extra calls) for C-API functions that need
> the strings (making the code a little easier to understand and
> speeding it up)
> * the objects are referenced from a fixed address in the static data
> section instead of the heap (speeding things up and allowing the C
> compiler to optimize better)
> * there is no lazy allocation (or lookup, etc.) so there are fewer
> possible failures when the objects get used (thus less error return
> checking)
> * saves memory (at little, at least)
> * if needed, the approach for per-interpreter is simpler
> * helps us get rid of several hundred static variables throughout the code
> base
> * allows us to get rid of _Py_IDENTIFIER() and a bunch of related
> C-API functions
> * "deep frozen" modules can use the global strings
> * commonly-used strings could be pre-allocated by adding
> _PyRuntimeState fields for them
>
> Cons:
>
> * a little less convenient: adding a global string requires modifying
> a separate file from the one where you actually want to use the string
> * strings can get "orphaned" (I'm planning on checking in CI)
> * some strings may never get used for any given ./python invocation
> (not that big a difference though)
>
> I have a PR up (https://github.com/python/cpython/pull/30928) that
> adds the global strings and replaces use of _Py_IDENTIFIER() in our
> code base, except for in non-builtin stdlib extension modules.  (Those
> will be handled separately if we proceed.)  The PR also adds a CI
> check for "orphaned" strings.  It leaves _Py_IDENTIFIER() for now, but
> disallows any Py_BUILD_CORE code from using it.
>
> With that change I'm seeing a 1% improvement in performance (see
> https://github.com/faster-cpython/ideas/issues/230).
>
> I'd also like to actually get rid of _Py_IDENTIFIER(), along with
> other related API including ~14 (private) C-API functions.  Dropping
> all that helps reduce maintenance costs.  However, at least one PyPI
> project (blender) is using _Py_IDENTIFIER().  So, before we could get
> rid of it, we'd first have to deal with that project (and any others).
>

datapoint: an internal code search turns up blender, multidict, and
typed_ast as open source users of _Py_IDENTIFIER .  Easy to clean up as
PRs.  There are a couple of internal uses as well, all of which are
similarly easy to address and are only in code that is expected to need API
cleanup tweaks between CPython versions.

Overall I think addressing the broader strategy question among
the performance focused folks is worthwhile though.

-gps


>
> To sum up, I wanted to see if there are any objections before I start
> merging anything.  Thanks!
>
> -eric
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BCQJ6AZMPTI2DGFQPC27RUIFJQGDIOQD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Steering-council] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Gregory P. Smith]

-cc: python-steering-council

On Fri, Mar 5, 2021 at 4:26 PM Guido van Rossum  wrote:

> On Fri, Mar 5, 2021 at 11:11 AM Brett Cannon  wrote:
>
>> Speaking for myself ...
>>
>
> Ditto ...
>
> On Fri, Mar 5, 2021 at 7:04 AM Mark Shannon  wrote:
>> [...]
>>
>>> In some cases, the PEP would have improved the situation.
>>>
>>> For example:
>>> sys.setrecursionlimit(5000)
>>> def f():
>>>  f()
>>>
>>> Currently, it raises a RecursionError on linux, but crashes the
>>> interpreter on Windows.
>>> With PEP 651 it would have raised a RecursionError on both platforms.
>>>
>>> Am I missing something here?
>>>
>>
>> So your example shows a user already comfortable in raising their
>> recursion limit to work around needing more stack space to reach
>> completion. What is stopping the user from continuing to raise the limit
>> until they still reach their memory limit even with PEP 651? If you're
>> worried about runaway recursion you will very likely hit that with the
>> default stack depth already, so I personally don't see how a decoupled
>> stack counter from the C stack specifically makes it any easier/better to
>> detect runaway recursion. And if I need more recursion than the default,
>> you're going to bump the recursion depth anyway, which weakens the
>> protection in either the C or decoupled counter scenarios. Sure, it's
>> currently platform-specific, but plenty of people want to push that limit
>> based on their machine anyway and don't need consistency on platforms they
>> will never run on, i.e. I don't see a huge benefit to being able to say
>> that an algorithm consistently won't go past 5000 calls on all platforms
>> compared to what the C stack protection already gives us (not to say
>> there's zero benefit, but it isn't massive or widespread either IMO). I
>> personally just don't see many people saying, "I really want to limit my
>> program to an exact call stack depth of 5000 on all platforms which is
>> beyond the default, but anything under is fine and anything over --
>> regardless of what the system can actually handle -- is unacceptable".
>>
>> Tack on the amount of changes required to give a cross-platform stack
>> count and limit check compared to the benefit being proposed, and to me
>> that pushes what the PEP is proposing into net-negative payoff.
>>
>
> To me, the point of that example is as a reminder that currently fiddling
> with the recursion limit can cause segfaults.
>
> Mark's PEP proposes two, somewhat independent, changes: (1) don't consume
> C stack on pure Python-to-Python (pp2p) function calls; (2) implement
> fool-proof C stack overflow checks.
>
> Change (2) makes it safe for users to mess with the stack overflow limit
> however they see fit. Despite (1), the limit for pp2p calls remains at 1000
> so that users who unintentionally write some naively recursive code don't
> have to wait until they fill up all of memory before they get a traceback.
> (Of course they could also write a while-True loop that keeps adding an
> item to a list and they'd end up in the same situation. But in my
> experience that situation is less painful to deal with than accidental
> stack overflow, and I'd shudder at the thought of a traceback of a million
> lines.)
>
> Given that we have (1), why is (2) still needed? Because there are ways to
> recursively call Python code that aren't pp2p calls. By a pp2p (pure
> Python-to-Python) call, I mean any direct call, e.g. a method call or a
> function call. But what about other operations that can invoke Python code?
> E.g. if we have a dict d and a class C, we could create an instance of C
> and use it to index d, e.g. d[C()]. This operation is not a p2pp call --
> the BINARY_SUBSCR opcode calls the dict's `__getitem__` method, and that
> calls the key's `__hash__` method. Here's a silly demonstration:
> ```
> def f(c):
> d = {}
> return d[c]
>
> class C:
> def __hash__(self):
> return f(self)
>
> f(C())
> ```
> Note that the "traceback compression" implemented for simple recursive
> calls fails here -- I just ran this code and got 2000 lines of output.
>
> The way I imagine Mark wants to implement pp2p calls means that in this
> case each recursion step *does* add several other C stack frames, and this
> would be caught by the limit implemented in (2). I see no easy way around
> this -- after all the C code involved in the recursion could be a piece of
> 3rd party C code that itself is not at fault.
>
> So we could choose to implement only (2), robust C stack overflow checks.
> This would require a bunch of platform-specific code, and there might be
> platforms where we don't know how to implement this (I vaguely recall a
> UNIX version where main() received a growable stack but each thread only
> had a fixed 64kb stack), but those would be no worse off than today.
>
> Or we could choose to implement only (1), eliminating C stack usage for
> pp2p calls. But in that case we'd still need a recursion limit for non-pp2p
> calls. 

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Ethan Smith]

On Sat, Jan 29, 2022 at 7:38 PM Inada Naoki  wrote:

> On Sun, Jan 30, 2022 at 12:03 PM Ethan Smith  wrote:
> >
> > As a non-committer, I want to make a plea for non-committer approval
> reviews, or at least that they have a place. When asked how outsiders can
> contribute I frequently see "review open PRs" as a suggested way of
> contributing to CPython. Not being able to approve PRs that are good would
> be a barrier to those contributions.
> >
> > Furthermore, I am collaborating with a couple of core devs, it would
> make collaboration more difficult if I couldn't review their work and say
> that I thought the changes looked good.
> >
>
> You can still write a review comment, including "LGTM".


Fair enough, I suppose commenting with "LGTM" works just as well.


> What you can
> not is labeling PR as "Approved."
> So I don't think it makes collaboration difficult.
>
By preventing approval from others, we can easily find PRs approved
> from core-devs or triage members but not merged yet.
>

> > I know "drive by approvals" are annoying but I think it is unfortunately
> part of open source projects.
> >
>
> Sorry, but I don't think so.
>

Well, if you disallow drive-by approvals, you will still get drive-by LGTM
comments. People seem to believe that adding their approval will expedite a
PR merge, for some reason (or want to bump a stale PR in hopes of a quicker
merge).


>
> --
> Inada Naoki  
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QJ5QBB4XCFU3DFSOFBRUJB5XC4UMX7TK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Ethan Smith]

As a non-committer, I want to make a plea for non-committer approval
reviews, or at least that they have a place. When asked how outsiders can
contribute I frequently see "review open PRs" as a suggested way of
contributing to CPython. Not being able to approve PRs that are good would
be a barrier to those contributions.

Furthermore, I am collaborating with a couple of core devs, it would make
collaboration more difficult if I couldn't review their work and say that I
thought the changes looked good.

I know "drive by approvals" are annoying but I think it is unfortunately
part of open source projects.


On Sat, Jan 29, 2022, 5:51 PM Inada Naoki  wrote:

> On Sun, Jan 30, 2022 at 10:39 AM Ethan Furman  wrote:
> >
> >
> >  > And lots of non-committer PR reviews that only approve.
> >
> > I have seen this.  Quite irritating.
> >
>
> We can prohibit approval from non core developers. Do we use this
> setting already?
>
> https://github.blog/2021-11-01-github-keeps-getting-better-for-open-source-maintainers/
>
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KHXSE2MSEC5JR2QB5F6HJUFYCC4SHGFF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BKIUK2KGP2YMN7GBEYYQVTEMECE2WTSD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Gregory P. Smith]

On Thu, Jan 20, 2022 at 8:16 PM Dan Stromberg  wrote:

>
> On Fri, Apr 16, 2021 at 11:13 AM Christian Heimes 
> wrote:
>
>> On 16/04/2021 19.14, redrad...@gmail.com wrote:
>> > My personal stop of contributing in CPython is that it is written in
>> pure C !!
>> > I wrote code in both: pure C and C++, but I like writing code in C++,
>> because it simplifies things without losing perfomance
>>
>> There are plenty of Open Source projects that could use more capable C++
>> developers. :)
>>
>> I'm not a fan of C++. It has its use cases, e.g. in UI. Python core
>> isn't the best fit. AFAIK most core devs are not fluent in C++. Despite
>> it's name, C++ is really a different language than C. It has a different
>> ABI and stdlib than C, too. In my personal opinion C++ won't give us any
>> net benefits. I'd much rather go for Rust than C++ to gain memory safety.
>>
> Agreed.
>
> Rust would be much better than C++.
>

At least one Python runtime has been written in Rust already:
https://github.com/RustPython/RustPython

The fundamental reason we haven't embarked on new language things in
CPython is compatibility and complexity.  If there were clear demonstrable
wins to adopting a language in addition to C for the CPython runtime, we
could consider it (ultimately as a PEP with reasoning, alternatives, and
rejected ideas all laid out). But it isn't going to be done on a whim as it
a new language doesn't magically solve problems, it just adds more problems.

Of the three things listed at the start of this thread, the first two don't
exist. From huge codebase experience with C++, it does not cause
significantly better (1) Readabillity or (2) Maintainability on its own
compared to C - both of those are entirely up to the software engineering
practices adopted within the project no matter what the language is.  (3)
RAII is the number one thing I would enjoy having in the CPython
internals.  Manually managing Py_DECREFs is inhumane.  As a result we have
built up infrastructure to detect leaks in our test suite so new refcount
leak problems *usually* don't happen - so long as test coverage is in place
(ex: 3.10.2...). And RAII isn't magic, even it can be used wrong. It's just
that the location and spelling of the wrong changes.

Rewrites might sound "easy" (emphasis on the air quotes! nobody should
think it's easy)... but only until you actually have to match API and ABI
and bug for bug compatibility with the entire corpus of existing real world
code. Just ask PyPy. ;)

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LTYWGK7FS7P62BK5OB4XVUCXLN2YFOJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Gregory P. Smith]

On Tue, Jan 18, 2022 at 6:24 AM Victor Stinner  wrote:

> Hi,
>
> My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on
> updating Python 3.10 to Python 3.11 in Fedora, but some specific
> Python 3.11 incompatible changes are causing more troubles than
> others:
> https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911
>
> We propose to revert the following 2 changes in Python 3.11 and
> postpone them in a later Python version, once most projects will be
> compatible with these changes:
>
> * Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages
> * Removals from configparser module (bpo-45173) - broke 28 Fedora packages
>
> --
>
> We reported the issue to many affected projects, or when possible, we
> also wrote a pull request to make the code compatible (lot of those
> were made by others, e.g. Hugo van Kemenade, thank you!).
>

+1 to rolling both of these back for 3.11.  Deprecation removals are hard.
Surfacing these to the impacted upstream projects to provide time for those
to integrate the changes is the right way to make these changes stick in
3.12 or later.  Thanks for doing a significant chunk of that work!

As you've done the work to clean up a lot of other OSS projects, I suggest
we defer this until 3.12 with the intent that we won't defer it again. That
doesn't mean we can't hold off on it, just that we believe pushing for this
now and proactively pushing for a bunch of cleanups has improved the state
of the world such that the future is brighter.  That's a much different
strategy than our passive aggressive DeprecationWarnings.

-gps


>
> The problem is that fixing a Fedora package requires multiple steps:
>
> * (1) Propose a pull request upstream
> * (2) Get the pull request merged upstream
> * (3) Wait for a new release upstream
> * (4) Update the Fedora package downstream, or backport the change in
> Fedora (only needed by Fedora)
>
> Identifying the Python incompatible changes causing most troubles took
> us a lot of time, but we did this work. Reverting the two Python 3.11
> incompatible changes (causing most troubles) will save us "bug triage"
> time, to get more time on updating projects to Python 3.11 for the
> other remaining incompatible changes.
>
> We are not saying that these incompatible changes are bad, it's just a
> matter of getting most projects ready before Python 3.11 final will be
> released.
>
> --
>
> By the way, before making a change known to be incompatible, it would
> be nice to run a code search on PyPI top 5000 projects to estimate how
> many projects will be broken, and try to update these projects
> *before* making the change.
>
> For example, only introduce an incompatible change into Python once
> less than 15 projects are affected. Getting a pull request merged is
> nice, but a release including the fix is way better for practical
> reasons: people use "pip install ".
>
> --
>
> Fedora work on Python 3.11 is public and tracked at:
> https://bugzilla.redhat.com/show_bug.cgi?id=PYTHON3.11
>
> Click on "depends on" to see current issues:
>
> https://bugzilla.redhat.com/buglist.cgi?bug_id=2016048_id_type=anddependson=tvp
>
> Example of bz #2025600: mom fails to build with Python 3.11:
> AttributeError: module 'configparser' has no attribute
> 'SafeConfigParser'.
>
>
> Victor Stinner -- in the name of the Python Red Hat / Fedora maintenance
> team
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GJTREADEXYAETECE5JDTPYWK4WMTKYGR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3J3VKNTKGPWACFVDWPRCS7FNED2A34R4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11

[Gregory P. Smith]

On Tue, Jan 18, 2022 at 10:58 PM Christopher Barker 
wrote:

> On Tue, Jan 18, 2022 at 10:30 AM Brett Cannon  wrote:
>
>>  I remember that "noisy by default" deprecation warnings were widely
>>> despised.
>>>
>>> One thought, what if they were off by default UNLESS you were doing unit
>>> tests?
>>>
>>
>> I believe pytest already does this.
>>
>
> Indeed it does, at least in recent versions (1-2 yrs ago?)
>
> And even that is pretty darn annoying. It's really helpful for my code,
> but they often get lost in the noise of all the ones I get from upstream
> packages.
>
> I suppose I need to take the time to figure out how to silence the ones I
> don't want.
>
> And it does prompt me to make sure that the upstream packages are working
> on it.
>
> Now we just need to get more people to use pytest :-)
>

Our stdlib unittest already enables warnings by default per
https://bugs.python.org/issue10535.

Getting the right people to pay attention to them is always the hard part.


>
> -CHB
>
> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LZAF3TOOIAAFXXK2KPAXA5V5SRBOSIIP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5YOEMAIVXMVWCORAY54LZQO62755HQGX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Sun, Jan 16, 2022 at 1:51 PM Mark Dickinson  wrote:

> On Sun, Jan 16, 2022 at 9:28 PM Guido van Rossum  wrote:
>
>> Does the optimization for //10 actually help in the real world? [...]
>>
>
> Yep, I don't know. If 10 is *not* the most common small divisor in real
> world code, it must at least rank in the top five. I might hazard a guess
> that division by 2 would be more common, but I've no idea how one would go
> about establishing that.
>

All of the int constants relating to time and date calculations show up
frequently as well.  But I'd assume -fprofile-values isn't likely to pick
many to specialize on to avoid adding branches so maybe 10 is ironically
it.  --enable-optimizations with clang doesn't trigger value specialization
(I'm pretty sure they support the concept, but I've never looked at how).

>
> The reason that the divisor of 10 is turning up from the PGO isn't a
> particularly convincing one - it looks as though it's a result of our
> testing the builtin int-to-decimal-string conversion by comparing with an
> obviously-correct repeated-division-by-10 algorithm.
>
> Then again I'm not sure what's *lost* even if this optimization is
>> pointless -- surely it doesn't slow other divisions down enough to be
>> measurable.
>>
>
> Agreed. That at least is testable. I can run some timings (but not
> tonight).
>

BTW, I am able to convince clang 11 and higher to produce a 64:32 divide
instruction with a modified version of the code. Basically just taking your
assembly divl variant as an example and writing that explicitly as the
operations in C:

https://godbolt.org/z/63eWPczjx

Taking that code and turning it into an actual test within CPython itself,
it appears to deliver the desired speedup in gcc9 as well.
https://github.com/python/cpython/pull/30626 for
https://bugs.python.org/issue46406.

20% faster microbenchmarking with x//1 or x//17 or other non-specialized
divide values.  similar speedup even in --enable-optimizations builds.
with both gcc9 and clang13.

The compilers seem happier optimizing that code.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FJRKRXSXPF24C3NHGYZMVPB3ZZPCBI6A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Fri, Jan 14, 2022 at 9:50 AM Mark Dickinson  wrote:

> On Sun, Jan 2, 2022 at 10:35 AM Mark Dickinson  wrote:
>
>> Division may still be problematic.
>>
>
> On that note: Python divisions are somewhat crippled even on x64. Assuming
> 30-bit digits, the basic building block that's needed for multi-precision
> division is a 64-bit-by-32-bit unsigned integer division, emitting a 32-bit
> quotient (and ideally also a 32-bit remainder). And there's an x86/x64
> instruction that does exactly that, namely DIVL. But without using inline
> assembly, current versions of GCC and Clang apparently can't be persuaded
> to emit that instruction from the longobject.c source - they'll use DIVQ (a
> 128-bit-by-64-bit division, albeit with the top 64 bits of the dividend set
> to zero) on x64, and the __udivti3 or __udivti4 intrinsic on x86.
>
> I was curious to find out what the potential impact of the failure to use
> DIVL was, so I ran some timings. A worst-case target is division of a large
> (multi-digit) integer by a single-digit integer (where "digit" means digit
> in the sense of PyLong digit, not decimal digit), since that involves
> multiple CPU division instructions in a fairly tight loop.
>
> Results: on my laptop (2.7 GHz Intel Core i7-8559U, macOS 10.14.6,
> non-optimised non-debug Python build), a single division of 10**1000 by 10
> takes ~1018ns on the current main branch and ~722ns when forced to use the
> DIVL instruction (by inserting inline assembly into the inplace_divrem1
> function). IOW, forcing use of DIVL instead of DIVQ, in combination
> with getting the remainder directly from the DIV instruction instead of
> computing it separately, gives a 41% speedup in this particular worst case.
> I'd expect the effect to be even more marked on x86, but haven't yet done
> those timings.
>
> For anyone who wants to play along, here's the implementation of the
> inplace_divrem1 (in longobject.c) that I was using:
>
> static digit
> inplace_divrem1(digit *pout, digit *pin, Py_ssize_t size, digit n)
> {
> digit remainder = 0;
>
> assert(n > 0 && n <= PyLong_MASK);
> while (--size >= 0) {
> twodigits dividend = ((twodigits)remainder << PyLong_SHIFT) | pin[size];
> digit quotient, high, low;
> high = (digit)(dividend >> 32);
> low = (digit)dividend;
> __asm__("divl %2\n"
> : "=a" (quotient), "=d" (remainder)
> : "r" (n), "a" (low), "d" (high)
> );
> pout[size] = quotient;
> }
> return remainder;
> }
>
>
> I don't know whether we *really* want to open the door to using inline
> assembly for performance reasons in longobject.c, but it's interesting to
> see the effect.
>

That only appears true in default boring -O2 builds.  Use `./configure
--enable-optimizations` and the C version is *much* faster than your asm
one...

250ns for C vs 370ns for your asm divl one using old gcc 9.3 on my zen3
when compiled using --enable-optimizations.

tested using ` -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r ==
10**999, r' 'x//10' `

Use of __asm__ appears to have prevented the compiler from being able to
fully optimize that code in PGO/FDO mode.

I trust the compiler toolchain to know what's close to best here.
Optimizing use of a div instruction isn't entirely straight forward as on
many microarchitectures the time required varies based on the inputs as
they'll internally implement looping when values exceed the bits with which
their hw operates.  there's probably interesting ways to optimize bignum
division using opencl and vector hardware as well - for the case when you
know you've got over a dozen digits; but that's what numpy et. al. are for.
Bignums in python are a convenience. Most values normal code deals with are
less than 2**100.

-Greg


>
> --
> Mark
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZWGPO3TMCI7WNLC3EMS26DIKI5D3ZWMK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAZGSHPS7LBN4QNYVVSUG2RC26322L5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Minor inconvenience: f-string not recognized as docstring

[Eric V. Smith]

On 1/11/2022 3:44 PM, Brett Cannon wrote:



On Tue, Jan 11, 2022 at 10:40 AM Gregory P. Smith  wrote:


On Tue, Jan 11, 2022 at 10:29 AM Guido van Rossum
 wrote:

I personally think F-strings should not be usable as
docstrings. If you want a dynamically calculated docstring you
should assign it dynamically, not smuggle it in using a
string-like expression. We don't allow "blah {x}
blah".format(x=1) as a docstring either, not "foo %s bar" % x.


Agreed.  If we wanted to remove the wart of constant f-strings
happening to work as an implementation detail in this context,
that /could/ be made into a warning.  But that kind of check may
be best left to a linter for /all/ of these dynamic situations
that don't wind up populating __doc__.


Agreed on not supporting it and linters catching such a mistake.


Just to be clear, we don't support this.

>>> class C: 'foo'
...
>>> C.__doc__ == 'foo'
True

>>> class C: f'foo'
...
>>> C.__doc__ == 'foo'
False
>>> C.__doc__ is None
True

And there's a test to make sure constant f-strings are not doc strings: 
https://github.com/python/cpython/blob/dce642f24418c58e67fa31a686575c980c31dd37/Lib/test/test_fstring.py#L406


Eric



-Brett


-gps


On Tue, Jan 11, 2022 at 8:12 AM Antoine Pitrou
 wrote:

    On Tue, 11 Jan 2022 10:58:03 -0500
"Eric V. Smith"  wrote:
> Constant f-strings (those without substitutions) as doc
strings used to
> work, since the compiler turns them into normal strings.
>
> I can't find exactly where it was removed, but there was
definitely
> discussion about it. See
https://bugs.python.org/issue28739 for at least
> part of the discussion.

Ah, sorry for the misunderstanding.  While the example I
showed doesn't
have any substitutions, I'm interested in the non-trivial
(non-constant)
case actually :-)

Regards

Antoine.


>
> Eric
>
> On 1/11/2022 8:41 AM, Antoine Pitrou wrote:
> > Hello,
> >
> > Currently, a f-string is not recognized as a docstring:
> >
> >>>> class C: f"foo"
> >>>> C.__doc__
> >>>>
> > This means you need to use a (admittedly easy) workaround:
> >
> >>>> class C: __doc__ = f"foo"
> >>>> C.__doc__
> > 'foo'
> >
> > Shouldn't the former be allowed for convenience?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to
python-dev-le...@python.org
> >
https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/UALMEMQ4QW7W4HE2PIBARWYBKFWJZFB4/
> > Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/37YAHCREZYZFSV4BRDKUEQAX4ZF4JTI6/
Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)

/Pronouns: he/him //(why is my pronoun here?)/

<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/35R3DCNPIQJ7ZCHTLP64IP2XZCK7QSLJ/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to p

[Python-Dev] Re: Minor inconvenience: f-string not recognized as docstring

[Gregory P. Smith]

On Tue, Jan 11, 2022 at 10:29 AM Guido van Rossum  wrote:

> I personally think F-strings should not be usable as docstrings. If you
> want a dynamically calculated docstring you should assign it dynamically,
> not smuggle it in using a string-like expression. We don't allow "blah {x}
> blah".format(x=1) as a docstring either, not "foo %s bar" % x.
>

Agreed.  If we wanted to remove the wart of constant f-strings happening to
work as an implementation detail in this context, that *could* be made into
a warning.  But that kind of check may be best left to a linter for *all*
of these dynamic situations that don't wind up populating __doc__.

-gps


>
> On Tue, Jan 11, 2022 at 8:12 AM Antoine Pitrou  wrote:
>
>> On Tue, 11 Jan 2022 10:58:03 -0500
>> "Eric V. Smith"  wrote:
>> > Constant f-strings (those without substitutions) as doc strings used to
>> > work, since the compiler turns them into normal strings.
>> >
>> > I can't find exactly where it was removed, but there was definitely
>> > discussion about it. See https://bugs.python.org/issue28739 for at
>> least
>> > part of the discussion.
>>
>> Ah, sorry for the misunderstanding.  While the example I showed doesn't
>> have any substitutions, I'm interested in the non-trivial (non-constant)
>> case actually :-)
>>
>> Regards
>>
>> Antoine.
>>
>>
>> >
>> > Eric
>> >
>> > On 1/11/2022 8:41 AM, Antoine Pitrou wrote:
>> > > Hello,
>> > >
>> > > Currently, a f-string is not recognized as a docstring:
>> > >
>> > >>>> class C: f"foo"
>> > >>>> C.__doc__
>> > >>>>
>> > > This means you need to use a (admittedly easy) workaround:
>> > >
>> > >>>> class C: __doc__ = f"foo"
>> > >>>> C.__doc__
>> > > 'foo'
>> > >
>> > > Shouldn't the former be allowed for convenience?
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> > >
>> > > ___
>> > > Python-Dev mailing list -- python-dev@python.org
>> > > To unsubscribe send an email to python-dev-le...@python.org
>> > > https://mail.python.org/mailman3/lists/python-dev.python.org/
>> > > Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/UALMEMQ4QW7W4HE2PIBARWYBKFWJZFB4/
>> > > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/37YAHCREZYZFSV4BRDKUEQAX4ZF4JTI6/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/35R3DCNPIQJ7ZCHTLP64IP2XZCK7QSLJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QFGCXW25TZOMEN2DRVLDQ4XQQSYNNTI7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Minor inconvenience: f-string not recognized as docstring

[Eric V. Smith]

Constant f-strings (those without substitutions) as doc strings used to 
work, since the compiler turns them into normal strings.


I can't find exactly where it was removed, but there was definitely 
discussion about it. See https://bugs.python.org/issue28739 for at least 
part of the discussion.


Eric

On 1/11/2022 8:41 AM, Antoine Pitrou wrote:

Hello,

Currently, a f-string is not recognized as a docstring:


class C: f"foo"
C.__doc__


This means you need to use a (admittedly easy) workaround:


class C: __doc__ = f"foo"
C.__doc__

'foo'

Shouldn't the former be allowed for convenience?

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UALMEMQ4QW7W4HE2PIBARWYBKFWJZFB4/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NCJKOGE4UKHTGBWZOLPAAWZ2DVU4FLZZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: About vulnerabilities in Cpython native code

[Eric V. Smith]

This is also at https://bugs.python.org/issue46280. Please direct 
comments there.


Eric

On 1/6/2022 8:22 AM, lxr1210--- via Python-Dev wrote:

Hi all,

I am currently doing some research on the security of CPython. I used 
the open source vulnerability analysis engine, 
Infer(https://fbinfer.com/), to scan the native code of CPython 3.10.0.


The scan results show that there are still a number of vulnerabilities 
in the CPython native code, such as Null dereference, Uninitialized 
variable, Resource/Memory leak, etc. Moreover, I found that some of 
the vulnerabilities are related to Python/C API. I enclose the 
vulnerability report for your reference.


Based on the research of the result, I tried to design a tool to 
automatically detect and repair vulnerabilities in CPython and make 
this tool available. See:


https://github.com/PVMPATCH/PVMPatch

Python is my favourite programming language. I sincerely hope that I 
can help Python become stronger and safer. I hope this discovery can 
be useful for you to develop Python in the future.


Thank you for your time and consideration!



Lin


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WQ2TVXPWFWP7V34WOGL2IDSDNAUQGEJU/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5BJNES7RVFLMLA2KJHSVF6AXMHLLQGSP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Sanity check about ctypes

[Gregory P. Smith]

On Wed, Jan 5, 2022 at 3:17 PM Yonatan Zunger  wrote:

> Hey everyone.
>
> Quick sanity check: The ctypes docs
>  refer to
> _CData as a non-public class which is in the module, but _ctypes.c doesn't
> actually export it
> .
> (I discovered this because it turns out that typeshed *is* referencing
> _CData, e.g. in its type annotations for RawIOBase
> 
> )
>
> Is this intended behavior in CPython (in which case the docs are a bit off
> and typeshed has a bug), or is it unexpected to people on this list (in
> which case it's an issue in _ctypes.c)?
>

typeshed is presumably referring to itself. It defines an interface for
ctypes._CData in
https://github.com/python/typeshed/blob/master/stdlib/ctypes/__init__.pyi#L82

The CPython ctypes docs *seem* reasonable to me. There is such a class. It
is not public, so you cannot access ctypes._CData in any direct manner.
That it gets called a class may be somewhat historical - its purpose is to
provide a common interface. What code would ever actually care that it used
class mechanisms as an internal implementation detail to do that?

-gps


>
> Yonatan
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7L6DNNI3MJ4UIM3C7A7KAIWHX562MRZL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ENKP334GO73ISSQ5XM5UWTOQYGTROK4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Tue, Jan 4, 2022 at 3:24 AM Tal Einat  wrote:

> I have a spare RPi zero that I could try to set up as a buildbot. Would
> that be useful?
>

No need. We've already got a 32-bit raspbian bot, adding another wouldn't
add value. The rpi0/1/2 are too slow to compile on anyways.

-gps

On Tue, Jan 4, 2022 at 10:59 AM Antoine Pitrou  wrote:
>
>> On Mon, 3 Jan 2022 22:40:25 -0800
>> "Gregory P. Smith"  wrote:
>> >
>> > rerunning a mere few of those in --rigorous mode for more runs does not
>> > significantly improve the stddev so I'm not going to let that finish.
>>
>> The one benchmark that is bigint-heavy is pidigits AFAIK, so you might
>> want to re-run that one if you want a more rigorous confirmation that
>> there is no regression.
>>
>> > my recommendation: proceed with removing 15-bit bignum digit support.
>> > 30-bit only future with simpler better code here we come.
>>
>> Sounds reasonable to me as well.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/MRE5HF3XRKJBEHF3YJNGXWXECLX7GQGG/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FPYXF33EKE23AUYPALAOKDLGKE7LGUGX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FZ7WAQGF6UVPVNG7SHIYTYWYBZXAPJSV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Sun, Jan 2, 2022 at 2:37 AM Mark Dickinson  wrote:

> On Sat, Jan 1, 2022 at 9:05 PM Antoine Pitrou  wrote:
>
>> Note that ARM is merely an architecture with very diverse
>> implementations having quite differing performance characteristics.  [...]
>>
>
> Understood. I'd be happy to see timings on a Raspberry Pi 3, say. I'm not
> too worried about things like the RPi Pico - that seems like it would be
> more of a target for MicroPython than CPython.
>
> Wikipedia thinks, and the ARM architecture manuals seem to confirm, that
> most 32-bit ARM instruction sets _do_ support the UMULL
> 32-bit-by-32-bit-to-64-bit multiply instruction. (From
> https://en.wikipedia.org/wiki/ARM_architecture#Arithmetic_instructions:
> "ARM supports 32-bit × 32-bit multiplies with either a 32-bit result or
> 64-bit result, though Cortex-M0 / M0+ / M1 cores don't support 64-bit
> results.") Division may still be problematic.
>

It's rather irrelevant anyways, the pi zero/one is the lowest spec arm that
matters at all. Nobody is ever going to ship something worse than that
capable of running CPython.

Anyways I ran actual benchmarks on a pi3. On 32-bit raspbian I build
CPython 3.10 with no configure flags and with --enable-big-digits (or
however that's spelled) for 30-bit digits and ran pyperformance 1.0.2 on
them.

Caveat: This is not a good system to run benchmarks on.  widely variable
performance (it has a tiny heatsink which never meaningfully got hot), and
the storage is a random microsd card. Each full pyperformance run took 6
hours. :P

Results basically say: no notable difference.  Most do not change and the
variability (look at those stddev's and how they overlap on the few things
that produced a "significant" result at all) is quite high.  Things wholly
unrelated to integers such as the various regex benchmarks showing up as
faster demonstrate the unreliability of the numbers.  And also at how
pointless caring about this fine level of detail for performance is on this
platform.

```
pi@pi3$ pyperf compare_to 15bit.json 30bit.json
2to3: Mean +- std dev: [15bit] 7.88 sec +- 0.39 sec -> [30bit] 8.02 sec +-
0.36 sec: 1.02x slower
crypto_pyaes: Mean +- std dev: [15bit] 3.22 sec +- 0.34 sec -> [30bit] 3.40
sec +- 0.22 sec: 1.06x slower
fannkuch: Mean +- std dev: [15bit] 13.4 sec +- 0.5 sec -> [30bit] 13.8 sec
+- 0.5 sec: 1.03x slower
pickle_list: Mean +- std dev: [15bit] 74.7 us +- 22.1 us -> [30bit] 85.7 us
+- 15.5 us: 1.15x slower
pyflate: Mean +- std dev: [15bit] 19.6 sec +- 0.6 sec -> [30bit] 19.9 sec
+- 0.6 sec: 1.01x slower
regex_dna: Mean +- std dev: [15bit] 2.99 sec +- 0.24 sec -> [30bit] 2.81
sec +- 0.22 sec: 1.06x faster
regex_v8: Mean +- std dev: [15bit] 520 ms +- 71 ms -> [30bit] 442 ms +- 115
ms: 1.18x faster
scimark_monte_carlo: Mean +- std dev: [15bit] 3.31 sec +- 0.24 sec ->
[30bit] 3.22 sec +- 0.24 sec: 1.03x faster
scimark_sor: Mean +- std dev: [15bit] 6.42 sec +- 0.34 sec -> [30bit] 6.27
sec +- 0.33 sec: 1.03x faster
spectral_norm: Mean +- std dev: [15bit] 4.85 sec +- 0.31 sec -> [30bit]
4.74 sec +- 0.20 sec: 1.02x faster
unpack_sequence: Mean +- std dev: [15bit] 1.42 us +- 0.42 us -> [30bit]
1.60 us +- 0.33 us: 1.13x slower

Benchmark hidden because not significant (47): chameleon, chaos, deltablue,
django_template, dulwich_log, float, go, hexiom, json_dumps, json_loads,
logging_format, logging_silent, logging_simple, mako, meteor_contest,
nbody, nqueens, pathlib, pickle, pickle_dict, pickle_pure_python, pidigits,
python_startup, python_startup_no_site, raytrace, regex_compile,
regex_effbot, richards, scimark_fft, scimark_lu, scimark_sparse_mat_mult,
sqlalchemy_declarative, sqlalchemy_imperative, sqlite_synth, sympy_expand,
sympy_integrate, sympy_sum, sympy_str, telco, tornado_http, unpickle,
unpickle_list, unpickle_pure_python, xml_etree_parse, xml_etree_iterparse,
xml_etree_generate, xml_etree_process
```

rerunning a mere few of those in --rigorous mode for more runs does not
significantly improve the stddev so I'm not going to let that finish.

my recommendation: proceed with removing 15-bit bignum digit support.
30-bit only future with simpler better code here we come.

-gps


>
> --
> Mark
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/F53IZRZPNAKB4DUPOVYWGMQDC4DAWLTF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5RJGI6THWCDYTTEPXMWXU7CK66RQUTD4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

Regarding ABI issues, I don't see anything obvious either. I was probably
misremembering the potential marshal issue, which was addressed.

struct _longobject (the implementation details behind the public
PyLongObject typedef name) and the digit definition are excluded from
Py_LIMITED_API.  So per https://docs.python.org/3.10/c-api/stable.html we
are free to change the struct layout.  yay.

regardless, I have confirmed that, sys.getsizeof(0) returns the same value
(12) on a 32-bit build both with 15-bit and 30-bit (--enable-big-digits)
builds on 32-bit architectures (I checked arm and x86).

So it'd only "break" something depending on non-limited minor version
specific ob_digit definitions and using it on the wrong Python version.
not a big deal.  People wanting that need to use Py_LIMITED_API in their
extension code as per our existing policy.

The getsizeof increments go from 12 14 16 18 20 to 0digits=12 1digit=16
2digit=20 as expected when doubling the digit size, but this isn't a
problem.  memory allocator wise, the same amount of ram is going to be
consumed by the same magnitude int regardless of how it gets built.
nothing allocates and tracks at a 2-byte granularity.

Perhaps I missed it, but maybe an action item would be to add a
>> buildbot which configures for 15-bit PyLong digits.
>>
>
> Yep, good point. I was wrong to say that  "15-bit builds don't appear to
> be exercised by the buildbots": there's a 32-bit Gentoo buildbot that's
> (implicitly) using 15-bit digits, and the GitHub Actions Windows/x86 build
> also uses 15-bit digits. I don't think we have anything that's explicitly
> using the `--enable-big-digits` option, though.
>

My raspbian bot covers the 32-bit use case we primarily care about.  (I
should promote that to one to stable)

I suggest just going for it. Remove 15-bit digit support and clean up the
code. My guess is that there will not be a meaningful performance impact on
32-bit hosts. I'm happy to run some tests on a rpi once you've got a PR up
if you don't already have a dozen of those laying around. :)

-gps


>
> --
> Mark
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZIR2UF7KHYJ2W5Z4A3OS5BDRI3DS5QTM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UAYEAALSRWZGZWEB7W3QE2LFRCGT5USR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Thu, Dec 30, 2021 at 12:42 PM Gregory P. Smith  wrote:

>
> On Thu, Dec 30, 2021 at 4:47 AM Mark Dickinson  wrote:
>
>> tl;dr: I'd like to deprecate and eventually remove the option to use
>> 15-bit digits in the PyLong implementation. Before doing so, I'd like to
>> find out whether there's anyone still using 15-bit PyLong digits, and if
>> so, why they're doing so.
>>
>> History: the use of 30-bit digits in PyLong was introduced for Python 3.1
>> and Python 2.7, to improve performance of int (Python 3) / long (Python 2)
>> arithmetic. At that time, we retained the option to use 15-bit digits, for
>> two reasons:
>>
>> - (1) use of 30-bit digits required C99 features (uint64_t and friends)
>> at a time when we hadn't yet committed to requiring C99
>> - (2) it wasn't clear whether 30-bit digits would be a performance win on
>> 32-bit operating systems
>>
>> Twelve years later, reason (1) no longer applies, and I suspect that:
>>
>> - No-one is deliberately using the 15-bit digit option.
>> - There are few machines where using 15-bit digits is faster than using
>> 30-bit digits.
>>
>> But I don't have solid data on either of these suspicions, hence this
>> post.
>>
>> Removing the 15-bit digit option would simplify the code (there's
>> significant mental effort required to ensure we don't break things for
>> 15-bit builds when modifying Objects/longobject.c, and 15-bit builds don't
>> appear to be exercised by the buildbots), remove a hidden compatibility
>> trap (see b.p.o. issue 35037), widen the applicability of the various fast
>> paths for arithmetic operations, and allow for some minor fast-path
>> small-integer optimisations based on the fact that we'd be able to assume
>> that presence of *two* extra bits in the C integer type rather than just
>> one. As an example of the latter: if `a` and `b` are PyLongs that fit in a
>> single digit, then with 15-bit digits and a 16-bit `digit` and `sdigit`
>> type, `a + b` can't currently safely (i.e., without undefined behaviour
>> from overflow) be computed with the C type `sdigit`. With 30-bit digits and
>> a 32-bit `digit` and `sdigit` type, `a + b` is safe.
>>
>> Mark
>>
>
> tying the thread together: this is https://bugs.python.org/issue45569
>
> Check 32-bit builds.  When I pushed for the 30-bit digit implementation, I
> wanted it for all builds but if I recall correctly it *might* have
> changed the minimum structure size for PyLong which could've been an ABI
> issue?  double check that.  32-bit is still important. Raspbian. rpi, rpi
> zero, and first rev rpi2 are 32-bit arm architectures so even with 64-bit
> raspbian on the horizon, that won't be the norm.  and for those, memory
> matters so a 32-bit userspace on 64-bit capable hardware is still preferred
> for small pointer sizes on the majority which have <=4GiB ram.
>
> I believe performance was the other concern, 30-bit happens to perform
> great on 32-bit x86 as it has 32*32->64 multiply hardware.  Most 32-bit
> architectures do not AFAIK, making 30 bit digit multiplies less efficient.
> And 32-bit x86 was clearly on its way out by the time we adopted 30-bit so
> it was simpler to just not do it on that dying snowflake of a platform.
> (test it on raspbian - it's the one that matters)
>
> Regardless of possible issues to work out, I'd love us to have a simpler
> 30-bit only implementation.
>
> Granted, modern 64-bit hardware often has 64*64->128 bit multiply hardware
> so you can imagine going beyond 30 and winding up in complexity land
> again.  at least the extra bits would be >=2 at that point.  The reason for
> digits being a multiple of 5 bits should be revisited vs its original
> intent and current state of the art "bignum optimized for mostly small
> numbers" at some point as well.
>
> -gps
>
>
Historical context of adding the 30-bit support (also driven primarily by
Mark, no surprise!) in late 2008 early 2009:
https://bugs.python.org/issue4258 (and https://codereview.appspot.com/14105)

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PRL7TJWUDYAMBQKVDLAIKD2OS2VOQPMQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Thu, Dec 30, 2021 at 4:47 AM Mark Dickinson  wrote:

> tl;dr: I'd like to deprecate and eventually remove the option to use
> 15-bit digits in the PyLong implementation. Before doing so, I'd like to
> find out whether there's anyone still using 15-bit PyLong digits, and if
> so, why they're doing so.
>
> History: the use of 30-bit digits in PyLong was introduced for Python 3.1
> and Python 2.7, to improve performance of int (Python 3) / long (Python 2)
> arithmetic. At that time, we retained the option to use 15-bit digits, for
> two reasons:
>
> - (1) use of 30-bit digits required C99 features (uint64_t and friends) at
> a time when we hadn't yet committed to requiring C99
> - (2) it wasn't clear whether 30-bit digits would be a performance win on
> 32-bit operating systems
>
> Twelve years later, reason (1) no longer applies, and I suspect that:
>
> - No-one is deliberately using the 15-bit digit option.
> - There are few machines where using 15-bit digits is faster than using
> 30-bit digits.
>
> But I don't have solid data on either of these suspicions, hence this post.
>
> Removing the 15-bit digit option would simplify the code (there's
> significant mental effort required to ensure we don't break things for
> 15-bit builds when modifying Objects/longobject.c, and 15-bit builds don't
> appear to be exercised by the buildbots), remove a hidden compatibility
> trap (see b.p.o. issue 35037), widen the applicability of the various fast
> paths for arithmetic operations, and allow for some minor fast-path
> small-integer optimisations based on the fact that we'd be able to assume
> that presence of *two* extra bits in the C integer type rather than just
> one. As an example of the latter: if `a` and `b` are PyLongs that fit in a
> single digit, then with 15-bit digits and a 16-bit `digit` and `sdigit`
> type, `a + b` can't currently safely (i.e., without undefined behaviour
> from overflow) be computed with the C type `sdigit`. With 30-bit digits and
> a 32-bit `digit` and `sdigit` type, `a + b` is safe.
>
> Mark
>

tying the thread together: this is https://bugs.python.org/issue45569

Check 32-bit builds.  When I pushed for the 30-bit digit implementation, I
wanted it for all builds but if I recall correctly it *might* have changed
the minimum structure size for PyLong which could've been an ABI issue?
double check that.  32-bit is still important. Raspbian. rpi, rpi zero, and
first rev rpi2 are 32-bit arm architectures so even with 64-bit raspbian on
the horizon, that won't be the norm.  and for those, memory matters so a
32-bit userspace on 64-bit capable hardware is still preferred for small
pointer sizes on the majority which have <=4GiB ram.

I believe performance was the other concern, 30-bit happens to perform
great on 32-bit x86 as it has 32*32->64 multiply hardware.  Most 32-bit
architectures do not AFAIK, making 30 bit digit multiplies less efficient.
And 32-bit x86 was clearly on its way out by the time we adopted 30-bit so
it was simpler to just not do it on that dying snowflake of a platform.
(test it on raspbian - it's the one that matters)

Regardless of possible issues to work out, I'd love us to have a simpler
30-bit only implementation.

Granted, modern 64-bit hardware often has 64*64->128 bit multiply hardware
so you can imagine going beyond 30 and winding up in complexity land
again.  at least the extra bits would be >=2 at that point.  The reason for
digits being a multiple of 5 bits should be revisited vs its original
intent and current state of the art "bignum optimized for mostly small
numbers" at some point as well.

-gps



>
>
> *References*
>
> Related b.p.o. issue: https://bugs.python.org/issue45569
> MinGW compatibility issue: https://bugs.python.org/issue35037
> Introduction of 30-bit digits: https://bugs.python.org/issue4258
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZICIMX5VFCX4IOFH5NUPVHCUJCQ4Q7QM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WS5WHRCRHO4DOQPRW7EZYG3PRF4UIKAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: issues-test-2 spam?

[Eric V. Smith]

On 12/27/2021 12:43 PM, Zachary Ware wrote:

On Mon, Dec 27, 2021 at 11:29 AM Matti Picus  wrote:

You may want to try the experiments in a private repo under your username 
rather than under the python organization. I think this will prevent anyone 
except you from getting notified. You could even go further and set up a dummy 
bpo-migrator github user, then no one should get notifications. Thanks for 
doing the work.


Frankly, with as many other emails as constantly come in from GitHub,
these are a trivially ignorable drop in the bucket and not worth
derailing Ezio's work to avoid.  If you receive them and they bother
you, filter them out in your client.


I completely agree. And I'm sure that testing in an official python repo 
is a better test than using a private repo. I don't see the harm in an 
occasional message.


Thanks for your work, Ezio.

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G3NBGIVSAED67UOFX2ON6JLOZWNAXKSJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

[Nathaniel Smith]

On Wed, Dec 15, 2021 at 3:07 AM Victor Stinner  wrote:
> I wrote https://bugs.python.org/issue39511 and
> https://github.com/python/cpython/pull/18301 to have per-interpreter
> None, True and False singletons. My change is backward compatible on
> the C API: you can still use "Py_None" in your C code. The code gets
> the singleton object from the current interpreter with a function
> call:
>
> #define Py_None Py_GetNone()
>
> Py_GetNone() is implemented as: "return _PyInterpreterState_GET()->none;"

It's backward compatible for the C API, but not for the stable C ABI
-- that exports Py_None directly as a symbol.

You also need a solution for all the static global PyTypeObjects in C
extensions. I don't think there's any API-compatible way to make those
heap-allocated.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HS3NENZXKYSBCN4OVUOZ2CUX55Z5CXG5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

[Nathaniel Smith]

On Wed, Dec 15, 2021 at 2:21 AM Antoine Pitrou  wrote:
>
> On Wed, 15 Dec 2021 10:42:17 +0100
> Christian Heimes  wrote:
> > On 14/12/2021 19.19, Eric Snow wrote:
> > > A while back I concluded that neither approach would work for us.  The
> > > approach I had taken would have significant cache performance
> > > penalties in a per-interpreter GIL world.  The approach that modifies
> > > Py_INCREF() has a significant performance penalty due to the extra
> > > branch on such a frequent operation.
> >
> > Would it be possible to write the Py_INCREF() and Py_DECREF() macros in
> > a way that does not depend on branching? For example we could use the
> > highest bit of the ref count as an immutable indicator and do something like
> >
> >  ob_refcnt += !(ob_refcnt >> 63)
> >
> > instead of
> >
> >  ob_refcnt++
>
> Probably, but that would also issue spurious writes to immortal
> refcounts from different threads at once, so might end up worse
> performance-wise.

Unless the CPU is clever enough to skip claiming the cacheline in
exclusive-mode for a "+= 0". Which I guess is something you'd have to
check empirically on every microarch and instruction pattern you care
about, because there's no way it's documented. But maybe? CPUs are
very smart, except when they aren't.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZLARVPQCPZXWVHGYOZNSDRTCNNJ67ANM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python release announcement format

[Gregory P. Smith]

On Tue, Dec 14, 2021 at 9:06 AM Yann Droneaud  wrote:

> Hi,
>
> I'm not familiar with the Python release process, but looking at the latest 
> release
> https://www.python.org/downloads/release/python-3101/
>
> we can see MD5 is still used ... which doesn't sound right in 2021 ...
> especially since we proved it's possible to build different .tar.gz that have
> the same MD5
>
> https://twitter.com/ydroneaud/status/1448659749604446211https://twitter.com/angealbertini/status/1449736035110461443
>
> You would reply there's OpenPGP / GnuPG signature. But then I would like to 
> raise
> another issue regarding the release process:
>
> As the announcement on comp.lang.python.announce / 
> python-announce-l...@python.org
> doesn't record the release digest / release signature, the operator 
> behindhttps://www.python.org/downloads/release/python-3101/ are free to 
> change the release
> content at any time, provided there's a valid signature. And there will no 
> way for
> us to check the release wasn't modified after the announcement.
>
>
For source archives, one can diff the contents of the source download vs
those of the equivalent tag in the git repository. For binaries, well,
there's already a ton of trust involved in accepting a binary from anyone.
But agreed having the currently secure hashes in the announcement email
would be good.


> It would be great if https://www.python.org/dev/peps/pep-0101/ would be 
> improved
> from the naive:
>
>  "Write the announcement for the mailing lists.  This is the fuzzy bit 
> because not
>   much can be automated.  You can use an earlier announcement as a template, 
> but
>   edit it for content!"
>
> to require the release announcement to record release archives digests as 
> SHA-2 256
> (added point if the announcement is signed), or the armored OpenPGP 
> signatures (but's
> that a lot of base64 characters).
>
> Should I open a bug for this issue ?
>
>
Makes sense, it is a pretty small change to make to the announcement
format. Filed. https://bugs.python.org/issue46077

-gps

Regards.
>
> --
> Yann Droneaud
> OPTEYA
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6NI6V7DHTXCTUTNC2C5YSGOB6UJRFUDR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OXS2TK43QKH2M54R5HHECOZ6HYCQGJON/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Nathaniel Smith]

Whoops, never mind, I see that you started the "immortal objects"
thread to discuss this.

On Tue, Dec 14, 2021 at 4:54 PM Nathaniel Smith  wrote:
>
> How did you end up solving the issue where Py_None is a static global
> that's exposed as part of the stable C ABI?
>
> On Tue, Dec 14, 2021 at 9:13 AM Eric Snow  wrote:
> >
> > Hi all,
> >
> > I'm still hoping to land a per-interpreter GIL for 3.11.  There is
> > still a decent amount of work to be done but little of it will require
> > solving any big problems:
> >
> > * pull remaining static globals into _PyRuntimeState and PyInterpreterState
> > * minor updates to PEP 554
> > * finish up the last couple pieces of the PEP 554 implementation
> > * maybe publish a companion PEP about per-interpreter GIL
> >
> > There are also a few decisions to be made.  I'll open a couple of
> > other threads to get feedback on those.  Here I'd like your thoughts
> > on the following:
> >
> > Do we need a PEP about per-interpreter GIL?
> >
> > I haven't thought there would be much value in such a PEP.  There
> > doesn't seem to be any decision that needs to be made.  At best the
> > PEP would be an explanation of the project, where:
> >
> > * the objective has gotten a lot of support (and we're working on
> > addressing the concerns of the few objectors)
> > * most of the required work is worth doing regardless (e.g. improve
> > runtime init/fini, eliminate static globals)
> > * the performance impact is likely to be a net improvement
> > * it is fully backward compatible and the C-API is essentially unaffected
> >
> > So the value of a PEP would be in consolidating an explanation of the
> > project into a single document.  It seems like a poor fit for a PEP.
> >
> > (You might wonder, "what about PEP 554?"  I purposefully avoided any
> > discussion of the GIL in PEP 554.  It's purpose is to expose
> > subinterpreters to Python code.)
> >
> > However, perhaps I'm too close to it all.  I'd like your thoughts on the 
> > matter.
> >
> > Thanks!
> >
> > -eric
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QDXHLDVHIW4B3GN7PNEL4LPZRJVGGW2R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Nathaniel Smith]

How did you end up solving the issue where Py_None is a static global
that's exposed as part of the stable C ABI?

On Tue, Dec 14, 2021 at 9:13 AM Eric Snow  wrote:
>
> Hi all,
>
> I'm still hoping to land a per-interpreter GIL for 3.11.  There is
> still a decent amount of work to be done but little of it will require
> solving any big problems:
>
> * pull remaining static globals into _PyRuntimeState and PyInterpreterState
> * minor updates to PEP 554
> * finish up the last couple pieces of the PEP 554 implementation
> * maybe publish a companion PEP about per-interpreter GIL
>
> There are also a few decisions to be made.  I'll open a couple of
> other threads to get feedback on those.  Here I'd like your thoughts
> on the following:
>
> Do we need a PEP about per-interpreter GIL?
>
> I haven't thought there would be much value in such a PEP.  There
> doesn't seem to be any decision that needs to be made.  At best the
> PEP would be an explanation of the project, where:
>
> * the objective has gotten a lot of support (and we're working on
> addressing the concerns of the few objectors)
> * most of the required work is worth doing regardless (e.g. improve
> runtime init/fini, eliminate static globals)
> * the performance impact is likely to be a net improvement
> * it is fully backward compatible and the C-API is essentially unaffected
>
> So the value of a PEP would be in consolidating an explanation of the
> project into a single document.  It seems like a poor fit for a PEP.
>
> (You might wonder, "what about PEP 554?"  I purposefully avoided any
> discussion of the GIL in PEP 554.  It's purpose is to expose
> subinterpreters to Python code.)
>
> However, perhaps I'm too close to it all.  I'd like your thoughts on the 
> matter.
>
> Thanks!
>
> -eric
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KRGRUHLHKBN5QQUY4VBO33DUS2Z2H5A4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should dataclasses add__set__ (and possibly __get __) descriptors ?

[Eric V. Smith]

On 12/13/2021 1:43 AM, Vioshim wrote:

Hello good morning.

I've decided to open a discussion of a topic that I consider should be part of 
dataclasses, but not sure how suggestions work and many threads recommend to 
check python dev first so-.

Anyways, at the moment that I write this message in python3.10.1, It happens 
that when making a class with the dataclasses module, this class can't actually 
be used in Multiple inheritance for Enum purposes, this is mostly to avoid code 
repetition by having to write the methods over again.

Here's an example

from dataclasses import dataclass
from enum import Enum

@dataclass
class Foo:
 a: int = 0

class Entries(Foo, Enum):
 ENTRY1 = Foo(1)
 ENTRY2 = Foo(2)
 ENTRY3 = Foo(3)


assert Entries.ENTRY1.a == 1

This raises AssertionError, due to the values not being defined at that point, 
as the class is currently just using the default paremeters. This is why I 
think it'd be essential to implement __set__ in the module.


I don't understand what you're trying to achieve.

Entries.ENTRY1.a is Foo object. Why would you expect it to be 1? And I 
don't understand why you're inheriting from Foo. What are you trying to 
accomplish with that?


Could you explain in more detail why you're using this pattern?

Eric



Currently there's a workaround to have this work, which is defining a __set__ 
but doing this, in more complex cases, can get to be very tedious and 
repetitive coding wise

from dataclasses import dataclass
from enum import Enum

@dataclass
class Foo:
 a: int = 0

 def __set__(self, instance, value: int):
 instance.a = value


class Entries(Foo, Enum):
 ENTRY1 = Foo(1)
 ENTRY2 = Foo(2)
 ENTRY3 = Foo(3)


assert Entries.ENTRY1.a == 1


Have a great day, and hopefully this post enables a good discussion.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U2YDIU5T67GFV4PAOHBMOGURG6TYUHP7/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P64KFE6VITRI74FCFTR7KAAP7G25MVEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Eric V. Smith]

Here's a use case for runtime type annotations: dataclasses use 
annotations to define fields. With the exception of ClassVar, the actual 
type is ignored. There's code in dataclasses.py to deal with stringized 
annotations, specifically just looking for ClassVar. I'd like to see 
this special case go away, assuming 649 is accepted.


And while dataclasses itself doesn't care about the types, apparently 
users do, and want to be able to call get_type_hints on the annotations. 
Many people actually want dataclasses to call this for them, but I've 
resisted. See https://bugs.python.org/issue36643


There are plenty of bugs around this issue, some of which have been 
fixed. But there are also at least a couple that haven't been fixed, and 
I'm not sure they will be. I'm hoping that 649 gets accepted and I don't 
have to worry about the problem at all. Here's an example of an open bug 
around get_type_hints with dataclasses: https://bugs.python.org/issue45524


Eric

On 11/29/2021 6:00 PM, Barry Warsaw wrote:

On Nov 26, 2021, at 01:13, Paul Moore  wrote:


I'd therefore interpret Barry's plea as being for *anyone* with a use
for annotations to provide their feedback (at least, anyone who
accepts that annotations are types), with particular emphasis on
people who want to use the types declared in annotations to affect
runtime behaviour, as that's the most under-represented group at the
moment (and it's not clear whether it's under-represented because
there aren't many such uses, or because the users aren't being heard
from).

Spot on.

-Barry


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4522JDS2ZOYXXFGPVXRGJ67IB3YLCEFM/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ENVEVQUEMEDKT2YUI2LQOBZDJC2YMNQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Eric V. Smith]

On 11/28/2021 7:52 PM, Christopher Barker wrote:
I will frequently do simple computation with literals to make my code 
more clear:


t = 2 * 3600  # 2 hours in seconds


That is optimized as you'd hope. Tested in 3.8:

>>> dis.dis("t = 2 * 3600")
  1   0 LOAD_CONST   0 (7200)
  2 STORE_NAME   0 (t)
  4 LOAD_CONST   1 (None)
  6 RETURN_VALUE

Eric



But I see no need to optimize this kind of thing -- it would never be 
in a tight loop.


-CHB


On Sun, Nov 28, 2021 at 3:06 PM Rob Cliffe via Python-Dev 
 wrote:


I am slightly surprised that it seems to be *easier* to fold selected
constant expressions than to have more generic code to fold them all.
Or at least, all those that don't contain containers, such as
 1 in [0,1,2]
Rob Cliffe

On 28/11/2021 21:10, Eric V. Smith wrote:
>> On Nov 28, 2021, at 3:03 PM, Serhiy Storchaka
 wrote:
>>
>> 28.11.21 17:13, Skip Montanaro пише:
>>>> That is not entirely true:
https://github.com/python/cpython/pull/29639#issuecomment-974146979
>>> The only places I've seen "if 0:" or "if False:" in live code
was for
>>> debugging. Optimizing that hardly seems necessary. In any
case, the
>>> original comment was about comparisons of two constants. I suppose
>>> sweeping up all of that into a constant expression
folding/elimination
>>> step performed on the AST and/or during peephole optimization
would
>>> cover both cases.
>> "if 0:" and "if False:" is already optimized by the compiler.
The OP
>> proposes to optimize "2 < 1", and I cannot imagine any use case
for this.
> I agree. I suggest we don’t add this optimization.
>
> Eric
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/JP6FF2RHDQSIOS6ZAI45S7X6UXWGPBKR/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/LD3XHINO7VPDE5EMNFD5ON4FQIJLY4DI/
Code of Conduct: http://python.org/psf/codeofconduct/



--
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/NJWCKNV4ULEBHSLR44G3HNVROE4IAM47/
Code of Conduct:http://python.org/psf/codeofconduct/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YEG3B44GJRHACEB4U6U2VAHT5E2SVXTO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Eric V. Smith]

> On Nov 28, 2021, at 3:03 PM, Serhiy Storchaka  wrote:
> 
> 28.11.21 17:13, Skip Montanaro пише:
>>> That is not entirely true: 
>>> https://github.com/python/cpython/pull/29639#issuecomment-974146979
>> 
>> The only places I've seen "if 0:" or "if False:" in live code was for
>> debugging. Optimizing that hardly seems necessary. In any case, the
>> original comment was about comparisons of two constants. I suppose
>> sweeping up all of that into a constant expression folding/elimination
>> step performed on the AST and/or during peephole optimization would
>> cover both cases.
> 
> "if 0:" and "if False:" is already optimized by the compiler. The OP
> proposes to optimize "2 < 1", and I cannot imagine any use case for this.

I agree. I suggest we don’t add this optimization. 

Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JP6FF2RHDQSIOS6ZAI45S7X6UXWGPBKR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

[Eric V. Smith]

On 11/14/2021 11:39 AM, Eric V. Smith wrote:


For things that really are removed (and I won't get in to the reasons 
for why something must be removed), I think a useful stance is "we 
won't remove anything that would make it hard to support a single code 
base across all supported python versions". We'd need to define 
"hard", maybe "no hasattr calls" would be part of it.


On second thought, I guess the existing policy already does this. Maybe 
we should make it more than 2 versions for deprecations? I've written 
libraries where I support 4 or 5 released versions. Although maybe I 
should just trim that back.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XXPSVAC2CYJWBQFMZ6VO6FSEOOAEZ5MZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

[Eric V. Smith]

On 11/12/2021 5:55 AM, Petr Viktorin wrote:


If deprecation now means "we've come up with a new way to do things, 
and you have two years to switch", can we have something else that 
means "there's now a better way to do things; the old way is a bit 
worse but continues to work as before"?


I think optparse is a good example of this (not that I love argparse).

For things that really are removed (and I won't get in to the reasons 
for why something must be removed), I think a useful stance is "we won't 
remove anything that would make it hard to support a single code base 
across all supported python versions". We'd need to define "hard", maybe 
"no hasattr calls" would be part of it.


Reliable tools to make the migration between versions would help, too.

I could live with this, although I write systems that support old 
versions of python. We just moved to 3.7, for example.


Eric

PS: Someone else said that my estimate of tens of thousands of dollars 
to deal with deprecations is too high. If anything, I think it's too 
low. For all of the testing and signoffs we do for a single release, 
I've calculated that it costs us $5k just to release code to prod, even 
if there are no actual code changes. Could that be improved? Sure. Will 
it? Unlikely. Maybe I'm an outlier, but I doubt it.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5WCUZD3W3RDZXL4VKMGGD6D6ATFWAEQA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Having Sorted Containers in stdlib?

[Eric V. Smith]

On 11/12/2021 11:31 AM, Chris Angelico wrote:


I think that proves the value of download counts: very little. The
highest on the list is a thing called botocore, which I've never heard
of. What is it? It's a dependency of a number of Amazon web services.
My guess is that it's a dependency of popular tools - or maybe of
automatically-installed tools, even - while not itself being well
known. Actually, quite a few of the most popular packages on that list
are Amazon-related, so quite possibly they're all being installed as a
set in response to one single *actual* dependency.


botocore and boto3 are the Amazon-provided low- and high-level 
interfaces, respectively, to AWS. They're extremely import and well 
known in that environment.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V4GSZINECFIAZ4MGDZNM3L354LJDSX2Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Gregory P. Smith]

On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista 
wrote:

> Hello everyone!
>
> I need to pack a long list of numbers into shared memory, so I thought
> about using `struct.pack_into`.
>
> Its signature is
>
> struct.pack_into(format, buffer, offset, v1, v2, ...)
>
> I have a long list of nums (several millions), ended up doing the
> following:
>
> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
>
> However, passing all nums as `*args` is very inefficient [0]. So I
> started wondering why we don't have something like:
>
> struct.pack_into(format, buffer, offset, values=values)
>
> which would receive the list of values directly.
>
> Is that because my particular case is very uncommon? Or maybe we *do*
> want this but we don't have it yet? Or do we already have a better way
> of doing this?
>
> Thanks!
>
> [0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4


My first reaction on seeing things like this is "Why not use a numpy.array?"

Does what you have really need to be a long list?  If so, that's already a
huge amount of Python object storage as it is. Is it possible for your
application to have kept that in a numpy array for the entirety of the data
lifetime?
https://numpy.org/doc/stable/reference/routines.array-creation.html

I'm not saying the stdlib shouldn't have a better way to do this by not
abusing *args as an API, just that other libraries solve the larger problem
of data-memory-inefficiency in their own way already.

*(neat tricks from others regarding stdlib array, shm, & memoryview even
if... not ideal)*

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YD7WP6F3LEXFSEITVPZUNONVUBK3AKUW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Nathaniel Smith]

On Thu, Oct 7, 2021 at 7:54 PM Sam Gross  wrote:
> Design overview:
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

Whoa, this is impressive work.

I notice the fb.com address -- is this a personal project or something
facebook is working on? what's the relationship to Cinder, if any?

Regarding the tricky lock-free dict/list reads: I guess the more
straightforward approach would be to use a plain ol' mutex that's
optimized for this kind of fine-grained per-object lock with short
critical sections and minimal contention, like WTF::Lock. Did you try
alternatives like that? If so, I assume they didn't work well -- can
you give more details?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XAZYWRYXKIVUSMRSMFAETKQDLGL27L7X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Gregory P. Smith]

On Sun, Oct 3, 2021 at 10:47 AM Łukasz Langa  wrote:

>
>  I know it's a bit late for bikeshedding this thing so if we want to be
> conservative and stick to the current syntactical options already defined
> in PEP 654, I'm voting Option 2 (given the awkwardness of the *(E1, E2)
> example).
>

+1 on the `except* E` Option 2 syntax. It better conveys its uniqueness and
non-relation to other meanings of *.

Someone mentioned allowing both and letting people decide.  Whatever is
chosen, please not that.  There should be only one way to write this.  That
avoids style arguments when no auto-formatter is involved.

-gps


>
> - Ł
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/T7QZJ575RFNYZ5KMYD66YMR2ZLNDVF56/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T5FOZVPJ2YWIQ25HRSQ5NZ6U3YMDKI7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric V. Smith]

On 9/28/2021 9:17 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 09:14:38 -0400
"Eric V. Smith"  wrote:

On 9/28/2021 9:10 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 08:55:05 -0400
"Eric V. Smith"  wrote:

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.

I agree with Victor here: I'd rather have #1.

As a compromise, how about go with #1, but print a warning if python
detects that it's not built with optimizations or is run from a source
tree (the conditions in #2 and #3)? The warning could suggest running
with "-X frozen_modules=off". I realize that it will probably be ignored
over time, but maybe it will provide enough of a reminder if someone is
debugging and sees the warning.

What would be the point of printing a warning instead of doing just
what the user is expecting?

To me, the point would be to get the same behavior no matter which
python executable I run, and without regard to where I run it from.

But why do you care about this?  What does it change *concretely*?


It reduces the number of things I have to remember which are different 
based on where I'm running python (or which executable I'm running).


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2FN2OCYGMLU5XST44N7SFDUAI426VRB3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric V. Smith]

On 9/28/2021 9:10 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 08:55:05 -0400
"Eric V. Smith"  wrote:

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.

I agree with Victor here: I'd rather have #1.

As a compromise, how about go with #1, but print a warning if python
detects that it's not built with optimizations or is run from a source
tree (the conditions in #2 and #3)? The warning could suggest running
with "-X frozen_modules=off". I realize that it will probably be ignored
over time, but maybe it will provide enough of a reminder if someone is
debugging and sees the warning.

What would be the point of printing a warning instead of doing just
what the user is expecting?


To me, the point would be to get the same behavior no matter which 
python executable I run, and without regard to where I run it from.


Eric



Freezing the stdlib is a startup performance optimization. It doesn't
need to be turned on when hacking on the Python source code... And
having to type "-X frozen_modules=off" is much more of a nuisance than
losing 10 ms in startup time (which you won't even notice in most
cases).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IJEP37TCGR4SVRPGAHIILA42BSUKZZ2O/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CEKYHC2Z66DAASLQEHCT4PEQ35LWUE7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric V. Smith]

On 9/28/2021 8:36 AM, Victor Stinner wrote:

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  wrote:

We've frozen most of the stdlib modules imported during "python -c
pass" [1][2], to make startup a bit faster.  Import of those modules
is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
to "off" but we'd like to default to "on".  The blocker is the impact
on contributors.  I expect many will make changes to a stdlib module
and then puzzle over why those changes aren't getting used.  That's an
annoyance we can avoid, which is the point of this thread.

Possible solutions:

1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)
3. default to "on" unless running from the source tree

Thoughts?

Honestly, for me, #1: always on, is the most reasonable choice.

I dislike when Python behaves differently depending on subtle things
like "was it built with optimizations" or "is Python started from its
source tree"?

When I built Python without optimization and/or from its source tree,
I do that to debug an issue. If the bug goes away in this case, it can
waste my time.

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.


I agree with Victor here: I'd rather have #1.

As a compromise, how about go with #1, but print a warning if python 
detects that it's not built with optimizations or is run from a source 
tree (the conditions in #2 and #3)? The warning could suggest running 
with "-X frozen_modules=off". I realize that it will probably be ignored 
over time, but maybe it will provide enough of a reminder if someone is 
debugging and sees the warning.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5QKEUKJSUQON6G3WOT2GS2WGLHLSQSKS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Tue, Sep 28, 2021 at 12:40 AM Guido van Rossum  wrote:
>
> What I have heard repeatedly, from people who are paid to know, is that most 
> users don’t care about the latest features, and would rather stick to a 
> release until it becomes unsupported. (Extreme example: Python 2.)
>
> Numpy isn’t random, it’s at the bottom of the food chain for a large 
> ecosystem or two — if it doesn’t support a new Python release, none of its 
> dependent packages can even start porting. (I guess only Cython is even 
> lower, but it’s a build-time tool. And indeed it has supported 3.10 for a 
> long time.)

Well, no, it wasn't entirely random :-).

Being on the bottom of the food chain is important, but I don't think
it's the full story -- Tensorflow is also at the bottom of a huge
ecosystem. I think it's also related to NumPy being mostly
volunteer-run, which means they're sensitive to feedback from
individual enthusiasts, and enthusiasts are the most aggressive early
adopters. OTOH Tensorflow is a huge commercial collaboration, and
companies *hate* upgrading.

Either way though, it doesn't seem to be anything to do with CPython's
ABI stability or release cadence.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNE7TY7ZFGBOOOG3H3YNB5MFGX5IWZ76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Sun, Sep 26, 2021 at 3:38 AM  wrote:
> Open3D is an example. Will finally move to Python 3.9 some time the coming 
> month. Its dependency graph contains about 70 other packages.
>
> In this specific case, the underlying problem was that TensorFlow was stuck 
> at 3.8. The TensorFlow codebase got ported in November 2020, then released 
> early 2021. Then Open3D included the new Tensorflow (plus whatever else 
> needed to be adapted) in their codebase in May. They’re now going through 
> their release schedule, and their 0.14 release should be up on PyPI soon.

I took a minute to look up the release dates to fill in this timeline:

Python 3.9 released: October 2020
Tensorflow adds 3.9 support: November 2020
Tensorflow v2.5.0 released with the new 3.9 support: May 2021
Open3d adds 3.9 support: May 2021
First Open3d release to include the new 3.9 support: ~October 2021

So it seems like in this case at least, the year long delay consists
of ~1 month of porting work, and ~11 months of projects letting the
finished code sit in their repos without shipping to users.

It seems like the core problem here is that these projects don't
consider it important to keep up with the latest Python release. I'm
not sure what CPython upstream can do about that. Maybe you could
lobby these projects to ship releases more promptly?

By contrast, to pick a random library that uses the unstable C API
extensively, NumPy is already shipping wheels for 3.10 -- and 3.10
isn't even out yet. So it's certainly possible to do, even for
projects with a tiny fraction of Tensorflow's engineering budget.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGBANTGGJAI2ZM5SYSLCWFGRIML5VWUL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Sat, Sep 25, 2021 at 5:40 PM  wrote:
> PyPI packages and wheels are targeted to specific Python versions, which 
> means that any project that depends on some of the larger extension packages 
> (of which there are many, and many of which are must-have for many projects) 
> now start lagging Python versions by years, because somewhere deep down in 
> the dependency graph there is something that is still stuck at Python 3.8 
> (for example).

Can you give some examples of the packages you're thinking of, that
are prominent/must-have and stuck on years-old Pythons?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF7Z4EDN3DHBKPMYCIJFTI2JDTIJYHXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing exception text in micro releases

[Eric V. Smith]

On 9/24/2021 4:53 PM, Guido van Rossum wrote:

In this case I am inclined not to backport.

I agree. I decided not to backport it.
In general we should look at existing usage before making changes. 
Somebody’s code might break — but does it matter? That depends on a 
lot of factors. E.g. if parsing an error message has become a common 
way to get useful info out of the error that is not easily available 
otherwise, we have to tread lightly.


In this case I don't think anyone could be parsing it, because the 
existing version doesn't contain any information, it's always the string 
"Invalid format specifier". But I agree we should be cautious, so this 
will wait until 3.11. The original bug was reported 7.5 years ago, so 
it's not like we're in any rush.


Thanks for the input.

Eric



—Guido

On Fri, Sep 24, 2021 at 09:59 Terry Reedy <mailto:tjre...@udel.edu>> wrote:


On 9/24/2021 10:46 AM, Eric V. Smith wrote:
> On 9/24/2021 10:39 AM, Ethan Furman wrote:
>> On 9/24/21 6:51 AM, Eric V. Smith wrote:
>>
>>
>> > This is clearly an improvement. My question is: is it okay to
>> backport this to 3.9 and 3.10? I
>> > think we have a rule that it's okay to change the exception
text,
>> but I'm not sure how that
>> > applies to micro releases (3.9.x, 3.10.1).
>>
>> "better" != "bug fix", so I wouldn't change it in a micro release.
>
> Yeah, as much as I'd love to see this included, you're right.
Thanks for
> the "adult" take on it.

Exception messages are subject to change in each new version, without
deprecating the old version.  Changes are sometimes backported
when the
existing message is sufficiently erroneous. 'Sufficiently' means
something like 'harm of error outweighs harm of changing'. I don't
think a simple, easily recognized, typo or grammar error qualifies.
Among active developers, I believe Guido has the most continuity of
experience with such discussions and decisions.  Perhaps something
could
usefully be added to the devguide, but I am not sure.


-- 
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-le...@python.org
<mailto:python-dev-le...@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
<https://mail.python.org/mailman3/lists/python-dev.python.org/>
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/KKZVPESKEAIL4DYEIGSTPLKE34KH7XTQ/

<https://mail.python.org/archives/list/python-dev@python.org/message/KKZVPESKEAIL4DYEIGSTPLKE34KH7XTQ/>
Code of Conduct: http://python.org/psf/codeofconduct/
<http://python.org/psf/codeofconduct/>

--
--Guido (mobile)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQWMP7BL5AJH3ZAEU22LPZLO4R4OW6RY/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QARFSFOLDLKOOIURMM7V2MKXXVH5ICSI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing exception text in micro releases

[Eric V. Smith]

On 9/24/2021 10:39 AM, Ethan Furman wrote:

On 9/24/21 6:51 AM, Eric V. Smith wrote:


> This is clearly an improvement. My question is: is it okay to 
backport this to 3.9 and 3.10? I
> think we have a rule that it's okay to change the exception text, 
but I'm not sure how that

> applies to micro releases (3.9.x, 3.10.1).

"better" != "bug fix", so I wouldn't change it in a micro release.


Yeah, as much as I'd love to see this included, you're right. Thanks for 
the "adult" take on it.


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SXIUK2DELUVAUESWLUYJBUA6IVYBTX2C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Changing exception text in micro releases

[Eric V. Smith]

This PR https://github.com/python/cpython/pull/28310 changes the message 
for some exceptions.


Currently:

>>> format('', '%M')
Traceback (most recent call last):
  File "", line 1, in 
ValueError: Invalid format specifier

With the proposed change:
>>> format('', '%M')
Traceback (most recent call last):
  File "", line 1, in 
ValueError: Invalid format specifier '%M' for object of type 'str'

This also applies to str.format() and f-strings.

This is clearly an improvement. My question is: is it okay to backport 
this to 3.9 and 3.10? I think we have a rule that it's okay to change 
the exception text, but I'm not sure how that applies to micro releases 
(3.9.x, 3.10.1).


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G6H3M7USSQ2YCFMATMU24J5BJMDJPDVW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

On 9/21/2021 7:15 PM, Guido van Rossum wrote:
On Tue, Sep 21, 2021 at 4:08 PM Steve Dower <mailto:steve.do...@python.org>> wrote:


On 9/21/2021 7:42 PM, Eric V. Smith wrote:
> I don't recall exactly why, but I disallowed backslashes inside
> expressions at the last minute before 3.6 was released. It might
have
> been because I was interpreting them in a way that didn't make
sense if
> a "real" parser were inspecting f-strings. The idea, even back
then, was
> to re-allow them when/if we moved f-string parsing into the parser
> itself. I think it's time.

Yeah, we were still trying to figure out whether escapes like "\\n"
would be evaluated as "\\n" or "\n" in the expression, and decided to
decide later. If we can clearly articulate which it is now, then
let's
go ahead and enable it.


That would seem easy enough, right?

f"spam {'xyz'.replace('y', '\\n')} spam"

should be equal to

"spam x\\ny spam"

and print as

spam x\ny spam

(i.e. a literal backslash followed by 'n', not a newline).

Yes, I think that's the desired behavior. Before I removed this in 3.6 
(during the betas, IIRC), it would have produced an actual newline, 
because of where the f-string 'parser' was able to insert itself into 
the process. I/we didn't like that behavior, and it was too late to 
change it.


We could add this now in the bespoke f-string parser, although I don't 
know off the top of my head how much work it would be. But if we're 
going to switch to Pablo's parser then I don't think there's any point.


You shouldn't have to double the \ in the interpolated expression just 
because it's in an f-string.

Right.
I presume it was trickier at the time because we were coming from 
"{xxx}".format(...), where the parser doesn't know that the string is 
a format string.


Yes, that was part of it.

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FKMSE7BPAO5R4R3IKZ2K5CFEG6PYKV6D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

To bring this back on track, I'll try and answer the questions from your 
original email.


On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:
I have started a project to move the parsing off-strings to the parser 
and the grammar. Appart
from some maintenance improvements (we can drop a considerable 
amount of hand-written code),
there are some interesting things we **could** (emphasis on could) get 
out of this and I wanted

to discuss what people think about them.


I think this is all awesome.

My position is that if we make zero syntactic changes to f-strings, and 
leave the functionality exactly as it is today, I think we should still 
move the logic into the parser and grammar, as you suggested. As you 
say, this would eliminate a lot of code, and in addition likely get us 
better error messages. As for the things we could possibly add:


* The parser will likely have "\n" characters and backslashes in 
f-strings expressions, which currently is impossible:


>>> f"blah blah {'\n'} blah"
  File "", line 1
    f"blah blah {'\n'} blah"
                            ^
SyntaxError: f-string expression part cannot include a backslash


I think supporting backslashes in strings inside of f-string expression 
(the part inside {}) would be a big win, and should be the first thing 
we allow. I often have to do this:


nl = '\n'
x = f"blah {nl if condition else ' '}"

Being able to write this more naturally would be a big win.

I don't recall exactly why, but I disallowed backslashes inside 
expressions at the last minute before 3.6 was released. It might have 
been because I was interpreting them in a way that didn't make sense if 
a "real" parser were inspecting f-strings. The idea, even back then, was 
to re-allow them when/if we moved f-string parsing into the parser 
itself. I think it's time.


* The parser will allow nesting quote characters. This means that we 
**could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"
I'm okay with this, with the caveat that I raised in another email: the 
effect on non-Python tools and alternate Python implementations. To 
restate that here: as long as we survey some (most?) of the affected 
parties and they're okay with it (or at least it doesn't cause them a 
gigantic amount of work), then I'm okay with it. This will of course be 
subjective. My big concern is tools that today use regex's (or similar) 
to recognize f-strings, and then completely ignore what's inside them. 
They just want to "skip over" f-strings in the source code, maybe 
because they're doing some sort of source-to-source transpiling, and 
they're just going to output the f-strings as-is. It seems to me we're 
creating a lot of work for such tools. Are there a lot of such tools? I 
don't know: maybe there are none.
* The parser will naturally allow more control over error messages and 
AST positions.

This would be a good win.
* The **grammar** of the f-string will be fully specified without 
ambiguity. Currently, the "grammar" that we have in the docs
(https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals 
) 
is not really formal grammar because
not only is mixing lexing details with grammar details (the definition 
of "literal_char") but also is not compatible with the current python
lexing schema (for instance, it recognizes "{{" as its own token, 
which the language doesn't allow because something like "{{a:b}:c}"
is tokenized as "{", "{", "a" ... not as "{{", "a". Adding a formal 
grammar could help syntax highlighters, IDEs, parsers and other tools

to make sure they properly recognize everything that there is.

Also a big win.

There may be some other advantages that we have not explored still.

The work is at a point where the main idea works (all the grammar is 
already there and working), but we need to make sure that all existing
errors and specifics are properly ported to the new code, which is a 
considerable amount of work still so I wanted to make sure we are on the
same page before we decide to invest more time on this (Batuhan is 
helping me with this and Lyssandros will likely join us). We are doing
this work in this branch: 
https://github.com/we-like-parsers/cpython/blob/fstring-grammar 



Tell me what you think.

P.S. If you are interested to help with this project, please reach out 
to me. If we decide to go ahead we can use your help! :)


I'm interested in helping.

Thanks for your work on this.

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

On 9/20/2021 11:19 AM, Terry Reedy wrote:

On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:

there are some interesting things we **could** (emphasis on could) 
get out of this and I wanted

to discuss what people think about them.

* The parser will allow nesting quote characters. This means that we 
**could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"


I believe that this will disable regex-based processing, such as 
syntax highlighters, as in IDLE.  I also think that it will be 
sometimes confusing to human readers.


When I initially wrote f-strings, it was an explicit design goal to be 
just like existing strings, but with a new prefix. That's why there are 
all of the machinations in the parser for scanning within f-strings: the 
parser had already done its duty, so there needed to be a separate stage 
to decode inside the f-strings. Since they look just like regular 
strings, most tools could add the lowest possible level of support just 
by adding 'f' to existing prefixes they support: 'r', 'b', 'u'. The 
upside is that if you don't care about what's inside an f-string, your 
work is done.


I definitely share your concern about making f-strings more complicated 
to parse for tool vendors: basically all editors, alternative 
implementations, etc.: really anyone who parses python source code. But 
maybe we've already crossed this bridge with the PEG parser. Although I 
realize there's a difference between lexing and parsing. While the PEG 
parser just makes parsing more complicated, this change would make what 
was lexing into a more sophisticated parsing problem.


In 2018 or 2019 at PyCon in Cleveland I talked to several tool vendors. 
It's been so long ago that I don't remember who, but I'm pretty sure it 
was PyCharm and 2 or 3 other editors. All of them supported making this 
change, even understanding the complications it would cause them. I 
don't recall if I talked to anyone who maintains an alternative 
implementation, but we should probably discuss it with MicroPython, 
Cython, PyPy, etc., and understand where they stand on it.


In general I'm supportive of this change, because as Pablo points out 
there are definite benefits. But I think if we do accept it we should 
understand what sort of burden we're putting on tool and implementation 
authors. It would probably be a good idea to discuss it at the upcoming 
dev sprints.


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DN5HB7CBS7I2FXI74UBM4ZZVMSNVDQ57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

On 9/20/2021 11:21 AM, Terry Reedy wrote:

On 9/20/2021 8:46 AM, Serhiy Storchaka wrote:

20.09.21 14:18, Pablo Galindo Salgado пише:

* The parser will likely have "\n" characters and backslashes in
f-strings expressions, which currently is impossible:


What about characters "\x7b", "\x7d", "\x5c", etc?

What about newlines in single quotes? Currently this works:

f'''{1 +
2}'''

But this does not:

f'{1 +
2}'


The later is an error with or without the 'f' prefix and I think that 
this should continue to be the case.


The thought is that anything that's within braces {} and is a valid 
expression should be allowed. Basically, the opening brace puts you in 
"parse expression" mode. Personally, I'd be okay with this particular 
change.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QWCISJYDR6LYXOD4DAKUTA3EYV3XQQIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should PEP 8 be updated for Python 3 only?

[Eric V. Smith]

I think we’re better off removing them. 2.7 is completely unsupported by us. 

Why do you think they should still be kept?

--
Eric

> On Aug 25, 2021, at 1:32 PM, Mike Miller  wrote:
> 
> 
> How about moving Python 2 notes into a section at the bottom?
> 
> 
> -Mike
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/76JIO4AF7WMJ5475FTIHTK6GR2O52OOF/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/235UK4OXI4CVBDQ5AFTS54OFCATSO7X4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Gregory P. Smith]

Just adding a datapoint, searching our internal codebase at work, I only
found two things that I'd consider to be uses of PyEval_GetLocals() outside
of CPython itself:

https://github.com/earwig/mwparserfromhell/blob/develop/src/mwparserfromhell/parser/ctokenizer/tokenizer.c#L220
https://github.com/numba/numba/blob/master/numba/_dispatcher.cpp#L664

The bulk of uses are naturally with CPython's own ceval.c and similar.

-gps

On Mon, Aug 23, 2021 at 9:02 AM Guido van Rossum  wrote:

> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>
>> Hi Guido,
>>
>> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
>> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > > > wrote:
>> >
>> > Hi Nick,
>> >
>> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
>> >
>> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
>> > true then
>> >  > I would be more sympathetic to his proposal, but I know it isn't
>> > true,
>> >  > because it still works fine in the PEP 558 implementation (it
>> even
>> >  > immediately sees changes made via proxies, and proxies see
>> > changes to
>> >  > extra variables). The only truly unfixable public API is
>> >  > PyFrame_LocalsToFast().
>> >
>> > You are making claims that seem inconsistent with each other.
>> > Namely, you are claiming that:
>> >
>> > 1. That the result of locals() is ephemeral.
>> > 2. That PyEval_GetLocals() returns a borrowed reference.
>> >
>> > This seems impossible, as you can't return a borrowed reference to
>> > an emphemeral object. That's just a pointer to freed memory.
>> >
>> > Do `locals()` and `PyEval_GetLocals()` behave differently?
>> >
>> >
>> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
>> > dict, the Python-level f_locals property returns a fresh proxy that has
>> > no state except a pointer to the frame, and PyEval_GetLocals() returns
>> a
>> > borrowed reference to the dict that's stored on the frame's C-level
>> > f_locals attribute
>>
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>>
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
>> )
>>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very
> confusing) and I took it from him. And honestly it would be nice to have a
> specific short name for it, rather than circumscribing it with "an internal
> dynamic snapshot stored on the frame object " :-)
>
>
>> >
>> > (In my "crazy" proposal all that is the same.)
>>
>> >
>> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>> >
>> >
>> > I wouldn't call it a cache -- deleting it would affect the semantics,
>> > not just the performance. But yes, it returns a reference to an object
>> > that is owned by the frame, just as it does in 3.10 and before.
>> >
>> > If that were the case, then it is a bit confusing, but could work.
>> >
>> >
>> > Yes, see my "crazy" proposal.
>> >
>> > Would PyEval_GetLocals() be defined as something like this?
>> >
>> > (add _locals_cache attribute to the frame which is initialized to
>> NULL).
>> >
>> > def PyEval_GetLocals():
>> >   frame._locals_cache attribute = locals()
>> >   return borrow(frame._locals_cache attribute)
>> >
>> >
>> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
>> > C-level f_locals attribute, which is consulted by the Python-level
>> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
>> > Nick's latest version it is also still used to cache by that proxy.
>> > Nick's locals() just returns dict(sys._getframe().f_locals).
>>
>> The "extra" variables must be distinct from the result of locals() as
>> that includes both extras and "proper" variables.
>> If we want to cache the locals(), it needs to be distinct from the extra
>> variables.
>>
>
> I don't care that much about caching locals(), but it seems we're bound to
> cache any non-NULL result from PyEval_GetLocals(), since it returns a
> borrowed reference. So they may be different things, with different
> semantics, if we don't cache locals().
>
>
>> A debugger setting extra variables in a function that that is also
>> accessed by a C call to PyEval_GetLocals() is going to be incredibly
>> rare. Let's not worry about efficiency here.
>>
>
> Agreed.
>
> >
>> > None of this is clear (at least not to me) from PEP 558.
>> >
>> >
>> > One problem with PEP 558 is that it's got too many words, and it's
>> > lacking a section that crisply describes the semantics of the proposed
>> > implementation. I've suggested to Nick that he add a section with
>> > pseudo-code for the implementation, like you did in yours.
>> >
>> > (PS, did you read my PS 

[Python-Dev] Re: PEP 467 feedback from the Steering Council

[Gregory P. Smith]

On Tue, Aug 10, 2021 at 3:48 PM Christopher Barker 
wrote:

> On Tue, Aug 10, 2021 at 3:00 PM  wrote:
>
>> The history of bytes/bytearray is a dual-purpose view.  It can be used in
>> a string-like way to emulate Python 2 string handling (hence all the usual
>> string methods and a repr that displays in a string-like fashion).  It can
>> also be used as an array of numbers, 0 to 255 (hence the list methods and
>> having an iterator of ints).  ISTM that the authors of this PEP reject or
>> want to discourage the latter use cases.
>>
>
> I didn't read it that way, but if so, please no, I"d rather see the former
> use cases discouraged. ISTM that the Py2 string handling is still needed
> for working with mixed binary / text data -- but that should be a pretty
> specialized use case. spelling the way to create a byte, byte() sure makes
> more sense in any other context.
>
>
>> ... anything where a C programmer would an array of unsigned chars).
>>
>
> or any programmer would use an array of unsigned 8bit integers :-) numpy
> spells it: `np.uint8`, and the the type in the C99 stdint.h is `uint8_t`.
> My point is that for anyone not an "old time" C programmer, or even a
> Python2 programmer, the  "character is an unsigned 8 bit int" concept is
> alien and confusing, not a helpful mnemonic.
>
>
>> For example, creating a single byte with bytes([0x1f]) isn't pleasant,
>> obvious, or fast.
>>
>
> no, though bytes([31]) isn't horrible ;-)   (despite coding for over four
> decades, I'm still not comfortable with hex notation)
>
> I say it's not horrible, because bytes is a Sequence of bytes (or integer
> values between 0 and 255), initializing it with an iterable seems pretty
> reasonable, that's how we initialize most (all?) other sequences after all.
> And compatible with array.array and numpy arrays.
>

I consider bytes([31]) notation to be horrible API design because a simple
easy to make typo of omitting the [] or using () and forgetting the
tupleizing comma turns it into a different valid call with an entirely
different meaning.  bytes([31]) vs bytes((31)) vs bytes(31).

It's also ugly to anyone who thinks about what bytecode is generated and
executed in order to do it.  an entire new list object with a single
element referring to a tiny int is created and destroyed just to create a
b'\037' object?  An optimizer pass to fix that up at the bytecode level
isn't easy as it can only be done when it can prove that `bytes` has not
been reassigned to something other than the builtin.  Near impossible in a
lot of code.  bytes.fromint(31) isn't much better in the bytecode regard,
but at least a temporary list is not being created.

As much as I think that bytes(size: int) was a bad idea to have as an API -
bytearray(size: int) is fine and useful as it is mutable - that ship sailed
and getting rid of it would break some odd code.  It doesn't have much use,
so adding fromsize(size: int) methods don't sound very compelling as it
just adds yet another way to do the same thing.  we should just live with
that specific wart.

`bchr` as a builtin... I'm with the others on saying no to any new builtin
that isn't expected to see frequent use.  bchr won't see frequent use.

`bytes.fromint` seems fine.  others are proposing `bytes.byte` for that.  I
don't *like* to argue over names (the last stage of anything) but I do need
to point out how that sounds to read.  It falls victim to API stuttering.
"bytes dot byte" or "bytes byte" doesn't convey much to a reader in English
as the difference is a subtle "s".  "bytes dot from int" or "bytes from
int" is quite clear.  (avoiding stuttering in API design was popularized by
golang - it's a good thing to strive for in any language)  It's times like
this that i wish Python had chosen consistent camelCase, CapWords, or
snake_case in all API names as conjoinedwords aren't great. But they are
sadly consistent with our past sins.

One thing never mentioned in the PEP.  If you expect a primary use of the
fromint (aka bchr builtin that isn't going to happen) to be called on
constant values often.  Why are we adding name lookups and function calls
to this?  Why not address the elephant in the room and allow for decimal
values to be written as an escape sequence within bytes literals?

b'\d31' for example to say "decimal byte 31".  Proposal: Only values 0-255
with no leading zero should be accepted when parsing such an escape.  (Do
not bother adding the same feature for codepoints in unicode strs; leave
that to later if someone shows actual demand).  This can't address the
bytearray need, but that's been true of bytearray for ages, a common way to
create them is via a copy from transient bytes objects.  bytearray(b'\d31')
isn't much different than bytearray.fromint(31).  one less name lookup.

Why not add a \d escape? Introducing a new escape is fraught with peril as
existing \d's within b'' literals in code could change meaning.  backwards
compatibility fail.  But one that is easy to check for