[Python-Dev] Re: Python history: origin of the arrow annotation

[Guido van Rossum]

Good question. I don't think anyone has ever asked this before... Given the
variants you propose, I'd say that the 3-character ones would be more
effort to type without real benefits, and `=>` would at the time (and
perhaps still :-) be seen as too close to `>=`.

Could it be that there were already other languages using `->` for return
types? I do know that we needed something there -- from the start I was a
fan of `x: int` because it's the same as Pascal (the language I used most
intensely in college), and if it had been syntactically possible I would
have used 'def f(): int' for the return type as well, but the LL(1) parser
in use in 1999-2000 would interpret that as a very short function body.

It's also possible that it comes from the `->` operator in C, which would
be the only "ASCII art arrow" that I was familiar with at the time.

Note that a slide deck I presented in 2000 on the topic still exists:
https://gvanrossum.github.io/static-typing/ This shows something pretty
close to the modern notation for function and variable annotations, but
with a `decl` keyword thrown in.

On Fri, Mar 5, 2021 at 3:08 AM Steven D'Aprano  wrote:

> I was curious how and why return annotations use the arrow `->` symbol,
> so I went spelunking into the depths of the Python-Ideas and Python-Dev
> mailing lists.
>
> Much to my surprise, I couldn't find any discussion or debate about it.
>
> Eventually I tracked the discussion back to a mailing list I didn't even
> know existed, Types-Sig, all the way back to 13th Dec 1999, where I
> found this email by Tim Hochberg:
>
> https://mail.python.org/pipermail/types-sig/1999-December/000281.html
>
> suggesting the arrow symbol for all type declarations. This appears to
> be the first suggestion of the arrow symbol for type annotations in
> Python.
>
> I am surprised that type checking was being discussed so long ago.
> Python was not even a decade old, and if my calculations are correct,
> the versions at the time would have been 1.5.2 and 1.6.0.
>
> But I am especially amazed that the choice of symbol seems to have been
> accepted with very little argument or debate about alternative colours
> for the bike-shed. I can't find any suggestions for alternative symbols
> such as `=>` `-->` `==>` `->>` etc, and very little for keywords such as
> `as`. The day after Tim posted, Tony Lownds suggested `as` and then
> immediately suggested using Tim's arrow symbol for the return type only:
>
> https://mail.python.org/pipermail/types-sig/1999-December/000335.html
>
> and then as far as I can tell, we've never looked back.
>
> If there is anyone whose memory reaches back that far, does that sound
> right?
>
>
> --
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4ZZP5DJARHHKN4HBC4JHHRIV5Z2EZOVT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RUNYLIQ3NAV7RRRKR4XCAAWA2S37ESQ2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Steering-council] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Guido van Rossum]

On Fri, Mar 5, 2021 at 11:11 AM Brett Cannon  wrote:

> Speaking for myself ...
>

Ditto ...

On Fri, Mar 5, 2021 at 7:04 AM Mark Shannon  wrote:
> [...]
>
>> In some cases, the PEP would have improved the situation.
>>
>> For example:
>> sys.setrecursionlimit(5000)
>> def f():
>>  f()
>>
>> Currently, it raises a RecursionError on linux, but crashes the
>> interpreter on Windows.
>> With PEP 651 it would have raised a RecursionError on both platforms.
>>
>> Am I missing something here?
>>
>
> So your example shows a user already comfortable in raising their
> recursion limit to work around needing more stack space to reach
> completion. What is stopping the user from continuing to raise the limit
> until they still reach their memory limit even with PEP 651? If you're
> worried about runaway recursion you will very likely hit that with the
> default stack depth already, so I personally don't see how a decoupled
> stack counter from the C stack specifically makes it any easier/better to
> detect runaway recursion. And if I need more recursion than the default,
> you're going to bump the recursion depth anyway, which weakens the
> protection in either the C or decoupled counter scenarios. Sure, it's
> currently platform-specific, but plenty of people want to push that limit
> based on their machine anyway and don't need consistency on platforms they
> will never run on, i.e. I don't see a huge benefit to being able to say
> that an algorithm consistently won't go past 5000 calls on all platforms
> compared to what the C stack protection already gives us (not to say
> there's zero benefit, but it isn't massive or widespread either IMO). I
> personally just don't see many people saying, "I really want to limit my
> program to an exact call stack depth of 5000 on all platforms which is
> beyond the default, but anything under is fine and anything over --
> regardless of what the system can actually handle -- is unacceptable".
>
> Tack on the amount of changes required to give a cross-platform stack
> count and limit check compared to the benefit being proposed, and to me
> that pushes what the PEP is proposing into net-negative payoff.
>

To me, the point of that example is as a reminder that currently fiddling
with the recursion limit can cause segfaults.

Mark's PEP proposes two, somewhat independent, changes: (1) don't consume C
stack on pure Python-to-Python (pp2p) function calls; (2) implement
fool-proof C stack overflow checks.

Change (2) makes it safe for users to mess with the stack overflow limit
however they see fit. Despite (1), the limit for pp2p calls remains at 1000
so that users who unintentionally write some naively recursive code don't
have to wait until they fill up all of memory before they get a traceback.
(Of course they could also write a while-True loop that keeps adding an
item to a list and they'd end up in the same situation. But in my
experience that situation is less painful to deal with than accidental
stack overflow, and I'd shudder at the thought of a traceback of a million
lines.)

Given that we have (1), why is (2) still needed? Because there are ways to
recursively call Python code that aren't pp2p calls. By a pp2p (pure
Python-to-Python) call, I mean any direct call, e.g. a method call or a
function call. But what about other operations that can invoke Python code?
E.g. if we have a dict d and a class C, we could create an instance of C
and use it to index d, e.g. d[C()]. This operation is not a p2pp call --
the BINARY_SUBSCR opcode calls the dict's `__getitem__` method, and that
calls the key's `__hash__` method. Here's a silly demonstration:
```
def f(c):
d = {}
return d[c]

class C:
def __hash__(self):
return f(self)

f(C())
```
Note that the "traceback compression" implemented for simple recursive
calls fails here -- I just ran this code and got 2000 lines of output.

The way I imagine Mark wants to implement pp2p calls means that in this
case each recursion step *does* add several other C stack frames, and this
would be caught by the limit implemented in (2). I see no easy way around
this -- after all the C code involved in the recursion could be a piece of
3rd party C code that itself is not at fault.

So we could choose to implement only (2), robust C stack overflow checks.
This would require a bunch of platform-specific code, and there might be
platforms where we don't know how to implement this (I vaguely recall a
UNIX version where main() received a growable stack but each thread only
had a fixed 64kb stack), but those would be no worse off than today.

Or we could choose to implement only (1), eliminating C stack usage for
pp2p calls. But in that case we'd still need a recursion limit for non-pp2p
calls. We could eliminate the recursion limit entirely for pp2p calls, but
then we'd probably end up with user complaints -- I could imagine a
framework that executes arbitrary user code and attempts recovery after
receiving an exception (for exam

[Python-Dev] Re: [Steering-council] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Brett Cannon]

Speaking for myself ...

On Fri, Mar 5, 2021 at 7:04 AM Mark Shannon  wrote:

> Hi,
>
> Thanks for taking the time to consider the PEP.
>
> Although the PEP was rejected, I still believe that the safety
> guarantees in PEP 651 are worth adding to Python in the future.
>
> To do that (maybe for 3.11), I need to understand your concerns better.
>
> Would you clarify a few points for me?
>
> On 03/03/2021 7:19 pm, Python Steering Council wrote:
> > Hi Mark,
> >
> > Thank you for submitting PEP 651. The Steering Council has spent the
> past two weeks reviewing PEP 651. After careful consideration, we have
> decided to reject the PEP. The following were the key points that led us to
> this decision:
> >
> > * The benefits are not compelling enough. Deep recursion is not a common
> tool in
> >Python, and even with PEP 651 it would not be efficient enough to
> make it a common
> >tool.
> >
> > * The benefit of PEP 651 is negated as soon as a non-Python function is
> involved in the
> >recursion, making the likelihood of it being useful even smaller. It
> also creates
> >easy pitfalls for users who do end up relying on recursion.
>
> Could you give an example pitfall?
>
> >
> > * We believe the PEP understates the disruption created by the technical
> solution of
> >multiple Python stack frames per C call. Although this may be
> solvable, it will
> >certainly cause substantial disruption to existing debuggers,
> tracers, and state
> >inspection tools as they need to adapt to this change (which may not
> be trivial).
>
> This is presumably the key objection.
> Is there a particular tool that you feel would be problematic?
> I have only looked at gdb and py-spy.
>

There's also any debugger that has an extension module component, e.g.
debugpy.


>
> >
> > * As the way to approach this will be platform-specific (as some parts
> of the proposal
> >are not portable), this can cause generic Python code to behave
> differently on
> >different platforms, making this kind of code less portable and less
> predictable.
>
>
> There are two issues here. Portability and changes to behaviour.
>
> Regarding portability, I have to admit that PEP is rather vague.
> That's my fault; I should have done more implementation first :(
> FWIW, I have an implementation that should be portable.
>
> https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation
>
> Regarding changes to behaviour, I don't see how "generic" Python code
> would behave differently on different platforms, except for cases where
> it already does.
>
> In some cases, the PEP would have improved the situation.
>
> For example:
> sys.setrecursionlimit(5000)
> def f():
>  f()
>
> Currently, it raises a RecursionError on linux, but crashes the
> interpreter on Windows.
> With PEP 651 it would have raised a RecursionError on both platforms.
>
> Am I missing something here?
>

So your example shows a user already comfortable in raising their recursion
limit to work around needing more stack space to reach completion. What is
stopping the user from continuing to raise the limit until they still reach
their memory limit even with PEP 651? If you're worried about runaway
recursion you will very likely hit that with the default stack depth
already, so I personally don't see how a decoupled stack counter from the C
stack specifically makes it any easier/better to detect runaway recursion.
And if I need more recursion than the default, you're going to bump the
recursion depth anyway, which weakens the protection in either the C or
decoupled counter scenarios. Sure, it's currently platform-specific, but
plenty of people want to push that limit based on their machine anyway and
don't need consistency on platforms they will never run on, i.e. I don't
see a huge benefit to being able to say that an algorithm consistently
won't go past 5000 calls on all platforms compared to what the C stack
protection already gives us (not to say there's zero benefit, but it isn't
massive or widespread either IMO). I personally just don't see many people
saying, "I really want to limit my program to an exact call stack depth of
5000 on all platforms which is beyond the default, but anything under is
fine and anything over -- regardless of what the system can actually handle
-- is unacceptable".

Tack on the amount of changes required to give a cross-platform stack count
and limit check compared to the benefit being proposed, and to me that
pushes what the PEP is proposing into net-negative payoff.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QLVLXHHGTR3Z4ILS5D5FZWMEHXST7HZO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2021-02-26 - 2021-03-05)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7444 (+14)
  closed 47702 (+66)
  total  55146 (+80)

Open issues with patches: 2959 


Issues opened (53)
==

#43331: [Doc][urllib.request] Explicit the fact that header keys are s
https://bugs.python.org/issue43331  opened by axel3rd

#43332: http/client.py: - uses multiple network writes, possibly causi
https://bugs.python.org/issue43332  opened by zveinn

#4: utf8 in BytesGenerator
https://bugs.python.org/issue4  opened by darcy.beurle

#43334: venv does not install libpython
https://bugs.python.org/issue43334  opened by anuppari

#43336: document whether io.TextIOBase.readline(size>0) will always re
https://bugs.python.org/issue43336  opened by calestyo

#43337: export the set newline value on TextIOBase/TextIOWrapper
https://bugs.python.org/issue43337  opened by calestyo

#43338: [feature request] Please provide offical installers for securi
https://bugs.python.org/issue43338  opened by zby1234

#43340: json.load() can raise UnicodeDecodeError, but this is not docu
https://bugs.python.org/issue43340  opened by mattheww

#43341: functools.partial missing __weakref__ descriptor?
https://bugs.python.org/issue43341  opened by bup

#43346: subprocess.run() sometimes ignores timeout in Windows
https://bugs.python.org/issue43346  opened by eryksun

#43347: IDLE crashes in macOS Apple chip, maybe completions
https://bugs.python.org/issue43347  opened by rhettinger

#43348: XMLRPC behaves strangely under pythonw, not under python
https://bugs.python.org/issue43348  opened by tim_magee

#43350: [sqlite3] Active statements are reset twice in _pysqlite_query
https://bugs.python.org/issue43350  opened by erlendaasland

#43351: `RecursionError` during deallocation
https://bugs.python.org/issue43351  opened by andrewvaughanj

#43352: Add a Barrier object in asyncio lib
https://bugs.python.org/issue43352  opened by yduprat

#43353: Document that logging.getLevelName() can return a numeric valu
https://bugs.python.org/issue43353  opened by felixxm

#43354: xmlrpc.client: Fault.faultCode erroneously documented to be a 
https://bugs.python.org/issue43354  opened by jugmac00

#43355: __future__.annotations breaks inspect.signature()
https://bugs.python.org/issue43355  opened by 1ace

#43356: PyErr_SetInterrupt should have an equivalent that takes a sign
https://bugs.python.org/issue43356  opened by pitrou

#43357: Python memory cleaning
https://bugs.python.org/issue43357  opened by absvsb

#43358: Bad free in assemble function
https://bugs.python.org/issue43358  opened by alex.henrie

#43359: Dead assignment in Py_UniversalNewlineFgets
https://bugs.python.org/issue43359  opened by alex.henrie

#43360: Dead initialization in parse_abbr function
https://bugs.python.org/issue43360  opened by alex.henrie

#43361: Dead assignment in idna_converter function
https://bugs.python.org/issue43361  opened by alex.henrie

#43362: Bad free in py_sha3_new_impl function
https://bugs.python.org/issue43362  opened by alex.henrie

#43364: Windows: Make UTF-8 mode more accessible
https://bugs.python.org/issue43364  opened by methane

#43365: Operation conflict between time package and file in python 3.8
https://bugs.python.org/issue43365  opened by minaki_2525

#43366: Unclosed bracket bug in code.interact prevents identifying syn
https://bugs.python.org/issue43366  opened by aroberge

#43367: submodule of c-extension module is quirky
https://bugs.python.org/issue43367  opened by mattip

#43371: Mock.assert_has_calls works strange
https://bugs.python.org/issue43371  opened by dmitriy.mironiyk

#43372: ctypes: test_frozentable fails when make regen-frozen
https://bugs.python.org/issue43372  opened by hroncok

#43374: Apple refuses apps written in Python
https://bugs.python.org/issue43374  opened by adigeo

#43377: _PyErr_Display should be available in the CPython-specific API
https://bugs.python.org/issue43377  opened by Maxime Belanger

#43378: Pattern Matching section in tutorial refers to | as or
https://bugs.python.org/issue43378  opened by ramalho

#43379: Pasting multiple lines in the REPL is broken since 3.9
https://bugs.python.org/issue43379  opened by romainfv

#43380: Assigning function parameter to class attribute by the same na
https://bugs.python.org/issue43380  opened by jennydaman

#43381: add small test for frozen module line number table
https://bugs.python.org/issue43381  opened by nascheme

#43382: github CI blocked by the Ubuntu CI with an SSL error
https://bugs.python.org/issue43382  opened by gregory.p.smith

#43384: Include regen-stdlib-module-names in regen-all
https://bugs.python.org/issue43384  opened by nascheme

#43387: Enable pydoc to run as background job
https://bugs.python.org/issue43387  opened by digitaldragon

#43388: shutil._fastcopy_sendfile() makes wro

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Antoine Pitrou]

Hi Mark,

Le 05/03/2021 à 18:06, Mark Shannon a écrit :

Hi Antoine,

On 05/03/2021 4:07 pm, Antoine Pitrou wrote:

On Fri, 5 Mar 2021 15:03:59 +
Mark Shannon  wrote:


There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague.
That's my fault; I should have done more implementation first :(
FWIW, I have an implementation that should be portable.
https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


I looked through this diff and I'm not sure how this works robustly.

This seems to assume:
- each thread's stack size is known and is a compile-time constant

We just need to know that the stack size is at least the compile-time
constant, we don't need to know exactly what it is.

If the stack size is less than the compile-time constant, then a crash
is a possibility. However, since PEP 651 would reduce stack consumption,
a crash would have been less likely than it currently is.

For all supported platforms, there is a default stack size and it is
considerably larger than the chosen value of 512k.


Right, but Python more or less works on a large range of non-formally 
supported platforms.  With your proposal and a hardcoded 512 kB minimum 
stack size, Python may crash almost immediately on such platforms.


I don't know if we care about that (we could tell third-party 
maintainers to change the #define to an appropriate value), but it's 
still something to will make porting a bit more cumbersome.



- Python owns the thread and can compute the stack limit reliably from
the current stack frame

There are two points of entry into Python code. From the high level API
and thread_run().


Hmm, actually, there are many entry points into Python code.  C code can 
create its own thread and call arbitrary CPython APIs from it (after 
having initialized the tstate using the PyGILState API).  That call 
inside a C-created thread can happen after an arbitrary amount of stack 
was already consumed by the calling C code.



But, this is no worse than the status quo, and probably better due to
reduced C stack use.


I'm not sure what you mean with "no worse than the statu quo".

Supporting a smaller recursion limit than advertised is one thing, 
crashing almost immediately when executing Python code (because of 
trying to access memory outside beyond the reserved stack area) is 
another.  Currently, you would only crash when recursing a lot.



- it's ok to write memory beyond the current stack top (I can imagine
verification tools such as Valgrind complaining about that, though
this can be worked around using suppressions)

It's not much of a stack if you can't grow it :)


The stack size is typically limited (see `ulimit -s`).

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MKLBLT4OGAYGPJFIATSGQVD4MV7TEYC3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Mark Shannon]

Hi Antoine,

On 05/03/2021 4:07 pm, Antoine Pitrou wrote:

On Fri, 5 Mar 2021 15:03:59 +
Mark Shannon  wrote:


There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague.
That's my fault; I should have done more implementation first :(
FWIW, I have an implementation that should be portable.
https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


I looked through this diff and I'm not sure how this works robustly.

This seems to assume:
- each thread's stack size is known and is a compile-time constant
We just need to know that the stack size is at least the compile-time 
constant, we don't need to know exactly what it is.


If the stack size is less than the compile-time constant, then a crash 
is a possibility. However, since PEP 651 would reduce stack consumption, 
a crash would have been less likely than it currently is.


For all supported platforms, there is a default stack size and it is 
considerably larger than the chosen value of 512k.



- Python owns the thread and can compute the stack limit reliably from
   the current stack frame
There are two points of entry into Python code. From the high level API 
and thread_run().
Since thread_run is called when a thread is started, it is guaranteed to 
have the full stack available.
The high level API (PyRun_SimpleStringFlags and friends) is a bit more 
problematic. If there isn't sufficient stack space, then a crash is a 
possibility.
But, this is no worse than the status quo, and probably better due to 
reduced C stack use.



- the OS allocates stack pages in units of BLOCK_SIZE or more (currently
   8kiB)
The BLOCK_SIZE is just a number. A larger numbers means fewer slow 
checks, but wastes more stack. It doesn't matter what the underlying 
platform's granularity is.



- the OS doesn't use a segmentation scheme that limits stack accesses
   with a finer granularity than page (or "block") size

I don't think that would matter.


- it's ok to write memory beyond the current stack top (I can imagine
   verification tools such as Valgrind complaining about that, though
   this can be worked around using suppressions)

It's not much of a stack if you can't grow it :)
I wouldn't be surprised if valgrind complains, though.



I would be curious if you could elaborate on these points.


The above scheme does require some care from users of the high-level 
C-API and should be tested thoroughly for new platforms.

But it is a *lot* more robust than what we have now.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WT6I75DELQM43VP5QYAYQGMG3RFPHA3F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] CFLAGS_NODIST and -qalias=noansi

[Michael Felt]

Dear all,

After several months of absence - my first manual build surprised me by 
the addition of the -qalias=noansi.


Before I open an issue - maybe it is not that important - I am trying to 
find what brought it in. It looks to be a change in behavior in 
configure(.ac) - starting with Python3.9.


Historical builds - and make.out:

py38-3.8.7/.buildaix/make.out:0
py39-3.9.0/.buildaix/make.out:239
py39-3.9.1.0/.buildaix/make.out:254
py39-3.9.1/.buildaix/make.out:148
py3a-3.10/.buildaix/make.out:244

Prior to Python3.9

xlc_r -c  -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 
-I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64    
-I../src/py38-3.8.7/Include/internal


i.e., the flags became:

xlc_r -c  -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 
-I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64  -qalias=noansi 
-qmaxmem=-1 -I../src/py38-3.8.7/Include/internal


Note the flags have been repeating themselves for a long time, but now - 
after the repeated flags `-qalias=noansi -qmaxmem=-1` is being added. my 
gut repsonse is that I do not like seeing 'noansi' being passed as a 
flag to the compiler and I am trying to understand why - also how.


Fyi: using `git log` I have tried to get any clue re: CFLAGS_NODIST 
and/or -qalias=noansi - but I do not seem to be skilled enough to find 
that information.


Help is appreciated - and, thereafter, as needed, I'll open an issue.

Regards,

M Felt aka aixtools



OpenPGP_0x722BFDB61F396FC2.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LNGKVRQWYQUXGHAP57UYSTC4RPCXVFSP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Baptiste Carvello]

Hi,

Le 05/03/2021 à 14:46, Irit Katriel via Python-Dev a écrit :
> 
> 
> Whether you have "as" or not, the value of sys.exc_info() (which is what
> would be attached as the context to anything you raise in the except
> block) is the same. So these are not two different cases -- the only
> difference is whether or not you have a local variable set to
> sys.exc_info().

I don't really understand how this contradicts the behavior I proposed
(i.e. that bare "raise" raises something else than what is bound to the
"as" variable).

Specifically, which of the following statements are hard and unbreakable
rules, even when exception groups are involved:

1) sys.exc_info() must be the same object that is bound to the "as"
variable, if any;

2) sys.exc_info() must be the same object that is attached as __context__;

3) sys.exc_info() must be the same object that is raised by a bare "raise".

I'd say 1 is a hard rule because both objects are visible to user code,
so backwards compatibility applies. Rules 2 and 3, however, are internal
to the exception handling machinery, so I'm not so sure.

Cheers,
Baptiste

> On Thu, Mar 4, 2021 at 4:46 PM Baptiste Carvello
>  > wrote:
> 
> Hi,
> 
> I'll take a shoot at this, just to see how it tastes… So, let's say:
> 
> When an exception group reaches a set of traditional "except" clauses,
> those are examined one after the other, in the order they are in the
> code. That way, exceptions matched by several clauses will cause the
> first one to run, same as today.
> 
> A subgroup is built with the subset of exceptions matched by the
> examined clause, as the PEP specifies for "except*". If this subgroup is
> None, the clause is not selected, and the next clause, if any, is
> examined. On the contrary, if the subgroup contains at least one matched
> exception, the clause is selected and no other clause will run (again,
> same as today). Exceptions not part of the subgroup are discarded.
> 
> The clause body is then run just once (so the boss only gets one email
> about KeyboardInterrupt). If the clause uses the "as" form, the "as"
> variable is bound to one exception in the subgroup, which one is
> unspecified (at least for now). The other ones are discarded, except if
> a bare "raise" is reached (below).
> 
> If a bare "raise" is reached while executing the body, the selected
> subgroup propagates out of the "try-except" construct. Justification:
> the whole group cannot propagate, because today a bare "raise" cannot
> reraise exceptions of a type not matched by the clause. However, if a
> single-type exception group is handled similar to a single exception in
> traditional "except" clauses, it is acceptable to let it propagate.
> 
> So you would have:
> 
> try:
>     g=BaseExceptionGroup(
>         [ValueError(), KeyboardInterrupt(), KeyboardInterrupt()])
>     raise g
> except RuntimeError:               # doesn't match
>     log_the_error()
> except  KeyboardInterrupt as e: # builds s=g.subgroup(KeyboardInterrupt)
>     email_the_boss(e)           # tells the boss of any one error
>     raise                       # reraises s
> except BaseException:           # would match, but doesn't run
>     launch_nuclear_attack()
> 
> # BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt()])
> # propagates further, a traditional "except KeyboardInterrupt"
> # would catch it. The ValueError is discarded.
> 
> An interesting feature would be: when the matching clause has no "as",
> "except" behaves the same as "except*", apart from the fact that only
> one clause may run.
> 
> Cheers,
> Baptiste
> ___
> Python-Dev mailing list -- python-dev@python.org
> 
> To unsubscribe send an email to python-dev-le...@python.org
> 
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> 
> https://mail.python.org/archives/list/python-dev@python.org/message/EXC6BQVHQXUXBHIEWHLSLU6FY7SJKIF3/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LXAATJDDWXYYQ3P62SAZ4H7CEZGX2TUW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Antoine Pitrou]

On Fri, 5 Mar 2021 15:03:59 +
Mark Shannon  wrote:
> 
> There are two issues here. Portability and changes to behaviour.
> 
> Regarding portability, I have to admit that PEP is rather vague.
> That's my fault; I should have done more implementation first :(
> FWIW, I have an implementation that should be portable.
> https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation

I looked through this diff and I'm not sure how this works robustly.

This seems to assume:
- each thread's stack size is known and is a compile-time constant
- Python owns the thread and can compute the stack limit reliably from
  the current stack frame
- the OS allocates stack pages in units of BLOCK_SIZE or more (currently
  8kiB)
- the OS doesn't use a segmentation scheme that limits stack accesses
  with a finer granularity than page (or "block") size
- it's ok to write memory beyond the current stack top (I can imagine
  verification tools such as Valgrind complaining about that, though
  this can be worked around using suppressions)

I would be curious if you could elaborate on these points.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HFV33HIVTWLIJXI4WQAGK6ZLWYQHKJPX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Integer concatenation to byte string

[Martin (gzlist) via Python-Dev]

On Tue, 2 Mar 2021 at 21:44, Memz  wrote:
>
> foo+= 255 # Works the same as 
> bytesvariable+=b"ÿ"

foo = b"%b%d" % (foo, 255)

> foo+= a"\x255\x00"  # Concatenation with itself

foo = b"%b%b" % (foo, foo)

See PEP461:

Adding % formatting to bytes and bytearray

"With Python 3 and the split between str and bytes, one small but
important area of programming became slightly more difficult, and much
more painful -- wire format protocols."

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TM2XN4P6JMOFP3RUSFJLFPTDC5J77J7O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Mark Shannon]

Hi,

Thanks for taking the time to consider the PEP.

Although the PEP was rejected, I still believe that the safety 
guarantees in PEP 651 are worth adding to Python in the future.


To do that (maybe for 3.11), I need to understand your concerns better.

Would you clarify a few points for me?

On 03/03/2021 7:19 pm, Python Steering Council wrote:

Hi Mark,

Thank you for submitting PEP 651. The Steering Council has spent the past two 
weeks reviewing PEP 651. After careful consideration, we have decided to reject 
the PEP. The following were the key points that led us to this decision:

* The benefits are not compelling enough. Deep recursion is not a common tool in
   Python, and even with PEP 651 it would not be efficient enough to make it a 
common
   tool.

* The benefit of PEP 651 is negated as soon as a non-Python function is 
involved in the
   recursion, making the likelihood of it being useful even smaller. It also 
creates
   easy pitfalls for users who do end up relying on recursion.


Could you give an example pitfall?



* We believe the PEP understates the disruption created by the technical 
solution of
   multiple Python stack frames per C call. Although this may be solvable, it 
will
   certainly cause substantial disruption to existing debuggers, tracers, and 
state
   inspection tools as they need to adapt to this change (which may not be 
trivial).


This is presumably the key objection.
Is there a particular tool that you feel would be problematic?
I have only looked at gdb and py-spy.



* As the way to approach this will be platform-specific (as some parts of the 
proposal
   are not portable), this can cause generic Python code to behave differently 
on
   different platforms, making this kind of code less portable and less 
predictable.



There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague.
That's my fault; I should have done more implementation first :(
FWIW, I have an implementation that should be portable.
https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation

Regarding changes to behaviour, I don't see how "generic" Python code 
would behave differently on different platforms, except for cases where 
it already does.


In some cases, the PEP would have improved the situation.

For example:
sys.setrecursionlimit(5000)
def f():
f()

Currently, it raises a RecursionError on linux, but crashes the 
interpreter on Windows.

With PEP 651 it would have raised a RecursionError on both platforms.

Am I missing something here?


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L3U7FRGRFAXOVLA6YEAG3TSQB7EDQ2TU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Thu, Mar 4, 2021 at 11:15 PM Glenn Linderman 
wrote:

> I like explicit, and avoiding magic.
>
> And this gives a compatibility story for outer loops that except:
> Exception, and even for others cases that are not recoded for
> ExceptionGroup handling.
>


It could help during the migration/mixed python version phase. But I'm not
convinced yet that this is what we want to end up with in the long term.

Waiting to hear more thoughts.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VITKAEOCPYMWN6EOZAGT23A3Y2ATOHC4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Thu, Mar 4, 2021 at 8:48 PM Sven R. Kunze  wrote:

> Hi Irit,
>
> makes sense. So, in case of a *mixed-type ExceptionGroup,* SystemExit
> wins and forces the program to exit.
>
>
> Could you add your reasoning to the PEP?
>

Good idea, I'll add "ExceptionGroup(BaseException)" as a rejected idea and
explain it there.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WJUKFBFDVRW5VQK74QKZVYFSAQO3FLBT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Whether you have "as" or not, the value of sys.exc_info() (which is what
would be attached as the context to anything you raise in the except block)
is the same. So these are not two different cases -- the only difference is
whether or not you have a local variable set to sys.exc_info().

On Thu, Mar 4, 2021 at 4:46 PM Baptiste Carvello <
devel2...@baptiste-carvello.net> wrote:

> Hi,
>
> I'll take a shoot at this, just to see how it tastes… So, let's say:
>
> When an exception group reaches a set of traditional "except" clauses,
> those are examined one after the other, in the order they are in the
> code. That way, exceptions matched by several clauses will cause the
> first one to run, same as today.
>
> A subgroup is built with the subset of exceptions matched by the
> examined clause, as the PEP specifies for "except*". If this subgroup is
> None, the clause is not selected, and the next clause, if any, is
> examined. On the contrary, if the subgroup contains at least one matched
> exception, the clause is selected and no other clause will run (again,
> same as today). Exceptions not part of the subgroup are discarded.
>
> The clause body is then run just once (so the boss only gets one email
> about KeyboardInterrupt). If the clause uses the "as" form, the "as"
> variable is bound to one exception in the subgroup, which one is
> unspecified (at least for now). The other ones are discarded, except if
> a bare "raise" is reached (below).
>
> If a bare "raise" is reached while executing the body, the selected
> subgroup propagates out of the "try-except" construct. Justification:
> the whole group cannot propagate, because today a bare "raise" cannot
> reraise exceptions of a type not matched by the clause. However, if a
> single-type exception group is handled similar to a single exception in
> traditional "except" clauses, it is acceptable to let it propagate.
>
> So you would have:
>
> try:
> g=BaseExceptionGroup(
> [ValueError(), KeyboardInterrupt(), KeyboardInterrupt()])
> raise g
> except RuntimeError:   # doesn't match
> log_the_error()
> except  KeyboardInterrupt as e: # builds s=g.subgroup(KeyboardInterrupt)
> email_the_boss(e)   # tells the boss of any one error
> raise   # reraises s
> except BaseException:   # would match, but doesn't run
> launch_nuclear_attack()
>
> # BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt()])
> # propagates further, a traditional "except KeyboardInterrupt"
> # would catch it. The ValueError is discarded.
>
> An interesting feature would be: when the matching clause has no "as",
> "except" behaves the same as "except*", apart from the fact that only
> one clause may run.
>
> Cheers,
> Baptiste
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EXC6BQVHQXUXBHIEWHLSLU6FY7SJKIF3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TN2DVSAATJSYJ6EVWVRK2Z4EBXWAYUCT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python history: origin of the arrow annotation

[Steven D'Aprano]

I was curious how and why return annotations use the arrow `->` symbol, 
so I went spelunking into the depths of the Python-Ideas and Python-Dev 
mailing lists.

Much to my surprise, I couldn't find any discussion or debate about it.

Eventually I tracked the discussion back to a mailing list I didn't even 
know existed, Types-Sig, all the way back to 13th Dec 1999, where I 
found this email by Tim Hochberg:

https://mail.python.org/pipermail/types-sig/1999-December/000281.html

suggesting the arrow symbol for all type declarations. This appears to 
be the first suggestion of the arrow symbol for type annotations in 
Python.

I am surprised that type checking was being discussed so long ago. 
Python was not even a decade old, and if my calculations are correct, 
the versions at the time would have been 1.5.2 and 1.6.0.

But I am especially amazed that the choice of symbol seems to have been 
accepted with very little argument or debate about alternative colours 
for the bike-shed. I can't find any suggestions for alternative symbols 
such as `=>` `-->` `==>` `->>` etc, and very little for keywords such as 
`as`. The day after Tim posted, Tony Lownds suggested `as` and then 
immediately suggested using Tim's arrow symbol for the return type only:

https://mail.python.org/pipermail/types-sig/1999-December/000335.html

and then as far as I can tell, we've never looked back.

If there is anyone whose memory reaches back that far, does that sound 
right?


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ZZP5DJARHHKN4HBC4JHHRIV5Z2EZOVT/
Code of Conduct: http://python.org/psf/codeofconduct/