[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Michał Górny]

On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
> I was reading a discussion thread 
>  about 
> various issues with the Debian packaged version of Python, and the following 
> statement stood out for me as shocking:
> 
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we 
> > don't want users to break their operating system with sudo pip install.
> 
> I don't think sudo pip install should break the operating system. And I think 
> if it does, that problem should be solved rather than merely advising users 
> against using it. And why is it, anyway, that distributions whose package 
> managers can't coexist with pip-installed packages don't ever seem to get the 
> same amount of flak for "damaging python's brand" as Debian is getting from 
> some of the people in the discussion thread? Why is it that this community is 
> resigned to recommending a workaround when distributions decide the 
> site-packages directory belongs to their package manager rather than pip, 
> instead of bringing the same amount of fiery condemnation of that practice as 
> we apparently have for *checks notes* splitting parts of the stdlib into 
> optional packages? Why demand that pip be present if we're not going to 
> demand that it works properly?
> 
> I think that installing packages into the actual python installation, both 
> via distribution packaging tools and pip [and using both simultaneously - the 
> Debian model of separated dist-packages and site-packages folders seems like 
> a reasonable solution to this problem] can and should be a supported 
> paradigm, and that virtual environments [or more extreme measures such as 
> shipping an entire python installation as part of an application's 
> deployment] should ideally be reserved for the rare corner cases where that 
> doesn't work for some reason.

The problem is a little deeper and the Debian solution doesn't really
solve all of it.

Yes, pip installing into the same directory as the package manager
is a problem.  It's a problem to the point that I'm patching pip
in Gentoo to explicitly block that.  We've gotten far too many bug
reports about people's systems suddenly being horribly broken after they
used pip.

While using two directories can prevent pip from directly overwriting
system packages, you still can't expect two independent package managers
to work simultaneously unless they can communicate with each other to
prevent conflicts.  If pip installs a different version of the same
package as the package manager, which one is supposed to be used? 
Whichever choice you make, you'll bound to eventually break dependency
graph of some package.

-- 
Best regards,
Michał Górny

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QRARBX6CJBV6ZVDFVOVLHKZJ5P44NPTC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

FWIW, Distro repacks are advantageous in comparison to
"statically-bundled" releases that for example bundle in an outdated
version of OpenSSL, because when you `apt-get upgrade -y` that should
upgrade the OpenSSL that all the other distro packages depend upon.

Here's something that doesn't get called as frequently as `apt-get upgrade -y`:

```bash
pip install -U certifi
```

https://github.com/certifi/python-certifi (Mozilla's CA bundle
extracted into a Python package)

```bash
apt-get install -y ca-certificates
dnf install -y ca-certificates
```

On 2/24/21, Wes Turner  wrote:
> On 2/23/21, Random832  wrote:
>> I was reading a discussion thread
>>  about
>> various issues with the Debian packaged version of Python, and the
>> following
>> statement stood out for me as shocking:
>>
>> Christian Heimes wrote:
>>> Core dev and PyPA has spent a lot of effort in promoting venv because we
>>> don't want users to break their operating system with sudo pip install.
>>
>> I don't think sudo pip install should break the operating system. And I
>> think if it does, that problem should be solved rather than merely
>> advising
>> users against using it. And why is it, anyway, that distributions whose
>> package managers can't coexist with pip-installed packages don't ever
>> seem
>> to get the same amount of flak for "damaging python's brand" as Debian is
>> getting from some of the people in the discussion thread? Why is it that
>> this community is resigned to recommending a workaround when
>> distributions
>> decide the site-packages directory belongs to their package manager
>> rather
>> than pip, instead of bringing the same amount of fiery condemnation of
>> that
>> practice as we apparently have for *checks notes* splitting parts of the
>> stdlib into optional packages? Why demand that pip be present if we're
>> not
>> going to demand that it works properly?
>>
>> I think that installing packages into the actual python installation,
>> both
>> via distribution packaging tools and pip [and using both simultaneously -
>> the Debian model of separated dist-packages and site-packages folders
>> seems
>> like a reasonable solution to this problem] can and should be a supported
>> paradigm, and that virtual environments [or more extreme measures such as
>> shipping an entire python installation as part of an application's
>> deployment] should ideally be reserved for the rare corner cases where
>> that
>> doesn't work for some reason.
>>
>> How is it that virtual environments have become so indispensable, that
>> no-one considers installing libraries centrally to be a viable model
>> anymore? Are library maintainers making breaking changes too frequently,
>> reasoning that if someone needs the old version they can just venv it? Is
>> there some other cause?
>
> First, pip+venv is not sufficient for secure software deployment:
> something must set appropriate permissions so that the application
> cannot overwrite itself and other core libraries (in order to
> eliminate W^X violations (which e.g. Android is solving by requiring
> all installed binaries to come from an APK otherwise they won't and
> can't be labeled with the SELinux extended file atrributes necessary
> for a binary to execute; but we don't have binaries, we have an
> interpreter and arbitrary hopefully-signed somewhere source code, at
> least)).
>
> Believe it or not, this is wrong:
>
> ```bash
> # python -m venv httpbin || virtualenv httpbin
> # source httpbin/bin/activate
> mkvirtualenv httpbin
>
> pip install httpbin gunicorn
> gunicorn -b 0.0.0.0:8001 httpbin:app
>
> # python -m webbrowser http://127.0.0.1:8001
> ```
>
> It's wrong - it's insecure - because the user executing the Python
> interpreter (through gunicorn, in this case) can overwrite the app.
> W^X: has both write and execute permissions. What would be better?
>
> This would be better because pip isn't running setup.py as root (with
> non-wheels) and httpbin_exec can't modify the app interpreter or the
> code it loads at runtime:
>
> ```bash
> useradd httpbin # also creates a group also named 'httpbin'
> sudo -u httpbin sh -c ' \
> python -m venv httpbin; \
> umask 0022; \
> ./httpbin/bin/python -m pip install httpbin gunicorn'
>
> useradd httpbin_exec -G httpbin
> sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'
> ```
>
> This would be better if it worked, though there are a few caveats:
>
> ```bash
> sudo apt-get install python-gunicorn python-httpbin
> sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app
> ```
>
> 1. Development is impossible:
> - You can't edit the code in /usr/lib/python3.n/site-package/ without
> root permissions.
> - You should not be running an editor as root.
> - You can edit distro-package files individually with e.g. sudoedit
> (and then the GPG-signed package file checksums will fail when you run
> `debsums` or `rpm -Va` because you've edited the fi

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

On 2/23/21, Random832  wrote:
> I was reading a discussion thread
>  about
> various issues with the Debian packaged version of Python, and the following
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
>> Core dev and PyPA has spent a lot of effort in promoting venv because we
>> don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> I think that installing packages into the actual python installation, both
> via distribution packaging tools and pip [and using both simultaneously -
> the Debian model of separated dist-packages and site-packages folders seems
> like a reasonable solution to this problem] can and should be a supported
> paradigm, and that virtual environments [or more extreme measures such as
> shipping an entire python installation as part of an application's
> deployment] should ideally be reserved for the rare corner cases where that
> doesn't work for some reason.
>
> How is it that virtual environments have become so indispensable, that
> no-one considers installing libraries centrally to be a viable model
> anymore? Are library maintainers making breaking changes too frequently,
> reasoning that if someone needs the old version they can just venv it? Is
> there some other cause?

First, pip+venv is not sufficient for secure software deployment:
something must set appropriate permissions so that the application
cannot overwrite itself and other core libraries (in order to
eliminate W^X violations (which e.g. Android is solving by requiring
all installed binaries to come from an APK otherwise they won't and
can't be labeled with the SELinux extended file atrributes necessary
for a binary to execute; but we don't have binaries, we have an
interpreter and arbitrary hopefully-signed somewhere source code, at
least)).

Believe it or not, this is wrong:

```bash
# python -m venv httpbin || virtualenv httpbin
# source httpbin/bin/activate
mkvirtualenv httpbin

pip install httpbin gunicorn
gunicorn -b 0.0.0.0:8001 httpbin:app

# python -m webbrowser http://127.0.0.1:8001
```

It's wrong - it's insecure - because the user executing the Python
interpreter (through gunicorn, in this case) can overwrite the app.
W^X: has both write and execute permissions. What would be better?

This would be better because pip isn't running setup.py as root (with
non-wheels) and httpbin_exec can't modify the app interpreter or the
code it loads at runtime:

```bash
useradd httpbin # also creates a group also named 'httpbin'
sudo -u httpbin sh -c ' \
python -m venv httpbin; \
umask 0022; \
./httpbin/bin/python -m pip install httpbin gunicorn'

useradd httpbin_exec -G httpbin
sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'
```

This would be better if it worked, though there are a few caveats:

```bash
sudo apt-get install python-gunicorn python-httpbin
sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app
```

1. Development is impossible:
- You can't edit the code in /usr/lib/python3.n/site-package/ without
root permissions.
- You should not be running an editor as root.
- You can edit distro-package files individually with e.g. sudoedit
(and then the GPG-signed package file checksums will fail when you run
`debsums` or `rpm -Va` because you've edited the file and that's
changed the hash).

- Non-root users cannot install python packages without having someone
repack (and sign it) for them.

- What do I need to do in order to patch the distro's signed repack of
the Python package released to PyPI?
  - I like how Fedora pkgs and conda-forge have per-package git repos now.
  - Conda-forge has a bot that watches PyPI for new releases and tries
sending an automated PR.
  - If I send a PR to the main branch of the source repo and it gets
merged, how long will it be before there's a distro repack built and
uploaded to the distro package index?

2. It should be installed in a chroot/jail/zone/container/context/vm
so that it cannot read other data on the machine.
The httpbin app does not need read access to 

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Tue, Feb 23, 2021 at 8:00 AM Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

>
>- In
>https://www.python.org/dev/peps/pep-0654/#programming-without-except,
>the natural way isn't shown:
>
> try:
> 
> except (MultiError, ValueError) as e:
> def _handle(e):
> if isinstance(e, ValueError):
> return None
> else:
> return exc
> MultiError.filter(_handle,e)
>
> So a statement that the current handling is "cumbersome" and "unintuitive"
> is unconvincing.
>
> If this results in lots of boilerplate code with isinstance(), filter()
> can be changed to e.g. accept a dict of exception types and handlers.
> Actually, I wonder why you didn't do that already if handling specific
> exception types and reraising the rest is the standard procedure!
> Then the code would be reduced to:
>
> try:
> 
> except (MultiError, ValueError) as e:
> MultiError.filter({ValueError: lambda _: None},
>   e)
>
>
>- https://github.com/python-trio/trio/issues/611 says that it somehow
>causes problems with 3rd-party libraries -- but there's no explanation how
>-- either there or in the PEP.
>
> If some code doesn't know about MultiError's, it should handle one like
> any other unknown exception that it cannot do anything intelligent about.
> If it wishes to handle them, you need to split MultiError into a separate
> library that anyone could use without having to pull the entire `trio`.
>

I think the main point our PEP tries to make is that having to define a
helper function to handle exceptions (and then having to call a utility to
call the helper) feels clumsy.

However there may be an omission in the PEP -- what if we want to do
something special for each suberror? If we just iterate over `eg.errors` we
would have to do something recursive for exceptions wrapped inside multiple
nested groups. We could add a helper method to ExceptionGroup that iterates
over suberrors, flattening nested groups. But we'd need to do something
special to recover the tracebacks there (groups share the common part of
the traceback). Or we could say "if you need the traceback you're going to
have to write something recursive" -- like we have to do in traceback.py.
But we might as well generalize whatever we're doing there. (Irit: I
suspect there's a piece of API design we missed here.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WLGKRP4OQXXW32HXBGG65OEFHZTQTOVQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Ethan Furman]

On 2/23/21 7:56 PM, Guido van Rossum wrote:

On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman wrote:



It sounds like the long-term goal is to move away from `except` and
replace it with `except *` -- is that correct?


I don't think so -- if we expected that to happen the extra '*' in the
syntax would be a nuisance. The premise of the PEP is rather that
raising and catching multiple exceptions at once is always going
to be an esoteric hobby. The most common case by far would be in async
frameworks, but we don't expect 'async def' to eventually become the
standard function definition either (nor 'await f()' the standard call :-).

And even in an async framework, I don't expect that every async function
would be "upgraded" to raise ExceptionGroup -- only specific APIs like
gather() or create_connection(). (Even for those, backwards compatibility
concerns will probably mean that we'll have to introduce *new* APIs that
can raise ExceptionGroup.) The PEP lists a handful of other use cases,
none of which seem to suggest to me that this is going to be a common thing.


I can see the value in `except *` for concurrent code; my concern is how it 
will operate with the standard `try/except` framework.  With the new 
non-Exception-derived ExceptionGroup class, existing try-excepts that catch 
Exception will fail for every ExceptionGroup that leaks through; the fix for 
that is a doubly-nested try-except/try-except* -- what do we get for that code 
churn?  What is the advantage in not deriving from Exception?  If I recall 
correctly, we have only three exception types that derive directly from 
BaseException: Exception, SystemExit, and KeyboardInterrupt -- with SystemExit 
and KeyboardInterrupt both being concerned with making sure an application can 
quit.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/445RAHP7XXLQFFGFKCXITPQPGAB55MDY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Tue, Feb 23, 2021 at 2:27 PM Damian Shaw 
wrote:

> A common example I see from the GitHub search is a catch all exception on
> some third party API. If the user wants to catch all exceptions from
> calling a third party API why would they want to let an ExceptionGroup go
> unhandled in the future? Once ExceptionGroups are introduced then libraries
> will surely implement them where appropriate and users would want to
> continue having their board exception handling working, would they not?
>

See my response to Ethan -- I think libraries should think twice before
raising ExceptionGroup.

Note that ExceptionGroup is a subclass of BaseException, so if you're
catching the latter you don't have to do anything special.

OTOH we might reconsider deriving ExceptionGroup from BaseException --
maybe it's better to inherit from Exception? I don't think the PEP
currently has a clear reason why it must derive from BaseException. I
believe it has to do with the possibility that ExceptionGroup might wrap a
BaseException instance (e.g. KeyboardInterrupt or SystemExit). But I
believe that this was more important in an earlier revision of the design.
Note that there is a section labeled "Should MultiError inherit from
BaseException, or from Exception? (Or something else?)" in Trio's
MultiError v2 issue (https://github.com/python-trio/trio/issues/611). I
think the key argument here is "However, if MultiError is an Exception,
then it can only contain Exception subclasses, because otherwise a
MultiError(KeyboardInterrupt) could get caught by an except Exception
handler, which would be bad." But those notes are from back in 2018, not
from the 2020 core dev sprint when we got much closer to the current
design. So I'm not sure the reasoning still applies. (Yury? Can you help us
out here?)


> I don't actually know if the code I wrote in the first email is correct
> though, would it catch all standard Exception / ExceptionGroups across
> Python 3.x if this PEP was implemented in the future? And would it be best
> practice for this pattern? If so it seems it would require many users to
> rewrite their code as it adds boilerplate and nuance for a pattern that is
> a relatively simple and very commonly used right now.
>
> I guess summarized my commentary is please don't dismiss this pattern out
> of hand, which this PEP currently disrupts, it is used widely in Python
> right now and has many valid use cases. I don't have anything further to
> add so I won't continue this discussion and I assume anything further you
> have to add to be considered and more informed than my own opinion.
>

I expect that if you want to do a good job of emulating `except
*ValueError` in Python versions that don't have ExceptionGroup you're going
to have to write more code. It's relatively straightforward if you just
want to ignore or log the errors collectively, but it'll be more complex if
you want to handle each suberror separately, because the PEP-654-aware code
will have to iterate over the errors, recursing into ExceptionGroup
instances. (Irit, maybe the PEP could use a worked-out example of how to do
the latter -- even with `except *` available it seems non-trivial.)

Thanks,
> Damian
>

Thanks Damian for getting us to think more about 'except Exception:'

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FS4IT2PEYU4ZD7UAUSMJ5IK2DWCZOEJT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman  wrote:

> On 2/22/21 4:24 PM, Irit Katriel via Python-Dev wrote:
>
>  > We would like to request feedback on PEP 654 -- Exception Groups and
>  > except*.
>  >
>  > https://www.python.org/dev/peps/pep-0654/
>  >
>  > It proposes language extensions that allow programs to raise and
>  > handle multiple unrelated exceptions simultaneously, motivated by the
>  > needs of asyncio and other concurrency libraries, but with other use
>  > cases as well.
>
> It sounds like the long-term goal is to move away from `except` and
> replace it with `except *` -- is that correct?
>

I don't think so -- if we expected that to happen the extra '*' in the
syntax would be a nuisance. The premise of the PEP is rather that raising
and catching multiple exceptions at once is always going to be an esoteric
hobby. The most common case by far would be in async frameworks, but we
don't expect 'async def' to eventually become the standard function
definition either (nor 'await f()' the standard call :-).

And even in an async framework, I don't expect that every async function
would be "upgraded" to raise ExceptionGroup -- only specific APIs like
gather() or create_connection(). (Even for those, backwards compatibility
concerns will probably mean that we'll have to introduce *new* APIs that
can raise ExceptionGroup.) The PEP lists a handful of other use cases, none
of which seem to suggest to me that this is going to be a common thing.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DVMATQ46G2IGRQWORUZUFD5NCSBKUWRH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Ethan Furman]

On 2/22/21 4:24 PM, Irit Katriel via Python-Dev wrote:

> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
>
> It proposes language extensions that allow programs to raise and
> handle multiple unrelatedexceptions simultaneously, motivated by the
> needs of asyncio and other concurrency libraries,but with other use
> cases as well.

It sounds like the long-term goal is to move away from `except` and 
replace it with `except *` -- is that correct?


--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IT56OKHUKVUNJHAF2UYNDXTK5MYPDNZ6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Emily Bowman]

On Tue, Feb 23, 2021 at 4:51 PM Random832  wrote:

> Why is it that this community is resigned to recommending a workaround
> when distributions decide the site-packages directory belongs to their
> package manager rather than pip, instead of bringing the same amount of
> fiery condemnation of that practice as we apparently have for *checks
> notes* splitting parts of the stdlib into optional packages? Why demand
> that pip be present if we're not going to demand that it works properly?
>

The alternative to virtualenv is the solution most software takes for, eg,
Java: Package a complete binary distribution internally to eliminate any
dependency on the system package -- and allow the software to pollute their
own package in any way they want without affecting the rest of the system,
while potentially becoming very out of date with security patches.
Virtualenv allows software to lockstep and pollute their own runtime of
Python almost all they want while still sharing the actual base install
that gets regular security and bugfix updates. (Of course, even the hint of
possibility that anything about the runtime could change and cause downtime
is too much for many enterprise Java systems, no matter how severe the
security update, so even ported to Python they wouldn't settle for that,
they'd still package the entire interpreter internally.)

Since telling application developers "no, don't do that, just get along"
doesn't work, no matter how loudly you say it, virtualenv is a happy middle
ground. A few core OS maintainers are much more likely to listen, so the
yelling about splitting up or overly customizing the stdlib achieves
results.

-Em
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S6BIVMQH32J5VVWITDBSAWWJARSAVIFF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Jonathan Goble]

On Tue, Feb 23, 2021 at 7:48 PM Random832  wrote:
>
> I was reading a discussion thread 
>  about 
> various issues with the Debian packaged version of Python, and the following 
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we 
> > don't want users to break their operating system with sudo pip install.
>
> [snip]
>
> How is it that virtual environments have become so indispensable, that no-one 
> considers installing libraries centrally to be a viable model anymore? Are 
> library maintainers making breaking changes too frequently, reasoning that if 
> someone needs the old version they can just venv it? Is there some other 
> cause?

I can't speak for distributors or maintainers [1], but I can speak for
myself as a user. I run Debian testing (currently bullseye as that is
preparing for release) as my daily OS on my personal laptop, used for
personal matters and school assignments (I'm a university computer
science student in my senior year).

I don't use the system Python for anything of my own, whether it's a
school assignment or a personal project, precisely because I don't
want to risk screwing something up. Rather, I maintain a clone/fork of
the official CPython GitHub repo, and periodically build from source
and `make altinstall` into `~/.local/`. The `python3` command
continues to refer to the system Python, while `python3.8`, etc. refer
to the ones installed in my home folder. To the latter I make symlinks
for `py38`, `py39`, etc., and just `py` (and `pip`) for the one I use
most often (usually the latest stable release). I typically have
multiple versions installed at once since different scripts/projects
run on different versions at different times. Given this setup, I can
just do a simple `pip install spam` command and I don't need either
`sudo` or `--user`, nor do I need virtual envs.

While the average person would probably not clone the GitHub repo and
build that way, it's not terribly unreasonable for an inclined person
to do the same with a tarball downloaded from python.org, and so I
doubt I'm the only one with this type of setup.

Just some food for thought.

[1] Technically I am a library maintainer since I have a couple
projects on PyPI, but those are mostly unused and more or less
abandoned at this point, and neither ever reached the point where I
could consider graduating them from beta status. Most of what I work
with these days is private personal code or school assignments.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M6OGQXKLG27Y6KA5L5UFBWA2NAZNBOHL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Have virtual environments led to neglect of the actual environment?

[Paul Bryan]

I think it's a classic case of dependency hell.

OS packagers are rebundling Python packages as OS packages and
expressing their own OS-package dependency graphs. Then, you sudo pip
install something that has a conflicting dependency, it bypasses OS
packaging, and *boom*.

I find tools like pipx go a long way to solve this, as they install a
Python package and all of its dependencies in its own venv. This is
great for Python apps, and (kinda) treats them like apps on platforms
like Android, where all app dependencies are bundled and isolated from
others.

I think it would great if OS vendors did something similar to pipx for
Python-based apps: bundle the app and all of its dependencies into its
own venv.

 
On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
> I was reading a discussion thread
> 
> about various issues with the Debian packaged version of Python, and
> the following statement stood out for me as shocking:
> 
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv
> > because we don't want users to break their operating system with
> > sudo pip install.
> 
> I don't think sudo pip install should break the operating system. And
> I think if it does, that problem should be solved rather than merely
> advising users against using it. And why is it, anyway, that
> distributions whose package managers can't coexist with pip-installed
> packages don't ever seem to get the same amount of flak for "damaging
> python's brand" as Debian is getting from some of the people in the
> discussion thread? Why is it that this community is resigned to
> recommending a workaround when distributions decide the site-packages
> directory belongs to their package manager rather than pip, instead
> of bringing the same amount of fiery condemnation of that practice as
> we apparently have for *checks notes* splitting parts of the stdlib
> into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
> 
> I think that installing packages into the actual python installation,
> both via distribution packaging tools and pip [and using both
> simultaneously - the Debian model of separated dist-packages and
> site-packages folders seems like a reasonable solution to this
> problem] can and should be a supported paradigm, and that virtual
> environments [or more extreme measures such as shipping an entire
> python installation as part of an application's deployment] should
> ideally be reserved for the rare corner cases where that doesn't work
> for some reason.
> 
> How is it that virtual environments have become so indispensable,
> that no-one considers installing libraries centrally to be a viable
> model anymore? Are library maintainers making breaking changes too
> frequently, reasoning that if someone needs the old version they can
> just venv it? Is there some other cause?
> ___
> Python-ideas mailing list -- python-id...@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-id...@python.org/message/KMRNKSVEPHAXA7BHFT4PWX4EKWYUF4G7/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2VYJAJHHGOOPFQA6U7ATBKREMAC66CXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Have virtual environments led to neglect of the actual environment?

[Random832]

I was reading a discussion thread 
 about various 
issues with the Debian packaged version of Python, and the following statement 
stood out for me as shocking:

Christian Heimes wrote:
> Core dev and PyPA has spent a lot of effort in promoting venv because we 
> don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think 
if it does, that problem should be solved rather than merely advising users 
against using it. And why is it, anyway, that distributions whose package 
managers can't coexist with pip-installed packages don't ever seem to get the 
same amount of flak for "damaging python's brand" as Debian is getting from 
some of the people in the discussion thread? Why is it that this community is 
resigned to recommending a workaround when distributions decide the 
site-packages directory belongs to their package manager rather than pip, 
instead of bringing the same amount of fiery condemnation of that practice as 
we apparently have for *checks notes* splitting parts of the stdlib into 
optional packages? Why demand that pip be present if we're not going to demand 
that it works properly?

I think that installing packages into the actual python installation, both via 
distribution packaging tools and pip [and using both simultaneously - the 
Debian model of separated dist-packages and site-packages folders seems like a 
reasonable solution to this problem] can and should be a supported paradigm, 
and that virtual environments [or more extreme measures such as shipping an 
entire python installation as part of an application's deployment] should 
ideally be reserved for the rare corner cases where that doesn't work for some 
reason.

How is it that virtual environments have become so indispensable, that no-one 
considers installing libraries centrally to be a viable model anymore? Are 
library maintainers making breaking changes too frequently, reasoning that if 
someone needs the old version they can just venv it? Is there some other cause?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KMRNKSVEPHAXA7BHFT4PWX4EKWYUF4G7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Petr,

Thank you for your careful reading and encouragement.

 > The `ExceptionGroup` class is final, i.e., it cannot be subclassed.
>
> What's the rationale for this?
>

ExceptionGroup.subgroup()/split() need to create new instances, and
subclassing
would make that complicated if we want the split results to have the same
type.

 > It is possible to catch the ExceptionGroup type with except, but not
> with except* because the latter is ambiguous
>
> What about `except *(TypeError, ExceptionGroup):`?
>

Good question. We should block that as well.


>  > Motivation: Errors in wrapper code
>
> This use case sticks out a bit: it's the only one where ExceptionGroup
> doesn't represent joining equivalent tasks.
> Consider code similar to bpo-40857:
>
>try:
>with TemporaryDirectory() as tempdir:
>os.rmdir(tempdir)
>n = 1 / 0
>except ArithmeticError:
># that error can be safely ignored!
>pass
>
> Instead of a FileNotFoundError with ArithmeticError for context you'd
> now get an ExceptionGroup. Neither is handled by `except
> ArithmeticError`. Where is the win?
>

I agree, if TemporaryDirectory() were ever to adopt ExceptionGroups then it
will probably
be through a new API (like an opt-in parameter) to avoid breaking current
code. Users of
such a TemporaryDirectory would need to wrap the calls with try-except*
(see the example
in my previous reply to Steve).


>  > Motivation: Multiple failures when retrying an operation
>
> This is somewhat similar to the TemporaryDirectory, except there's no
> `with` block that feels like it should be "transparent" w.r.t. user errors.
> If I currently have:
>
>  try:
>  create_connection(*addresses)
>  except (Timeout, NetworkNotConnected):
>  # that's OK, let's try later
>  pass
>
> what should happen after Python 3.10? Apart from adding a new function,
> I can see two possible changes:
> - create_connection() starts always raising ExceptionGroup on error,
> breaking backwards compatibility in the error case
> - create_connection() starts only raising ExceptionGroup only for 2+
> errors, breaking backwards compatibility in the 2+ errors case


> Both look like heisenbug magnets. IMO, the second one is worse; "code
> which is now *potentially* raising ExceptionGroup" (as mentioned in the
> Backwards Compatibility section; emphasis mine) should be discouraged.
>
> Arguably, this here is a problem with the create_connection function:
> the PEP adds a better way how it could have been designed, and that is
> virtuous. Still, having it in Motivation might be misleading.
>


I agree. Raising ExceptionGroups is an API breaking change, for any
library.
No function should just start raising ExceptionGroups.


 > long term plan to replace `except` by `catch`
>
> Whoa! Is that a real plan?
>


In the rejected ideas section, anything goes!

Irit




>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R7ONTCH5MUWKBOPHY2QYKVNPOT5MXPYZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Caleb,

On Tue, Feb 23, 2021 at 11:05 PM Caleb Donovick 
wrote:

> What is the motivation for returning `None` on empty splits?  I feel like
> this creates an unnecessary asymmetry.  I don't personally have a use
> case for the this feature so I may be missing something but it seems like
> it would force an annoying pattern:
>

Split is used by the interpreter to implement except*. It needs to check
whether the first value of the split is empty or not (this determines
whether the except* clause executes).
This is more efficiently done with None. And then there is creating an
object that you don't need.

The intention is that you use except* rather than do this:


> ```
> try:
> foo()
> except ExceptionGroup as eg:
> g1, g2 = eg.split(predicate)
> for e in g1:
> handle_true(e)
> for e in g2:
> handle_false(e)
> ```
>

(as an aside - exception groups ended up not being iterable - that is one
of the rejected ideas, see explanation there).


> Also this creates an subtle difference with subgroup:
>

> ```
> g1, g2 = eg.split(predicate)
> h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
> assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
> ```
>

An empty subgroup should also return None. I see this is not mentioned in
the PEP, so I will add it. Thanks.

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PW25VAAF3J3WQFNBC6NBYJXQIK63NTQO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Daniel Moisset]

In addition to the changes proposed here that go beyond PEP-634 (which
other people discuss), I find that many of the definitions fail to capture
some of the basic features of PEP-634, especially when nesting patterns.

Take for example: "case [int(), str()]". According to
https://www.python.org/dev/peps/pep-0653/#sequence-patterns that desugars
to:

if $kind & MATCH_SEQUENCE:
if $list is None:
$list = list($value)
if len($list) == len([int(), str()]):
int(), str() = $list   *# This is invalid!*
if guard:
   DONE
In general, the way these semantics are defined non-recursively over
patterns won't be able to work with nested patterns.

I value the idea of having a more accurate definition of what the semantics
are, but this approach doesn't seem to work.

Best,   D.


On Thu, 18 Feb 2021 at 16:45, Mark Shannon  wrote:

> Hi everyone,
>
> I'd like to announce a new PEP.
> https://www.python.org/dev/peps/pep-0653/
>
> It builds on PEP 634 (Structural Pattern Matching: Specification), adding:
>
> More precise semantics.
> A bit more customization for complex classes.
>
> Its also a bit more robust and should be faster (eventually).
>
> The syntax is unchanged, and in most cases the semantics are the same.
>
> As always, comments and suggestions are welcome.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/O4I345VHIQUXPDJWPDA54RWVXMYEEBRM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6RGSDHU7ZB77RTWUS7IBTMCZNLV4NBJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[MRAB]

On 2021-02-23 23:05, Caleb Donovick wrote:
What is the motivation for returning `None` on empty splits?  I feel 
like this creates an unnecessary asymmetry.  I don't personally have a 
use case for the this feature so I may be missing something but it seems 
like it would force an annoying pattern:


```
try:
     foo()
except ExceptionGroup as eg:
     g1, g2 = eg.split(predicate)
     for e in g1:
         handle_true(e)
     for e in g2:
         handle_false(e)
```
Would need to be written:
```
try:
     foo()
except ExceptionGroup as eg:
     g1, g2 = eg.split(predicate)
     if g1 is not None:
         for e in g1:
             handle_true(e)
     if g2 is not None:
         for e in g2:
             handle_false(e)
```

Also this creates an subtle difference with subgroup:

```
g1, g2 = eg.split(predicate)
h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
```


.subgroup returns an ExceptionGroup and .split returns 2 of them.

ExceptionGroup isn't iterable (it's mentioned in the "Rejected Ideas" 
section), so your code wouldn't work anyway.


On Mon, Feb 22, 2021 at 4:47 PM Irit Katriel via Python-Dev 
mailto:python-dev@python.org>> wrote:



Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and
except*.

https://www.python.org/dev/peps/pep-0654/


It proposes language extensions that allow programs to raise and
handle multiple unrelated
exceptions simultaneously, motivated by the needs of asyncio and
other concurrency libraries,
but with other use cases as well.

* A new standard exception type, ExceptionGroup, to represent
multiple exceptions with
   shared traceback.
* Updates to the traceback printing code to display (possibly
nested) ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:
https://github.com/iritkatriel/cpython/pull/10


Thank you for your help


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U7A4PZE7SSDFNKAFZZ3BZBLL7VGTOWVL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Caleb Donovick]

What is the motivation for returning `None` on empty splits?  I feel like
this creates an unnecessary asymmetry.  I don't personally have a use
case for the this feature so I may be missing something but it seems like
it would force an annoying pattern:

```
try:
foo()
except ExceptionGroup as eg:
g1, g2 = eg.split(predicate)
for e in g1:
handle_true(e)
for e in g2:
handle_false(e)
```
Would need to be written:
```
try:
foo()
except ExceptionGroup as eg:
g1, g2 = eg.split(predicate)
if g1 is not None:
for e in g1:
handle_true(e)
if g2 is not None:
for e in g2:
handle_false(e)
```

Also this creates an subtle difference with subgroup:

```
g1, g2 = eg.split(predicate)
h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
```

On Mon, Feb 22, 2021 at 4:47 PM Irit Katriel via Python-Dev <
python-dev@python.org> wrote:

>
> Hi all,
>
> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
>
> It proposes language extensions that allow programs to raise and handle
> multiple unrelated
> exceptions simultaneously, motivated by the needs of asyncio and other
> concurrency libraries,
> but with other use cases as well.
>
> * A new standard exception type,  ExceptionGroup, to represent multiple
> exceptions with
>   shared traceback.
> * Updates to the traceback printing code to display (possibly nested)
> ExceptionGroups.
> * A new syntax except* for handling ExceptionGroups.
>
> A reference implementation (unreviewed) can be found at:
> https://github.com/iritkatriel/cpython/pull/10
>
> Thank you for your help
>
> Kind regards
> Irit, Yury & Guido
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WWJ6KLJFZ627UBB4MUTIJQVWWO6LRISO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Steve,

Thank you for trying out the implementation. Please do let me know about
bugs you find.

This part of your code looks good:

--- a/Lib/tempfile.py
> +++ b/Lib/tempfile.py
> @@ -819,8 +819,14 @@ def __repr__(self):
>  def __enter__(self):
>  return self.name
>
> -def __exit__(self, exc, value, tb):
> -self.cleanup()
> +def __exit__(self, exc_cls, exc_value, tb):
> +try:
> +self.cleanup()
> +except Exception as clean_exc:
> +if exc_value is not None:
> +raise ExceptionGroup('Exception occurred during cleanup',
> [exc_value, clean_exc])
> +else:
> +raise
>
>  def cleanup(self):
>  if self._finalizer.detach():
>

def do_some_stuff():
with tempfile.TemporaryDirectory() as td:
os.rmdir(td)
pathlib.Path(td).write_text("Surprise!")
1/0


Then, use it like this:

if __name__ == '__main__':
try:
do_some_stuff()
except *NotADirectoryError as e:
print("Fail Site 2", repr(e))
except *Exception as e:
print("Fail Site 3", repr(e))
else:
print("No error")


Output:
   Fail Site 2 ExceptionGroup('Exception occurred during cleanup',
[NotADirectoryError(20, 'The directory name is invalid')])
   Fail Site 3 ExceptionGroup('Exception occurred during cleanup',
[ZeroDivisionError('division by zero')])

This works whether do_some_stuff raises naked exceptions or
ExceptionGroups, because "except *T"  catches a naked T (and wraps it in an
ExceptionGroup).
So if I comment out the 1/0 I get:

Fail Site 2 ExceptionGroup('', (NotADirectoryError(20, 'The directory
name is invalid'),))

There is no need for it to always raise ExceptionGroup. It can propagate
the user's exception as is if it has no exceptions to add. But the caller
needs to assume that it may raise an ExceptionGroup, and use except*.

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AI7YFJK7KSCVNBAMAKQWMLSIGTT3OJ5X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Damian Shaw]

Hi Irit,

A common example I see from the GitHub search is a catch all exception on
some third party API. If the user wants to catch all exceptions from
calling a third party API why would they want to let an ExceptionGroup go
unhandled in the future? Once ExceptionGroups are introduced then libraries
will surely implement them where appropriate and users would want to
continue having their board exception handling working, would they not?

I don't actually know if the code I wrote in the first email is correct
though, would it catch all standard Exception / ExceptionGroups across
Python 3.x if this PEP was implemented in the future? And would it be best
practice for this pattern? If so it seems it would require many users to
rewrite their code as it adds boilerplate and nuance for a pattern that is
a relatively simple and very commonly used right now.

I guess summarized my commentary is please don't dismiss this pattern out
of hand, which this PEP currently disrupts, it is used widely in Python
right now and has many valid use cases. I don't have anything further to
add so I won't continue this discussion and I assume anything further you
have to add to be considered and more informed than my own opinion.

Thanks,
Damian



On Tue, Feb 23, 2021 at 4:41 PM Irit Katriel 
wrote:

>
> Hi Damian,
>
> While I agree that there are a handful of use cases for catching all
> Exceptions, if you look at some of the 5 million hits you found in github,
> the vast majority are not such cases. They are mostly try/except blocks
> that wrap a particular operation (a dict access, a raw_input call, etc).
> You don't want to automatically translate "except Exception" to something
> that involves ExceptionGroups, because most of the time it is not needed --
> the code in the body can't raise ExceptionGroups.
>
> Your first email on this thread showed that you were able to quite
> quickly, just from what's already in the PEP, think of how to do what you
> need for your valid "except Exception" use case. My feeling is that people
> who need it will be able to do it quite easily. I'm more worried about
> people assuming that they need it when they don't.
>
> Irit
>
>
> On Tue, Feb 23, 2021 at 8:00 PM Damian Shaw 
> wrote:
>
>> Hi Irit,
>>
>> Catching exceptions like this is an extremely common pattern in the real
>> world, e.g. this pattern has over 5 million GitHub matches:
>> https://github.com/search?l=&q=%22except+Exception%22+language%3APython&type=code
>>
>> How common it is aside there are also many valid use cases for this
>> pattern, e.g. logging a final unhandled exception of a script, catching
>> exceptions inside an orchestration framework, exploring code where the list
>> of exceptions is unknown, etc.
>>
>> A description from the PEP on how to handle this kind of pattern,
>> especially for code that must support multiple versions of Python, would be
>> extremely helpful.
>>
>> Damian
>>
>>>
>>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4MSMXURT72BQCISBQVVBW3TZWXF5QC7P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Petr Viktorin]

On 2/23/21 1:24 AM, Irit Katriel via Python-Dev wrote:


Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and 
except*.


https://www.python.org/dev/peps/pep-0654/ 




Thank you for this PEP!

Also, thank you Nathaniel (and possibly other author(s) of Trio – sadly, 
I'm not following closely enough to track contributions) for doing 
things right even though it's hard (and slow), and documenting your work 
beautifully. I'm glad to see the ideas assimilated into Python & asyncio!


The PEP reminds me of PEP 380 (yield from): it looks like syntax sugar 
for code you could already write, but once you look closer, it turns out 
that there are so many details and corner cases to keep track of, 
getting it correct is very hard.



I kept notes as I read the PEP, then deleted most as I went through 
Rejected Ideas. These remained:



> The `ExceptionGroup` class is final, i.e., it cannot be subclassed.

What's the rationale for this?


> It is possible to catch the ExceptionGroup type with except, but not 
with except* because the latter is ambiguous


What about `except *(TypeError, ExceptionGroup):`?


> Motivation: Errors in wrapper code

This use case sticks out a bit: it's the only one where ExceptionGroup 
doesn't represent joining equivalent tasks.

Consider code similar to bpo-40857:

  try:
  with TemporaryDirectory() as tempdir:
  os.rmdir(tempdir)
  n = 1 / 0
  except ArithmeticError:
  # that error can be safely ignored!
  pass

Instead of a FileNotFoundError with ArithmeticError for context you'd 
now get an ExceptionGroup. Neither is handled by `except 
ArithmeticError`. Where is the win?



> Motivation: Multiple failures when retrying an operation

This is somewhat similar to the TemporaryDirectory, except there's no 
`with` block that feels like it should be "transparent" w.r.t. user errors.

If I currently have:

try:
create_connection(*addresses)
except (Timeout, NetworkNotConnected):
# that's OK, let's try later
pass

what should happen after Python 3.10? Apart from adding a new function, 
I can see two possible changes:
- create_connection() starts always raising ExceptionGroup on error, 
breaking backwards compatibility in the error case
- create_connection() starts only raising ExceptionGroup only for 2+ 
errors, breaking backwards compatibility in the 2+ errors case


Both look like heisenbug magnets. IMO, the second one is worse; "code 
which is now *potentially* raising ExceptionGroup" (as mentioned in the 
Backwards Compatibility section; emphasis mine) should be discouraged.


Arguably, this here is a problem with the create_connection function: 
the PEP adds a better way how it could have been designed, and that is 
virtuous. Still, having it in Motivation might be misleading.




> long term plan to replace `except` by `catch`

Whoa! Is that a real plan?



--

Also, one of the examples has such a missed opportunity to use 
print(f'{e1 = }')!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFGOEAX4A46BVQHBX6IECOAW63KQAFGU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Stestagg]

This pep sounded kinda scary to me, so I wanted to try it out.

The reference implementation appears to have some bugs in it (around
reraise star) , so it's not entirely clear what the expected behavior is
supposed to be, but by checking out ffc493b5 I got some OK results.

I had a look at the tempfile example given under the Motivations section,
and wanted to see how this would work (as it's the area of code I'm most
familiar with).

Would the proposed tempfile change look something like this? (functionally,
if not stylistically):

--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -819,8 +819,14 @@ def __repr__(self):
 def __enter__(self):
 return self.name

-def __exit__(self, exc, value, tb):
-self.cleanup()
+def __exit__(self, exc_cls, exc_value, tb):
+try:
+self.cleanup()
+except Exception as clean_exc:
+if exc_value is not None:
+raise ExceptionGroup('Exception occurred during cleanup',
[exc_value, clean_exc])
+else:
+raise

 def cleanup(self):
 if self._finalizer.detach():

If so, then the following code fails to catch the ZeroDivisionError (there
is an uncaught exception raised):

import tempfile, os, pathlib

def do_some_stuff():
with tempfile.TemporaryDirectory() as td:
os.rmdir(td)
pathlib.Path(td).write_text("Surprise!")
1/0

if __name__ == '__main__':
try:
do_some_stuff()
except Exception:
print("Something went wrong")
else:
print("No error")

The output I get:
"""
Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 7, in do_some_stuff
1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 824, in
__exit__
self.cleanup()
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 833, in
cleanup
self._rmtree(self.name)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 809, in
_rmtree
_shutil.rmtree(name, onerror=onerror)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 718, in
rmtree
_rmtree_safe_fd(fd, path, onerror)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 631, in
_rmtree_safe_fd
onerror(os.scandir, path, sys.exc_info())
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 627, in
_rmtree_safe_fd
with os.scandir(topfd) as scandir_it:
NotADirectoryError: [Errno 20] Not a directory: '/tmp/tmpeedn6r0n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 12, in 
do_some_stuff()
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 7, in do_some_stuff
1/0
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 827, in
__exit__
raise ExceptionGroup('Exception occurred during cleanup', [exc_value,
clean_exc])
ExceptionGroup: Exception occurred during cleanup
"""

===

So, to catch and handle errors raised from TemporaryDirectory safely, the
try-except has to be wrapped in a try-*except block?:


if __name__ == '__main__':
try:
try:
do_some_stuff()
except Exception:
print("Fail Site 1")
except *NotADirectoryError:
print("Fail Site 2")
except *Exception:
print("Fail Site 3")

In this situation, Sites 2 and 3 are called, but if there is no problem
during cleanup then Site 1 is called?

If, instead, the ExceptionGroup is explicitly handled:

if __name__ == '__main__':
try:
do_some_stuff()
except (Exception, ExceptionGroup):
print("Fail Site 1")

Then this actually works quite nicely for the 'except Exception' scenario,
but is much more complex if you want to catch and handle specific types of
exceptions:

if __name__ == '__main__':
try:
do_some_stuff()
except ExceptionGroup as exc_group:
zd_errors, others = exc_group(lambda e: isinstance(e,
ZeroDivisionError))
if zd_errors:
print("Fail Site 1")
if others:
raise others
except ZeroDivisionError:
print("Fail Site 2")

If the idea is that tempfile.TemporaryDirectory *always* raises an
ExceptionGroup (even if there was no cleanup exception) then any code that
calls anything that might eventually call TemporaryDirectory will have to
be aware that a different type of exception could appear that normal
handling doesn't catch (or for /every/ usage of TemporaryDirectory be
immediately wrapped in try-*except handling code).

It feels like ever letting an ExceptionGroup unwind more than 1 or two
stack frames is super dangerous, as it directly requires all 

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Damian,

While I agree that there are a handful of use cases for catching all
Exceptions, if you look at some of the 5 million hits you found in github,
the vast majority are not such cases. They are mostly try/except blocks
that wrap a particular operation (a dict access, a raw_input call, etc).
You don't want to automatically translate "except Exception" to something
that involves ExceptionGroups, because most of the time it is not needed --
the code in the body can't raise ExceptionGroups.

Your first email on this thread showed that you were able to quite quickly,
just from what's already in the PEP, think of how to do what you need for
your valid "except Exception" use case. My feeling is that people who need
it will be able to do it quite easily. I'm more worried about people
assuming that they need it when they don't.

Irit


On Tue, Feb 23, 2021 at 8:00 PM Damian Shaw 
wrote:

> Hi Irit,
>
> Catching exceptions like this is an extremely common pattern in the real
> world, e.g. this pattern has over 5 million GitHub matches:
> https://github.com/search?l=&q=%22except+Exception%22+language%3APython&type=code
>
> How common it is aside there are also many valid use cases for this
> pattern, e.g. logging a final unhandled exception of a script, catching
> exceptions inside an orchestration framework, exploring code where the list
> of exceptions is unknown, etc.
>
> A description from the PEP on how to handle this kind of pattern,
> especially for code that must support multiple versions of Python, would be
> extremely helpful.
>
> Damian
>
>>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5SOONPYY3COVU43QMOG37EBMHSE4JFSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw 
wrote:

>
> Firstly, if I have a library which supports multiple versions of Python
> and I need to catch all standard exceptions, what is considered the best
> practise after this PEP is introduced?
>
> Currently I might have code like this right now:
> try:
> ... # Code
> except Exception as e:
> ... # Logic to handle exception
>
>
Hi Damian,

Catching all exceptions in this way is not a common pattern. Typically you
have an idea what kind of exceptions you expect to get from an operation,
and which of those exceptions you are interested in handling. Most of your
try/except blocks will be targeted, like handling KeyError from d[k], an
operation that will never raise an ExceptionGroup.

ExceptionGroups will only be raised by specific APIs which will advertise
themselves as such. We don't expect that there will be a need for blanket
migrations of "except T" to "except (T, ExceptionGroup)" to "except *T".

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TWEGKU7H5GC3I4WHNVBQ2EPWQMSP633Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Damian Shaw]

Hi Irit,

Catching exceptions like this is an extremely common pattern in the real
world, e.g. this pattern has over 5 million GitHub matches:
https://github.com/search?l=&q=%22except+Exception%22+language%3APython&type=code

How common it is aside there are also many valid use cases for this
pattern, e.g. logging a final unhandled exception of a script, catching
exceptions inside an orchestration framework, exploring code where the list
of exceptions is unknown, etc.

A description from the PEP on how to handle this kind of pattern,
especially for code that must support multiple versions of Python, would be
extremely helpful.

Damian

On Tue, Feb 23, 2021 at 2:35 PM Irit Katriel 
wrote:

> On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw 
> wrote:
>
>>
>> Firstly, if I have a library which supports multiple versions of Python
>> and I need to catch all standard exceptions, what is considered the best
>> practise after this PEP is introduced?
>>
>> Currently I might have code like this right now:
>> try:
>> ... # Code
>> except Exception as e:
>> ... # Logic to handle exception
>>
>>
> Hi Damian,
>
> Catching all exceptions in this way is not a common pattern. Typically you
> have an idea what kind of exceptions you expect to get from an operation,
> and which of those exceptions you are interested in handling. Most of your
> try/except blocks will be targeted, like handling KeyError from d[k], an
> operation that will never raise an ExceptionGroup.
>
> ExceptionGroups will only be raised by specific APIs which will advertise
> themselves as such. We don't expect that there will be a need for blanket
> migrations of "except T" to "except (T, ExceptionGroup)" to "except *T".
>
> Irit
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XAPO475TVFICD77M3VRLC7OKA5O7WTOT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [SPAM] Re: Move support of legacy platforms/architectures outside Python

[Rob Boehne]

On 2/22/21, 4:06 PM, "Antoine Pitrou"  wrote:

On Mon, 22 Feb 2021 19:50:43 +
Rob Boehne  wrote:
> 
> The other thing that crept into this thread was the mention of test that 
intermittently fail.
> That's a huge problem because it suggests that applications will 
sometimes fail.
> I have usually seen these sort of issues because of
>   1) Uninitialized memory being used (read)
>   2) Threading problems
>   3) Resources used (files, networking, daemons) but unavailable
>   4) memory mis-management (buffer overrun that doesn't cause a crash)
> 
> #3 is probably best fixed by testing for resources and skipping when 
unavailable

5) Poor quality POSIX support in the target platform.  The Python test
suite is actually quite demanding in this regard (except on Windows).

Regards

Antoine.

   

Antoine:

I find this a bit of a surprise, can you give an example of poor POSIX support 
leading to intermittent test failures? 

Thanks,

Rob Boehne 


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5XXXKPQCF2OTXLKYOJ5AJIW27M3OUHY3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Ivan Pozdeev via Python-Dev]

 * In https://www.python.org/dev/peps/pep-0654/#programming-without-except, the 
natural way isn't shown:

try:
    
except (MultiError, ValueError) as e:
    def _handle(e):
    if isinstance(e, ValueError):
        return None
    else:
    return exc
    MultiError.filter(_handle,e)

So a statement that the current handling is "cumbersome" and "unintuitive" is 
unconvincing.

If this results in lots of boilerplate code with isinstance(), filter() can be changed to e.g. accept a dict of exception types and 
handlers. Actually, I wonder why you didn't do that already if handling specific exception types and reraising the rest is the standard 
procedure!

Then the code would be reduced to:

try:
    
except (MultiError, ValueError) as e:
    MultiError.filter({ValueError: lambda _: None},
  e)


 * https://github.com/python-trio/trio/issues/611 says that it somehow causes 
problems with 3rd-party libraries -- but there's no
   explanation how -- either there or in the PEP.

If some code doesn't know about MultiError's, it should handle one like any other unknown exception that it cannot do anything intelligent 
about.
If it wishes to handle them, you need to split MultiError into a separate library that anyone could use without having to pull the entire 
`trio`.



On 23.02.2021 3:24, Irit Katriel via Python-Dev wrote:


Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/ 


It proposes language extensions that allow programs to raise and handle 
multiple unrelated
exceptions simultaneously, motivated by the needs of asyncio and other 
concurrency libraries,
but with other use cases as well.

* A new standard exception type, ExceptionGroup, to represent multiple 
exceptions with
  shared traceback.
* Updates to the traceback printing code to display (possibly nested) 
ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:
https://github.com/iritkatriel/cpython/pull/10 


Thank you for your help

Kind regards
Irit, Yury & Guido



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
Code of Conduct: http://python.org/psf/codeofconduct/


--
Regards,
Ivan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DY7BGNVB6GBYSDBLL3IVZQ7GWYEQCEI5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Damian Shaw]

Apologies I'm not a core dev so this might be the wrong place to ask but I
have 2 small clarifying questions about the PEP.

Firstly, if I have a library which supports multiple versions of Python and
I need to catch all standard exceptions, what is considered the best
practise after this PEP is introduced?

Currently I might have code like this right now:
try:
... # Code
except Exception as e:
... # Logic to handle exception

Reading through the PEP, particularly on backwards compatibility it looks
like I would now need to do this:
try:
ExceptionGroup
except NameError:
standard_exceptions = Exception
else:
standard_exceptions = (Exception, ExceptionGroup)

try:
... # Code
except standard_exceptions as e:
... # Logic to handle exception

Secondly, once I only support versions of Python that support Exception
groups, what replaces the best practise for generally catching standard
exceptions? Is it this:
try:
... # Code
except *Exception as e:
... # Logic to handle exception

Thanks,
Damian (He/Him)

On Mon, Feb 22, 2021 at 7:48 PM Irit Katriel via Python-Dev <
python-dev@python.org> wrote:

>
> Hi all,
>
> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
>
> It proposes language extensions that allow programs to raise and handle
> multiple unrelated
> exceptions simultaneously, motivated by the needs of asyncio and other
> concurrency libraries,
> but with other use cases as well.
>
> * A new standard exception type,  ExceptionGroup, to represent multiple
> exceptions with
>   shared traceback.
> * Updates to the traceback printing code to display (possibly nested)
> ExceptionGroups.
> * A new syntax except* for handling ExceptionGroups.
>
> A reference implementation (unreviewed) can be found at:
> https://github.com/iritkatriel/cpython/pull/10
>
> Thank you for your help
>
> Kind regards
> Irit, Yury & Guido
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZFZOASIEUABFOJPUHXPZ3HRYKENE6VT/
Code of Conduct: http://python.org/psf/codeofconduct/