[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Wed, Feb 24, 2021 at 1:38 AM Irit Katriel 
wrote:

>
>
> On Wed, Feb 24, 2021 at 4:39 AM Guido van Rossum  wrote:
>
>>
>> OTOH we might reconsider deriving ExceptionGroup from BaseException --
>> maybe it's better to inherit from Exception? I don't think the PEP
>> currently has a clear reason why it must derive from BaseException. I
>> believe it has to do with the possibility that ExceptionGroup might wrap a
>> BaseException instance (e.g. KeyboardInterrupt or SystemExit).
>>
>
>
> That was the reason, and we also said that an ExceptionGroup escaping
> something that isn't supposed to be raising ExceptionGroups is a bug, so if
> you call an API that raises ExceptionGroups it is your responsibility to
> handle them.
>
> We could make ExceptionGroup be an Exception, and refuse to wrap anything
> which is not an Exception. So asyncio either raises KeyboardInterrupt or an
> ExceptionGroup of user exceptions.
>
> Those are quite different directions.
>
>>
Indeed. I don't think we require that all wrapped exceptions derive from
Exception; an important special case in asyncio is CancelledError, which
doesn't derive from Exception. (Ditto for trio.Cancelled.)

It is really worth carefully reading the section I mentioned in Trio's
MultiError v2 issue. But its "plan" seems complex, as it would require
asyncio to define `class TaskGroupError(ExceptionGroup, Exception)` -- that
was fine when we considered this *only* for asyncio, but now that we've
found several other use cases it really doesn't feel right.

Here's a potentially alternative plan, which is also complex, but doesn't
require asyncio or other use cases to define special classes. Let's define
two exceptions, BaseExceptionGroup which wraps BaseException instances, and
ExceptionGroup which only wraps Exception instances. (Names to be
bikeshedded.) They could share a constructor (always invoked via
BaseExceptionGroup) which chooses the right class depending on whether
there are any non-Exception instances being wrapped -- this would do the
right thing for split() and subgroup() and re-raising unhandled exceptions.

Then 'except Exception:' would catch ExceptionGroup but not
BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be
caught (even if there's also e.g. a ValueError among the wrapped errors).

Pseudo-code:

class BaseExceptionGroup(BaseException):
def __new__(cls, msg, errors):
if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e
in errors):
cls = ExceptionGroup
return BaseException.__new__(cls, msg, errors)

class ExceptionGroup(Exception, BaseExceptionGroup):
pass

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZFXN3GCNWC6GMZMC3UD56NLGAHGZBLTY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Wed, Feb 24, 2021 at 7:09 PM Emily Bowman 
wrote:

> After reading through the PEP and skimming the code (but I didn't build
> it), something I didn't see: What happens to a currently conforming except
> check?
>
> try:
> async with trio.open_nursery() as nursery:
> # Make two concurrent calls to child()
> nursery.start_soon(child)
> nursery.start_soon(child)
> except ValueError:
> pass
>
> I've removed the * from the example: Say the interface was built for 3.7,
> but the "trio" module has been upgraded to use ExceptionGroups which can't
> fall back to a standalone exception.
>
> Silently hand back the first exception, or the first matching exception?
> Deprecation warning? Silently skip? Context-specific error? Run the default
> error handler?
>
> I think that deserves a statement in the PEP.
>

The ExceptionGroup would bubble up. IIRC this Trio behavior is considered
problematic by the Trio developers, because it means that if *two* children
fail with ValueError, it will raise MultiError and the except clause will
not trigger (see the "MultiError v2" issue linked from the PEP).

But changing trio.open_nursery() to raise ExceptionGroup would be
considered backwards compatible. I would recommend introducing a new API
instead. This is discussed (though maybe not in enough detail?) under
Backwards Compatibility in the PEP.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OYKCVAOYAJKOI24DRWMUSJUJKGQ74CQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Wed, Feb 24, 2021 at 5:58 PM Jim J. Jewett  wrote:

> If it (compatible __new__ and __init__) needs to be checked at definition
> time, just try create an instance passing the same arguments you would pass
> to the base class.  If the creation it doesn't raise an exception, that is
> good enough.
>

User code can do that, the interpreter should not use such heuristics.


> This isn't about theoretical type safety against malice; it is about
> defining the minimal protocol for an ExceptionGrouping that has to be
> supported by someone who wants something other than the default flavor.
>

You still haven't shown a use case for wanting to subclass ExceptionGroup.
It's easy to allow this later if there's a valid use case (it wouldn't
require a PEP, just a well-reasoned use case). But if we allow it now and
in the future we discover it causes subtle bugs, it would be difficult to
roll back (requiring deprecation over several releases). Why would you want
a different flavor of ExceptionGroup?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SMEJMIMSIMWOXW47KDMRL3POCRHNTSXC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

On Wed, Feb 24, 2021 at 5:51 PM Jim J. Jewett  wrote:

> Are you saying that
>
> except *ValueError as e:
>
> will catch
>
> ValueError
> and
> ExceptionGroup(ValueError)
> but miss
> ExceptionGroup(ExceptionGroup(ValueError))
> ?
>

No, it will catch it regardless of how many levels of ExceptionGroup wrap
it (zero or more, in fact).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3XIBSH7IJDIAM2YMGI42ZWQSOIHTNW2R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

[Subthread: handling individual errors]

> Rather than iterator, I think we should add a visitor that calls a
function for each leaf exception,

I agree that we shouldn't add an `__iter__` method. But what if we added a
separate method that returns an iterator? Then you could write
```
for e in eg.something():
print("caught", e)
```
This looks cleaner to me than having to write
```
def handler(e):
print("caught", e)
eg.something(handler)
```
(Does my bias against callback functions shine through? :-)

> plus a utility that returns the traceback of a leaf exception
> (as a single list, copying frames from the tracebacks of ExceptionGroups
so that it's not destructive.)

Hm, what would the arguments for that utility be? Since ExceptionGroups can
be nested, if you pass in the root EG and the leaf exception, it would
require a full tree search to find the nodes that have all the needed
tracebacks.

Maybe the API for the iterator-returning method should be that it returns
an iterator of (exception, list-of-tracebacks) pairs? Or maybe there should
be two iterator-returning methods, one that yields just exceptions and one
that yields such pairs, so that if you don't need the tracebacks we don't
have to construct them. (Separate methods so the static type doesn't depend
on the value of a flag.) If in the end we were to go for something with a
handler callback we could do a similar thing.



On Wed, Feb 24, 2021 at 1:27 AM Irit Katriel 
wrote:

>
>
> On Wed, Feb 24, 2021 at 4:55 AM Guido van Rossum  wrote:
>
> However there may be an omission in the PEP -- what if we want to do
>> something special for each suberror? If we just iterate over `eg.errors` we
>> would have to do something recursive for exceptions wrapped inside multiple
>> nested groups. We could add a helper method to ExceptionGroup that iterates
>> over suberrors, flattening nested groups. But we'd need to do something
>> special to recover the tracebacks there (groups share the common part of
>> the traceback). Or we could say "if you need the traceback you're going to
>> have to write something recursive" -- like we have to do in traceback.py.
>> But we might as well generalize whatever we're doing there. (Irit: I
>> suspect there's a piece of API design we missed here.)
>>
>
> This is mentioned briefly where we talk about why ExceptionGroup is not
> iterable: https://www.python.org/dev/peps/pep-0654/#the-exceptiongroup-api
> We left it out pending a use case.
>
> Rather than iterator, I think we should add a visitor that calls a
> function for each leaf exception, plus a utility that returns the traceback
> of a leaf
> exception (as a single list, copying frames from the tracebacks of
> ExceptionGroups so that it's not destructive.) This way the expensive
> traceback
> construction is happening explicitly when it is needed.
>
> I think accessing one exception at a time to do something other than
> formatting it is more likely to be overused than actually needed.
> We did that in an earlier version of the PEP for the "filter OSErrors by
> type" example, which we did with iteration and now do with subgroup:
>
> try:
> low_level_os_operation()
> except *OSerror as errors:
> raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None
>
>
>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JEXBOJQKIGIEOYJZIBJSYTFNVUUBTGLL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Peter Wang]

On Wed, Feb 24, 2021 at 8:50 PM Mike Miller  wrote:

> I never understood the fear around version conflicts.


With binary extension modules, version conflicts lead to (at best) runtime
segfault and (at worst) subtle *data* bugs that return incorrect results.
There are also deeper concerns around security and reproducibility.


> Perhaps it has to do with
> the decline of sys-admin skills over the years?


Many millions of users of new Python users show up every year, using the
language and its powerful ecosystem for data analytics and scientific
computing, and they have no hope of having sys-admin skills.

-Peter
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TYZ6QT4BVQY7SVXV6E63YKNV6SCNFZ7V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving threadstate to thread-local storage.

[谢俊逸 via Python-Dev]

On MacOS & iOS, __thread variable is 35% faster than using pthread_getspecific. 

getSpecific cost: 0.000649
getTLS cost: 0.000423


- (void)test {
double t1 = CFAbsoluteTimeGetCurrent();
for (int i = 0; i < 10; i++) {
[self getSpecific];
}
double t2 = CFAbsoluteTimeGetCurrent();

printf("getSpecific cost: %f\n", t2 - t1);

double t3 = CFAbsoluteTimeGetCurrent();

for (int i = 0; i < 10; i++) {
[self getTLS];
}
double t4 = CFAbsoluteTimeGetCurrent();
printf("getTLS cost: %f\n", t4 - t3);

}

- (int)getSpecific {
int * value = pthread_getspecific(tlsKey);
return *value;
}
- (int)getTLS {
return *tlv_v5;
}___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KTHAKNENSBKURE7I2SRVXEPJ6NDNCACI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Emily Bowman]

After reading through the PEP and skimming the code (but I didn't build
it), something I didn't see: What happens to a currently conforming except
check?

try:
async with trio.open_nursery() as nursery:
# Make two concurrent calls to child()
nursery.start_soon(child)
nursery.start_soon(child)
except ValueError:
pass

I've removed the * from the example: Say the interface was built for 3.7,
but the "trio" module has been upgraded to use ExceptionGroups which can't
fall back to a standalone exception.

Silently hand back the first exception, or the first matching exception?
Deprecation warning? Silently skip? Context-specific error? Run the default
error handler?

I think that deserves a statement in the PEP.

-Em


On Mon, Feb 22, 2021 at 4:48 PM Irit Katriel via Python-Dev <
python-dev@python.org> wrote:

>
> Hi all,
>
> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
>
> It proposes language extensions that allow programs to raise and handle
> multiple unrelated
> exceptions simultaneously, motivated by the needs of asyncio and other
> concurrency libraries,
> but with other use cases as well.
>
> * A new standard exception type,  ExceptionGroup, to represent multiple
> exceptions with
>   shared traceback.
> * Updates to the traceback printing code to display (possibly nested)
> ExceptionGroups.
> * A new syntax except* for handling ExceptionGroups.
>
> A reference implementation (unreviewed) can be found at:
> https://github.com/iritkatriel/cpython/pull/10
>
> Thank you for your help
>
> Kind regards
> Irit, Yury & Guido
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2ABNKHUQK2GZZBPEOSOKZFVZTQCQ2QOK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Mike Miller]

On 2021-02-24 02:52, Stéfane Fermigier wrote:
I have currently 57 apps installed via pipx on my laptop, and the 57 
environments take almost 1 GB already.



I never understood the fear around version conflicts.  Perhaps it has to do with 
the decline of sys-admin skills over the years?  So, the strategy above feels 
like swatting a fly with a sledgehammer to me.  Same as with a venv for every 
app. :-)


Rather than installing every package this way, why not wait until a conflict 
actually occurs?


Personally, I rarely have such conflicts, maybe every few years or so.  When it 
happens, I fix them by uninstalling the offender and putting the more difficult 
one into the venv or pipx.  Right now I only have one, a giant app from work 
that uses pipenv, and it's fine.



Now what about sudo and all that?  Well, I used it in the old days because 
that's what the instructions said.  But, to be honest, it never made any sense. 
I haven't shared a computer in decades, and when we did we used NFS for common 
tools, so it never saved any disk space.


Pip (and easy_install?) dragged their feet for years to properly support user 
installs (should have been default) but once they did I didn't look back.  I 
dump almost all packages to user, which gets cleaned up every other year when 
the distro moves to the next version.  The strategy has been working well for a 
long time.


So, --user works at the low end, and containers for the high end.  Honestly, I 
don't have much of a use case any longer for venvs.


-Mike
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z2WSO5LC7WRX24XT5WERYHSA2UONQKB6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

Ideally, (at least) trivial subclasses could be declared, and the class itself 
would serve as the marker.  I would prefer regular subclasses, so that they 
could offer methods as well.  Alternatively, at least copy the instance 
__dict__ to the new ExceptionGroup instance.

By compatible __init__ and __new__, I mean "whatever you do in 
ExceptionGroup.subgroup to create a new instance with fewer contained 
Exceptions ... do the same with MyExceptionGroup."  I'm assuming that 
"whatever" is to call __new__ and __init__ with "a message string and a 
sequence of the nested exceptions, for example: ExceptionGroup('issues', 
[ValueError('bad value'), TypeError('bad type')])."

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SFE46EGTGLXLCJDXZ4EI6HOEATYREOL4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

If it (compatible __new__ and __init__) needs to be checked at definition time, 
just try create an instance passing the same arguments you would pass to the 
base class.  If the creation it doesn't raise an exception, that is good enough.

This isn't about theoretical type safety against malice; it is about defining 
the minimal protocol for an ExceptionGrouping that has to be supported by 
someone who wants something other than the default flavor.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JW4BFAR7SN23NF7S6TRGOROJIMPDGQEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

Are you saying that 

except *ValueError as e:

will catch

ValueError
and
ExceptionGroup(ValueError)
but miss
ExceptionGroup(ExceptionGroup(ValueError))
?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VLPYAILCWTGG25WYJXUA47LIQPULQ7ZM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

By the time a KeyboardInterrupt or SystemExit is being grouped into an 
ExceptionGroup, it already isn't being treated as an immediate interruption ... 
it has (presumably) already killed its own execution path, but it should not 
kill the program as a whole.  Whether ExceptionGroup inherits from Exception or 
BaseException shouldn't matter for deciding what it can group.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XSEL22SDNEX2BWT7GS7OUPDAA2UVMH42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

Would it be better to have an (os, os_release, arch) to
[site_packages_dirs] map within CPython or are the patches too varied to be
replaced by a centralized map and a runtime conditional?

For development convenience, having writeable site-packages dirs before
unwriteable site-packages is easy;

But for security and production deployment, site-packages maybe shouldn't
be writeable at all by default because that's doing it wrong?

On Wed, Feb 24, 2021, 15:34 Christian Heimes  wrote:

> On 24/02/2021 20.03, Christian Heimes wrote:
> > On 24/02/2021 19.17, Steve Dower wrote:
> >> On 2/24/2021 4:26 PM, Christian Heimes wrote:
> >>> On 24/02/2021 15.16, Random832 wrote:
>  On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> > Separate directories don't prevent clashes and system breakage. But
> > they
> > provide an easy way to *recover* from a broken system.
> 
>  I think it could be turned into a way to prevent them by A) having
>  site-packages always take precedence over dist-packages [i believe
>  this is already the case] in normal usage and B) providing an option
>  to the interpreter, used by system scripts, to exclude site-packages
>  entirely from the path.
> 
>  Basically, site-packages would effectively be layered on top of "Lib
>  + dist-packages" in a similar way to how a venv is layered on top of
>  the main python installation - the inverse of the suggestion someone
>  else in the thread made for the system python to be a venv. This
>  wouldn't *exactly* be a venv because it wouldn't imply the other
>  things that entering a venv does such as "python" [and script names
>  such as pip] being an alias for the correct version of python, but it
>  would provide the same kind of one-way isolation, whereby the "system
>  environment" can influence the "normal environment" and not
>  vice-versa, in the same way that packages installed in the main
>  installation affect a venv [unless system-site-packages is disabled]
>  but the venv obviously has no effect on the main installati
> >> on.
> >>>
> >>> Yes, you are describing one major aspect of my idea for a system Python
> >>> interpreter. I'm happy to read that other users are coming to similar
> >>> conclusions. Instead of an option I'd create a new executable to lock
> >>> down additional things (e.g. isolated mode, code verification hook). A
> >>> separate executable would also allow distros to provide a stripped down
> >>> interpreter that does not cause bad user experience.
> >>
> >> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
> >> option and PYTHONNOUSERSITE env variable to provide that one-way
> >> isolation).
> >>
> >> Is the problem that pip doesn't use it by default? Or that the distros
> >> went and made patches for the runtime rather than patching pip? (For
> >> Windows installs from the Store, where even admin rights can't do an
> >> all-users package install, I added a new config file location for pip to
> >> change this default, but a patch would also have worked.)
> >>
> >> Maybe we need an easier way to patch the location of user site packages?
> >> I also had to do this for the Store install on Windows, and it's a
> >> little bit of a hack... but maybe having an official recommendation
> >> would help encourage distributors to use the mechanism?
> >
> > (sorry for terse reply, I'm heading out.)
> >
> > Linux distros want an additional layer so the can differentiate between
> > Python packages installed by their package manager and Python packages
> > installed with "sudo pip install".
> >
> > Python has two default site-package directories (from highest to lowest
> > precedence):
> >
> > 1. ~/.local/lib/python3.9/site-packages
> > 2. /usr/lib/python3.9/site-packages
> >
> > (1) is defined by site.getusersitepackages(), (2) site.getsitepackages()
> >
> > Linux distro have additional site-package directories. My Fedora system
> > X86_64 system with multiarch has three additional candidates.
> >
> > 1. ~/.local/lib/python3.9/site-packages
> > 2. /usr/local/lib64/python3.9/site-packages
> > 3. /usr/local/lib/python3.9/site-packages
> > 4. /usr/lib64/python3.9/site-packages
> > 5. /usr/lib/python3.9/site-packages
> >
> > The "lib" directories are for pure Python packages whiel the "lib64"
> > directories contain X86_64 native code extensions (aka shared
> > libraries). The "/usr/lib*" directories are used for distro packages.
> > The downstream patch [1] ensures that "sudo pip install" installs
> > packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.
> >
> > Debian has a similar mechanism to provide
> > "/usr/lib/python3/dist-packages" [2].
> >
> > A hypothetical /usr/bin/system-python3 interpreter would only use (4)
> > and (5) as site-package directories. A user-facing /usr/bin/python3
> > would use (1) to (5).
> >
> > Christian
> >
> > [1]
> >
> 

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

Hi Ethan,

See Irit's response to my about why ExceptionGroup doesn't subclass
Exception -- I think this is still a little bit of an open issue.

Another response (which I also read somewhere in one of Irit's messages) is
that a library API shouldn't go from raising subclasses of Exception only
to raising ExceptionGroups. That would be a backwards incompatible change
to an API.

--Guido

On Tue, Feb 23, 2021 at 8:52 PM Ethan Furman  wrote:

> On 2/23/21 7:56 PM, Guido van Rossum wrote:
> > On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman wrote:
>
> >> It sounds like the long-term goal is to move away from `except` and
> >> replace it with `except *` -- is that correct?
> >
> > I don't think so -- if we expected that to happen the extra '*' in the
> > syntax would be a nuisance. The premise of the PEP is rather that
> > raising and catching multiple exceptions at once is always going
> > to be an esoteric hobby. The most common case by far would be in async
> > frameworks, but we don't expect 'async def' to eventually become the
> > standard function definition either (nor 'await f()' the standard call
> :-).
> >
> > And even in an async framework, I don't expect that every async function
> > would be "upgraded" to raise ExceptionGroup -- only specific APIs like
> > gather() or create_connection(). (Even for those, backwards compatibility
> > concerns will probably mean that we'll have to introduce *new* APIs that
> > can raise ExceptionGroup.) The PEP lists a handful of other use cases,
> > none of which seem to suggest to me that this is going to be a common
> thing.
>
> I can see the value in `except *` for concurrent code; my concern is how
> it will operate with the standard `try/except` framework.  With the new
> non-Exception-derived ExceptionGroup class, existing try-excepts that catch
> Exception will fail for every ExceptionGroup that leaks through; the fix
> for that is a doubly-nested try-except/try-except* -- what do we get for
> that code churn?  What is the advantage in not deriving from Exception?  If
> I recall correctly, we have only three exception types that derive directly
> from BaseException: Exception, SystemExit, and KeyboardInterrupt -- with
> SystemExit and KeyboardInterrupt both being concerned with making sure an
> application can quit.
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/445RAHP7XXLQFFGFKCXITPQPGAB55MDY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5JEJ2TM7VQIJPPH6SIJNTWX5BOCTS3CH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

Hi Jim,

Let me echo Irit's response here -- what's your use case for subclassing
ExceptionGroup?

Requiring that subclasses have a compatible __init__ or __new__ sounds like
a bug magnet, since this can't be checked when the subclass is being
defined, and it goes against conventional rules for those signatures.

--Guido

On Wed, Feb 24, 2021 at 6:22 AM Jim J. Jewett  wrote:

> The "no subclasses" seems pretty severe, particularly if you can't even
> use marker attributes because it isn't clear when a different instance of
> container ExceptionGroup will be substituted.
>
> The justification for this restriction was that .split() had to be be able
> to instantiate ... wouldn't it be enough to just say that subclasses need a
> compatible __init__ and __new__ for that reason, and then leave it to
> consenting adults?
>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5OLFDKJCNKLIJJQMNYYLQGV53Z7ZTRC5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BXW7E5UEWIJDCA2JN5QTBZXDFKQ4RFEK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi Jim,

On Wed, Feb 24, 2021 at 2:16 PM Jim J. Jewett  wrote:

> Petr:  What about except *(TypeError, ExceptionGroup):?
>
> Irit:  Good question. We should block that as well.
>
> But https://www.python.org/dev/peps/pep-0654/#backwards-compatibility
> seems to explicitly recommend the very similar:
>
> except (Exception, ExceptionGroup):



That's except, not except*.

Please include an example for:
>
> except *ExceptionGroup: pass
>

There is one here:
https://www.python.org/dev/peps/pep-0654/#forbidden-combinations
It raises a Runtime Error.


The "no subclasses" seems pretty severe, particularly if you can't even use
> marker attributes because it isn't clear when a different instance of
> container ExceptionGroup will be substituted.
>
> The justification for this restriction was that .split() had to be be able
> to instantiate ... wouldn't it be enough to just say that subclasses need a
> compatible __init__ and __new__ for that reason, and then leave it to
> consenting adults?


What do you mean by compatible?  How would one's marker attributes be
copied to the new instances?

split() currently copies (context, cause, traceback) from the original
ExceptionGroup to the new ones. We could have some protocol, but I'm not
sure what it would look like. I'm pretty sure there will be a cost, and I'm
not sure I understand why no subclassing is a serious enough limitation to
justify that.

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NMJSQW5D6O43KMLMOHY6X6Z2PXT5XYFB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 20.03, Christian Heimes wrote:
> On 24/02/2021 19.17, Steve Dower wrote:
>> On 2/24/2021 4:26 PM, Christian Heimes wrote:
>>> On 24/02/2021 15.16, Random832 wrote:
 On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> Separate directories don't prevent clashes and system breakage. But
> they
> provide an easy way to *recover* from a broken system.

 I think it could be turned into a way to prevent them by A) having
 site-packages always take precedence over dist-packages [i believe
 this is already the case] in normal usage and B) providing an option
 to the interpreter, used by system scripts, to exclude site-packages
 entirely from the path.

 Basically, site-packages would effectively be layered on top of "Lib
 + dist-packages" in a similar way to how a venv is layered on top of
 the main python installation - the inverse of the suggestion someone
 else in the thread made for the system python to be a venv. This
 wouldn't *exactly* be a venv because it wouldn't imply the other
 things that entering a venv does such as "python" [and script names
 such as pip] being an alias for the correct version of python, but it
 would provide the same kind of one-way isolation, whereby the "system
 environment" can influence the "normal environment" and not
 vice-versa, in the same way that packages installed in the main
 installation affect a venv [unless system-site-packages is disabled]
 but the venv obviously has no effect on the main installati
>> on.
>>>
>>> Yes, you are describing one major aspect of my idea for a system Python
>>> interpreter. I'm happy to read that other users are coming to similar
>>> conclusions. Instead of an option I'd create a new executable to lock
>>> down additional things (e.g. isolated mode, code verification hook). A
>>> separate executable would also allow distros to provide a stripped down
>>> interpreter that does not cause bad user experience.
>>
>> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
>> option and PYTHONNOUSERSITE env variable to provide that one-way
>> isolation).
>>
>> Is the problem that pip doesn't use it by default? Or that the distros
>> went and made patches for the runtime rather than patching pip? (For
>> Windows installs from the Store, where even admin rights can't do an
>> all-users package install, I added a new config file location for pip to
>> change this default, but a patch would also have worked.)
>>
>> Maybe we need an easier way to patch the location of user site packages?
>> I also had to do this for the Store install on Windows, and it's a
>> little bit of a hack... but maybe having an official recommendation
>> would help encourage distributors to use the mechanism?
> 
> (sorry for terse reply, I'm heading out.)
> 
> Linux distros want an additional layer so the can differentiate between
> Python packages installed by their package manager and Python packages
> installed with "sudo pip install".
> 
> Python has two default site-package directories (from highest to lowest
> precedence):
> 
> 1. ~/.local/lib/python3.9/site-packages
> 2. /usr/lib/python3.9/site-packages
> 
> (1) is defined by site.getusersitepackages(), (2) site.getsitepackages()
> 
> Linux distro have additional site-package directories. My Fedora system
> X86_64 system with multiarch has three additional candidates.
> 
> 1. ~/.local/lib/python3.9/site-packages
> 2. /usr/local/lib64/python3.9/site-packages
> 3. /usr/local/lib/python3.9/site-packages
> 4. /usr/lib64/python3.9/site-packages
> 5. /usr/lib/python3.9/site-packages
> 
> The "lib" directories are for pure Python packages whiel the "lib64"
> directories contain X86_64 native code extensions (aka shared
> libraries). The "/usr/lib*" directories are used for distro packages.
> The downstream patch [1] ensures that "sudo pip install" installs
> packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.
> 
> Debian has a similar mechanism to provide
> "/usr/lib/python3/dist-packages" [2].
> 
> A hypothetical /usr/bin/system-python3 interpreter would only use (4)
> and (5) as site-package directories. A user-facing /usr/bin/python3
> would use (1) to (5).
> 
> Christian
> 
> [1]
> https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-user-install-location.patch
> [2]
> https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

PS: Fedora patch [1] uses -s / PYTHONNOUSERSITE to exclude
"/usr/local/lib*" site-package directories.

$ python3 -c 'import site; print(site.getsitepackages())'
['/usr/local/lib64/python3.9/site-packages',
'/usr/local/lib/python3.9/site-packages',
'/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

$ python3 -s -c 'import site; print(site.getsitepackages())'
['/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']


___
Python-Dev mailing 

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 19.17, Steve Dower wrote:
> On 2/24/2021 4:26 PM, Christian Heimes wrote:
>> On 24/02/2021 15.16, Random832 wrote:
>>> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
 Separate directories don't prevent clashes and system breakage. But
 they
 provide an easy way to *recover* from a broken system.
>>>
>>> I think it could be turned into a way to prevent them by A) having
>>> site-packages always take precedence over dist-packages [i believe
>>> this is already the case] in normal usage and B) providing an option
>>> to the interpreter, used by system scripts, to exclude site-packages
>>> entirely from the path.
>>>
>>> Basically, site-packages would effectively be layered on top of "Lib
>>> + dist-packages" in a similar way to how a venv is layered on top of
>>> the main python installation - the inverse of the suggestion someone
>>> else in the thread made for the system python to be a venv. This
>>> wouldn't *exactly* be a venv because it wouldn't imply the other
>>> things that entering a venv does such as "python" [and script names
>>> such as pip] being an alias for the correct version of python, but it
>>> would provide the same kind of one-way isolation, whereby the "system
>>> environment" can influence the "normal environment" and not
>>> vice-versa, in the same way that packages installed in the main
>>> installation affect a venv [unless system-site-packages is disabled]
>>> but the venv obviously has no effect on the main installati
> on.
>>
>> Yes, you are describing one major aspect of my idea for a system Python
>> interpreter. I'm happy to read that other users are coming to similar
>> conclusions. Instead of an option I'd create a new executable to lock
>> down additional things (e.g. isolated mode, code verification hook). A
>> separate executable would also allow distros to provide a stripped down
>> interpreter that does not cause bad user experience.
> 
> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
> option and PYTHONNOUSERSITE env variable to provide that one-way
> isolation).
> 
> Is the problem that pip doesn't use it by default? Or that the distros
> went and made patches for the runtime rather than patching pip? (For
> Windows installs from the Store, where even admin rights can't do an
> all-users package install, I added a new config file location for pip to
> change this default, but a patch would also have worked.)
> 
> Maybe we need an easier way to patch the location of user site packages?
> I also had to do this for the Store install on Windows, and it's a
> little bit of a hack... but maybe having an official recommendation
> would help encourage distributors to use the mechanism?

(sorry for terse reply, I'm heading out.)

Linux distros want an additional layer so the can differentiate between
Python packages installed by their package manager and Python packages
installed with "sudo pip install".

Python has two default site-package directories (from highest to lowest
precedence):

1. ~/.local/lib/python3.9/site-packages
2. /usr/lib/python3.9/site-packages

(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()

Linux distro have additional site-package directories. My Fedora system
X86_64 system with multiarch has three additional candidates.

1. ~/.local/lib/python3.9/site-packages
2. /usr/local/lib64/python3.9/site-packages
3. /usr/local/lib/python3.9/site-packages
4. /usr/lib64/python3.9/site-packages
5. /usr/lib/python3.9/site-packages

The "lib" directories are for pure Python packages whiel the "lib64"
directories contain X86_64 native code extensions (aka shared
libraries). The "/usr/lib*" directories are used for distro packages.
The downstream patch [1] ensures that "sudo pip install" installs
packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.

Debian has a similar mechanism to provide
"/usr/lib/python3/dist-packages" [2].

A hypothetical /usr/bin/system-python3 interpreter would only use (4)
and (5) as site-package directories. A user-facing /usr/bin/python3
would use (1) to (5).

Christian

[1]
https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-user-install-location.patch
[2]
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RZACFNIDET7FZJG6LC25EAPWCR7ADFUE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Wed, 24 Feb 2021 at 18:25, Steve Dower  wrote:
> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
> option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
>
> Is the problem that pip doesn't use it by default?

Note that these days, pip *does* use usersite by default if
site-packages isn't writable. Of course, if users run pip under sudo,
that invalidates that mechanism :-(

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UE7W7IFPNAPTU64TNTHXZ5GUCLOSMUJV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Steve Dower]

On 2/24/2021 4:26 PM, Christian Heimes wrote:

On 24/02/2021 15.16, Random832 wrote:

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:

Separate directories don't prevent clashes and system breakage. But they
provide an easy way to *recover* from a broken system.


I think it could be turned into a way to prevent them by A) having 
site-packages always take precedence over dist-packages [i believe this is 
already the case] in normal usage and B) providing an option to the 
interpreter, used by system scripts, to exclude site-packages entirely from the 
path.

Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how 
a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread 
made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things 
that entering a venv does such as "python" [and script names such as pip] being an alias for the correct 
version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" 
can influence the "normal environment" and not vice-versa, in the same way that packages installed in the 
main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the 
main installation.


Yes, you are describing one major aspect of my idea for a system Python
interpreter. I'm happy to read that other users are coming to similar
conclusions. Instead of an option I'd create a new executable to lock
down additional things (e.g. isolated mode, code verification hook). A
separate executable would also allow distros to provide a stripped down
interpreter that does not cause bad user experience.


I mean, this is _precisely_ what PEP 370 defines (including the "-s" 
option and PYTHONNOUSERSITE env variable to provide that one-way isolation).


Is the problem that pip doesn't use it by default? Or that the distros 
went and made patches for the runtime rather than patching pip? (For 
Windows installs from the Store, where even admin rights can't do an 
all-users package install, I added a new config file location for pip to 
change this default, but a patch would also have worked.)


Maybe we need an easier way to patch the location of user site packages? 
I also had to do this for the Store install on Windows, and it's a 
little bit of a hack... but maybe having an official recommendation 
would help encourage distributors to use the mechanism?


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IHEECOXYDPJ6ZQJE2QTGPDOOCOP7J37A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python Language Summit 2021 Signups Are Now Open

[Łukasz Langa]

I’m happy to announce that we’ve opened the sign-up forms for the 2021 Python 
Language Summit!

TL;DR

When: Tuesday, May 11, 2021 (4 hours) and Wednesday, May 12, 2021 (4 hours). 
Exact times TBD depending on attendee timezones.
Where: Online via Zoom (link will be sent via email to attendees)
Co-chairs: Mariatta Wijaya & Łukasz Langa
Blogger: Joanna Jablonski
Sign up to attend and actively participate: https://forms.gle/cgmGnmQMDhD2mhHY8 
 (closes after March 22nd, 2021 AoE)
Propose a topic: https://forms.gle/Jui9mxsHrB4fVvAB8 
 (closes after March 22nd, 2021 AoE)
To get an idea of past Python Language Summits, you can read these blog posts:

2020: Python Software Foundation News: The 2020 Python Language Summit 

2019: http://pyfound.blogspot.com/2019/05/the-2019-python-language-summit.html 

2018: The 2018 Python Language Summit [LWN.net] 

2017: The 2017 Python Language Summit [LWN.net] 

Do I need to sign up if I’m a Python core developer?

Yes please! While in the past we have limited attendance to 50 people, this 
time, due to virtual format, we will be a bit more flexible, but will still 
keep it small and manageable. We aren’t planning to go beyond 80 participants. 
Please register to reserve your space.

Can I sign up if I’m not a Python core developer?

Yes you can. In the past, we had quite a number of participants who were not 
Python core devs. Among them were maintainers and representatives from BeeWare, 
CircuitPython, PSF board member, PyCharm, PyPA, etc. Register if you want to 
participate. Note that until you hear back from us, your attendance is not 
confirmed. As explained in the question above, our “space” is more flexible 
than usual, but in the interest of maintaining a vigorous discussion space, we 
might still be unable to invite everyone who signs up.

What kind of topics are covered?

Python Language Summit is a special event with very specific audience: Python 
core developers. Ideally your topic is not an “announcement” or “project 
status” but rather something that will encourage further discussion and 
questions. The more controversial, the better. An open issue, group of issues, 
or a PEP that is awaiting decision are all good topics to propose. You can also 
further explain why this is better discussed in person instead of online.

According to last year’s feedback, our audience prefer more discussions and 
shorter talks.

Who can present a talk?

Anyone, even if you’re not a Python core developer. However, please understand 
that we will have to be selective as space and time are limited. In particular, 
we are prioritizing active core contributors, as well as those who we believe 
will be able to improve the quality of the discussions at the event and bring a 
more diverse perspective to core Python developers. Note that your topic is not 
confirmed until you hear back from us.

Code of Conduct

PyCon’s Code of Conduct  
applies and will be enforced.

Thanks!

@mariatta  & @ambv 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J4AC33NZ3SUWYF2GOQRBHXLG7ZECIDBW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[John Paul Adrian Glaubitz]

Hello!

On 2/22/21 12:30 PM, Victor Stinner wrote:
>> The thing is you made assumptions about how downstream distributions use 
>> Python without doing some research first ("16-bit m68k-linux").
> 
> I'm talking about 16-bit memory alignment which causes SIGBUS if it's
> not respected on m68k. For example, unicodeobject.c requires special
> code just for this arch:
> 
> /*
>  * Issue #17237: m68k is a bit different from most architectures in
>  * that objects do not use "natural alignment" - for example, int and
>  * long are only aligned at 2-byte boundaries.  Therefore the assert()
>  * won't work; also, tests have shown that skipping the "optimised
>  * version" will even speed up m68k.
>  */
> #if !defined(__m68k__)
> (...)
> 
> Such issue is hard to guess when you write code and usually only spot
> it while actually running the code on such architecture.

Just as a heads-up: There is a PR by Jessica Clarke now [1] which gets rid of 
this
architecture-specific #ifdef. I think this is a good approach as it gets rid of 
one
of your complaining points.

I have already verified that these changes don't break on 32-bit PowerPC, 
64-bit SPARC
and, of course, M68k.

Thanks,
Adrian

> [1] https://github.com/python/cpython/pull/24624

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TQVOGEZ3NXQP6PN5RTOG5IDBCGTIPAU5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Wed, Feb 24, 2021 at 4:39 AM Guido van Rossum  wrote:

>
> OTOH we might reconsider deriving ExceptionGroup from BaseException --
> maybe it's better to inherit from Exception? I don't think the PEP
> currently has a clear reason why it must derive from BaseException. I
> believe it has to do with the possibility that ExceptionGroup might wrap a
> BaseException instance (e.g. KeyboardInterrupt or SystemExit).
>


That was the reason, and we also said that an ExceptionGroup escaping
something that isn't supposed to be raising ExceptionGroups is a bug, so if
you call an API that raises ExceptionGroups it is your responsibility to
handle them.

We could make ExceptionGroup be an Exception, and refuse to wrap anything
which is not an Exception. So asyncio either raises KeyboardInterrupt or an
ExceptionGroup of user exceptions.

Those are quite different directions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EO6Z3DRWFXCOMEYENLDVZSIN27T24BAI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 12:42 PM Paul Moore  wrote:

> On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier  wrote:
> > There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations.
>
> Ultimately the problem is that a general tool can't deal with
> conflicts (except by raising an error). If application A depends on
> lib==1.0 and application B depends on lib==2.0, you simply can't have
> a (consistent) environment that supports both A and B. But that's the
> rare case - 99% of the time, there are no conflicts. One env per app
> is a safe, but heavy handed, approach. Managing environments manually
> isn't exactly *hard*, but it's annoying manual work that pipx does an
> excellent job of automating, so it's a disk space vs admin time
> trade-off.
>

There are three ways to approach the question:

1) Fully isolated envs. The safest option but uses the most space.

2) Try to minimise the number of dependencies installed by interpreting the
requirements specification in the looser way possible. This is both
algorithmically hard (see
https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the
more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and
risky, as you've noted.

3) But the best way IMHO is to compute dependencies for each virtualenv
independently from the others, but still share the packages, using some
indirection mechanisms (hard links, symlinks or some Python-specific
constructs) when the versions match exactly.

The 3rd solution is probably the best of the 3, but the sharing mechanism
still needs to be specified (and, if needed, implemented) properly.

I've tried Christian's suggestions of using rdfind on my pipx installation,
and it claims to reduce the footprint by 30% (nice, but less than I
expected. This would however scale better with the number of installed
packages).

I'm not sure this would be practical in reality, OTOH, because I think
there is a serious risk of breakage each time I would upgrade one of the
packages (via 'pipx upgrade-all' for instance).

So IMHO the best way to implement solution 3 would be by using some variant
of the approach popularized by Nix (repository of immutable packages +
links to each virtualenv).

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6GV3EPYU2SW2QKAKLXZCG3JLNCVSQZWG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 11:43 AM Henk-Jaap Wagenaar <
wagenaarhenkj...@gmail.com> wrote:

>
>>
> I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been
> quite happy with that, and it basically does what Jonathan does manually:
> clone the github repo and build python from scratch.
>

Also using pyenv myself, on platforms where either, or both, new and old
version of Python are not available (so on Ubuntu I'm sticking with Anthony
Sottile's deadsnakes.)

But I'm not using it to manage virtual envs. For my own development
projects, I'm using VirtualFish since I've switched to Fish as my main
shell, but I was happy with VirtualenvWrapper when I was still using bash
or zsh. I can't live without VirtualFish's autoactivation plugin (which
activate a given virtualenv each time you enter your project). There was
something similar for Zsh though I don't remember which. Sometimes I forget
that I'm not in the right folder, or that I didn't create the virtualenv,
or that I removed it in the case of an old project, and I end up polluting
my default Python install after a 'pip install ...'. No big deal but always
a bit unpleasant.

And as I wrote before, I'm now using pipx to install the apps that I use.

This works not so bad (except for some quirks now and then) but it took
quite some time to select the right (for me) tools, to set up the proper
config files on my computer, and it can be annoying when working on a
server which has not been set up the same way. It also takes some time to
explain to collaborators, especially interns or juniors.

So obviously there is room for improvement, especially in terms of
simplicity for beginners.

  S.

___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OZRHP3Q6EZNCPCTEEXLZGOCTGY6B7YFW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X7GSLJIICBL5TJVUG32XGBGVLPLIHKIW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Wed, Feb 24, 2021 at 4:55 AM Guido van Rossum  wrote:

However there may be an omission in the PEP -- what if we want to do
> something special for each suberror? If we just iterate over `eg.errors` we
> would have to do something recursive for exceptions wrapped inside multiple
> nested groups. We could add a helper method to ExceptionGroup that iterates
> over suberrors, flattening nested groups. But we'd need to do something
> special to recover the tracebacks there (groups share the common part of
> the traceback). Or we could say "if you need the traceback you're going to
> have to write something recursive" -- like we have to do in traceback.py.
> But we might as well generalize whatever we're doing there. (Irit: I
> suspect there's a piece of API design we missed here.)
>

This is mentioned briefly where we talk about why ExceptionGroup is not
iterable: https://www.python.org/dev/peps/pep-0654/#the-exceptiongroup-api
We left it out pending a use case.

Rather than iterator, I think we should add a visitor that calls a function
for each leaf exception, plus a utility that returns the traceback of a leaf
exception (as a single list, copying frames from the tracebacks of
ExceptionGroups so that it's not destructive.) This way the expensive
traceback
construction is happening explicitly when it is needed.

I think accessing one exception at a time to do something other than
formatting it is more likely to be overused than actually needed.
We did that in an earlier version of the PEP for the "filter OSErrors by
type" example, which we did with iteration and now do with subgroup:

try:
low_level_os_operation()
except *OSerror as errors:
raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5VX7AG266VCSZUV4QYXSWD77W7LOC75X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

I love pipx and I'm glad it exists at this point because it make

The main issue is that each virtualenv takes space, lots of space.

I have currently 57 apps installed via pipx on my laptop, and the 57
environments take almost 1 GB already.

 ~  cd .local/pipx/venvs/
 ~/.l/p/venvs  ls
abilian-tools/  concentration/  gitlabber/  pygount/sphinx/
ansible/cookiecutter/   httpie/ pyinfra/tentakel/
assertize/  cruft/  isort/  pylint/ tlv/
autoflake/  cython/ jupyterlab/ pyre-check/ towncrier/
black/  dephell/lektor/ pytype/ tox/
borgbackup/ docformatter/   md2pdf/ pyupgrade/  twine/
borgmatic/  flake8/ medikit/radon/  virtualenv/
bpytop/ flit/   mypy/   re-ver/ virtualfish/
check-manifest/ flynt/  nox/sailboat/   vulture/
clone-github/   gh-clone/   pdoc3/  salvo/
cloneall/   ghtop/  pdocs/  shed/
com2ann/gitchangelog/   pybetter/   sixer/
 ~/.l/p/venvs  du -sh .
990M .
 ~/.l/p/venvs  ls | wc
  57  57 475

There is probably a clever way to reuse common packages (probably via
clever symlinking) and reduce the footprint of these installations.

Still, I'm glad that pipx exists as it is now, and that it has been
packaged on Ubuntu 20.04 and later (and probably other distros as well).

Having pipx (or something similar) installed by the distro, and the distro
focussed on packaging only the packages that are needed for its own sake,
means that we could go past the controversies between the Python community
and the Debian (or other distros) packagers community, which are based on
different goals and assumption, such as this one:
https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e

  S.

On Wed, Feb 24, 2021 at 2:28 AM Paul Bryan  wrote:

> I think it's a classic case of dependency hell.
>
> OS packagers are rebundling Python packages as OS packages and expressing
> their own OS-package dependency graphs. Then, you sudo pip install
> something that has a conflicting dependency, it bypasses OS packaging, and
> *boom*.
>
> I find tools like pipx  go a long
> way to solve this, as they install a Python package and all of its
> dependencies in its own venv. This is great for Python apps, and (kinda)
> treats them like apps on platforms like Android, where all app dependencies
> are bundled and isolated from others.
>
> I think it would great if OS vendors did something similar to pipx for
> Python-based apps: bundle the app and all of its dependencies into its own
> venv.
>
>
> On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
>
> I was reading a discussion thread <
> https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about
> various issues with the Debian packaged version of Python, and the
> following statement stood out for me as shocking:
>
> Christian Heimes wrote:
>
> Core dev and PyPA has spent a lot of effort in promoting venv because we
> don't want users to break their operating system with sudo pip install.
>
>
> I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> I think that installing packages into the actual python installation, both
> via distribution packaging tools and pip [and using both simultaneously -
> the Debian model of separated dist-packages and site-packages folders seems
> like a reasonable solution to this problem] can and should be a supported
> paradigm, and that virtual environments [or more extreme measures such as
> shipping an entire python installation as part of an application's
> deployment] should ideally be reserved for the rare corner cases where that
> doesn't work for some reason.
>
> How is it that virtual environments have become so indispensable, that
> no-one considers installing libraries centrally to be a viable model
> anymore? Are library maintainers making breaking changes too frequently,
> reasoning that if someone needs the old version they can just venv it? Is
> there some other cause?
> 

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 1:47 PM Stéfane Fermigier  wrote:

>
> So IMHO the best way to implement solution 3 would be by using some
> variant of the approach popularized by Nix (repository of immutable
> packages + links to each virtualenv).
>

Another benefit of this kind of approach, besides sparing disk space, could
be similar improvements in terms of installation time (and even bigger
improvements when reinstalling a package, which happens all the time when
developing).

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UO2AZHKZUNEQ34SGNKCR5RTD4GYI2SHF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 15.16, Random832 wrote:
> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>> Separate directories don't prevent clashes and system breakage. But they
>> provide an easy way to *recover* from a broken system.
> 
> I think it could be turned into a way to prevent them by A) having 
> site-packages always take precedence over dist-packages [i believe this is 
> already the case] in normal usage and B) providing an option to the 
> interpreter, used by system scripts, to exclude site-packages entirely from 
> the path.
> 
> Basically, site-packages would effectively be layered on top of "Lib + 
> dist-packages" in a similar way to how a venv is layered on top of the main 
> python installation - the inverse of the suggestion someone else in the 
> thread made for the system python to be a venv. This wouldn't *exactly* be a 
> venv because it wouldn't imply the other things that entering a venv does 
> such as "python" [and script names such as pip] being an alias for the 
> correct version of python, but it would provide the same kind of one-way 
> isolation, whereby the "system environment" can influence the "normal 
> environment" and not vice-versa, in the same way that packages installed in 
> the main installation affect a venv [unless system-site-packages is disabled] 
> but the venv obviously has no effect on the main installation.

Yes, you are describing one major aspect of my idea for a system Python
interpreter. I'm happy to read that other users are coming to similar
conclusions. Instead of an option I'd create a new executable to lock
down additional things (e.g. isolated mode, code verification hook). A
separate executable would also allow distros to provide a stripped down
interpreter that does not cause bad user experience.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F5NCGF7EWYDMBAHPNDFNWWKSE2GRKHLA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

Please include an example for:

except *ExceptionGroup: pass

I *think* it swallows all exceptions, instead of re-raising.
I *think* it also catches (and then swallows) bare exception that were not 
initially part of an ExceptionGroup.
I *think* it catches anything that gets raised, not just subclasses of 
Exception (or even BaseException).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NTQ22LUIKNRNA2YHKOLVPR5GI2KW6TLW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Random832]

On Wed, Feb 24, 2021, at 09:08, Paul Moore wrote:
> I don't use Linux much, and I'm definitely not familiar with Linux
> distribution tools, but from what I can gather Linux distributions
> have made the choices:
> 
> 1. Write key operating system utilities in Python.
> 2. Share the Python interpreter and libraries.
> 3. Expose that Python interpreter as the *user's* default Python.

I think 1 *partially* mischaracterizes the problem, because any "system python" 
would logically be used by *every application written in python [or that embeds 
python] distributed by the OS's package management*, not just by "key operating 
system utilities". To suggest otherwise implies that they should not distribute 
any python applications at all.

That also makes asking all of their package maintainers to change their #! line 
to point at a different interpreter [or to pass an option, as I had suggested 
in another post] a more significant ask than the "just change a few core 
utilities" that some people seem to be assuming it would be. It also means that 
making a "system python" does not remove the need to distribute the largish 
subset of python *libraries* that they distribute, because even when these 
libraries aren't used by key OS utilities, they are still used by packaged 
applications.

[this, in turn, means we don't want the user's default python environment to 
stand entirely separate from the system python, or we'll start getting 
complaints along the lines of "I apt-get installed numpy, why can't I import it 
in my python interpreter?" - particularly from users who don't really care if 
it runs a couple versions behind]
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F7YNRI7LWU2VLS6AU6PQZUHHA3M5WFQJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

The "no subclasses" seems pretty severe, particularly if you can't even use 
marker attributes because it isn't clear when a different instance of container 
ExceptionGroup will be substituted.

The justification for this restriction was that .split() had to be be able to 
instantiate ... wouldn't it be enough to just say that subclasses need a 
compatible __init__ and __new__ for that reason, and then leave it to 
consenting adults?

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5OLFDKJCNKLIJJQMNYYLQGV53Z7ZTRC5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Random832]

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> Separate directories don't prevent clashes and system breakage. But they
> provide an easy way to *recover* from a broken system.

I think it could be turned into a way to prevent them by A) having 
site-packages always take precedence over dist-packages [i believe this is 
already the case] in normal usage and B) providing an option to the 
interpreter, used by system scripts, to exclude site-packages entirely from the 
path.

Basically, site-packages would effectively be layered on top of "Lib + 
dist-packages" in a similar way to how a venv is layered on top of the main 
python installation - the inverse of the suggestion someone else in the thread 
made for the system python to be a venv. This wouldn't *exactly* be a venv 
because it wouldn't imply the other things that entering a venv does such as 
"python" [and script names such as pip] being an alias for the correct version 
of python, but it would provide the same kind of one-way isolation, whereby the 
"system environment" can influence the "normal environment" and not vice-versa, 
in the same way that packages installed in the main installation affect a venv 
[unless system-site-packages is disabled] but the venv obviously has no effect 
on the main installation.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K3DB2MSBWPDEK3O4UZ5VEQVKW7BFZDRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Jim J. Jewett]

Petr:  What about except *(TypeError, ExceptionGroup):?

Irit:  Good question. We should block that as well.

But https://www.python.org/dev/peps/pep-0654/#backwards-compatibility
seems to explicitly recommend the very similar:

except (Exception, ExceptionGroup):
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N22HZBABDVM2GCEIWHOCTPGJDRSRVGS5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Wed, 24 Feb 2021 at 13:12, Antoine Pitrou  wrote:
>
> On Wed, 24 Feb 2021 13:47:40 +0100
> Stéfane Fermigier  wrote:
> > The 3rd solution is probably the best of the 3, but the sharing mechanism
> > still needs to be specified (and, if needed, implemented) properly.
>
> I wouldn't want to repeat myself too often, but conda and conda-based
> distributions already have sharing through hardlinks (or, on Windows,
> whatever is available) baked-in, assuming you install your software
> from conda packages.
>
> That also applies to non-Python packages, and to python itself (which
> is just a package like any other).

I'm not sure conda solves the problem of *application* distribution,
though, so I think it's addressing a different problem. Specifically,
I don't think conda addresses the use case pipx is designed for.

Although to be fair, this conversation has drifted *way* off the
original topic. Going back to that, my view is that Python does not
have a good solution to the "write your application in Python, and
then distribute it" scenario. Shipping just the app to be run on an
independently installed runtime results in the conflicting
dependencies issue. Shipping the app with bundled dependencies is
clumsy, mostly because no-one has developed tools to make it easier.
It also misses opportunities for sharing libraries (reduced
maintenance, less disk usage...). Shipping the app with a bundled
interpreter and libraries is safest, but hard to do and even more
expensive than the "bundled libraries" approach.

I'd love to see better tools for this, but the community preferred
approach seems to be "ship your app as a PyPI package with a console
entry point" and that's the approach pipx supports.

I don't use Linux much, and I'm definitely not familiar with Linux
distribution tools, but from what I can gather Linux distributions
have made the choices:

1. Write key operating system utilities in Python.
2. Share the Python interpreter and libraries.
3. Expose that Python interpreter as the *user's* default Python.

IMO, the mistake is (3) - because the user wants to install Python
packages, and not all packages are bundled by the distribution (or if
they are, they aren't updated quickly enough for the user), users want
to be able to install packages using Python tools. That risks
introducing unexpected library versions and/or conflicts, which breaks
the OS utilities, which expect their requirements to be respected
(that's what the OS packaging tools do).

Hindsight is way too easy here, but if distros had a "system Python"
package that OS tools depend on, and which is reserved for *only* OS
tools, and a "user Python" package that users could write their code
against, we'd probably have had far fewer issues (and much less FUD
about the "using sudo pip breaks your OS" advice). But it's likely way
too late to argue for such a sweeping change.

*Shrug* I'm not the person to ask here. My view is that I avoid using
Python on Linux, because it's *way* too hard. I find it so much easier
to work on Windows, where I can install Python easily for myself, and
I don't have to fight with system package managers, or
distribution-patched tools that don't work the way I expect. And
honestly, on Windows, there's no "neglect of the system environment"
to worry about - if you want to install Python, and use pip to install
packages into that environment for shared use, it works fine. People
(including me) use virtual environments for *convenience* on Windows,
not because it's a requirement.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J7U6525AZAW4P5ZYH5WLK5IDF6TCH73O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier  wrote:
> There is probably a clever way to reuse common packages (probably via clever 
> symlinking) and reduce the footprint of these installations.

Ultimately the problem is that a general tool can't deal with
conflicts (except by raising an error). If application A depends on
lib==1.0 and application B depends on lib==2.0, you simply can't have
a (consistent) environment that supports both A and B. But that's the
rare case - 99% of the time, there are no conflicts. One env per app
is a safe, but heavy handed, approach. Managing environments manually
isn't exactly *hard*, but it's annoying manual work that pipx does an
excellent job of automating, so it's a disk space vs admin time
trade-off.

As far as I know, no-one has tried to work on the more complex option
of sharing things (pipx shares the copies of pip, setuptools and wheel
that are needed to support pipx itself, but doesn't extend that to
application dependencies). It would be a reasonable request for pipx
to look at, or for a new tool, but I suspect the cost of implementing
it simply outweighs the benefit ("disk space is cheap" :-))

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4EAGDI66MG4WPNCHCOVHT4VHBWQUNRNP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 11.52, Stéfane Fermigier wrote:
> I love pipx and I'm glad it exists at this point because it make 
> 
> The main issue is that each virtualenv takes space, lots of space.
> 
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.
> 
>  ~  cd .local/pipx/venvs/
>  ~/.l/p/venvs  ls
> abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/
> ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/
> assertize/      cruft/          isort/          pylint/         tlv/
> autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/
> black/          dephell/        lektor/         pytype/         tox/
> borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/
> borgmatic/      flake8/         medikit/        radon/          virtualenv/
> bpytop/         flit/           mypy/           re-ver/         virtualfish/
> check-manifest/ flynt/          nox/            sailboat/       vulture/
> clone-github/   gh-clone/       pdoc3/          salvo/
> cloneall/       ghtop/          pdocs/          shed/
> com2ann/        gitchangelog/   pybetter/       sixer/
>  ~/.l/p/venvs  du -sh .
> 990M.
>  ~/.l/p/venvs  ls | wc
>       57      57     475
> 
> There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations. 

There are tools like https://rdfind.pauldreik.se/rdfind.1.html that
create hard links to deduplicate files. Some files systems have
deduplicated baked in, too.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CO6GV2CRDKBJMLE7DZVVQ4AMIPSKPMCJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Christian Heimes]

On 24/02/2021 08.03, Michał Górny wrote:
> On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
>> I was reading a discussion thread 
>>  about 
>> various issues with the Debian packaged version of Python, and the following 
>> statement stood out for me as shocking:
>>
>> Christian Heimes wrote:
>>> Core dev and PyPA has spent a lot of effort in promoting venv because we 
>>> don't want users to break their operating system with sudo pip install.
>>
>> I don't think sudo pip install should break the operating system. And I 
>> think if it does, that problem should be solved rather than merely advising 
>> users against using it. And why is it, anyway, that distributions whose 
>> package managers can't coexist with pip-installed packages don't ever seem 
>> to get the same amount of flak for "damaging python's brand" as Debian is 
>> getting from some of the people in the discussion thread? Why is it that 
>> this community is resigned to recommending a workaround when distributions 
>> decide the site-packages directory belongs to their package manager rather 
>> than pip, instead of bringing the same amount of fiery condemnation of that 
>> practice as we apparently have for *checks notes* splitting parts of the 
>> stdlib into optional packages? Why demand that pip be present if we're not 
>> going to demand that it works properly?
>>
>> I think that installing packages into the actual python installation, both 
>> via distribution packaging tools and pip [and using both simultaneously - 
>> the Debian model of separated dist-packages and site-packages folders seems 
>> like a reasonable solution to this problem] can and should be a supported 
>> paradigm, and that virtual environments [or more extreme measures such as 
>> shipping an entire python installation as part of an application's 
>> deployment] should ideally be reserved for the rare corner cases where that 
>> doesn't work for some reason.
> 
> The problem is a little deeper and the Debian solution doesn't really
> solve all of it.
> 
> Yes, pip installing into the same directory as the package manager
> is a problem.  It's a problem to the point that I'm patching pip
> in Gentoo to explicitly block that.  We've gotten far too many bug
> reports about people's systems suddenly being horribly broken after they
> used pip.
> 
> While using two directories can prevent pip from directly overwriting
> system packages, you still can't expect two independent package managers
> to work simultaneously unless they can communicate with each other to
> prevent conflicts.  If pip installs a different version of the same
> package as the package manager, which one is supposed to be used? 
> Whichever choice you make, you'll bound to eventually break dependency
> graph of some package.

Thanks Michał, you have explained the situation well.

Most Linux distros have two separate directories for distro packages and
"sudo pip install" packages. This is a great thing!

Separate directories don't prevent clashes and system breakage. But they
provide an easy way to *recover* from a broken system. The package
manager of several Linux distributions is written in Python. If your
system Python is broken, your package manager is likely busted as well.
The split directories allow ysers to fix their system with a single
command like "sudo rm -rf /usr/local/lib/python*" (paths may vary).

The issue is: Split distro/pip directories are not baked into Python or
pip. Distros have their own set of downstream patches that modify the
behavior of Python. Some of the patches have caused issues in the past.

I would like to see this feature implemented in Python's core, so distro
packages have to do less patching and upstream pip can test against it.

Christian

PS: I wrote PEP 370 for "install --user" 13 years ago before we had
virtual envs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZYXP5FARQIYQQTW46UFWVAXQRTFNFRFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Henk-Jaap Wagenaar]

On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou  wrote:

> On Tue, 23 Feb 2021 20:29:52 -0500
> Jonathan Goble  wrote:
> >
> > I can't speak for distributors or maintainers [1], but I can speak for
> > myself as a user. I run Debian testing (currently bullseye as that is
> > preparing for release) as my daily OS on my personal laptop, used for
> > personal matters and school assignments (I'm a university computer
> > science student in my senior year).
> >
> > I don't use the system Python for anything of my own, whether it's a
> > school assignment or a personal project, precisely because I don't
> > want to risk screwing something up. Rather, I maintain a clone/fork of
> > the official CPython GitHub repo, and periodically build from source
> > and `make altinstall` into `~/.local/`.
>
> For the record, instead of building Python by hand, you could use a
> distribution such as Anaconda or the community-maintained conda-forge.
>
> Regards
>
> Antoine.
>
>
I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been
quite happy with that, and it basically does what Jonathan does manually:
clone the github repo and build python from scratch.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OZRHP3Q6EZNCPCTEEXLZGOCTGY6B7YFW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Antoine Pitrou]

On Tue, 23 Feb 2021 20:29:52 -0500
Jonathan Goble  wrote:
> 
> I can't speak for distributors or maintainers [1], but I can speak for
> myself as a user. I run Debian testing (currently bullseye as that is
> preparing for release) as my daily OS on my personal laptop, used for
> personal matters and school assignments (I'm a university computer
> science student in my senior year).
> 
> I don't use the system Python for anything of my own, whether it's a
> school assignment or a personal project, precisely because I don't
> want to risk screwing something up. Rather, I maintain a clone/fork of
> the official CPython GitHub repo, and periodically build from source
> and `make altinstall` into `~/.local/`.

For the record, instead of building Python by hand, you could use a
distribution such as Anaconda or the community-maintained conda-forge.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3TXDWMNBTQLCSEEG5XKMPJHJJQ4BRWWF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [SPAM] Re: Move support of legacy platforms/architectures outside Python

[Antoine Pitrou]

On Tue, 23 Feb 2021 16:45:27 +
Rob Boehne  wrote:
> On 2/22/21, 4:06 PM, "Antoine Pitrou"  wrote:
> 
> On Mon, 22 Feb 2021 19:50:43 +
> Rob Boehne  wrote:
> > 
> > The other thing that crept into this thread was the mention of test 
> that intermittently fail.
> > That's a huge problem because it suggests that applications will 
> sometimes fail.
> > I have usually seen these sort of issues because of
> >   1) Uninitialized memory being used (read)
> >   2) Threading problems
> >   3) Resources used (files, networking, daemons) but unavailable
> >   4) memory mis-management (buffer overrun that doesn't cause a crash)
> > 
> > #3 is probably best fixed by testing for resources and skipping when 
> unavailable  
> 
> 5) Poor quality POSIX support in the target platform.  The Python test
> suite is actually quite demanding in this regard (except on Windows).
> 
> Regards
> 
> Antoine.
> 
>
> 
> Antoine:
> 
> I find this a bit of a surprise, can you give an example of poor POSIX 
> support leading to intermittent test failures? 

Hmm, I don't have any precise examples off the top of my head (that was
several years ago), but issues mixing pthreads, signals and/or fork
come to mind.  OpenBSD was quite bad at that especially.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4TOL7P47UBAYOXOFM2C3RTWGTJFWMHYU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Stestagg]

On Tue, Feb 23, 2021 at 11:00 PM Irit Katriel 
wrote:

>
> Hi Steve,
>
> Thank you for trying out the implementation. Please do let me know about
> bugs you find.
>
>
Thanks you for your response!  It seems I had missed the (now) obvious
naked exception wrapping logic.  Apologies for adding noise to the
conversation.

I've added a comment to your fork's PR with the error detail.

Best of luck with the PEP

Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6R6CKUL3QDROZFIFVK7WJRD75DMZQ43X/
Code of Conduct: http://python.org/psf/codeofconduct/