[issue43976] Allow Python distributors to add custom site install schemes

[Stefano Rivera]

Change by Stefano Rivera :


--
nosy: +stefanor

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Frederik Rietdijk]

Frederik Rietdijk  added the comment:

In Nixpkgs we install every Python package under a unique prefix, a so-called 
Nix store path. If we were to use sysconfig for installing packages, then we'd 
need to be able to dynamically set the paths. This was also discussed as part 
of the Installer project. https://github.com/pradyunsg/installer/issues/98

We could use a custom scheme, however, we do need to be able to dynamically set 
a certain variable, e.g. `base`.

```
variables = {"installed_base": "$out", "base": "$out", "platbase": "$out", 
"installed_platbase": "$out"}
# Note there is no `sysconfig.get_default_scheme()`
sysconfig._expand_vars("posix_prefix", variables)
```

I could imagine we do something like

```
# check whether we're in a nix build and want to install in a prefix
if "IN_NIX_BUILD" in os.environ:
base = os.environ["out"]

scheme = {...}
```
We'd then need to update the base variable in sysconfig or partially expand our 
own scheme using this variable.

--
nosy: +Frederik Rietdijk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I don't have a good answer, but given the title of this issue (which is 
specifically scoped to site install schemes), I'm tempted to say we should deal 
with prefixes in a separate, perhaps broader issue, and there address the 
reported issue (that a user's prefix override isn't honored by the scheme) and 
maybe more broadly the issue that there's not a design/spec for python 
installations (and probably there should be).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Miro Hrončok]

Miro Hrončok  added the comment:

In Fedora 36+ / Python 3.10+ we now use an install_scheme that looks like this:

'purelib': '{base}/local/lib/python{py_version_short}/site-packages',
'platlib': 
'{platbase}/local/{platlibdir}/python{py_version_short}/site-packages',
'scripts': '{base}/local/bin',
'data': '{base}/local',
...

We got a user report [1] saying that `pip install --root ... --prefix /usr` the 
prefix is not respected at all.

That is, users expect that /usr/local is the prefix, and when they explicitly 
set it to /usr, the /local/ bit will not be there, while in reality, /local/ is 
not a part of the prefix, but it is a part of the installation scheme.

I can somehow relate to that assumption.

Now I wonder whether we should have adapted prefix instead of the installation 
scheme :/


Any ideas on how to approach this problem? I am quite clueless.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=2026979

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Jason R. Coombs]

Jason R. Coombs  added the comment:

Here's what I propose:

1. In pypa/distutils, add support for honoring the proposed install schemes 
(based on PR 25718). Merge with Setuptools.
2. Add whatever ugly hacks are needed to pypa/distutils to honor other 
Debian-specific behaviors (revive https://github.com/pypa/distutils/pull/4 but 
properly match Debian expectations). Mark these ugly hacks as deprecated.
3. In Debian, Fedora, etc, provide patches that configure the install schemes. 
Test with latest Setuptools and SETUPTOOLS_USE_DISTUTILS=local.
4. Formalize the install schemes support in CPython (as in PR 25718).
5. Ask Debian to propose more a cleaner interface for Debian-specific needs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Jason R. Coombs]

Jason R. Coombs  added the comment:

> In the short term, and possible for the long term, Debian can continue to 
> patch the install routine...

The problem with this approach is Setuptools is attempting to adopt distutils 
by exposing its own vendored copy of distutils as `distutils` (top-level name). 
By doing this, it bypasses the Debian's patching of distutils as found in 
CPython. Because this bypass behavior breaks distutils for Debian users, the 
functionality has been disabled (opt-in).

Setuptools would like to be able to present a version of distutils that, 
unpatched, runs on all the major platforms, and thus make it default.

That won't be possible until Debian can stop relying on its patching of 
distutils.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

Matthias, can you check if bpo-44982 solves your issues related to the 
conflicts you were experiencing for the Debian patching?

If so, it would unblock this issue.

I am probably gonna discuss this proposal with the conda-forge folks next week, 
to get some feedback from the Conda perspective.
I would like to have this unblocked so that we can move forward, as far as I 
know the only issues are the ones Debian is experiencing about this proposal 
alone not being able to completely replace the downstream patching.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[jakirkham]

Change by jakirkham :


--
nosy: +jakirkham

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[xrcg]

xrcg  added the comment:

s/possible/possibly/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[xrcg]

xrcg  added the comment:

I haven’t read everything in this convo, but it looks like the changes proposed 
here cover all known downstream users other than Debian.

Is that correct?

Would these changes break Debian’s customizations substantially more than 
they’ll already be broken by the distutils deprecation refactoring?

If not, and if you don’t want to support the customizations that Debian wants, 
why not proceed with this enhancement to cover every other case, then deal with 
Debian later? In the short term, and possible for the long term, Debian can 
continue to patch the install routine, just in the new appropriate places to 
patch it. Debian wants something no one else wants, so they have to put in 
extra effort.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

Hi Jason, thank you.

I already tried 1), but could not convince Matthias on an alternative way to 
achieve the result that Debian wants. If you want, maybe you and I could try 
restarting those discussions.

2) is fine, but I am not the one that will me maintaining it, so it is a 
question for the core devs. I mean, I could commit to working on it and dealing 
with issues, but I am not a core dev, so it would always need to involve 
someone else.

3) does not really make sense as of the distutils deprecation. What needs to be 
patched are the install schemes, which are now outside distutils/setuptools.
I had hoped that we were able to have the mechanism proposed here in 3.10, so 
that people could just replace their distutils patches with it, but it was not 
possible.

I am not sure how to proceed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Jason R. Coombs]

Change by Jason R. Coombs :


--
versions: +Python 3.11 -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Jason R. Coombs]

Jason R. Coombs  added the comment:

Thanks Filipe for the summary. I was unsure the status.

Even if we grant that Debian is too intrusive in its patching of distutils, 
it's Python and distutils that are making the change here, so it's not 
unreasonable for Python to maintain feature parity with the current regime.

It's plausible that you or I could convince Matthias to adapt Debian to a less 
intrusive approach by adapting the whole Debian ecosystem to a new approach. I 
suspect achieving that change would be a bit of a challenge.

The only other alternative I see is for distutils to provide some mechanism to 
enable Debian to achieve its current expectations without patching.

It may not be necessary for CPython to support this mechanism. It may be the 
case that CPython can support the site install schemes, but that any additional 
customizations remain a contract between Setuptools/distutils. I'm unsure what 
implications that would have for other build systems not based on distutils, 
but presumably it would be up to those systems to support Debian.

I suggest one of three courses here:

1. Convince Debian to drop their customizations.
2. Add support to the CPython customizations to support Debian.
3. Draft support for Setuptools/distutils to support Debian customizations and 
get confirmation from Debian that satisfies their needs.

Are there any other options? What direction would you like to pursue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

Yes, but in the process of deprecating disutils, some internals were changed to 
stop relying on it. This will cause a lot of downstream patches to break.
I you are not modifying CPython, you don't need to worry, but if you are, then 
you should have a look and figure out how/where to patch it now.

As far as the public distutils API is concerned, everything will still the same 
until 3.12, just with deprecation warnings.
Development has moved to [1] and it will eventually be merged into setuptools.

[1] https://github.com/pypa/distutils

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

s/still/stay/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[xrcg]

xrcg  added the comment:

@FFY00: Thanks for the info.

>From what I quickly read, 3.10 will only deprecate distutils, and the removal 
>is scheduled for 3.12. Is that correct? If so, the impending release of 3.10 
>shouldn’t be a problem, as there are 2 more releases before distutils will be 
>removed.

Or does deprecating distutils cause problems other than warnings?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

I find it very difficult. I would be a bit anxious merging the code this close 
to the final release, so I imagine the release managers would be much more.

Anyway, this is the current state of this issue from my perspective:

As far as I understand, progress is currently blocked because Matthias thinks 
this mechanism must allow Debian to replace all related downstream patches.
I disagree with him in the way Debian is patching Python, which IMO is way too 
intrusive and fragile. Given that this vendor config will essentially shift the 
burden of supporting these modifications to the Python upstream, I don't think 
it would be a good idea to allow the modifications Debian wants in this config.
For all other downstreams I have talked to, the proposal seems to be fine, 
though more testing would definitely be appreciated.

I think that at this point, we could really use a core dev to help push this to 
the state of being able to get merged, because I don't think I can do that 
alone.
I have already requested feedback from the community and the technical 
discussions on the PR went silent, so I do not see what else I can do :/
If needed, I would be available to brief anyone on my understanding of the 
issues and how this proposal handles that. So, any core dev, please feel free 
to reach out if you want to help move this forward.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[xrcg]

xrcg  added the comment:

What is the probability that custom site install schemes will be supported 
without requiring a patch before distutils is removed?

--
nosy: +xrcg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Steve: I think the point of discussing whether "pip install" can
be used to manage system wide packages is moot. It's been like that
for ages, not only for pip, but also for the distutils setup.py install
process and the old Makefile.pre.in approach before that. People
have their reasons, it's what you'd expect to work as a Unix sysadmin
and won't go away anytime soon :-)

So back to the original point...

Filipe: Could you please explain why patching sysconfig.py is not a
long term solution ?

This doesn't involve any changes on the CPython side, is as flexible
as you can get (you can also patch functions defined in sysconfig.py
to do the necessary magic, not only provide a static dict),
doesn't create overhead for Python's startup, works with all the
different command line options for limiting sys.path additions and
avoids security issues with the Python import logic.

It's already clear that sysconfig.py will be the new golden source
for installation related APIs and schemes (perhaps this could be
made even clearer in the docs), so 3rd party packages will adapt
to this once 3.10 is out.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

>> The best option for restricting the import while still having it be a Python
>> import is to find the file (if it's present in the expected location under
>> sys.whatever), and then use importlib to import it:
>> https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
>
> Right, though that requires also a new import, importlib, which may not be
> optimal. Considering that this module is meant to be private and basically all
> other private importable parts of Python suffer from the same issue, I am
> finding it hard to justify. If there's enough consensus that this approach 
> would
> be better, I am more than happy to change the implementation.

Another alternative would be to convert sysconfig into a directory and make the 
vendor patch a submodule. That's _very slightly_ more impactful for the 
unpatched case, but only really for scenarios where people are trying to do 
things they shouldn't. Or we can include the file in all distros and import it 
earlier (before taking environment variables, etc. into account).

In my opinion, the security implications alone suggest we shouldn't be 
importing this by name without knowing where it is coming from.

>> I'd rather not have a new option here, I would much prefer "-S" in this
>> context to mean "run Python with only core libraries" and "-s" to mean "run
>> Python with only core and distro libraries" (and neither to mean "run Python
>> with core, distro and user libraries").
>
> I don't think having an option to start Python with only the vendor modules
> would be *necessary*, though it would certainly be helpful. Among other 
> things,
> it would be super helpful to be able to tell users to run Python with the -D
> (made up) option to isolate issues with the vendor modules and the user Python
> environment.

But the user can already exclude their user-installed packages with -s, right? 
It's the site-installed packages that would require -S, but that also excludes 
vendor modules.

Why do we encourage users to install site-wide packages using pip? Why is it 
such an important scenario for a distro-provided Python to be able to modify 
its global install using non-distro-provided tools and non-distro-provided 
packages? What's wrong with saying "install for --user", or else "apt install 
some-different-python-bundle" first and use that?

(To be clear, I'm framing these as confrontational questions to help my 
understanding. I'm totally willing to accept an answer of "just because", 
provided whoever is giving that answer actually "owns" dealing with the 
fallout.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

We cannot change how `sudo pip install` fundamentally works because there are 
too many people depending on it, and even if we could, this is not the place :P

I think we went a little off-topic here, so let's get back to the discussion.

> The best option for restricting the import while still having it be a Python 
> import is to find the file (if it's present in the expected location under 
> sys.whatever), and then use importlib to import it: 
> https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

Right, though that requires also a new import, importlib, which may not be 
optimal. Considering that this module is meant to be private and basically all 
other private importable parts of Python suffer from the same issue, I am 
finding it hard to justify. If there's enough consensus that this approach 
would be better, I am more than happy to change the implementation.

> I'd rather not have a new option here, I would much prefer "-S" in this 
> context to mean "run Python with only core libraries" and "-s" to mean "run 
> Python with only core and distro libraries" (and neither to mean "run Python 
> with core, distro and user libraries").

I don't think having an option to start Python with only the vendor modules 
would be *necessary*, though it would certainly be helpful. Among other things, 
it would be super helpful to be able to tell users to run Python with the -D 
(made up) option to isolate issues with the vendor modules and the user Python 
environment.

> That may be a bigger change, but there's enough angst around this issue that 
> we would be better off getting it right this time, even if it changes things, 
> than continuing to preserve the system that people dislike so much.

This may be completely wrong for other people, but is my understanding. AFAIK 
those these issues come from lack of separation between the distro, system and 
user environments, causing a hell of conflicts and silent module shadowing that 
neither the system package manager or pip can fix. Almost every time I help 
people with Python I have to tell them to use a virtual env, which most people 
aren't expecting, and would likely run into issues had I not suggested it.
Considering that, I think this approach, including the CLI option, would be a 
step forward. How big would that step be, I am not sure, but probably not 
*that* big.

But yeah, this is, of course, my experience, and that can vary for other 
people, so there may be different perspectives here. So I'd very much like to 
hear other people on this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 05.05.2021 10:29, Christian Heimes wrote:
> 
> I mean that Steve and you are talking about different things.

Could be. I was addressing the point Steve made about not allowing
or making it hard to run "pip install" as root user.

> Neither Steve nor you or I are are Linux distro packaging experts. I suggest 
> that we listen to the expertise of downstream packagers like Filipe or Miro. 
> They deal with packaging on a daily basis.

Agreed.

> By the way you are assuming that all container solutions work like Docker and 
> that all Docker and non-Docker based container solutions allow you to run 
> code as unrestricted, unconfined root. That's a) a incorrect, and b) offtopic 
> for this ticket.

I gave the Docker example as proof that running "pip install" as
root is a rather common scenario and needs to be supported.

Linux distros have been supporting this for many years and just
because distutils is deprecated should not mean that we no longer
provide ways to support this kind of setup.

BTW: I'm aware that other container solutions work in different ways,
e.g. Podman, LXC, etc. but I have yet to find a solution that doesn't
offer root permissions inside the containers (I'm not talking
about how the container is run in the host system).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Christian Heimes]

Christian Heimes  added the comment:

I mean that Steve and you are talking about different things.

Neither Steve nor you or I are are Linux distro packaging experts. I suggest 
that we listen to the expertise of downstream packagers like Filipe or Miro. 
They deal with packaging on a daily basis.

By the way you are assuming that all container solutions work like Docker and 
that all Docker and non-Docker based container solutions allow you to run code 
as unrestricted, unconfined root. That's a) a incorrect, and b) offtopic for 
this ticket.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 05.05.2021 10:01, Christian Heimes wrote:
> 
>> "as root" imply that there's no user site-packages directory at all
>  ^  
> 
> Steve is talking about user site-packages, not global site-packages directory.

You mean "pip install --user" as root ? That's not how you typically
install Python packages as root in a Dockerfile, no, but, of course,
even as root, there is the possibility to install into /root/.local/.

The typical Unix way of installing non-system packages is either
into /usr/local, /opt/local or similar variants, not into /usr.
Python itself also defaults to /usr/local when running
"make install". System provided packages normally live
under /usr (or even directly under / for low level tools).

As a root user, I'd assume that "pip install" also installs into
a /usr/local based site-packages dir -- and that's what happens
at least on Debian based OSes. But it can only happen because
the distros patch the install scheme, since this would normally
install into the /usr based site-packages dir for a python binary
living in /usr/bin.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Christian Heimes]

Christian Heimes  added the comment:

> "as root" imply that there's no user site-packages directory at all
 ^  

Steve is talking about user site-packages, not global site-packages directory.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 04.05.2021 22:58, Steve Dower wrote:
>> "pip install as root" will need to continue to work and thus distros
>> need to get a way to make sure that it doesn't corrupt the system
>> installed packages
> 
> Excuse my ignorance, but does "as root" imply that there's no user 
> site-packages directory at all?

Why should there be no site-packages dir ? All non-core packages get
installed into site-packages (or a similar dir which holds such packages)
by distutils / setuptools.

However, distros usually split this up further into packages which are
managed by the distro packager and ones which are managed by distutils /
setuptools and this is why the install schemes need to be patched.

> I'm not imagining a solution that doesn't require *users* to change their 
> commands, so if they're currently running "sudo pip install" because they 
> need to, but we change it so they shouldn't, then I'm okay with them having 
> to remove the "sudo". (At least for this discussion - we can evaluate 
> transition plans separately.)

I'm not sure I understand what you're suggesting.

For Docker, the instructions from the Dockerfile are run as root, so
there is no sudo involved. Whether you use sudo or not or how pip is
invoked is really not relevant for the discussion. The main point is
that the target of the installation is the system installation, not
a local user installation or a venv. That installation layout is what
sysconfig.py defines in the install schemes.

> And yeah, patching sysconfig.py seems easier. But then, adding a file to the 
> distro is even easier, and if it's easiest for Linux distros to do that via 
> configure than to add a copy step into their build (which is how I'll do it 
> for Windows distros that need it), then I'll leave that to others to 
> decide/implement.

You mean: put something like...

from _sysconfig_site import *
install_sysconfig_site()

at the end of sysconfig.py and then have distros add a
_sysconfig_site module ?

That would work as well, but details will have to be hashed out, since
this can be abused to hijack the system installation of Python (python
-S would have no effect on this). Patching sysconfig.py is definitely
safer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

> "pip install as root" will need to continue to work and thus distros
> need to get a way to make sure that it doesn't corrupt the system
> installed packages

Excuse my ignorance, but does "as root" imply that there's no user 
site-packages directory at all?

I'm not imagining a solution that doesn't require *users* to change their 
commands, so if they're currently running "sudo pip install" because they need 
to, but we change it so they shouldn't, then I'm okay with them having to 
remove the "sudo". (At least for this discussion - we can evaluate transition 
plans separately.)

And yeah, patching sysconfig.py seems easier. But then, adding a file to the 
distro is even easier, and if it's easiest for Linux distros to do that via 
configure than to add a copy step into their build (which is how I'll do it for 
Windows distros that need it), then I'll leave that to others to 
decide/implement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 04.05.2021 22:29, Steve Dower wrote:
> 
> Would "pip install --user ..." in a Docker container also work, though? 
> Presumably all the filesystem paths are being redirected anyway, so is there 
> a difference?
> 
> (My assumption is that "--user" would essentially become the default if 
> you're using the OS provided pip/Python. If you do your own build/install of 
> it then you obviously get "default" behaviour, for better or worse.)

More modern Docker setups run the application itself under a non-root
user, but still install the packages and other dependencies as root.

See eg. Zammad's Dockerfile:
https://github.com/zammad/zammad-docker/blob/master/Dockerfile

Not sure whether that answers your question, though.

It's rather uncommon to install venvs inside Docker containers: one of the
main reasons for using containers is the added isolation, but it doesn't
make a lot of sense to add another layer of isolation inside the container.

"pip install as root" will need to continue to work and thus distros
need to get a way to make sure that it doesn't corrupt the system
installed packages. And perhaps distros can also patch pip to not
output those silly warnings anymore when using the system pip package :-)

Regarding the proposed solution: I'm not sure whether a new configure
option is the right way to go about this. Distros could simply patch
sysconfig.py, since that's the golden source of this information from
Python 3.10 onward.

setuptools' distutils version (and other packages which ship distutils)
will have to use this information instead of the copy which is/was
backed into distutils/sysconfig.py on Python 3.10+

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

Would "pip install --user ..." in a Docker container also work, though? 
Presumably all the filesystem paths are being redirected anyway, so is there a 
difference?

(My assumption is that "--user" would essentially become the default if you're 
using the OS provided pip/Python. If you do your own build/install of it then 
you obviously get "default" behaviour, for better or worse.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue43976] Allow Python distributors to add custom site install schemes

[M.-A. Lemburg]

On 04.05.2021 22:07, Steve Dower wrote:
> 
> Perhaps what I'm suggesting here is that I don't see any reason for "sudo pip 
> install ..." into a distro-installed Python to ever need to work, and would 
> be quite happy for it to just fail miserably every time (which is already the 
> case for the Windows Store distro of Python).

The "pip install" into a root environment approach is the standard way
to setup Docker (and similar) containers, so I think trying to break
this on purpose will not do Python a good service.

The pip warning about this kind of setup which apparently got added
in one of the more recent versions of pip already is causing a lot
of unnecessary noise when building containers and doesn't make Python
look good in that environment.

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

> I'd rather not have a new option here ...

Perhaps what I'm suggesting here is that I don't see any reason for "sudo pip 
install ..." into a distro-installed Python to ever need to work, and would be 
quite happy for it to just fail miserably every time (which is already the case 
for the Windows Store distro of Python).

Admin installed all-user packages is the expert scenario here, and can be as 
twisted as possible. Pip installed per-user packages and system-tool installed 
packages are the defaults, and the more easily those can be overridden by a 
file in the distro, the better.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

The best option for restricting the import while still having it be a Python 
import is to find the file (if it's present in the expected location under 
sys.whatever), and then use importlib to import it: 
https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

I'd rather not have a new option here, I would much prefer "-S" in this context 
to mean "run Python with only core libraries" and "-s" to mean "run Python with 
only core and distro libraries" (and neither to mean "run Python with core, 
distro and user libraries").

That may be a bigger change, but there's enough angst around this issue that we 
would be better off getting it right this time, even if it changes things, than 
continuing to preserve the system that people dislike so much.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

> My biggest concern is with the bare "import _vendor_config", which I'd prefer 
> to have restricted to a fixed location, rather than being influenced by 
> environment variables and other options. We already have an issue with 
> readline being imported from anywhere it can be found.

Oh, I share the same concern! Though users could already mess up Python pretty 
badly by shadowing/overwriting parts of it, so I didn't thought it would be 
that big of an issue. Is there a way to achieve this while still allowing us to 
do everything we want?

> Sorry for not getting to this sooner, but 5 days is really tight for such a 
> change.

No worries. It was my fault, I should have been more attentive to the Python 
release timeline.

> With -S/-I, It would be great if sys.path only included packages installed as 
> part of the OS, and not those installed by `sudo pip`. (Or `pip --user`, but 
> that's covered).

Perhaps we could add an option to enable only vendor site schemes?

> It seems that with the current patch, pip will install into site-packages and 
> there's no way to disable/change site-packages. Is that the case?

I mean, there is, though not as straightforward as -S/-I. I was planning on 
using it to build the distro entrypoint scripts, so that they only include the 
distro packages.

$ python -S
> site.addsitedir(sysconfig.get_path('purelib', 'vendor'))
> site.addsitedir(sysconfig.get_path('platlib', 'vendor'))

As I mentioned above, we could add a cli flag to do essentially the same.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Petr Viktorin]

Petr Viktorin  added the comment:

Sorry for not getting to this sooner, but 5 days is really tight for such a 
change.


With -S/-I, It would be great if sys.path only included packages installed as 
part of the OS, and not those installed by `sudo pip`. (Or `pip --user`, but 
that's covered).

It seems that with the current patch, pip will install into site-packages and 
there's no way to disable/change site-packages. Is that the case?

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

Yes, I saw some of the latest changes in the PR.

My biggest concern is with the bare "import _vendor_config", which I'd prefer 
to have restricted to a fixed location, rather than being influenced by 
environment variables and other options. We already have an issue with readline 
being imported from anywhere it can be found.

A native flag to suppress it (i.e. something in sys.flags) could also become 
important for embedders, though it may matter more at a higher level (i.e. 
should an embedded CPython *ever* be using sysconfig? Probably not...). I 
wouldn't add a new flag for it right now, but I feel like sys.flags.isolated 
should probably imply that this should be ignored.

Though then we hit the issue again that these patches are about changing the 
"safe default" behaviour, which is what you want to get back when you run with 
-S or -I. And I'm not totally sure how to resolve this.

So basically, my concerns are:
* don't import arbitrary files
* ensure -S/-I options remain useful (or become even more useful)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Filipe Laíns]

Filipe Laíns  added the comment:

FYI, I have change the implementation to split the extra install schemes and 
extra schemes activated on site. This still makes sense over sitecustomize 
because we want the packages to be included in site.getsitepackages -- we want 
the vendor packages to essentially be the same as site-packages.

I have also moved sysconfig._get_preferred_schemes to the vendor config, 
instead of asking distributors to patch sysconfig -- this is why I prefer 
having it as executable code, we customize using functions, etc.
https://docs.python.org/3.10/library/sysconfig.html#sysconfig._get_preferred_schemes

A config taking advantage of all these mechanisms should look like this:

```
EXTRA_INSTALL_SCHEMES = {
'vendor': {
'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
'purelib': '{base}/lib/python{py_version_short}/vendor-packages',
'platlib': 
'{platbase}/{platlibdir}/python{py_version_short}/vendor-packages',
'include':
'{installed_base}/include/python{py_version_short}{abiflags}',
'platinclude':
'{installed_platbase}/include/python{py_version_short}{abiflags}',
'scripts': '{base}/bin',
'data': '{base}',
},
}

EXTRA_SITE_INSTALL_SCHEMES = [
'vendor',
]

def get_preferred_schemes(...):
...
```

Do you have any thoughts on this?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Steve Dower]

Steve Dower  added the comment:

> Making sysconfig look at sitecustomize seems like the wrong approach.

I mean, you're literally customizing the site, so having it be done from 
sitecustomize doesn't seem terribly wrong. But I agree, I'd rather see the code 
in sitecustomize poke paths into sysconfig, rather than the other way around.

The problem then would be that -S bypasses the path configuration entirely, 
which is likely going to point at non-existent paths. So yeah, for this case 
you need an override that isn't tied to the site module. Having a 
similar-but-different mechanism in sysconfig seems fine. I have a *slight* 
preference for non-executable code, mostly to avoid the risk of import 
hijacking, but it's only slight.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43976] Allow Python distributors to add custom site install schemes

[Terry J. Reedy]

Change by Terry J. Reedy :


--
title: Introduce mechanism to allow Python distributors to add custom site 
install schemes -> Allow Python distributors to add custom site install schemes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com