[Python-ideas] Re: [Python-Dev] Have virtual environments led to neglect of the actual environment?

2021-02-27 Thread Nick Coghlan 
On Wed, 24 Feb 2021 at 10:49, Random832  wrote:
>
> I was reading a discussion thread 
>  about 
> various issues with the Debian packaged version of Python, and the following 
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we 
> > don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I think 
> if it does, that problem should be solved rather than merely advising users 
> against using it. And why is it, anyway, that distributions whose package 
> managers can't coexist with pip-installed packages don't ever seem to get the 
> same amount of flak for "damaging python's brand" as Debian is getting from 
> some of the people in the discussion thread? Why is it that this community is 
> resigned to recommending a workaround when distributions decide the 
> site-packages directory belongs to their package manager rather than pip, 
> instead of bringing the same amount of fiery condemnation of that practice as 
> we apparently have for *checks notes* splitting parts of the stdlib into 
> optional packages? Why demand that pip be present if we're not going to 
> demand that it works properly?

The reason venv is promoted as heavily as it is is because it's the
only advice that can be given that is consistently correct regardless
of the operating system the user is running locally, whereas safely
using a system-wide Python installation varies a lot depending on
whether you're on Windows, Mac OS X, or Linux (let alone some other
platform outside the big 3 desktop clients).

conda is also popular for the same reason: while the instructions for
installing conda in the first place are OS-dependent, once it is up
and running you can use consistent platform independent conda commands
rather than having to caveat all your documentation with
platform-specific instructions.

Apple moved all of their dynamic language interpreter implementations
to inaccessible-by-default locations so Mac OS X users would stop
using them to run their own code.

Alongside that, we *have* worked with the Linux distro vendors to help
make "sudo pip install" safe (e.g [1]), but that only helps if a user
is running a new enough version of a distro that has participated in
that work.

However, while the option of running "platform native" environments
will never go away, and work will continue to make it less error
prone, the level of knowledge of your specific OS's idiosyncrasies
that it requires is almost certainly going to remain too high for it
to ever again become the default recommendation that it used to be.

Cheers,
Nick.

[1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note:
this change mitigated some aspects of the problem in a way similar to
what Debian does, but still doesn't solve it completely, as custom
Python builds may still make arbitrary changes)

P.S. "But what about user site-packages?" you ask. Until relatively
recently, Debian didn't put the user's local bin directory on the
system path by default, so commands provided by user level package
installs didn't work without the user adjusting their PATH. The
CPython Windows installer also doesn't adjust PATH by default (for
good reasons). And unlike a venv, "python -m" doesn't let you ensure
that the code executed is the version installed in user site-packages
- it could be coming from a directory earlier in sys.path.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RDLEH6DUF57UB6U4HNL2QRVAJY4KDSSJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Have virtual environments led to neglect of the actual environment?

2021-02-23 Thread Wes Turner 
FWIW, Distro repacks are advantageous in comparison to
"statically-bundled" releases that for example bundle in an outdated
version of OpenSSL, because when you `apt-get upgrade -y` that should
upgrade the OpenSSL that all the other distro packages depend upon.

Here's something that doesn't get called as frequently as `apt-get upgrade -y`:

```bash
pip install -U certifi
```

https://github.com/certifi/python-certifi (Mozilla's CA bundle
extracted into a Python package)

```bash
apt-get install -y ca-certificates
dnf install -y ca-certificates
```

On 2/24/21, Wes Turner  wrote:
> On 2/23/21, Random832  wrote:
>> I was reading a discussion thread
>>  about
>> various issues with the Debian packaged version of Python, and the
>> following
>> statement stood out for me as shocking:
>>
>> Christian Heimes wrote:
>>> Core dev and PyPA has spent a lot of effort in promoting venv because we
>>> don't want users to break their operating system with sudo pip install.
>>
>> I don't think sudo pip install should break the operating system. And I
>> think if it does, that problem should be solved rather than merely
>> advising
>> users against using it. And why is it, anyway, that distributions whose
>> package managers can't coexist with pip-installed packages don't ever
>> seem
>> to get the same amount of flak for "damaging python's brand" as Debian is
>> getting from some of the people in the discussion thread? Why is it that
>> this community is resigned to recommending a workaround when
>> distributions
>> decide the site-packages directory belongs to their package manager
>> rather
>> than pip, instead of bringing the same amount of fiery condemnation of
>> that
>> practice as we apparently have for *checks notes* splitting parts of the
>> stdlib into optional packages? Why demand that pip be present if we're
>> not
>> going to demand that it works properly?
>>
>> I think that installing packages into the actual python installation,
>> both
>> via distribution packaging tools and pip [and using both simultaneously -
>> the Debian model of separated dist-packages and site-packages folders
>> seems
>> like a reasonable solution to this problem] can and should be a supported
>> paradigm, and that virtual environments [or more extreme measures such as
>> shipping an entire python installation as part of an application's
>> deployment] should ideally be reserved for the rare corner cases where
>> that
>> doesn't work for some reason.
>>
>> How is it that virtual environments have become so indispensable, that
>> no-one considers installing libraries centrally to be a viable model
>> anymore? Are library maintainers making breaking changes too frequently,
>> reasoning that if someone needs the old version they can just venv it? Is
>> there some other cause?
>
> First, pip+venv is not sufficient for secure software deployment:
> something must set appropriate permissions so that the application
> cannot overwrite itself and other core libraries (in order to
> eliminate W^X violations (which e.g. Android is solving by requiring
> all installed binaries to come from an APK otherwise they won't and
> can't be labeled with the SELinux extended file atrributes necessary
> for a binary to execute; but we don't have binaries, we have an
> interpreter and arbitrary hopefully-signed somewhere source code, at
> least)).
>
> Believe it or not, this is wrong:
>
> ```bash
> # python -m venv httpbin || virtualenv httpbin
> # source httpbin/bin/activate
> mkvirtualenv httpbin
>
> pip install httpbin gunicorn
> gunicorn -b 0.0.0.0:8001 httpbin:app
>
> # python -m webbrowser http://127.0.0.1:8001
> ```
>
> It's wrong - it's insecure - because the user executing the Python
> interpreter (through gunicorn, in this case) can overwrite the app.
> W^X: has both write and execute permissions. What would be better?
>
> This would be better because pip isn't running setup.py as root (with
> non-wheels) and httpbin_exec can't modify the app interpreter or the
> code it loads at runtime:
>
> ```bash
> useradd httpbin # also creates a group also named 'httpbin'
> sudo -u httpbin sh -c ' \
> python -m venv httpbin; \
> umask 0022; \
> ./httpbin/bin/python -m pip install httpbin gunicorn'
>
> useradd httpbin_exec -G httpbin
> sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'
> ```
>
> This would be better if it worked, though there are a few caveats:
>
> ```bash
> sudo apt-get install python-gunicorn python-httpbin
> sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app
> ```
>
> 1. Development is impossible:
> - You can't edit the code in /usr/lib/python3.n/site-package/ without
> root permissions.
> - You should not be running an editor as root.
> - You can edit distro-package files individually with e.g. sudoedit
> (and then the GPG-signed package file checksums will fail when you run
> `debsums` or `rpm -Va` because you've edited the

[Python-ideas] Re: [Python-Dev] Have virtual environments led to neglect of the actual environment?

2021-02-23 Thread Wes Turner 
On 2/23/21, Random832  wrote:
> I was reading a discussion thread
>  about
> various issues with the Debian packaged version of Python, and the following
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
>> Core dev and PyPA has spent a lot of effort in promoting venv because we
>> don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I
> think if it does, that problem should be solved rather than merely advising
> users against using it. And why is it, anyway, that distributions whose
> package managers can't coexist with pip-installed packages don't ever seem
> to get the same amount of flak for "damaging python's brand" as Debian is
> getting from some of the people in the discussion thread? Why is it that
> this community is resigned to recommending a workaround when distributions
> decide the site-packages directory belongs to their package manager rather
> than pip, instead of bringing the same amount of fiery condemnation of that
> practice as we apparently have for *checks notes* splitting parts of the
> stdlib into optional packages? Why demand that pip be present if we're not
> going to demand that it works properly?
>
> I think that installing packages into the actual python installation, both
> via distribution packaging tools and pip [and using both simultaneously -
> the Debian model of separated dist-packages and site-packages folders seems
> like a reasonable solution to this problem] can and should be a supported
> paradigm, and that virtual environments [or more extreme measures such as
> shipping an entire python installation as part of an application's
> deployment] should ideally be reserved for the rare corner cases where that
> doesn't work for some reason.
>
> How is it that virtual environments have become so indispensable, that
> no-one considers installing libraries centrally to be a viable model
> anymore? Are library maintainers making breaking changes too frequently,
> reasoning that if someone needs the old version they can just venv it? Is
> there some other cause?

First, pip+venv is not sufficient for secure software deployment:
something must set appropriate permissions so that the application
cannot overwrite itself and other core libraries (in order to
eliminate W^X violations (which e.g. Android is solving by requiring
all installed binaries to come from an APK otherwise they won't and
can't be labeled with the SELinux extended file atrributes necessary
for a binary to execute; but we don't have binaries, we have an
interpreter and arbitrary hopefully-signed somewhere source code, at
least)).

Believe it or not, this is wrong:

```bash
# python -m venv httpbin || virtualenv httpbin
# source httpbin/bin/activate
mkvirtualenv httpbin

pip install httpbin gunicorn
gunicorn -b 0.0.0.0:8001 httpbin:app

# python -m webbrowser http://127.0.0.1:8001
```

It's wrong - it's insecure - because the user executing the Python
interpreter (through gunicorn, in this case) can overwrite the app.
W^X: has both write and execute permissions. What would be better?

This would be better because pip isn't running setup.py as root (with
non-wheels) and httpbin_exec can't modify the app interpreter or the
code it loads at runtime:

```bash
useradd httpbin # also creates a group also named 'httpbin'
sudo -u httpbin sh -c ' \
python -m venv httpbin; \
umask 0022; \
./httpbin/bin/python -m pip install httpbin gunicorn'

useradd httpbin_exec -G httpbin
sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'
```

This would be better if it worked, though there are a few caveats:

```bash
sudo apt-get install python-gunicorn python-httpbin
sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app
```

1. Development is impossible:
- You can't edit the code in /usr/lib/python3.n/site-package/ without
root permissions.
- You should not be running an editor as root.
- You can edit distro-package files individually with e.g. sudoedit
(and then the GPG-signed package file checksums will fail when you run
`debsums` or `rpm -Va` because you've edited the file and that's
changed the hash).

- Non-root users cannot install python packages without having someone
repack (and sign it) for them.

- What do I need to do in order to patch the distro's signed repack of
the Python package released to PyPI?
  - I like how Fedora pkgs and conda-forge have per-package git repos now.
  - Conda-forge has a bot that watches PyPI for new releases and tries
sending an automated PR.
  - If I send a PR to the main branch of the source repo and it gets
merged, how long will it be before there's a distro repack built and
uploaded to the distro package index?

2. It should be installed in a chroot/jail/zone/container/context/vm
so that it cannot read other data on the machine.
The httpbin app does not need read access to

[Python-ideas] Re: [Python-Dev] Have virtual environments led to neglect of the actual environment?

2021-02-23 Thread Jonathan Goble 
On Tue, Feb 23, 2021 at 7:48 PM Random832  wrote:
>
> I was reading a discussion thread 
>  about 
> various issues with the Debian packaged version of Python, and the following 
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we 
> > don't want users to break their operating system with sudo pip install.
>
> [snip]
>
> How is it that virtual environments have become so indispensable, that no-one 
> considers installing libraries centrally to be a viable model anymore? Are 
> library maintainers making breaking changes too frequently, reasoning that if 
> someone needs the old version they can just venv it? Is there some other 
> cause?

I can't speak for distributors or maintainers [1], but I can speak for
myself as a user. I run Debian testing (currently bullseye as that is
preparing for release) as my daily OS on my personal laptop, used for
personal matters and school assignments (I'm a university computer
science student in my senior year).

I don't use the system Python for anything of my own, whether it's a
school assignment or a personal project, precisely because I don't
want to risk screwing something up. Rather, I maintain a clone/fork of
the official CPython GitHub repo, and periodically build from source
and `make altinstall` into `~/.local/`. The `python3` command
continues to refer to the system Python, while `python3.8`, etc. refer
to the ones installed in my home folder. To the latter I make symlinks
for `py38`, `py39`, etc., and just `py` (and `pip`) for the one I use
most often (usually the latest stable release). I typically have
multiple versions installed at once since different scripts/projects
run on different versions at different times. Given this setup, I can
just do a simple `pip install spam` command and I don't need either
`sudo` or `--user`, nor do I need virtual envs.

While the average person would probably not clone the GitHub repo and
build that way, it's not terribly unreasonable for an inclined person
to do the same with a tarball downloaded from python.org, and so I
doubt I'm the only one with this type of setup.

Just some food for thought.

[1] Technically I am a library maintainer since I have a couple
projects on PyPI, but those are mostly unused and more or less
abandoned at this point, and neither ever reached the point where I
could consider graduating them from beta status. Most of what I work
with these days is private personal code or school assignments.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/M6OGQXKLG27Y6KA5L5UFBWA2NAZNBOHL/
Code of Conduct: http://python.org/psf/codeofconduct/