[issue33499] Environment variable to set alternate location for pycache tree

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fc96437db4fa95dff99d43d4500beaf12bad5bff by Victor Stinner in 
branch 'master':
bpo-33499: Fix pymain_init_pycache_prefix() (GH-8596)
https://github.com/python/cpython/commit/fc96437db4fa95dff99d43d4500beaf12bad5bff


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +8106

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Nick Coghlan  added the comment:


New changeset 16eb3bcdb22be4d82dc597b92b7154fcb11c6479 by Nick Coghlan in 
branch 'master':
bpo-33499: PYTHONPYCACHEPREFIX What's New entry (GH-7749)
https://github.com/python/cpython/commit/16eb3bcdb22be4d82dc597b92b7154fcb11c6479


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Change by Nick Coghlan :


--
pull_requests: +7357

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Nick Coghlan  added the comment:

Merged as PYTHONPYCACHEPREFIX=path, -X pycache_prefix=path and 
sys.pycache_prefix :)

I'll also update PEP 304 with a note saying a variant of the idea was 
eventually accepted for Python 3.8.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Nick Coghlan  added the comment:


New changeset b193fa996a746111252156f11fb14c12fd6267e6 by Nick Coghlan (Carl 
Meyer) in branch 'master':
bpo-33499: Add PYTHONPYCACHEPREFIX env var for alt bytecode cache location. 
(GH-6834)
https://github.com/python/cpython/commit/b193fa996a746111252156f11fb14c12fd6267e6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Carl Meyer  added the comment:

Cool, thanks for the pointer on -X. PR is updated with `-X bytecode_path=PATH`; 
don't think it's critical to have it, but it wasn't that hard to add.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Nick Coghlan  added the comment:

I believe the main argument for -X options is the fact that cmd on Windows 
doesn't offer a nice way of setting environment variables as part of the 
command invocation (hence "-X utf8", for example).

As far as setting values for X options goes, `sys._xoptions` in CPython is a 
str:Union[bool,str] dict, with the command args split on "=":

$ python3 -X arg=value -c "import sys; print(sys._xoptions)"

  
{'arg': 'value'}

If no value is given for the arg, then it's just set to the boolean True.

The _xoptions entry shouldn't be the public API though - it's just a way of 
shuttling settings from the command line through to CPython-specific 
initialisation code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Honestly, I don't think there's a strong argument for a CLI option.  I'm 
perfectly happy with just an environment variable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Carl Meyer  added the comment:

Can we have a named -X option that also takes a parameter? I don't see any 
existing examples of that. This option needs to take the path where bytecode 
should be written.

Are there strong use-cases for having a CLI arg for this? I don't mind doing 
the implementation work if there are, but right now I'm struggling to think of 
any case where it would be better to run `python -C /tmp/bytecode` than 
`PYTHONBYTECODEPATH=/tmp/bytecode python`. Our existing "takes a path" env 
variables (`PYTHONHOME` and `PYTHONPATH`) do not have CLI equivalents.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 17, 2018, at 08:14, Nick Coghlan  wrote:
> 
> If we did add an option, then a named -X option would probably make the most 
> sense.

+1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Nick Coghlan]

Nick Coghlan  added the comment:

Regarding environment variables, note that they get used in two *very* 
different ways:

1. The "persistent shell setting" case that Raymond describes. While setting 
PYTHONBYTECODEPATH to always point to a RAM disk could make quite a bit of 
sense for some developers, it's more likely that this case would be associated 
with tools like `pipenv shell`.

2. The "inheritable process setting" case, where you prepend the environment 
variable setting to a shell command, or add it to the env dict in a Python 
subprocess call.

Anywhere that I used this setting, I'd want it to be passed along to child 
processes, so an environment variable would be a lot more useful than a command 
line option.

If we did add an option, then a named -X option would probably make the most 
sense.

Regarding the state caching: having this be read once at startup would help 
avoid a lot of potential for weird state inconsistencies where some modules 
were loaded from one cache directory, while later modules were loaded from a 
different one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On May 15, 2018, at 22:58, Carl Meyer  wrote:

> Our use case includes a webserver process that embeds Python; I'm not sure if 
> we could pass a CLI arg to it or not.

I think you pretty much have to have an environment variable, as there are just 
too many places where you’re going to invoke Python without the ability to set 
the command line.  We have precedence for having both a switch and environment 
variable, and I think that makes sense here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> Environment variables aren't system-wide, they are 
> per-process (though they can be inherited by child processes).

Yes, that is how they work.  It is not how they are used.  Environment 
variables are commonly set in shell start-up scripts such as .bashrc and the 
results then affect every python application that gets run in any shell session.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Carl Meyer  added the comment:

> a system-wide environment variable

Environment variables aren't system-wide, they are per-process (though they can 
be inherited by child processes).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Carl Meyer  added the comment:

Environment variable seems to make a bit more sense for this, since it's not 
per-invocation; there's no point writing bytecode cache to a particular 
location unless the next invocation reads the cache from there.

Our use case includes a webserver process that embeds Python; I'm not sure if 
we could pass a CLI arg to it or not.

Python has lots of precedent for similar environment variables (e.g. 
`PYTHONHOME`, `PYTHONDONTWRITEBYTECODE`, `PYTHONPATH`, etc). Compared to those, 
`PYTHONBYTECODEPATH` is pretty much harmless if it "leaks" to an unintended 
process.

I asked Brett Cannon in the sprints if I should add a CLI flag in addition to 
the env var; he suggested it wasn't worth it. I'm not opposed to adding the CLI 
flag, but I think removing the env var option would be a mistake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Raymond Hettinger]

Raymond Hettinger  added the comment:

FWIW, I've had issues with environment variables in that they affect every 
version of Python running on a system and seem to defy isolation.  So, if one 
application needs the environment variable set, it will affect every 
application, even if it wants to keeps its contents private and not leak 
outside of a virtual environment.

Can your needs be met with just CLI flag rather than a system-wide environment 
variable?

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Łukasz Langa]

Change by Łukasz Langa :


--
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Change by Carl Meyer :


--
keywords: +patch
pull_requests: +6517
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

Carl Meyer  added the comment:

Per vstinner Python prefers to not have underscores in environment variable 
names, for historical reasons. So I'm using `PYTHONBYTECODEPATH` as the env var.

Other open questions: 

1) Does there need to be a corresponding CLI flag, or is env-var-only 
sufficient?

2) Is it OK to check the environ every time, or do we need to cache its value 
in a `sys` flag at startup?

Will push an initial version for review that has no CLI flag nor `sys` 
attribute.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33499] Environment variable to set alternate location for pycache tree

[Carl Meyer]

New submission from Carl Meyer :

We would like to set an environment variable that would cause Python to read 
and write `__pycache__` directories from a separate location on the filesystem 
(outside the source code tree). We have two reasons for this:

1. In our development setup (with a webserver running in a container on the 
dev-tree code), the `__pycache__` directories end up root-owned, and managing 
permissions on them so that they don't disrupt VCS operations on the code repo 
is untenable. (Currently we use PYTHONDONTWRITEBYTECODE as a workaround, but we 
have enough code that this costs us multiple seconds of developer time on every 
restart; we'd like to take advantage of cached bytecode without requiring that 
it pollute the code tree.)

2. In addition to just _having_ cached bytecode, we'd like to keep it on a 
ramdisk to minimize filesystem overhead.

Proposal: a `PYTHON_BYTECODE_PATH` environment variable. If set, 
`source_from_cache` and `cache_from_source` in `importlib._bootstrap_external` 
will respect it, creating a directory tree under that prefix that mirrors the 
source tree.

--
messages: 316518
nosy: brett.cannon, carljm, eric.snow, lukasz.langa, ncoghlan
priority: normal
severity: normal
status: open
title: Environment variable to set alternate location for pycache tree
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com