[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Change by Eryk Sun :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:


New changeset 5ab27cc518f614a0b954ff3eb125290f264242d5 by Paul Moore in branch 
'master':
bpo-42041: Clarify how subprocess searches for the executable (GH-22715)
https://github.com/python/cpython/commit/5ab27cc518f614a0b954ff3eb125290f264242d5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Mark Keller]

Mark Keller  added the comment:

+1 to what Steve said

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

I like that, I've updated the PR accordingly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Steve Dower]

Steve Dower  added the comment:

Proposed alternative based on Eryk's:

  For maximum reliability, use a fully-qualified path for the executable.
  To search for an unqualified name on :envvar:`PATH`, use
  :meth:`shutil.which`. On all platforms, passing :data:`sys.executable`
  is the recommended way to launch the current Python interpreter again,
  and use the ``-m`` command-line format to launch an installed module.

  Resolving the path of *executable* (or the first item of *args*) is
  platform dependent. For POSIX, see :meth:`os.execvpe`, and note that
  when resolving or searching for the executable path, *cwd* overrides the
  current working directory and *env* can override the ``PATH``
  environment variable. For Windows, see the documentation of the
  ``lpApplicationName`` and ``lpCommandLine`` parameters of WinAPI
  ``CreateProcess``, and note that when resolving or searching for the
  executable path with ``shell=False``, *cwd* does not override the
  current working directory and *env* cannot override the ``PATH``
  environment variable. Using a full path avoids all of these variations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> My biggest concern with the suggested wording ... is that it 
> reintroduces the issue with the redirector.

I thought I addressed that in second paragraph by recommending sys.executable. 
It could be emphasized that running "python[x][.y]" is unreliable, without 
going into detailed examples of where it's unreliable on various platforms 
(i.e. avoid going into detail about issues with naming of binaries, embedding, 
the search path, redirectors, etc). A separate cross-platform note in venv 
could advise scripts to use sys.executable, with a link to the subprocess.Popen 
docs and a reference to the note about the platform-dependent search behavior 
and unreliability of running "python[x][.y]".

If the redirector issue is mentioned anywhere, I think it should be in the venv 
docs. It can include a note about the Windows implementation detail to use a 
redirector for non-symlink virtual environments. Of concern to me here is that 
the process handle and PID returned by CreateProcess is for the redirector. One 
can't use the returned process handle or PID with DuplicateHandle or 
WSADuplicateSocket (i.e. socket.socket.share) to share handles and sockets with 
a script that's running as a child process. It might seem to still be working, 
purely by accident, because the parent script is executing the base "python" 
instead of the redirector, but in that case the child script isn't using the 
virtual environment. There isn't an official way to support running 
sys._base_executable with the __PYVENV_LAUNCHER__ environment variable, as 
multiprocessing implements internally. Maybe the workaround should be 
incorporated implicitly in subprocess.Popen if `executable`, `args`, or args[0] 
is equa
 l to sys.executable and sys._base_executable is different. If the latter is 
implemented, and using sys.executable is strongly advised, it strengthens the 
case to avoid discussing the redirector entirely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

My biggest concern with the suggested wording (which in broad terms looks OK to 
me) is that it reintroduces the issue with the redirector.

By specifically saying that the behaviour in `CreateProcess` applies, we lead 
the user to the statement that the search path includes "the directory 
containing the running application", which the user will interpret as the 
directory of `sys.executable`. So we need to qualify this somewhere by 
clarifying that "the running application" may not actually be `sys.executable` 
(without saying what it actually *is*, as that would mean documentin what the 
redirector does, which Steve wants to avoid).

I don't have a good answer here - we have two conflicting preferences (document 
the search behaviour but don't document the way the redirector works) and 
something needs to give. (To be fair, there's a third conflicting priority 
here, which is me wanting everything to be explicit and clear. I'm willing to 
give that up if you and Steve can agree on something, though).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> I'd read it because I'm interested, but that is probably too much detail
> for someone who is trying to get something done quickly.

I did say that I don't know how much should be documented. I tend to dump a lot 
of information in issues so that people can make informed decisions. Personally 
I would keep it limited. I do want more help, and generic help, in the warning, 
but I don't want to spam the already exhausting and MEGO-prone (i.e. my eyes 
glaze over) subprocess docs.

With regard to the use of a platform search, note that subprocess actually 
implements its own search in POSIX. (subprocess supports posix_spawn -- but not 
posix_spawnp -- on macOS and on Linux with glibc 2.24+, but it's limited to a 
narrow subset of Popen arguments, excluding `cwd`, and it requires a qualified 
`executable` path.) A name containing a slash is resolved against `cwd` or the 
process current working directory. An unqualified name is searched for in the 
os.get_exec_path(env) search path, based on PATH from `env` or os.environ, else 
os.defpath. This allows `env` to override the POSIX search path, just as `cwd` 
in POSIX overrides the current working directory when resolving a qualified 
args[0] path and `executable`. This is sort of documented by saying that POSIX 
Popen() works "os.execvp()-like", but to be more correct it should say 
"os.execvpe()-like". The final "e" in "execvpe" is fundamentally important 
since PATH can be sourced from `env`.

In Windows, the search path and the working directory used for resolving paths 
can't be overridden without passing shell=True. subprocess incorrectly 
documents this by claiming in general that "the function looks for `executable` 
(or for the first item in `args`) relative to `cwd` if the executable path is a 
relative path". That wording isn't even correct for POSIX, for which the 
working directory only applies to args[0] if it contains a slash.

In light of this, how about inserting the following warning:

  Resolving the path of *executable* or the first item of *args* is
  platform dependent even with ``shell=False``. For POSIX, see
  :meth:`os.execvpe`, and note that when resolving or searching for the
  executable path, *cwd* overrides the current working directory and *env*
  can override the ``PATH`` environment variable. For Windows, see the
  documentation of the ``lpApplicationName`` and ``lpCommandLine``
  parameters of WinAPI ``CreateProcess``, and note that when resolving or
  searching for the executable path with ``shell=False``, *cwd* does not
  override the current working directory and *env* cannot override the
  ``PATH`` environment variable. Cross-platform usage can improve
  reliability by using a fully-qualified path for the executable, such as
  from :meth:`shutil.which`.

  On all platforms, using :data:`sys.executable` as the path of the
  executable is the recommended and reliable way to run the current Python
  interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Mark Keller]

Mark Keller  added the comment:

Paul explained precisely of what I was going through when I reached out to you 
with this:

> then the "running executable" is "path/to/venv/scripts/python.exe", and so 
> the path containing the running executable is "path/to/venv/scripts", so a 
> search for "python" will locate "path/to/venv/scripts/python.exe", because 
> that's how Windows path search rules work. The problem is that that exe is a 
> redirector, and the script is *actually* being run by the system Python.

However, after reading through your messages and thinking about this for a few 
days I think it'd be sufficient to point out sys.executable's existence early 
on in the subprocess docs and then say something like how python is not 
reliable. Similarly to what Paul said here: 
https://bugs.python.org/issue42041#msg378727

I used python in my subprocess call because I thought it would always resolve 
to the the current interpreter. Clearly I don't work a whole lot with Windows...

Eryk - I really appreciate your explanation and I think that level of detail 
should be documented. I'd read it because I'm interested, but that is probably 
too much detail for someone who is trying to get something done quickly.

I think both the quick explanation (and pointing out sys.executable's 
existence) and the detailed why would have it's place in the docs.
Maybe the details could be documented for educational purposes and not as a 
guarantee of implementation?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> For platform semantics, I'd prefer a link to the CreateProcessW docs, 
> with advice to read about the lpApplicationName parameter with 
> respect to `executable` and lpCommandLine with respect to `args` 
> and the platform search semantics. 

For example, let's help someone figure out that because "python3.8" has a ".8" 
filename 'extension', ".exe" may or may not be appended in a search.

>>> print(shutil.which('python3.8'))
C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8.EXE

>>> subprocess.call(['python3.8.exe', '-V'])
Python 3.8.6
0

SearchPathW (called internally by CreateProcessW) won't append the ".exe" 
default extension to a name that already has a ".8" extension:

>>> try: subprocess.call(['python3.8', '-V'])
... except OSError as e: print(e)
...
[WinError 2] The system cannot find the file specified

But with shell=True it works because CMD always appends the PATHEXT extensions 
(thankfully there isn't a "python3.8.com" file to get in the way, since .COM is 
usually listed before .EXE in PATHEXT):

>>> subprocess.call('python3.8 -V', shell=True)
Python 3.8.6
0

SearchPathW does append the default ".exe" extension for a qualified path:

>>> 
subprocess.call([r'C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8',
 '-V'])
Python 3.8.6
0

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> You can build a cross-platform wrapper on top of native behaviour 
> (witness `shutil.which`) but you can't do the opposite.

Nothing would prevent adding a parameter to use the platform semantics, if that 
had been in the design from the outset. But it's not something we should change 
at this stage. It's too fundamental.

> I'm not convinced that using `shutil.which` is the right 
> answer for anything other than "python".

Actually, I think shutil.which() is the right answer generally (within the 
filesystem scope) except for "python", for which sys.executable is usually what 
people want, i.e. the current interpreter. A PATH search via 
shutil.which('python') doesn't necessarily find the current interpreter, if 
anything at all. So I'm not arguing against advice to use sys.executable, but 
rather that recommmending shutil.which() should be done in a more generic way 
that's separate from the "python" problem. 

For platform semantics, I'd prefer a link to the CreateProcessW docs, with 
advice to read about the lpApplicationName parameter with respect to 
`executable` and lpCommandLine with respect to `args` and the platform search 
semantics. The CreateProcessW docs are rather long and off-putting, so I think 
it helps to narrow it down for people, and explicitly map between Popen() and 
CreateProcessW parameters.

Over the years, I've come across many forum questions in which novice users 
waste days on problems with respect to Windows quirks that can be resolved in a 
minute or so by someone with knowledge and experience. They get lots of advice 
with good intentions from programmers who only have POSIX experience (the 
Python ecosystem online is still heavily dominated by POSIX, despite the number 
of Windows users), and it's mostly wrong advice and a waste of their time (not 
even a learning experience). Documenting platform inconsistencies helps 
experienced programmers to use the docs in order to help novice programmers. 
It's not necessarily about helping novices directly. Often just a nudge in the 
right direction is enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

Exactly. Why isn't the current directory searched? Why isn't foo/bar searched 
for on PATH? Why is it possible for the user to accidentally remove system 
commands from PATH and lose access to them? Any system is confusing and 
surprising to users only familiar with another system.

And I've had very bad experiences in the past with languages/applications that 
presume to define a "cross-platform" abstraction that ends up just "not 
behaving how I expect a native app to". You can build a cross-platform wrapper 
on top of native behaviour (witness `shutil.which`) but you can't do the 
opposite.

Maybe there's scope for a section in the documentation that discusses how to 
use `subprocess` in a platform-agnostic manner. I'd be OK with that, although 
I'd want it to read along the lines of "these are places where Windows and 
POSIX behave differently" and not "here's some weird stuff Windows does that 
POSIX users need to be aware of" which was how your comment read to me.

Never mind, I think we can just agree to differ on this. It's not likely to 
impact this issue or the PR for it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> Also, why would we document the Windows rules, but not the POSIX 
> rules? They are arguably just as strange to someone who doesn't 
> know them.

POSIX rules are simply to search PATH for the filename as passed, and if it has 
a slash in it, the path is resolved against the working directory. There is no 
implicit search of the application directory, current directory, and system 
directories. There is no search for "dir/file" in *every* search path directory 
instead of just resolving against the working directory. There is no figuring 
out when ".exe" or PATHEXT extensions will be appended or how to search for a 
filename that has no extension by appending a trailing "." to the name. And 
there is no massive inconsistency between the search semantics of shell=True 
and shell=False. What happens with "platform semantics" in Windows is 
complicated compared to POSIX. I'm more comfortable telling people to search 
via shutil.which() than relying on the platform search. I'd be much more 
comfortable if that's what subprocess.Popen() just did on its own. But we're 
locked into platform semantics, and I don't see that changing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

Well, I'm not convinced that using `shutil.which` is the right answer for 
anything other than "python". And even there, I'd recommend using 
`sys.executable` over `shutil.which` in pretty much every case.

My view is that if we ignore the "weirdness" introduced by the redirector[1], 
for everything else `subprocess` is fine - it searches for the executable using 
the normal platform semantics. That's nearly always the same across platforms, 
and when it isn't, users may well prefer consistency with the platform over 
identical cross-platform behaviour. And if they *do* want cross-platform 
consistency, at the cost of occasional differences between their program's 
behaviour and (for example) the shell, then they have `shutil.which` available.

Saying that `subprocess` follows platform semantics is easy, and clear. People 
who need to know platform details can find them out. People who want 
Python-defined cross-platform semantics have `shutil.which`.

That's the basis on which I feel that documenting the Windows rules is 
unnecessary. Also, why would we document the Windows rules, but not the POSIX 
rules? They are arguably just as strange to someone who doesn't know them.

[1] I would characterise that weirdness as being that the redirector impacts 
the behaviour of how the platform locates the "python" command under the 
platform search rules of the platform the redirector is used on - even though 
we don't document the behaviour of the redirector so the user cannot infer the 
actual rules without knowing implementation details.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

> I don't think we should document this level of detail

But a lot of it -- most of it -- is also strange behavior that no one would 
expect without reading about it somewhere. Most users of subprocess.Popen() 
will never wade through the documentation of CreateProcessW, SearchPathW, 
ShellExecuteExW, and cmd.exe -- to the extent that the behavior is usefully and 
correctly documented by Microsoft.

One of the reasons I bother writing it out in detail here is to make the case 
for always using shutil.which(), regardless of the value of `shell` (unless 
shell=True is being used beyond the filesystem scope). We have complete control 
of the implementation and documentation of shutil.which(). I don't want that 
advice to be narrowed down to just talking about running "python".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

I don't think we should document this level of detail, both because it's 
basically Windows standard behaviour and therefore not up to us to document, 
and because it's *way* too overwhelming for the average user.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

I don't know how much should be documented for subprocess.Popen, but here are 
the details for how searching works with shell=False (default) and shell=True.

For shell=False, the search path used by WinAPI CreateProcessW checks 
%__APPDIR__%; %__CD__% (unless %NoDefaultCurrentDirectoryInExePath% is defined 
and the name to search contains no backslashes); %SystemRoot%\System32; 
%SystemRoot%\System; %SystemRoot%; and then the %PATH% directories. The search 
path is passed to WinAPI SearchPathW, with a default ".exe" extension. 
SearchPathW tries to resolve a relative path (but not a drive-relative or 
rooted-relative path) against every directory in the search path, unless it 
explicitly begins with a "." or ".." component. For the relative case, it 
appends the default ".exe" extension to a name if and only if it has no 
extension (a trailing "." counts as an extension). For the non-relative case, 
it first tries to find the name as given and then with the default extension 
appended, even if the filename already has an extension (no exception is made 
for a trailing dot, i.e. searching for "C:\Temp\spam." will check for "spam." 
and then "spam..exe"
 ).

>From the POSIX perspective, the implicit inclusion of the application 
>directory, working directory, and system directories is strange and, regarding 
>the inclusion of the working directory, troubling. The fact that searching for 
>a relative name with one or more slashes, such as "spam\\python", is not 
>resolved against *only* the working directory is strange and not documented. 
>The rules governing when ".exe" will be appended are complicated and 
>incorrectly documented (e.g. the claim "if the file name contains a path, .exe 
>is not appended").

With shell=True, the CMD shell simply checks %__CD__% (unless 
%NoDefaultCurrentDirectoryInExePath% is defined and the name to search contains 
no slashes) and %PATH%. Support for forward slash in the name to search is 
wonky; it works only for quoted paths. But at least a relative path that 
contains slashes is only resolved against the current working directory instead 
of every directory in the search path. CMD's rules for appending default 
extensions are simpler than SearchPathW in some ways, but also more complicated 
because it's generalized as the PATHEXT list of extensions. In each directory, 
CMD always looks first for the searched name as given and then the name plus 
each extension in PATHEXT, regardless of the filepath type or whether the 
searched name already has an extension. It will not find a name that has no 
extension unless PATHEXT includes a "." entry for the empty extension. (This is 
consistent with the desktop shell API, which supports defining an association 
for the "."
  filetype.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

OK, PR updated as per discussion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

> We'd need Mark to weigh in (or possibly track down and ask a colleague), but 
> I expect the assumption is that subprocess.Popen("python") searches PATH 
> first.

I'm sure it was, TBH. The executable directory side of things is really just on 
digging a bit deeper - "well, Windows actually uses the executable directory, 
but that's not relevant because I ran the venv Python". That's when the 
mistaken assumption happens.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Steve Dower]

Steve Dower  added the comment:

> I'm not asking that we guarantee any behaviour, or that we commit ourselves 
> to anything. Just that we note that people ought not to be making a specific 
> assumption

Which is a fine intent, it's just the running assumption is that if it's in the 
documentation, it's supported (e.g. the distutils discussion). So no matter how 
carefully we document something as unsupported, people will still argue that 
it's supported because it's in the docs 路‍♂️

> which it appears from this issue, people *have* been making

We'd need Mark to weigh in (or possibly track down and ask a colleague), but I 
expect the assumption is that subprocess.Popen("python") searches PATH first. 
I've honestly never encountered anyone who deliberately depended on the 
application directory search (at least the first time, some who have discovered 
it have used it, though it tends to surprise colleagues when they do).

> I'll update the PR to add this statement or something similar to the 
> subprocess docs in the next couple of days.

No rush. We've got the sprints next week, and I don't have any major time 
consuming projects, so will probably get a few easier PRs done. Looks like I 
can edit your PR, so I can finish it up if you want to just move on :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

> This is still true, though, as much as it's ever been.

But it's *not*, if I understand the problem here. Windows resolves a bare 
"python" command by searching the directory containing the running executable 
first. I would reasonably assume that if I type

path/to/venv/scripts/python.exe foo.py

then the "running executable" is "path/to/venv/scripts/python.exe", and so the 
path containing the running executable is "path/to/venv/scripts", so a search 
for "python" will locate "path/to/venv/scripts/python.exe", because that's how 
Windows path search rules work. The problem is that that exe is a redirector, 
and the script is *actually* being run by the system Python.

Of course, the assumption I made is flawed, because there's never been anything 
saying that a program called "python.exe" can't do anything it likes in the 
process of running a script, including redirecting. But it's been the case for 
a long time, and IMO the introduction of the redirector counts as a 
user-visible behaviour change, and we should therefore explicitly point it out.

I really don't see why you are so reluctant to include this. I'm not asking 
that we guarantee any behaviour, or that we commit ourselves to anything. Just 
that we note that people ought not to be making a specific assumption (which it 
appears from this issue, people *have* been making).

> we could phrase it more positively

Cool, I'll update the PR to add this statement or something similar to the 
subprocess docs in the next couple of days.

I'm not going to fight to keep the comments in the venv documentation. I'm 
disappointed, and I feel like we'll end up with people relying on *more* of the 
implementation details because we're keeping things vague, but I have more 
important things to debate, so I'll drop this :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Steve Dower]

Steve Dower  added the comment:

> The bit that I *do* think is a venv gotcha is that it's entirely reasonable 
> for a user to expect that if they run path\to\venv\Scripts\python.exe, then 
> their Python script will be run by that executable.

This is still true, though, as much as it's ever been. The gotcha is that 
launching "python" may not launch "path\to\venv\Scripts\python.exe" even if 
that's the first entry on PATH.

For the documentation, we could phrase it more positively as "If you need the 
path to the Python executable, use sys.executable. Relying on explicit or 
implicit PATH resolution may result in the wrong version of Python being used, 
especially when the user has launched from a virtual environment."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

Fair enough. I think we have to be careful here - people do rely on how the 
"internal" aspects of virtual environments work, like it or not, and we've 
definitely broken people's code in the past as a result of taking a hardline 
"it's not documented, so it's not guaranteed" approach. (Disclaimer - we broke 
some of my code, so I'm not unbiased when I say the above ;-))

I take your point that this is a cross-platform issue, and as such would be 
worth putting in the `subprocess` documentation. When I looked, though, I 
couldn't really find anywhere that felt appropriate. I'll take a second look, 
but if you have any suggestions I'd appreciate it.

The bit that I *do* think is a venv gotcha is that it's entirely reasonable for 
a user to expect that if they run path\to\venv\Scripts\python.exe, then their 
Python script will be run by that executable. The redirector breaks that 
assumption, and while I'm fine with the idea that we don't want to document all 
of the details of the redirector, I *do* think we should document that you must 
not make that assumption.

How about this:

   The mechanism by which virtual enviroments are implemented means that
   you should not assume that the executable in the virtual environment
   is the one that will ultimately run your script. As a result, you
   should always use `sys.executable` to identify "this script's Python
   interpreter", as that is guaranteed to be set as you would expect.

I'm not completely happy with the above, but that's mostly because I'd rather 
just document how the process works (even if it's with a disclaimer "this is 
implementation detail and may change without notice"). But that's not my call 
to make, so I'll defer to you on that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Steve Dower]

Steve Dower  added the comment:

Thanks for doing the PR, Paul.

Honestly, I'd rather document in subprocess that Popen("python") is not a good 
idea, and should use either sys.executable or shutil.which depending on what 
you mean. This is a platform-independent gotcha, as it's very easy to reproduce 
the failure (e.g. run on Linux where "python"==Python 2.x).

Describing the current state of how venv works in the venv docs is fine, and 
what you've got there is accurate (might be easier to follow on first reading 
if it leads with Python picking up the venv based on the launched executable, 
rather than anything in the terminal's environment).

My concern is that we don't make this level of detail a documented guarantee, 
and can still change it between major versions without the deprecation period. 
I'm happy to guarantee intended uses, and it might be worth figuring out what 
those are (e.g. does double-clicking in Explorer matter? what about the Run 
window? Desktop shortcuts? With/without activation? etc.) and likely we'd have 
to clarify which ones don't work the same for the symlink vs. redirector 
options.

IOW, if someone writes an actual spec (and we all agree on it), we can 
guarantee that. Without it, I'd rather implementation internals remain internal.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

Eryk - I suggest that we modify (or remove) the note in venv that my PR adds as 
part of such a change, rather than try to qualify the text now (which would 
probably only make it harder for people to understand the issue).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Mark Keller]

Mark Keller  added the comment:

This sounds good to me Paul, thank you again!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Eryk Sun]

Eryk Sun  added the comment:

PR 22715 is accurate for the current implementation of subprocess. There has 
been discussion in past issues about implementing a PATH search that's similar 
to what the CMD shell does -- as in calling os.system() or Popen() with 
shell=True -- instead of relying on the search that WinAPI CreateProcessW 
implements. The resolved path would be passed as the `executable` argument, 
which becomes the lpApplicationName argument of CreateProcessW.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

I've created https://github.com/python/cpython/pull/22715 as a suggested 
documentation fix. Mark, does this cover what you would have needed to know? 
Steve, is what I've written technically accurate?

--
stage: patch review -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Change by Paul Moore :


--
keywords: +patch
pull_requests: +21683
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22715

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Mark Keller]

Mark Keller  added the comment:

Thank you so much for the prompt responses.

I would say that this should be documented on the venv page 
(https://docs.python.org/3/library/venv.html) as I find the subprocess page too 
dense of information already, but I do agree that venv page is most likely not 
where developers would look first.

I'm a little embarrassed to say this, but I didn't even know that 
sys.executable existed.

So maybe the better way to document this would be to mention sys.executable on 
the subprocess page, just as a sure way find where the current interpreter 
lives and then go into detail why this behavior difference is on the venv page. 
Maybe have them linked to each other?

What do you think about this?

--
components:  -Documentation
versions: +Python 3.7, Python 3.8 -Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Bernat Gabor]

Bernat Gabor  added the comment:

Taking a look at the existing documentation this should probably be mentioned 
under the args part of 
https://docs.python.org/3/library/subprocess.html#popen-constructor, not? 
(that's where other similar gotchas are explained too)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Paul Moore]

Paul Moore  added the comment:

Agreed, this is just "normal Windows behaviour". It's definitely surprising and 
non-obvious (I saw this issue and it never even occurred to me that this was 
what was happening) so I'd support documenting it somewhere. My instinct is 
that the subprocess documentation is the right place, but I suspect that in 
practice, people hitting this issue with a virtual environment won't spot 
anything we write there, so it'll end up just being a "well, we told you so" 
measure rather than helping users in a practical sense.

So I'd agree with Steve's comment - Mark, if you can suggest anywhere we could 
document this which you *would* have spotted when you encountered this issue, 
that would be great.

Hmm, maybe we could document in venv that the Python executable in a venv 
redirects to the base Python, so that the "application directory" for a script 
run from a venv is the base executable's directory, not the venv - with a link 
to the MS documentation that explains how this affects path searches? (IIRC, 
we've been hesitant to document the redirection as it's an "implementation 
detail", but this is clearly a case where it affects user-visible behaviour, so 
we can't really use that argument any more...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Steve Dower]

Steve Dower  added the comment:

This is due to how Windows resolves relative paths when creating a new process. 
It *always* looks in the current application directory first, which with this 
setup will be the original Python executable rather than the venv redirector.

The best fix is to pass sys.executable instead of "python", which is more 
reliable on every OS and under every configuration. I've heard multiple people 
say that they prefer to launch venvs directly rather than modifying their 
environment in order to put it first on PATH.

If you are deliberately trying to use PATH to resolve the user's default 
"python" (which may not be in any way related to the one that's currently 
running), then use shutil.which() first and pass the full path to subprocess.

There's no bug to fix here, but if someone wants to add a section to the docs 
(and ideally, Mark or someone who has hit this problem can tell us which docs 
they read where they might have seen it) explaining that passing "python" to 
subprocess is a bad idea, feel free.

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 3.10, Python 3.9 -Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Bernat Gabor]

Bernat Gabor  added the comment:

I wonder if this is an interaction issue with the builtin Windows provided 
python executable?

--
nosy: +eryksun, gaborjbernat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42041] venv subprocess call to python resolves to wrong interpreter

[Mark Keller]

New submission from Mark Keller :

Hey,

Wanted to report this weird behavior I saw recently. Let me first explain how 
to reproduce and then talk about where I think the issue comes from.

I attached a tar.gz file with 2 Python files in it.

Here's how to reproduce:
0. Make sure that pytest is unavailable in your command line, or please edit to 
another library that you don't have available from another Python install (make 
sure to update the import statement in `script1.py`).
1. Untar the 2 scripts into current working directory. Note: I reproduced on a 
Windows VM (Version 10.0.18363 Build 18363).
2. Make a Python 3.7, or 3.8 environment venv. Note: I used Python 3.8.2.
3. Activate venv and install pytest in it.
4. Run `python run_scripts.py` and observe the error.

Note: The same thing works on Python 3.6 as expected.

My team observed this issue when virtualenv released this Monday. It changed 
how it works with Windows and Python 3.7+, before it was not using the Windows 
redirect script generated by venv and just worked like how older versions on 
Windows do.
But what we are seeing is that if you start a subprocess call to `python` then 
that gets resolved to the system wide Python binary without the venv 
site-packages being in PYTHONPATH.
So the subprocess will fail when importing pytest in the subprocess.

If this is not clear please don't hesitate to ask me for clarification, or 
refer to https://github.com/pypa/virtualenv/issues/1981 for more debugging info.

Another interesting thing that I noticed is that if you make a subprocess call 
to `pytest` instead then that gets resolved correctly.

--
components: Windows
files: repro.tar.gz
messages: 378665
nosy: keller00, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware
priority: normal
severity: normal
status: open
title: venv subprocess call to python resolves to wrong interpreter
versions: Python 3.7, Python 3.8
Added file: https://bugs.python.org/file49520/repro.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com