[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Eryk Sun]

Eryk Sun  added the comment:

> I don't understand the wording proposed (that seem backwards to me?)

Thanks. Looks like I inverted the logic of the quoted paragraph. It should have 
been a "relative path with a slash in it" is resolved against the current 
working directory, not "without a slash".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[ban]

ban  added the comment:

>> "In particular, the function looks for executable (or for the 
>> first item in args) relative to cwd if the executable path is 
>> a relative path." 
>
> For POSIX, this should be stated as a "relative path without a slash in
> it" or a "relative path without a directory in it". An unqualified
> filename is a relative path that won't be resolved against `cwd`,
> unless there's a "." entry in PATH.

While I don't understand the wording proposed (that seem backwards to me?), I 
do think it would be important to fix this.

I just got puzzled and spent some effort writing a workaround *not to look in 
cwd* for a bare command name, before I got so skeptical I actually tried 
empirically what Python would do -- and see the behavior was sensible and `cwd` 
was taken into account *only if the executable had a path component in it*.

Any other behavior is annoying IMO as it means using a `cwd` *requires* one to 
pass in an absolute path to the executable not to risk running an unexpected 
executable, which basically makes support for looking up executable in the 
system PATH unusable if using a `cwd`.
It would also be somewhat inconsistent with the idea that `cwd` only *changes* 
the current directory prior to execution, as it would suggest the behavior is 
not the same when using `cwd=None` and `cwd=os.getcwd()`.

So I'd suggest amending the wording in some way, maybe something like
"In particular, the function looks for executable (or for the first item in 
args) relative to cwd if the executable has an explicit relative path."
or something like that, possibly even dropping in an example.
The problem with the current wording is that usually an unqualified name *is* a 
relative path, whereas here one should somehow understand that it means "if it 
has a path component and that path component is relative", or the more 
down-to-earthy "if the executable starts with './' or '../'".

--
nosy: +ban

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

For our subprocess docs, Eryk's text:

"""
For POSIX, ``executable`` should be stated as a "relative path without a slash 
in it" or a "relative path without a directory in it". An unqualified filename 
is a relative path that won't be resolved against ``cwd``, unless there's a "." 
entry in PATH.

For Windows, the use of CreateProcess() is documented. It could be stated more 
explicitly that ``executable``, ``args`` / ``list2cmdline(args)``, ``env``, and 
``cwd`` are passed directly to CreateProcess() as lpApplicationName, 
lpCommandLine, lpEnvironment, and lpCurrentDirectory.
"""

is quite reasonable.  I wouldn't include your long notes.  But a link to a MSDN 
article explaining that would be useful at the end of the Windows paragraph.

For the POSIX case we should describe which PATH is used.  The current one, or 
the one set in a ``env`` dict.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Eryk Sun]

Eryk Sun  added the comment:

> Python is conceptually multi-platform, so its behavior on 
> Linux and Windows should be as much consistent as possible.

It's not expected for the behavior of all Popen() parameters to be the same on 
all platforms. For example, the behavior and capabilities of shell=True are 
different in Windows vs POSIX. But I think a basic set of parameters should 
have been singled out for cross-platform consistency -- at least in the default 
case. Unfortunately it wasn't designed that way. New behavior that's consistent 
with POSIX can be implemented, but at this point it would have to be enabled by 
a parameter. 

> "In particular, the function looks for executable (or for the 
> first item in args) relative to cwd if the executable path is 
> a relative path." 

For POSIX, this should be stated as a "relative path without a slash in it" or 
a "relative path without a directory in it". An unqualified filename is a 
relative path that won't be resolved against `cwd`, unless there's a "." entry 
in PATH.

For Windows, the use of CreateProcess() is documented. It could be stated more 
explicitly that `executable`, `args` / list2cmdline(args), `env`, and `cwd` are 
passed directly to CreateProcess() as lpApplicationName, lpCommandLine, 
lpEnvironment, and lpCurrentDirectory.

Here are some notes and corrections about the documentation of lpCommandLine, 
in particular the following paragraph:

If the file name does not contain an extension, .exe is appended.
Therefore, if the file name extension is .com, this parameter must
include the .com extension. If the file name ends in a period (.)
with no extension, or if the file name contains a path, .exe is
not appended. If the file name does not contain a directory path,
the system searches for the executable file in the following
sequence [1][2][3]:

* %__APPDIR__%
* %__CD__% [4]
* %SystemRoot%\System32
* %SystemRoot%\System
* %SystemRoot%
* %PATH% (machine/user extended search sequence)

[1] The search sequence is rewritten here succinctly using
environment variables in the current process, including the
virtual variables __APPDIR__ (application directory) and
__CD__ (current directory), which are supported by WinAPI
GetEnvironmentVariableW().

[2] A path name is resolved by searching for it if it isn't fully
qualified and doesn't explicitly begin with the current
directory (".") or its parent (".."). Note that, unlike POSIX,
a relative path name is resolved by searching for it even if
it contains a directory component (i.e. a slash or backslash).
For example, "spam\eggs.exe" is resolved by looking for
r"%__APPDIR__%\spam\eggs.exe" and so on.

[3] If a path name has to be resolved by searching, and its final
component does not contain a "." character, then ".exe" is
appended to the name. On the other hand, if a path name does
not need to be resolved by searching, because it's fully
qualified or the first component is "." or "..", then if the 
given path name doesn't exist, it also looks for the name with
".exe" appended, even if the final component of the path name
contains a "." character.

[4] If "NoDefaultCurrentDirectoryInExePath" is defined in the
environment and the path name does not contain a directory
component (i.e. no slash or backslash), then the current
directory is excluded from the search sequence.

That %__APPDIR__% always takes precedence means that 
subprocess.Popen(['python']) runs the Python version of the current process, 
regardless of PATH, unless shell=True is used. 

The implementation of lpApplicationName (executable) and lpCurrentDirectory 
(cwd) means that argv[0] in the child process, as parsed from its command line, 
does not necessarily resolve to a name in the filesystem. Windows supports 
GetModuleFileNameW(NULL, ...) to get the path of the application executable.

--
versions: +Python 3.10, Python 3.9 -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee:  -> gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Chris Jerdonek]

Change by Chris Jerdonek :


--
assignee: chris.jerdonek -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Jan Lachnitt]

Jan Lachnitt  added the comment:

Nobody responds yet, so I will.

I think that the basic proposal was made by Chris Jerdonek in msg171692 already 
on 2012-10-01: First document both behaviors and then discuss the possible 
harmonization. I think the proposal was good and further discussion has 
confirmed this.

Chris Jerdonek, to whom this issue is assigned, last commented it on 
2012-12-18. Shouldn't the issue be assigned to somebody else?

By the way, the related env issue is #8557.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Thanks for pointing me at this issue Ned.  It sounds like there is a behavior 
difference between Windows and POSIX systems related to the current directory 
and/or which process environment is used by the system call that launches the 
new process to find the executable.

It seems to have existed "forever" in subprocess module API terms, so I don't 
know if we should reconcile the corner cases when cwd and/or env are supplied 
to a single cross platform behavior as that could break existing code.  Such a 
behavior change _could_ be made but be 3.8 specific.  BUT:  If we did that, it 
becomes a challenge for people writing code that needs to work on multiple 
Python versions.  Popen growing yet another bool flag parameter to choose the 
new behavior is possible, but quite ugly (and unusable on multi-python version 
code).

I think we should start out by living with the difference - document these 
platform specific corner case behaviors to minimize surprise.

If we want to provide a way for people to have the same behavior on both, we 
should document a recommended way to do that.  I believe that entails telling 
people get an absolute path to their executable themselves before launching the 
subprocess as that should work the same no matter the cwd or environment?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Ned Deily]

Change by Ned Deily :


--
nosy: +gregory.p.smith
versions: +Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Damon Atkins]

Damon Atkins  added the comment:

>From https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425.aspx

Note Python is using CreateProcess(), consider using CreateProcessW()

The Unicode version of this function, CreateProcessW, can modify the contents 
of this string. Therefore, this parameter cannot be a pointer to read-only 
memory (such as a const variable or a literal string). If this parameter is a 
constant string, the function may cause an access violation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Jan Lachnitt]

Jan Lachnitt  added the comment:

@eryksun: Sorry for my late reply, apparently I did not have time to reply in 
2017. I see your point, but still I think that Python is conceptually 
multi-platform, so its behavior on Linux and Windows should be as much 
consistent as possible.

I am not the one to decide which one of the two possible behaviors shall be the 
correct one. The current documentation 
 describes 
the behavior on Linux: "In particular, the function looks for executable (or 
for the first item in args) relative to cwd if the executable path is a 
relative path." If this is chosen as the correct behavior, then the behavior on 
Windows is incorrect.

@Damon Atkins: Thank you for reminding this issue, but I suspect your proposed 
solution of being thread-unsafe. I propose another solution: On Windows, Python 
should resolve the executable path itself (taking cwd and env into account) and 
then pass the absolute path to CreateProcess().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Damon Atkins]

Damon Atkins  added the comment:

See also https://bugs.python.org/msg262399

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Damon Atkins]

Damon Atkins  added the comment:

I see from this. That this is still an issue
https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1146

Is it not a solution to
save current directory location
chdir(cwd) before calling  _winapi.CreateProcess()
restore the original directory.

This will result in the cwd being searched for the executable, which most 
people would expect to happen.  It seems CreateProcess does not change to cwd 
until after the file is checked for existence or loaded.

--
nosy: +Damon Atkins

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Eryk Sun]

Eryk Sun added the comment:

The Unix implementation of subprocess.Popen follows the behavior of os.execvpe, 
which is an outlier. Other execvpe implementations, such as the one added to 
glibc in 2009, search PATH in the current environment instead of the passed 
environment. As such, and given the natural expectations of a Windows 
programmer, I do not see the current behavior of the Windows implementation as 
incorrect. It's a documentation bug.

On a related note, the Popen documentation for Windows should also mention that 
defining the environment variable NoDefaultCurrentDirectoryInExePath removes 
the current directory from the executable search path, in both CreateProcess 
and cmd.exe (i.e. w/ shell=True). This feature was introduced in Windows Vista, 
so it applies to Python 3.5+. 

> Python actually executes the program, but argv[0] is inconsistent with
> cwd. Imagine that the called program wants to resolve its own path:
> It joins cwd and argv[0] and gets 
> "C:\Users\Jenda\Bug 
> reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe"

A Windows program would call GetModuleFileName with hModule as NULL, which 
returns the path of the process executable. There's also the pseudo-environment 
variable __APPDIR__. 

Using argv[0] from the command line would be unreliable. For example:

>>> _ = run('"spam & eggs" /c echo %__APPDIR__%',
... executable=os.environ['ComSpec'])
C:\Windows\system32\

>>> _ = run('"spam & eggs" -m calendar 2017 2',
... executable=sys.executable)
   February 2017
Mo Tu We Th Fr Sa Su
   1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows

[Martin Panter]

Changes by Martin Panter :


--
components: +Windows
stage: test needed -> needs patch
title: subprocess.Popen(cwd) documentation -> subprocess.Popen(cwd) 
documentation: Posix vs Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com