[issue8557] subprocess PATH semantics and portability
Jan Lachnitt added the comment: A related issue exists with cwd: #15533. -- nosy: +pepalogik ___ Python tracker <https://bugs.python.org/issue8557> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows
Jan Lachnitt added the comment: Nobody responds yet, so I will. I think that the basic proposal was made by Chris Jerdonek in msg171692 already on 2012-10-01: First document both behaviors and then discuss the possible harmonization. I think the proposal was good and further discussion has confirmed this. Chris Jerdonek, to whom this issue is assigned, last commented it on 2012-12-18. Shouldn't the issue be assigned to somebody else? By the way, the related env issue is #8557. -- ___ Python tracker <https://bugs.python.org/issue15533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation: Posix vs Windows
Jan Lachnitt added the comment: @eryksun: Sorry for my late reply, apparently I did not have time to reply in 2017. I see your point, but still I think that Python is conceptually multi-platform, so its behavior on Linux and Windows should be as much consistent as possible. I am not the one to decide which one of the two possible behaviors shall be the correct one. The current documentation <https://docs.python.org/3/library/subprocess.html#subprocess.Popen> describes the behavior on Linux: "In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." If this is chosen as the correct behavior, then the behavior on Windows is incorrect. @Damon Atkins: Thank you for reminding this issue, but I suspect your proposed solution of being thread-unsafe. I propose another solution: On Windows, Python should resolve the executable path itself (taking cwd and env into account) and then pass the absolute path to CreateProcess(). -- ___ Python tracker <https://bugs.python.org/issue15533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Jan Lachnitt added the comment:
Thank Wolfgang Maier for reminding this issue and providing various details and
observations. Having taken a look at my old comments (and at others' comments,
too), I feel that the cwd issue deserves a clearer description.
Let's use the following simple C program as the callee:
#include
#include
int main(int argc, char* argv[]) {
char cwd[FILENAME_MAX+1];
for (int i = 0; i < argc; ++i)
printf("argv[%d] = %s\n", i, argv[i]);
getcwd(cwd, FILENAME_MAX);
printf("cwd = %s\n", cwd);
return 0;
}
As is evident, this program merely prints its arguments and working directory.
I have built it using gcc, called it "print_argv+cwd", and placed it in the
"subdir" subdirectory of the current directory.
Next, let's use the following Python 3 script for testing:
import os
from subprocess import run # substitute run->call in Python < 3.5
prg_name = 'print_argv+cwd'
if os.name == 'nt':
prg_name += '.exe'
else:
prg_name = os.path.join('.',prg_name)
dir_name = 'subdir'
def execute(path, cwd):
print('Executing "{}" in "{}":'.format(path,cwd))
try:
run([path], cwd=cwd) # substitute run->call in Python < 3.5
except Exception as err:
print(type(err).__qualname__+':', err)
print('Script\'s cwd =', os.getcwd())
execute(prg_name, dir_name)
execute(os.path.join(dir_name,prg_name), dir_name)
execute(os.path.abspath(os.path.join(dir_name,prg_name)), dir_name)
Output on Linux with Python 3.5.2:
Script's cwd = /home/jenda/Bug reports/Python/subprocess
Executing "./print_argv+cwd" in "subdir":
argv[0] = ./print_argv+cwd
cwd = /home/jenda/Bug reports/Python/subprocess/subdir
Executing "subdir/./print_argv+cwd" in "subdir":
FileNotFoundError: [Errno 2] No such file or directory:
'subdir/./print_argv+cwd'
Executing "/home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd" in
"subdir":
argv[0] = /home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd
cwd = /home/jenda/Bug reports/Python/subprocess/subdir
Output on Windows with Python 3.5.2:
Script's cwd = C:\Users\Jenda\Bug reports\Python\subprocess
Executing "print_argv+cwd.exe" in "subdir":
FileNotFoundError: [WinError 2] Systém nemůže nalézt uvedený soubor
Executing "subdir\print_argv+cwd.exe" in "subdir":
argv[0] = subdir\print_argv+cwd.exe
cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir
Executing "C:\Users\Jenda\Bug
reports\Python\subprocess\subdir\print_argv+cwd.exe" in "subdir":
argv[0] = C:\Users\Jenda\Bug reports\Python\subprocess\subdir\print_argv+cwd.exe
cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir
Summary: On Linux, subprocess.run (or call or Popen) behaves correctly, in
accordance with current documentation. On Windows, both possible relative paths
produce incorrect results. With the first one, relative to "subdir", Python
fails to find the executable. With the other one, relative to the script's cwd,
Python actually executes the program, but argv[0] is inconsistent with cwd.
Imagine that the called program wants to resolve its own path: It joins cwd and
argv[0] and gets "C:\Users\Jenda\Bug
reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe", which is an
incorrect (and nonexistent) path. This is why the cwd issue is not just a
documentation issue.
The only option working correctly on Windows is the last one, using absolute
path of the executable.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue15533>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
Jan Lachnitt added the comment: Knowing that the problem is related to the internal representation of the strings, I have written a short script which reproduces the problem. It is this simple: import os name = 'sub-fcc' wrkdir = 'D:\\Bug reports\\Python\\test' dirname = wrkdir+os.sep+name print(dirname) print(ascii(dirname.encode(unicode_internal))) dirname += os.sep+'bulk' print(dirname) print(ascii(dirname.encode(unicode_internal))) Output: D:\Bug reports\Python\test\sub-fcc b'D\x00:\x00\\\x00B\x00u\x00g\x00 \x00r\x00e\x00p\x00o\x00r\x00t\x00s\x00\\\x00P\x00y\x00t\x00h\x00o\x00n\x00\\\x00t\x00e\x00s\x00t\x00\\\x00s\x00u\x00b\x00-\x00f\x00c\x00c\x00' D:\Bug reports\Python\test\sub-fcc\bulk b'D\x00:\x00\\\x00B\x00u\x00g\x00 \x00r\x00e\x00p\x00o\x00r\x00t\x00s\x00\\\x00P\x00y\x00t\x00h\x00o\x00n\x00\\\x00t\x00e\x00s\x00t\x00\\\x00s\x00u\x00b\x00-\x00f\x00c\x00c\x00\x00\x00\x00\x00\xd8\xa3\x90\x02\x00\x00' The end of the output varies from run to run. It works correctly in Python 3.2.3 64-bit on Windows 8. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17137 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
Jan Lachnitt added the comment: On Windows XP 32-bit: 3.2.3 works, 3.3.0 fails. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17137 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
Jan Lachnitt added the comment: ... print(ctypes.sizeof(ctypes.c_wchar)) _PyObject_Dump=ctypes.pythonapi._PyObject_Dump _PyObject_Dump.argtypes=(ctypes.py_object,) print(_PyObject_Dump(dirname)) print(list(dirname)) in Python 3.3.0 64-bit on Windows 8 produces: 2 object : 'D:\\Bug reports\\Python\\test\\sub-fcc\\bulk' type: str refcount: 3 address : 028AC298 54 ['D', ':', '\\', 'B', 'u', 'g', ' ', 'r', 'e', 'p', 'o', 'r', 't', 's', '\\', 'P', 'y', 't', 'h', 'o', 'n', '\\', 't', 'e', 's', 't', '\\', 's', 'u', 'b', '-', 'f', 'c', 'c', '\\', 'b', 'u', 'l', 'k'] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17137 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
Jan Lachnitt added the comment:
Here is a part of my code (with some comments added):
for struct in ini_structures:
dirname = wrkdir+os.sep+struct.name
if not os.path.isdir(dirname): # This works fine. If the directory
doesn't exist,...
try:
os.mkdir(dirname) # ... it is created here.
except OSError:
raise AutoLEEDError('Cannot create directory '+dirname+'.')
dirname += os.sep+'bulk'# This defines a subdirectory.
if not os.path.isdir(dirname): ## Though it doesn't exist,
os.path.isdir returns True,...
try:
os.mkdir(dirname) # ... so it is not created here.
except OSError:
raise AutoLEEDError('Cannot create directory '+dirname+'.')
fn = dirname+os.sep+'cluster.i' # This defines a filename.
print('Writing file '+fn+'...')
straos = struct.write_bulk_cluster(fn,at_rad) # Here it fails (cannot
write to file).
According to Victor's post, I have inserted these lines before the line marked
## (and added necessary imports):
print('dirname =', dirname)
print('os.path.isdir(dirname) =', os.path.isdir(dirname))
print('nt._isdir(dirname) =', nt._isdir(dirname))
print('stat.S_ISDIR(os.stat(dirname).st_mode) =',
stat.S_ISDIR(os.stat(dirname).st_mode))
print(ascii(dirname.encode(unicode_internal)))
print(ascii(dirname.encode(utf-8)))
Here is the output of these lines (that directory really does not exist but its
parent directory does):
dirname = D:\Bug reports\Python\AutoLEED\default\sub-fcc\bulk
os.path.isdir(dirname) = True
nt._isdir(dirname) = True
stat.S_ISDIR(os.stat(dirname).st_mode) = True
b'D\x00:\x00\\\x00B\x00u\x00g\x00
\x00r\x00e\x00p\x00o\x00r\x00t\x00s\x00\\\x00P\x00y\x00t\x00h\x00o\x00n\x00\\\x00A\x00u\x00t\x00o\x00L\x00E\x00E\x00D\x00\\\x00d\x00e\x00f\x00a\x00u\x00l\x00t\x00\\\x00s\x00u\x00b\x00-\x00f\x00c\x00c\x00\x00\x002\x00\x03\x00\x00\x00\x00\x00'
b'D:\\Bug reports\\Python\\AutoLEED\\default\\sub-fcc\\bulk'
Yeah, the result of ascii(dirname.encode(unicode_internal)) seems to be wrong
(at the end).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17137
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
Jan Lachnitt added the comment: print(ascii(struct.name)) print(ascii(struct.name.encode(unicode_internal))) print(ascii(struct.name.encode(utf-8))) produces: 'sub-fcc' b's\x00u\x00b\x00-\x00f\x00c\x00c\x00' b'sub-fcc' and that looks correct. struct.name originally comes from an ini-file: cp = configparser.ConfigParser(interpolation=None) try: cp.read(filename) ... The ini-file is encoded in pure ASCII (while my Python sources are in UTF-8 with the identification bytes at the beginning of the file). struct.name is the name of a section in this file, as provided by cp.sections() . The name gets through several objects. I am not pasting all the relevant code pieces here because there are too many relevant pieces but they do nothing special (just passing and copying the name). I do not use ctypes. wrkdir is generated from inp_file_name, which is 'default.ini', by this statement: wrkdir = os.path.splitext(os.path.abspath(inp_file_name))[0] BTW, ascii(dirname.encode(unicode_internal)) result is different in this run: b'D\x00:\x00\\\x00B\x00u\x00g\x00 \x00r\x00e\x00p\x00o\x00r\x00t\x00s\x00\\\x00P\x00y\x00t\x00h\x00o\x00n\x00\\\x00A\x00u\x00t\x00o\x00L\x00E\x00E\x00D\x00\\\x00d\x00e\x00f\x00a\x00u\x00l\x00t\x00\\\x00s\x00u\x00b\x00-\x00f\x00c\x00c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17137 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17137] Malfunctioning compiled code in Python 3.3 x64
New submission from Jan Lachnitt: Python 3.3 64-bit seems to compile one of my files incorrectly. Specifically, os.path.isdir returns True for a nonexistent folder. The important point is that the code works correctly when it is performed step-by-step in pdb. Python version: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 OS: Windows 8 64-bit The code works fine in Python 3.2.3 32-bit on Windows XP. My project is quite complex and it interacts with other software packages. I tried to make a reduced test-case but I could not reproduce the problem this way. What files do you need for processing this bug report? Will e.g. the source file in question and the corresponding compiled file (*.pyc) be enough? Or should I upload the whole project here, along with the instructions on how to run it? -- messages: 181465 nosy: pepalogik priority: normal severity: normal status: open title: Malfunctioning compiled code in Python 3.3 x64 type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17137 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Jan Lachnitt added the comment: Hi all, I have solved the problem by using absolute path of the executable. The reason why the executable didn't work properly may be that the executable's relative path was inconsistent with current directory. See the following example (I have made an executable which shows its argv and cwd). If it is called normally, then: argv[0] = phsh0.exe cwd = D:\Jenda\AutoLEED\TESTING\default But if it is called by Python's subprocess.call from D:\Jenda\AutoLEED\TESTING as I want, then: argv[0] = default\phsh0.exe cwd = D:\Jenda\AutoLEED\TESTING\default The executable may be confused by this inconsistency. So it is not the documentation, but Python itself what should be changed. The executable should be searched in cwd on any platform to avoid the inconsistency. I have not yet updated my Python installation, so my results apply to 3.2.3. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Jan Lachnitt added the comment: Hi, I am using Python 3.2.3 on Windows XP. I encountered a problem with subprocess.call when using the cwd parameter. I used Google to look for a solution and I found this issue in Python tracker. But this issue seems absolutely reversed! The subprocess documentation says this: In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path. But this is NOT true. If I use the path relative to cwd, I get Windows Error 2 (system cannot find the file). If I change the executable's path to be relative to the current directory, where the script is running (i.e. NOT the cwd parameter passed to subprocess.call), it works fine. -- nosy: +pepalogik ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Jan Lachnitt added the comment: EDIT: No, it doesn't work fine: the executable reports stack overflow. I thought this had nothing to do with Python, hence I didn't mention it. But If I run the executable without Python, THEN it works fine. But this may be another issue. I'll update to 3.3.0 and then tell you if this has changed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Jan Lachnitt added the comment: Hi Chris, thank for your reply, I didn't see it while writing the edit. Does it mean that on Linux, it will use the cwd? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
