I have also raised an issue on the cpython manuals, http://bugs.python.org/issue29829, requesting a note be added to the subprocess documentation.
Steve On 16/03/2017 17:24, Coyot Linden (Glenn Glazer) wrote: > No, I hadn't seen that. > > Thanks for the pointer! > > Best, > > coyot > > > On 3/16/17 10:07, Jones, Bryan wrote: >> This link might be relevant >> -- https://github.com/pyinstaller/pyinstaller/wiki/Recipe-subprocess >> >> Have you tried those suggestions? >> >> On Thu, Mar 16, 2017 at 10:47 AM, Coyot Linden (Glenn Glazer) >> <[email protected] <mailto:[email protected]>> wrote: >> >> We discovered that code which ran perfectly correctly by executing >> the scripts in the POSIX environment or via python on the Windows >> command prompt would fail when compiled using the -w flag to >> PyInstaller. This flag prevents the application from launching a >> console window when the application starts, which we clearly did >> not want with our client. The problem occurs when a caller >> executes Python's subprocess and the caller tries to implicitly or >> explictly access stderr or stdin on Windows. >> >> Because this originally occurred in our code with a caller of an >> imported module our test code used that mode. We believe it can >> be reproduced without the caller. >> >> We also believe that this error is a deep down result in the >> pythonw implementation on Windows and PyInstaller just makes it >> more obvious. Some trials suggest that pythonw produces the same >> failure cases, albeit with somewhat different output. >> >> To the PyInstaller team: we found (and is reproducible with the >> code in the appendices below) that even with -w, a console window >> flashes for a tiny fraction of a second. Is there any way to >> eliminate that completely? >> >> ----- >> >> So beginning with code that fails that shows the example, we have: >> >> Sample problematic caller code: >> >> #!/usr/bin/env python >> >> import cgitb >> import os.path >> import subwrapper >> import sys >> >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) >> cgitb.enable(logdir=cwd, format='text') >> >> print subwrapper.getMachineID() >> >> Sample problematic imported module code: >> >> #!/usr/bin/env python >> >> import subprocess >> >> def getMachineID(): >> return subprocess.check_output(['wmic','csproduct','get','UUID']) >> >> Sample output under python: >> >> >python caller.py >> UUID >> D8F59000-4F39-0000-0000-000000000000 >> >> Sample output when compiled with the -y --clean --onefile flags: >> >> >caller.exe >> UUID >> D8F59000-4F39-0000-0000-000000000000 >> >> Sample output when compiled with the -y -w --clean --onefile flags: >> >> No output. Failure to execute script window appears (the name >> depends on the script name): >> >> >> >> Note that this error message is specific to PyInstaller, it comes >> from >> >> https://github.com/pyinstaller/pyinstaller/blob/a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411 >> >> <https://github.com/pyinstaller/pyinstaller/blob/a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411> >> . >> >> As noted above, we ran the toy scripts under cgitb tracing to >> obtain detailed call stack information which does not normally >> appear when that Window appears. Full cgitb output is below, but >> the important part is: >> >> Traceback (most recent call last): >> File "caller.py", line 11, in <module> >> File "subwrapper.py", line 6, in getMachineID >> File "subprocess.py", line 566, in check_output >> File "subprocess.py", line 702, in __init__ >> File "subprocess.py", line 823, in _get_handles >> WindowsError: [Error 6] The handle is invalid >> >> Researching that error message led us to: >> >> >> https://github.com/incuna/django-wkhtmltopdf/issues/91#issuecomment-179080434 >> >> <https://github.com/incuna/django-wkhtmltopdf/issues/91#issuecomment-179080434> >> >> which points at the problem being related to the passing of >> filehandles, but suggests hacking subprocess.py, which we were >> extremely reluctant to do for production software. >> >> We also found: >> >> >> http://stackoverflow.com/questions/337870/python-subprocess-call-fails-when-using-pythonw-exe >> >> <http://stackoverflow.com/questions/337870/python-subprocess-call-fails-when-using-pythonw-exe> >> >> which states >> >> /"sys.stdin and sys.stdout handles are invalid because pythonw >> does not provide console support as it runs as a deamon, so >> default arguments of subprocess.call() are failing.// >> // >> //Deamon (sic) programs close stdin/stdout/stderr purposedly and >> use logging instead, so that you have to manage this yourself: I >> would suggest to use subprocess.PIPE."/ >> >> The last comment at the end of that StackOverflow indicates that >> people have been running into this problem with pythonw and >> PyInstaller since 2014. >> >> Putting these things together, we realized that -w is destroying >> the stderr filehandle. We now do subprocess calls this way: >> >> with open(os.path.join(cwd,'output'),'a') as bar: >> with open(os.devnull) as nullin: >> try: >> foo = >> subprocess.check_output(['wmic','csproduct','get','UUID'], >> stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin) >> print >>bar, foo >> >> which is to say that we never, ever use stdin or stderr on Windows >> for anything, explicitly OR implicitly through default options. >> E.g., passing None to check_output() for a handle or not >> specifying a handle causes it to inherit from the parent and there >> is nothing to inherit and hence the invalid handle error message. >> No print statements, either. All debugging must go to logs. >> >> Best, >> >> coyot >> >> Appendix A - Correct Version of caller.py >> >> #!/usr/bin/env python >> >> import cgitb >> import os.path >> import subwrapper >> import sys >> >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) >> cgitb.enable(logdir=cwd, format='text') >> >> with open(os.path.join(cwd,'caller_output'),'a') as bar: >> print >>bar, subwrapper.getMachineID() >> >> Appendix B - Correct Version of subwrapper.py >> >> Note that while this uses subprocess.PIPE for stderr, in practice >> we use a file handle to the log file to capture any potentially >> diagnostic output. >> >> #!/usr/bin/env python >> >> import cgitb >> import os >> import os.path >> import subprocess >> import sys >> >> def getMachineID(): >> #return subprocess.check_output(['wmic','csproduct','get','UUID']) >> foo = "" >> #note sys.executable works in the compiled environment. If you >> try this with >> #pythonw caller.py, it will write its log files to wherever >> python is and probably not >> #what you want >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) >> cgitb.enable(logdir=cwd, format='text') >> with open(os.path.join(cwd,'output'),'a') as bar: >> with open(os.devnull) as nullin: >> try: >> foo = >> subprocess.check_output(['wmic','csproduct','get','UUID'], >> stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin) >> except Exception, e: >> print >>bar, repr(e) >> return foo >> >> if __name__ == "__main__": >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) >> with open(os.path.join(cwd,'output'),'a') as bar: >> print >>bar, getMachineID() >> >> Appendix C - output from caller_output >> >> >cat caller_output >> UUID >> D8F59000-4F39-0000-0000-000000000000 >> >> Appendix D - cgitb output >> >> N.B., the code lives in a subdirectory of my cygwin homedir, but >> everything was executed from the Windows command prompt. >> >> <type 'exceptions.WindowsError'> >> Python 2.7.11: c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.exe >> Wed Mar 15 10:36:25 2017 >> >> A problem occurred in a Python script. Here is the sequence of >> function calls leading up to the error, in the order they occurred. >> >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.py in <module>() >> >> >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subwrapper.py in >> getMachineID() >> >> >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in >> check_output(*popenargs=(['wmic', 'csproduct', 'get', 'UUID'],), >> **kwargs={}) >> >> >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in >> __init__(self=<subprocess.Popen object>, args=['wmic', >> 'csproduct', 'get', 'UUID'], bufsize=0, executable=None, >> stdin=None, stdout=-1, stderr=None, preexec_fn=None, >> close_fds=False, shell=False, cwd=None, env=None, >> universal_newlines=False, startupinfo=None, creationflags=0) >> >> >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in >> _get_handles(self=<subprocess.Popen object>, stdin=None, >> stdout=-1, stderr=None) >> >> <type 'exceptions.WindowsError'>: [Error 6] The handle is invalid >> __class__ = <type 'exceptions.WindowsError'> >> __delattr__ = <method-wrapper '__delattr__' of >> exceptions.WindowsError object> >> __dict__ = {} >> __doc__ = 'MS-Windows OS system call failed.' >> __format__ = <built-in method __format__ of >> exceptions.WindowsError object> >> __getattribute__ = <method-wrapper '__getattribute__' of >> exceptions.WindowsError object> >> __getitem__ = <method-wrapper '__getitem__' of >> exceptions.WindowsError object> >> __getslice__ = <method-wrapper '__getslice__' of >> exceptions.WindowsError object> >> __hash__ = <method-wrapper '__hash__' of >> exceptions.WindowsError object> >> __init__ = <method-wrapper '__init__' of >> exceptions.WindowsError object> >> __new__ = <built-in method __new__ of type object> >> __reduce__ = <built-in method __reduce__ of >> exceptions.WindowsError object> >> __reduce_ex__ = <built-in method __reduce_ex__ of >> exceptions.WindowsError object> >> __repr__ = <method-wrapper '__repr__' of >> exceptions.WindowsError object> >> __setattr__ = <method-wrapper '__setattr__' of >> exceptions.WindowsError object> >> __setstate__ = <built-in method __setstate__ of >> exceptions.WindowsError object> >> __sizeof__ = <built-in method __sizeof__ of >> exceptions.WindowsError object> >> __str__ = <method-wrapper '__str__' of exceptions.WindowsError >> object> >> __subclasshook__ = <built-in method __subclasshook__ of type >> object> >> __unicode__ = <built-in method __unicode__ of >> exceptions.WindowsError object> >> args = (6, 'The handle is invalid') >> errno = 9 >> filename = None >> message = '' >> strerror = 'The handle is invalid' >> winerror = 6 >> >> The above is a description of an error in a Python program. Here is >> the original traceback: >> >> Traceback (most recent call last): >> File "caller.py", line 11, in <module> >> File "subwrapper.py", line 6, in getMachineID >> File "subprocess.py", line 566, in check_output >> File "subprocess.py", line 702, in __init__ >> File "subprocess.py", line 823, in _get_handles >> WindowsError: [Error 6] The handle is invalid >> >> -- >> You received this message because you are subscribed to the Google >> Groups "PyInstaller" group. >> To unsubscribe from this group and stop receiving emails from it, >> send an email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/pyinstaller >> <https://groups.google.com/group/pyinstaller>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> >> >> -- >> Bryan A. Jones, Ph.D. >> Associate Professor >> Department of Electrical and Computer Engineering >> 231 Simrall / PO Box 9571 >> Mississippi State University >> Mississippi state, MS 39762 >> http://www.ece.msstate.edu/~bjones <http://www.ece.msstate.edu/%7Ebjones> >> bjones AT ece DOT msstate DOT edu >> voice 662-325-3149 >> fax 662-325-2298 > > -- > You received this message because you are subscribed to the Google > Groups "PyInstaller" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/pyinstaller. > For more options, visit https://groups.google.com/d/optout. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. -- You received this message because you are subscribed to the Google Groups "PyInstaller" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/pyinstaller. For more options, visit https://groups.google.com/d/optout.
