[PyInstaller] Python, Windows and no console environments

[Coyot Linden (Glenn Glazer)]

We discovered that code which ran perfectly correctly by executing the scripts in the POSIX environment or via python on the Windows command prompt would fail when compiled using the -w flag to PyInstaller. This flag prevents the application from launching a console window when the application starts, which we clearly did not want with our client. The problem occurs when a caller executes Python's subprocess and the caller tries to implicitly or explictly access stderr or stdin on Windows. Because this originally occurred in our code with a caller of an imported module our test code used that mode.  We believe it can be reproduced without the caller.  We also believe that this error is a deep down result in the pythonw implementation on Windows and PyInstaller just makes it more obvious.  Some trials suggest that pythonw produces the same failure cases, albeit with somewhat different output. To the PyInstaller team: we found (and is reproducible with the code in the appendices below) that even with -w, a console window flashes for a tiny fraction of a second.  Is there any way to eliminate that completely? ----- So beginning with code that fails that shows the example, we have: Sample problematic caller code: #!/usr/bin/env python import cgitb import os.path import subwrapper import sys cwd = os.path.dirname(os.path.realpath(str(sys.executable))) cgitb.enable(logdir=cwd, format='text') print subwrapper.getMachineID() Sample problematic imported module code: #!/usr/bin/env python import subprocess def getMachineID():    return subprocess.check_output(['wmic','csproduct','get','UUID'])  Sample output under python: >python caller.py UUID D8F59000-4F39-0000-0000-000000000000 Sample output when compiled with the -y --clean --onefile flags: >caller.exe UUID D8F59000-4F39-0000-0000-000000000000 Sample output when compiled with the -y -w --clean --onefile flags: No output.  Failure to execute script window appears (the name depends on the script name): Note that this error message is specific to PyInstaller, it comes from https://github.com/pyinstaller/pyinstaller/blob/a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411 . As noted above, we ran the toy scripts under cgitb tracing to obtain detailed call stack information which does not normally appear when that Window appears.  Full cgitb output is below, but the important part is: Traceback (most recent call last):   File "caller.py", line 11, in <module>   File "subwrapper.py", line 6, in getMachineID   File "subprocess.py", line 566, in check_output   File "subprocess.py", line 702, in __init__   File "subprocess.py", line 823, in _get_handles WindowsError: [Error 6] The handle is invalid Researching that error message led us to: https://github.com/incuna/django-wkhtmltopdf/issues/91#issuecomment-179080434 which points at the problem being related to the passing of filehandles, but suggests hacking subprocess.py, which we were extremely reluctant to do for production software. We also found: http://stackoverflow.com/questions/337870/python-subprocess-call-fails-when-using-pythonw-exe which states "sys.stdin and sys.stdout handles are invalid because pythonw does not provide console support as it runs as a deamon, so default arguments of subprocess.call() are failing. Deamon (sic) programs close stdin/stdout/stderr purposedly and use logging instead, so that you have to manage this yourself: I would suggest to use subprocess.PIPE." The last comment at the end of that StackOverflow indicates that people have been running into this problem with pythonw and PyInstaller since 2014. Putting these things together, we realized that -w is destroying the stderr filehandle. We now do subprocess calls this way:     with open(os.path.join(cwd,'output'),'a') as bar:        with open(os.devnull) as nullin:            try:                foo = subprocess.check_output(['wmic','csproduct','get','UUID'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin)                print >>bar, foo which is to say that we never, ever use stdin or stderr on Windows for anything, explicitly OR implicitly through default options.  E.g., passing None to check_output() for a handle or not specifying a handle causes it to inherit from the parent and there is nothing to inherit and hence the invalid handle error message.  No print statements, either.  All debugging must go to logs. Best, coyot Appendix A - Correct Version of caller.py #!/usr/bin/env python import cgitb import os.path import subwrapper import sys cwd = os.path.dirname(os.path.realpath(str(sys.executable))) cgitb.enable(logdir=cwd, format='text') with open(os.path.join(cwd,'caller_output'),'a') as bar:     print >>bar, subwrapper.getMachineID() Appendix B - Correct Version of subwrapper.py Note that while this uses subprocess.PIPE for stderr, in practice we use a file handle to the log file to capture any potentially diagnostic output. #!/usr/bin/env python import cgitb import os import os.path import subprocess import sys def getMachineID():    #return subprocess.check_output(['wmic','csproduct','get','UUID'])    foo = ""    #note sys.executable works in the compiled environment.  If you try this with    #pythonw caller.py, it will write its log files to wherever python is and probably not    #what you want    cwd = os.path.dirname(os.path.realpath(str(sys.executable)))    cgitb.enable(logdir=cwd, format='text')    with open(os.path.join(cwd,'output'),'a') as bar:        with open(os.devnull) as nullin:            try:                foo = subprocess.check_output(['wmic','csproduct','get','UUID'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin)            except Exception, e:                 print >>bar, repr(e)    return foo if __name__ == "__main__":     cwd = os.path.dirname(os.path.realpath(str(sys.executable)))     with open(os.path.join(cwd,'output'),'a') as bar:         print >>bar, getMachineID() Appendix C - output from caller_output >cat caller_output UUID D8F59000-4F39-0000-0000-000000000000 Appendix D - cgitb output N.B., the code lives in a subdirectory of my cygwin homedir, but everything was executed from the Windows command prompt. <type 'exceptions.WindowsError'> Python 2.7.11: c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.exe Wed Mar 15 10:36:25 2017 A problem occurred in a Python script.  Here is the sequence of function calls leading up to the error, in the order they occurred.  c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.py in <module>()  c:\cygwin64\home\coyot\hg\stderrtoy\dist\subwrapper.py in getMachineID()  c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in check_output(*popenargs=(['wmic', 'csproduct', 'get', 'UUID'],), **kwargs={})  c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in __init__(self=<subprocess.Popen object>, args=['wmic', 'csproduct', 'get', 'UUID'], bufsize=0, executable=None, stdin=None, stdout=-1, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)  c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in _get_handles(self=<subprocess.Popen object>, stdin=None, stdout=-1, stderr=None) <type 'exceptions.WindowsError'>: [Error 6] The handle is invalid     __class__ = <type 'exceptions.WindowsError'>     __delattr__ = <method-wrapper '__delattr__' of exceptions.WindowsError object>     __dict__ = {}     __doc__ = 'MS-Windows OS system call failed.'     __format__ = <built-in method __format__ of exceptions.WindowsError object>     __getattribute__ = <method-wrapper '__getattribute__' of exceptions.WindowsError object>     __getitem__ = <method-wrapper '__getitem__' of exceptions.WindowsError object>     __getslice__ = <method-wrapper '__getslice__' of exceptions.WindowsError object>     __hash__ = <method-wrapper '__hash__' of exceptions.WindowsError object>     __init__ = <method-wrapper '__init__' of exceptions.WindowsError object>     __new__ = <built-in method __new__ of type object>     __reduce__ = <built-in method __reduce__ of exceptions.WindowsError object>     __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.WindowsError object>     __repr__ = <method-wrapper '__repr__' of exceptions.WindowsError object>     __setattr__ = <method-wrapper '__setattr__' of exceptions.WindowsError object>     __setstate__ = <built-in method __setstate__ of exceptions.WindowsError object>     __sizeof__ = <built-in method __sizeof__ of exceptions.WindowsError object>     __str__ = <method-wrapper '__str__' of exceptions.WindowsError object>     __subclasshook__ = <built-in method __subclasshook__ of type object>     __unicode__ = <built-in method __unicode__ of exceptions.WindowsError object>     args = (6, 'The handle is invalid')     errno = 9     filename = None     message = ''     strerror = 'The handle is invalid'     winerror = 6 The above is a description of an error in a Python program.  Here is the original traceback: Traceback (most recent call last):   File "caller.py", line 11, in <module>   File "subwrapper.py", line 6, in getMachineID   File "subprocess.py", line 566, in check_output   File "subprocess.py", line 702, in __init__   File "subprocess.py", line 823, in _get_handles WindowsError: [Error 6] The handle is invalid -- You received this message because you are subscribed to the Google Groups "PyInstaller" group. To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller+unsubscr...@googlegroups.com. To post to this group, send email to pyinstaller@googlegroups.com. Visit this group at https://groups.google.com/group/pyinstaller. For more options, visit https://groups.google.com/d/optout.