Steve, Great idea -- thanks!
Bryan On Thu, Mar 16, 2017 at 12:51 PM, Steve Barnes <gadgetst...@live.co.uk> wrote: > I have also raised an issue on the cpython manuals, > http://bugs.python.org/issue29829, requesting a note be added to the > subprocess documentation. > > Steve > > On 16/03/2017 17:24, Coyot Linden (Glenn Glazer) wrote: > > No, I hadn't seen that. > > > > Thanks for the pointer! > > > > Best, > > > > coyot > > > > > > On 3/16/17 10:07, Jones, Bryan wrote: > >> This link might be relevant > >> -- https://github.com/pyinstaller/pyinstaller/wiki/Recipe-subprocess > >> > >> Have you tried those suggestions? > >> > >> On Thu, Mar 16, 2017 at 10:47 AM, Coyot Linden (Glenn Glazer) > >> <co...@lindenlab.com <mailto:co...@lindenlab.com>> wrote: > >> > >> We discovered that code which ran perfectly correctly by executing > >> the scripts in the POSIX environment or via python on the Windows > >> command prompt would fail when compiled using the -w flag to > >> PyInstaller. This flag prevents the application from launching a > >> console window when the application starts, which we clearly did > >> not want with our client. The problem occurs when a caller > >> executes Python's subprocess and the caller tries to implicitly or > >> explictly access stderr or stdin on Windows. > >> > >> Because this originally occurred in our code with a caller of an > >> imported module our test code used that mode. We believe it can > >> be reproduced without the caller. > >> > >> We also believe that this error is a deep down result in the > >> pythonw implementation on Windows and PyInstaller just makes it > >> more obvious. Some trials suggest that pythonw produces the same > >> failure cases, albeit with somewhat different output. > >> > >> To the PyInstaller team: we found (and is reproducible with the > >> code in the appendices below) that even with -w, a console window > >> flashes for a tiny fraction of a second. Is there any way to > >> eliminate that completely? > >> > >> ----- > >> > >> So beginning with code that fails that shows the example, we have: > >> > >> Sample problematic caller code: > >> > >> #!/usr/bin/env python > >> > >> import cgitb > >> import os.path > >> import subwrapper > >> import sys > >> > >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) > >> cgitb.enable(logdir=cwd, format='text') > >> > >> print subwrapper.getMachineID() > >> > >> Sample problematic imported module code: > >> > >> #!/usr/bin/env python > >> > >> import subprocess > >> > >> def getMachineID(): > >> return subprocess.check_output(['wmic','csproduct','get','UUID' > ]) > >> > >> Sample output under python: > >> > >> >python caller.py > >> UUID > >> D8F59000-4F39-0000-0000-000000000000 > >> > >> Sample output when compiled with the -y --clean --onefile flags: > >> > >> >caller.exe > >> UUID > >> D8F59000-4F39-0000-0000-000000000000 > >> > >> Sample output when compiled with the -y -w --clean --onefile flags: > >> > >> No output. Failure to execute script window appears (the name > >> depends on the script name): > >> > >> > >> > >> Note that this error message is specific to PyInstaller, it comes > >> from > >> https://github.com/pyinstaller/pyinstaller/blob/ > a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411 > >> <https://github.com/pyinstaller/pyinstaller/blob/ > a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411> > >> . > >> > >> As noted above, we ran the toy scripts under cgitb tracing to > >> obtain detailed call stack information which does not normally > >> appear when that Window appears. Full cgitb output is below, but > >> the important part is: > >> > >> Traceback (most recent call last): > >> File "caller.py", line 11, in <module> > >> File "subwrapper.py", line 6, in getMachineID > >> File "subprocess.py", line 566, in check_output > >> File "subprocess.py", line 702, in __init__ > >> File "subprocess.py", line 823, in _get_handles > >> WindowsError: [Error 6] The handle is invalid > >> > >> Researching that error message led us to: > >> > >> https://github.com/incuna/django-wkhtmltopdf/issues/91# > issuecomment-179080434 > >> <https://github.com/incuna/django-wkhtmltopdf/issues/91# > issuecomment-179080434> > >> > >> which points at the problem being related to the passing of > >> filehandles, but suggests hacking subprocess.py, which we were > >> extremely reluctant to do for production software. > >> > >> We also found: > >> > >> http://stackoverflow.com/questions/337870/python- > subprocess-call-fails-when-using-pythonw-exe > >> <http://stackoverflow.com/questions/337870/python- > subprocess-call-fails-when-using-pythonw-exe> > >> > >> which states > >> > >> /"sys.stdin and sys.stdout handles are invalid because pythonw > >> does not provide console support as it runs as a deamon, so > >> default arguments of subprocess.call() are failing.// > >> // > >> //Deamon (sic) programs close stdin/stdout/stderr purposedly and > >> use logging instead, so that you have to manage this yourself: I > >> would suggest to use subprocess.PIPE."/ > >> > >> The last comment at the end of that StackOverflow indicates that > >> people have been running into this problem with pythonw and > >> PyInstaller since 2014. > >> > >> Putting these things together, we realized that -w is destroying > >> the stderr filehandle. We now do subprocess calls this way: > >> > >> with open(os.path.join(cwd,'output'),'a') as bar: > >> with open(os.devnull) as nullin: > >> try: > >> foo = > >> subprocess.check_output(['wmic','csproduct','get','UUID'], > >> stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin) > >> print >>bar, foo > >> > >> which is to say that we never, ever use stdin or stderr on Windows > >> for anything, explicitly OR implicitly through default options. > >> E.g., passing None to check_output() for a handle or not > >> specifying a handle causes it to inherit from the parent and there > >> is nothing to inherit and hence the invalid handle error message. > >> No print statements, either. All debugging must go to logs. > >> > >> Best, > >> > >> coyot > >> > >> Appendix A - Correct Version of caller.py > >> > >> #!/usr/bin/env python > >> > >> import cgitb > >> import os.path > >> import subwrapper > >> import sys > >> > >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) > >> cgitb.enable(logdir=cwd, format='text') > >> > >> with open(os.path.join(cwd,'caller_output'),'a') as bar: > >> print >>bar, subwrapper.getMachineID() > >> > >> Appendix B - Correct Version of subwrapper.py > >> > >> Note that while this uses subprocess.PIPE for stderr, in practice > >> we use a file handle to the log file to capture any potentially > >> diagnostic output. > >> > >> #!/usr/bin/env python > >> > >> import cgitb > >> import os > >> import os.path > >> import subprocess > >> import sys > >> > >> def getMachineID(): > >> #return subprocess.check_output(['wmic','csproduct','get','UUID' > ]) > >> foo = "" > >> #note sys.executable works in the compiled environment. If you > >> try this with > >> #pythonw caller.py, it will write its log files to wherever > >> python is and probably not > >> #what you want > >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) > >> cgitb.enable(logdir=cwd, format='text') > >> with open(os.path.join(cwd,'output'),'a') as bar: > >> with open(os.devnull) as nullin: > >> try: > >> foo = > >> subprocess.check_output(['wmic','csproduct','get','UUID'], > >> stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin) > >> except Exception, e: > >> print >>bar, repr(e) > >> return foo > >> > >> if __name__ == "__main__": > >> cwd = os.path.dirname(os.path.realpath(str(sys.executable))) > >> with open(os.path.join(cwd,'output'),'a') as bar: > >> print >>bar, getMachineID() > >> > >> Appendix C - output from caller_output > >> > >> >cat caller_output > >> UUID > >> D8F59000-4F39-0000-0000-000000000000 > >> > >> Appendix D - cgitb output > >> > >> N.B., the code lives in a subdirectory of my cygwin homedir, but > >> everything was executed from the Windows command prompt. > >> > >> <type 'exceptions.WindowsError'> > >> Python 2.7.11: c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.exe > >> Wed Mar 15 10:36:25 2017 > >> > >> A problem occurred in a Python script. Here is the sequence of > >> function calls leading up to the error, in the order they occurred. > >> > >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.py in <module>() > >> > >> > >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subwrapper.py in > >> getMachineID() > >> > >> > >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in > >> check_output(*popenargs=(['wmic', 'csproduct', 'get', 'UUID'],), > >> **kwargs={}) > >> > >> > >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in > >> __init__(self=<subprocess.Popen object>, args=['wmic', > >> 'csproduct', 'get', 'UUID'], bufsize=0, executable=None, > >> stdin=None, stdout=-1, stderr=None, preexec_fn=None, > >> close_fds=False, shell=False, cwd=None, env=None, > >> universal_newlines=False, startupinfo=None, creationflags=0) > >> > >> > >> c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in > >> _get_handles(self=<subprocess.Popen object>, stdin=None, > >> stdout=-1, stderr=None) > >> > >> <type 'exceptions.WindowsError'>: [Error 6] The handle is invalid > >> __class__ = <type 'exceptions.WindowsError'> > >> __delattr__ = <method-wrapper '__delattr__' of > >> exceptions.WindowsError object> > >> __dict__ = {} > >> __doc__ = 'MS-Windows OS system call failed.' > >> __format__ = <built-in method __format__ of > >> exceptions.WindowsError object> > >> __getattribute__ = <method-wrapper '__getattribute__' of > >> exceptions.WindowsError object> > >> __getitem__ = <method-wrapper '__getitem__' of > >> exceptions.WindowsError object> > >> __getslice__ = <method-wrapper '__getslice__' of > >> exceptions.WindowsError object> > >> __hash__ = <method-wrapper '__hash__' of > >> exceptions.WindowsError object> > >> __init__ = <method-wrapper '__init__' of > >> exceptions.WindowsError object> > >> __new__ = <built-in method __new__ of type object> > >> __reduce__ = <built-in method __reduce__ of > >> exceptions.WindowsError object> > >> __reduce_ex__ = <built-in method __reduce_ex__ of > >> exceptions.WindowsError object> > >> __repr__ = <method-wrapper '__repr__' of > >> exceptions.WindowsError object> > >> __setattr__ = <method-wrapper '__setattr__' of > >> exceptions.WindowsError object> > >> __setstate__ = <built-in method __setstate__ of > >> exceptions.WindowsError object> > >> __sizeof__ = <built-in method __sizeof__ of > >> exceptions.WindowsError object> > >> __str__ = <method-wrapper '__str__' of exceptions.WindowsError > >> object> > >> __subclasshook__ = <built-in method __subclasshook__ of type > >> object> > >> __unicode__ = <built-in method __unicode__ of > >> exceptions.WindowsError object> > >> args = (6, 'The handle is invalid') > >> errno = 9 > >> filename = None > >> message = '' > >> strerror = 'The handle is invalid' > >> winerror = 6 > >> > >> The above is a description of an error in a Python program. Here is > >> the original traceback: > >> > >> Traceback (most recent call last): > >> File "caller.py", line 11, in <module> > >> File "subwrapper.py", line 6, in getMachineID > >> File "subprocess.py", line 566, in check_output > >> File "subprocess.py", line 702, in __init__ > >> File "subprocess.py", line 823, in _get_handles > >> WindowsError: [Error 6] The handle is invalid > >> > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "PyInstaller" group. > >> To unsubscribe from this group and stop receiving emails from it, > >> send an email to pyinstaller+unsubscr...@googlegroups.com > >> <mailto:pyinstaller+unsubscr...@googlegroups.com>. > >> To post to this group, send email to pyinstaller@googlegroups.com > >> <mailto:pyinstaller@googlegroups.com>. > >> Visit this group at https://groups.google.com/group/pyinstaller > >> <https://groups.google.com/group/pyinstaller>. > >> For more options, visit https://groups.google.com/d/optout > >> <https://groups.google.com/d/optout>. > >> > >> > >> > >> > >> -- > >> Bryan A. Jones, Ph.D. > >> Associate Professor > >> Department of Electrical and Computer Engineering > >> 231 Simrall / PO Box 9571 > >> Mississippi State University > >> Mississippi state, MS 39762 > >> http://www.ece.msstate.edu/~bjones <http://www.ece.msstate.edu/% > 7Ebjones> > >> bjones AT ece DOT msstate DOT edu > >> voice 662-325-3149 > >> fax 662-325-2298 > > > > -- > > You received this message because you are subscribed to the Google > > Groups "PyInstaller" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to pyinstaller+unsubscr...@googlegroups.com > > <mailto:pyinstaller+unsubscr...@googlegroups.com>. > > To post to this group, send email to pyinstaller@googlegroups.com > > <mailto:pyinstaller@googlegroups.com>. > > Visit this group at https://groups.google.com/group/pyinstaller. > > For more options, visit https://groups.google.com/d/optout. > > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > -- > You received this message because you are subscribed to the Google Groups > "PyInstaller" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pyinstaller+unsubscr...@googlegroups.com. > To post to this group, send email to pyinstaller@googlegroups.com. > Visit this group at https://groups.google.com/group/pyinstaller. > For more options, visit https://groups.google.com/d/optout. > -- Bryan A. Jones, Ph.D. Associate Professor Department of Electrical and Computer Engineering 231 Simrall / PO Box 9571 Mississippi State University Mississippi state, MS 39762 http://www.ece.msstate.edu/~bjones bjones AT ece DOT msstate DOT edu voice 662-325-3149 fax 662-325-2298 Our Master, Jesus Christ, is on his way. He'll show up right on time, his arrival guaranteed by the Blessed and Undisputed Ruler, High King, High God. - 1 Tim. 6:14b-15 (The Message) -- You received this message because you are subscribed to the Google Groups "PyInstaller" group. To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller+unsubscr...@googlegroups.com. To post to this group, send email to pyinstaller@googlegroups.com. Visit this group at https://groups.google.com/group/pyinstaller. For more options, visit https://groups.google.com/d/optout.
