No, I hadn't seen that.

Thanks for the pointer!

Best,

coyot


On 3/16/17 10:07, Jones, Bryan wrote:
This link might be relevant -- https://github.com/pyinstaller/pyinstaller/wiki/Recipe-subprocess

Have you tried those suggestions?

On Thu, Mar 16, 2017 at 10:47 AM, Coyot Linden (Glenn Glazer) <co...@lindenlab.com> wrote:
We discovered that code which ran perfectly correctly by executing the scripts in the POSIX environment or via python on the Windows command prompt would fail when compiled using the -w flag to PyInstaller. This flag prevents the application from launching a console window when the application starts, which we clearly did not want with our client. The problem occurs when a caller executes Python's subprocess and the caller tries to implicitly or explictly access stderr or stdin on Windows.

Because this originally occurred in our code with a caller of an imported module our test code used that mode.  We believe it can be reproduced without the caller. 

We also believe that this error is a deep down result in the pythonw implementation on Windows and PyInstaller just makes it more obvious.  Some trials suggest that pythonw produces the same failure cases, albeit with somewhat different output.

To the PyInstaller team: we found (and is reproducible with the code in the appendices below) that even with -w, a console window flashes for a tiny fraction of a second.  Is there any way to eliminate that completely?

-----

So beginning with code that fails that shows the example, we have:

Sample problematic caller code:

#!/usr/bin/env python

import cgitb
import os.path
import subwrapper
import sys

cwd = os.path.dirname(os.path.realpath(str(sys.executable)))
cgitb.enable(logdir=cwd, format='text')

print subwrapper.getMachineID()


Sample problematic imported module code:

#!/usr/bin/env python

import subprocess

def getMachineID():
   return subprocess.check_output(['wmic','csproduct','get','UUID'])


 Sample output under python:

>python caller.py
UUID
D8F59000-4F39-0000-0000-000000000000


Sample output when compiled with the -y --clean --onefile flags:

>caller.exe
UUID
D8F59000-4F39-0000-0000-000000000000


Sample output when compiled with the -y -w --clean --onefile flags:

No output.  Failure to execute script window appears (the name depends on the script name):



Note that this error message is specific to PyInstaller, it comes from https://github.com/pyinstaller/pyinstaller/blob/a70b20e4de6a6817987d28ca9f3201c8105fd858/bootloader/src/pyi_launch.c#L411 .

As noted above, we ran the toy scripts under cgitb tracing to obtain detailed call stack information which does not normally appear when that Window appears.  Full cgitb output is below, but the important part is:

Traceback (most recent call last):
  File "caller.py", line 11, in <module>
  File "subwrapper.py", line 6, in getMachineID
  File "subprocess.py", line 566, in check_output
  File "subprocess.py", line 702, in __init__
  File "subprocess.py", line 823, in _get_handles
WindowsError: [Error 6] The handle is invalid


Researching that error message led us to:

https://github.com/incuna/django-wkhtmltopdf/issues/91#issuecomment-179080434

which points at the problem being related to the passing of filehandles, but suggests hacking subprocess.py, which we were extremely reluctant to do for production software.

We also found:

http://stackoverflow.com/questions/337870/python-subprocess-call-fails-when-using-pythonw-exe

which states

"sys.stdin and sys.stdout handles are invalid because pythonw does not provide console support as it runs as a deamon, so default arguments of subprocess.call() are failing.

Deamon (sic) programs close stdin/stdout/stderr purposedly and use logging instead, so that you have to manage this yourself: I would suggest to use subprocess.PIPE."

The last comment at the end of that StackOverflow indicates that people have been running into this problem with pythonw and PyInstaller since 2014.

Putting these things together, we realized that -w is destroying the stderr filehandle. We now do subprocess calls this way:

    with open(os.path.join(cwd,'output'),'a') as bar:
       with open(os.devnull) as nullin:
           try:
               foo = subprocess.check_output(['wmic','csproduct','get','UUID'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin)
               print >>bar, foo


which is to say that we never, ever use stdin or stderr on Windows for anything, explicitly OR implicitly through default options.  E.g., passing None to check_output() for a handle or not specifying a handle causes it to inherit from the parent and there is nothing to inherit and hence the invalid handle error message.  No print statements, either.  All debugging must go to logs.

Best,

coyot

Appendix A - Correct Version of caller.py

#!/usr/bin/env python

import cgitb
import os.path
import subwrapper
import sys

cwd = os.path.dirname(os.path.realpath(str(sys.executable)))
cgitb.enable(logdir=cwd, format='text')

with open(os.path.join(cwd,'caller_output'),'a') as bar:
    print >>bar, subwrapper.getMachineID()


Appendix B - Correct Version of subwrapper.py

Note that while this uses subprocess.PIPE for stderr, in practice we use a file handle to the log file to capture any potentially diagnostic output.

#!/usr/bin/env python

import cgitb
import os
import os.path
import subprocess
import sys

def getMachineID():
   #return subprocess.check_output(['wmic','csproduct','get','UUID'])
   foo = ""
   #note sys.executable works in the compiled environment.  If you try this with
   #pythonw caller.py, it will write its log files to wherever python is and probably not
   #what you want
   cwd = os.path.dirname(os.path.realpath(str(sys.executable)))
   cgitb.enable(logdir=cwd, format='text')
   with open(os.path.join(cwd,'output'),'a') as bar:
       with open(os.devnull) as nullin:
           try:
               foo = subprocess.check_output(['wmic','csproduct','get','UUID'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=nullin)
           except Exception, e:
                print >>bar, repr(e)
   return foo

if __name__ == "__main__":
    cwd = os.path.dirname(os.path.realpath(str(sys.executable)))
    with open(os.path.join(cwd,'output'),'a') as bar:
        print >>bar, getMachineID()


Appendix C - output from caller_output

>cat caller_output
UUID
D8F59000-4F39-0000-0000-000000000000

Appendix D - cgitb output

N.B., the code lives in a subdirectory of my cygwin homedir, but everything was executed from the Windows command prompt.

<type 'exceptions.WindowsError'>
Python 2.7.11: c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.exe
Wed Mar 15 10:36:25 2017

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 c:\cygwin64\home\coyot\hg\stderrtoy\dist\caller.py in <module>()


 c:\cygwin64\home\coyot\hg\stderrtoy\dist\subwrapper.py in getMachineID()


 c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in check_output(*popenargs=(['wmic', 'csproduct', 'get', 'UUID'],), **kwargs={})


 c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in __init__(self=<subprocess.Popen object>, args=['wmic', 'csproduct', 'get', 'UUID'], bufsize=0, executable=None, stdin=None, stdout=-1, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)


 c:\cygwin64\home\coyot\hg\stderrtoy\dist\subprocess.py in _get_handles(self=<subprocess.Popen object>, stdin=None, stdout=-1, stderr=None)

<type 'exceptions.WindowsError'>: [Error 6] The handle is invalid
    __class__ = <type 'exceptions.WindowsError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.WindowsError object>
    __dict__ = {}
    __doc__ = 'MS-Windows OS system call failed.'
    __format__ = <built-in method __format__ of exceptions.WindowsError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.WindowsError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.WindowsError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.WindowsError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.WindowsError object>
    __init__ = <method-wrapper '__init__' of exceptions.WindowsError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.WindowsError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.WindowsError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.WindowsError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.WindowsError object>
    __setstate__ = <built-in method __setstate__ of exceptions.WindowsError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.WindowsError object>
    __str__ = <method-wrapper '__str__' of exceptions.WindowsError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.WindowsError object>
    args = (6, 'The handle is invalid')
    errno = 9
    filename = None
    message = ''
    strerror = 'The handle is invalid'
    winerror = 6

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "caller.py", line 11, in <module>
  File "subwrapper.py", line 6, in getMachineID
  File "subprocess.py", line 566, in check_output
  File "subprocess.py", line 702, in __init__
  File "subprocess.py", line 823, in _get_handles
WindowsError: [Error 6] The handle is invalid


--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller+unsubscribe@googlegroups.com.
To post to this group, send email to pyinstaller@googlegroups.com.
Visit this group at https://groups.google.com/group/pyinstaller.
For more options, visit https://groups.google.com/d/optout.



--
Bryan A. Jones, Ph.D.
Associate Professor
Department of Electrical and Computer Engineering
231 Simrall / PO Box 9571
Mississippi State University
Mississippi state, MS 39762
http://www.ece.msstate.edu/~bjones
bjones AT ece DOT msstate DOT edu
voice 662-325-3149
fax 662-325-2298

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller+unsubscr...@googlegroups.com.
To post to this group, send email to pyinstaller@googlegroups.com.
Visit this group at https://groups.google.com/group/pyinstaller.
For more options, visit https://groups.google.com/d/optout.

Reply via email to