Re: [Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

Hey,

Anatoly, you are free to modify the PEP and code. I do not have any
plans to work on this right now.

Eric

On Mon, Dec 24, 2012 at 04:42:20PM +0300, anatoly techtonik wrote:
 What should I do in case Eric lost interest after his GSoC project for PSF
 appeared as useless for python-dev community? Should I rewrite the proposal
 from scratch?
 
 On Thu, Dec 20, 2012 at 11:18 PM, Brett Cannon br...@python.org wrote:
 
  You cannot rewrite an existing PEP if you are not one of the original
  owners, nor can you add yourself as an author to a PEP without permission
  from the original authors.
 
  And please do not CC the peps mailing list on discussions. It should only
  be used to mail in new PEPs or acceptable patches to PEPs.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

I'm bumping this PEP again in hopes of getting some feedback.

Thanks,
Eric

On Tue, Sep 8, 2009 at 23:52, Eric Pruitt eric.pru...@gmail.com wrote:
 PEP: 3145
 Title: Asynchronous I/O For subprocess.Popen
 Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
 Type: Standards Track
 Content-Type: text/plain
 Created: 04-Aug-2009
 Python-Version: 3.2

 Abstract:

    In its present form, the subprocess.Popen implementation is prone to
    dead-locking and blocking of the parent Python script while waiting on data
    from the child process.

 Motivation:

    A search for python asynchronous subprocess will turn up numerous
    accounts of people wanting to execute a child process and communicate with
    it from time to time reading only the data that is available instead of
    blocking to wait for the program to produce data [1] [2] [3].  The current
    behavior of the subprocess module is that when a user sends or receives
    data via the stdin, stderr and stdout file objects, dead locks are common
    and documented [4] [5].  While communicate can be used to alleviate some of
    the buffering issues, it will still cause the parent process to block while
    attempting to read data when none is available to be read from the child
    process.

 Rationale:

    There is a documented need for asynchronous, non-blocking functionality in
    subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the
    utility of the Python standard library that can be used on Unix based and
    Windows builds of Python.  Practically every I/O object in Python has a
    file-like wrapper of some sort.  Sockets already act as such and for
    strings there is StringIO.  Popen can be made to act like a file by simply
    using the methods attached the the subprocess.Popen.stderr, stdout and
    stdin file-like objects.  But when using the read and write methods of
    those options, you do not have the benefit of asynchronous I/O.  In the
    proposed solution the wrapper wraps the asynchronous methods to mimic a
    file object.

 Reference Implementation:

    I have been maintaining a Google Code repository that contains all of my
    changes including tests and documentation [9] as well as blog detailing
    the problems I have come across in the development process [10].

    I have been working on implementing non-blocking asynchronous I/O in the
    subprocess.Popen module as well as a wrapper class for subprocess.Popen
    that makes it so that an executed process can take the place of a file by
    duplicating all of the methods and attributes that file objects have.

    There are two base functions that have been added to the subprocess.Popen
    class: Popen.send and Popen._recv, each with two separate implementations,
    one for Windows and one for Unix based systems.  The Windows
    implementation uses ctypes to access the functions needed to control pipes
    in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
    the Python interface for file control serves the same purpose.  The
    different implementations of Popen.send and Popen._recv have identical
    arguments to make code that uses these functions work across multiple
    platforms.

    When calling the Popen._recv function, it requires the pipe name be
    passed as an argument so there exists the Popen.recv function that passes
    selects stdout as the pipe for Popen._recv by default.  Popen.recv_err
    selects stderr as the pipe by default. Popen.recv and Popen.recv_err
    are much easier to read and understand than Popen._recv('stdout' ... and
    Popen._recv('stderr' ... respectively.

    Since the Popen._recv function does not wait on data to be produced
    before returning a value, it may return empty bytes.  Popen.asyncread
    handles this issue by returning all data read over a given time
    interval.

    The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
    allow a process to act like a file so that there are no blocking issues
    that can arise from using the stdout and stdin file objects produced from
    a subprocess.Popen call.


 References:

    [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
        http://mail.python.org/pipermail/python-bugs-list/2006-December/
          036524.html

    [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
        http://ivory.idyll.org/blog/feb-07/problems-with-subprocess

    [3] How can I run an external command asynchronously from Python? - Stack
        Overflow
        http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
          command-asynchronously-from-python

    [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
        http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

    [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
        http://docs.python.org/library/subprocess.html

[Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

Alright, I will re-submit with the contents pasted. I never use double
backquotes as I think them rather ugly; that is the work of an editor
or some automated program in the chain. Plus, it also messed up my
line formatting and now I have lines with one word on them... Anyway,
the contents of PEP 3145:

PEP: 3145
Title: Asynchronous I/O For subprocess.Popen
Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
Type: Standards Track
Content-Type: text/plain
Created: 04-Aug-2009
Python-Version: 3.2

Abstract:

In its present form, the subprocess.Popen implementation is prone to
dead-locking and blocking of the parent Python script while waiting on data
from the child process.

Motivation:

A search for python asynchronous subprocess will turn up numerous
accounts of people wanting to execute a child process and communicate with
it from time to time reading only the data that is available instead of
blocking to wait for the program to produce data [1] [2] [3].  The current
behavior of the subprocess module is that when a user sends or receives
data via the stdin, stderr and stdout file objects, dead locks are common
and documented [4] [5].  While communicate can be used to alleviate some of
the buffering issues, it will still cause the parent process to block while
attempting to read data when none is available to be read from the child
process.

Rationale:

There is a documented need for asynchronous, non-blocking functionality in
subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the
utility of the Python standard library that can be used on Unix based and
Windows builds of Python.  Practically every I/O object in Python has a
file-like wrapper of some sort.  Sockets already act as such and for
strings there is StringIO.  Popen can be made to act like a file by simply
using the methods attached the the subprocess.Popen.stderr, stdout and
stdin file-like objects.  But when using the read and write methods of
those options, you do not have the benefit of asynchronous I/O.  In the
proposed solution the wrapper wraps the asynchronous methods to mimic a
file object.

Reference Implementation:

I have been maintaining a Google Code repository that contains all of my
changes including tests and documentation [9] as well as blog detailing
the problems I have come across in the development process [10].

I have been working on implementing non-blocking asynchronous I/O in the
subprocess.Popen module as well as a wrapper class for subprocess.Popen
that makes it so that an executed process can take the place of a file by
duplicating all of the methods and attributes that file objects have.

There are two base functions that have been added to the subprocess.Popen
class: Popen.send and Popen._recv, each with two separate implementations,
one for Windows and one for Unix based systems.  The Windows
implementation uses ctypes to access the functions needed to control pipes
in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
the Python interface for file control serves the same purpose.  The
different implementations of Popen.send and Popen._recv have identical
arguments to make code that uses these functions work across multiple
platforms.

When calling the Popen._recv function, it requires the pipe name be
passed as an argument so there exists the Popen.recv function that passes
selects stdout as the pipe for Popen._recv by default.  Popen.recv_err
selects stderr as the pipe by default. Popen.recv and Popen.recv_err
are much easier to read and understand than Popen._recv('stdout' ... and
Popen._recv('stderr' ... respectively.

Since the Popen._recv function does not wait on data to be produced
before returning a value, it may return empty bytes.  Popen.asyncread
handles this issue by returning all data read over a given time
interval.

The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
allow a process to act like a file so that there are no blocking issues
that can arise from using the stdout and stdin file objects produced from
a subprocess.Popen call.


References:

[1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
http://mail.python.org/pipermail/python-bugs-list/2006-December/
  036524.html

[2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
http://ivory.idyll.org/blog/feb-07/problems-with-subprocess

[3] How can I run an external command asynchronously from Python? - Stack
Overflow
http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
  command-asynchronously-from-python

[4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
http://docs.python.org/library/subprocess.html

[Python-Dev] PEP 3145

[Eric Pruitt]

Hello all,

I have been working on adding asynchronous I/O to the Python
subprocess module as part of my Google Summer of Code project. Now
that I have finished documenting and pruning the code, I present PEP
3145 for its inclusion into the Python core code. Any and all feedback
on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP Submission

[Eric Pruitt]

Several days ago, around the time the python.org servers went down, I
submitted a PEP to edi...@python.org. When things to have been worked,
I submitted the PEP again. I have not seen any activity on the PEP in
Python-Dev or any reply acknowledging that it was received. Did I
misunderstand the process of submitting a PEP?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (try-except) conditional expression similar to (if-else) conditional (PEP 308)

[Eric Pruitt]

What about catching specific error numbers? Maybe an option so that
the dictionary elements can also be dictionaries with integers as the
keys:

filedata = try_3(open, randomfile, except = { IOError,  {2: None} } )

If it isn't found in the dictionary, then we raise the error.

On Thu, Aug 6, 2009 at 07:03, ilyailya.nikokos...@gmail.com wrote:
 I took a look at the options 1 and 2:

    x = float(string) except float('nan') if ValueError
    y = float(string) except ValueError: float('nan')

 and I think this can be done just as easily with existing syntax:

    x = try_1(float, string, except_ = float('nan'), if_ = ValueError)
    y = try_2(float, string, { ValueError: float('nan') })

 Here's the full example:

 - example starts -

 def try_1(func, *args, except_ = None, if_ = None):
    try:
        return func(*args)
    except if_ as e:
        return except_

 def try_2(func, *args):
    'The last argument is a dictionary {exception type: return value}.'
    dic = args[-1]
    try:
        return func(*args[:-1])
    except Exception as e:
        for k,v in dic.items():
            if isinstance(e, k):
                return v
        raise

 for string in ['5', 'five']:
    #   x = float(string) except float('nan') if ValueError
    x = try_1(float, string, except_ = float('nan'), if_ = ValueError)
    #   y = float(string) except ValueError: float('nan')
    y = try_2(float, string, { ValueError: float('nan') })
    print(x, y)

 - example ends -

 As a side note, if I just subscribed to python-dev, is it possible to
 quote an old email? Below is my manual cut-and-paste quote:

 -- my quote --

 Nick Coghlan wrote:
 P.J. Eby wrote:
 At 05:59 PM 8/5/2009 -0700, Raymond Hettinger wrote:
 [Jeffrey E. McAninch, PhD]
 I very often want something like a try-except conditional expression
 similar
 to the if-else conditional.

 An example of the proposed syntax might be:
    x = float(string) except float('nan')
 or possibly
    x = float(string) except ValueError float('nan')
 +1 I've long wanted something like this.
 One possible spelling is:

   x = float(string) except ValueError else float('nan')
 I think 'as' would be better than 'else', since 'else' has a different
 meaning in try/except statements, e.g.:

    x = float(string) except ValueError, TypeError as float('nan')

 Of course, this is a different meaning of 'as', too, but it's not as
 contradictory, IMO...  ;-)

 (We're probably well into python-ideas territory at this point, but I'll
 keep things where the thread started for now)

 The basic idea appears sound to me as well. I suspect finding an
 acceptable syntax is going to be the sticking point.

 Breaking the problem down, we have three things we want to separate:

 1. The expression that may raise the exception
 2. The expression defining the exceptions to be caught
 3. The expression to be used if the exception actually is caught

From there it is possible to come up with all sorts of variants.

 Option 1:

 Change the relative order of the clauses by putting the exception
 definition last:

   x = float(string) except float('nan') if ValueError
   op(float(string) except float('nan') if ValueError)

 I actually like this one (that's why I listed it first). It gets the
 clauses out of order relative to the statement, but the meaning still
 seems pretty obvious to me.

 A further extension (if we need it):

     result = foo(arg) except float('inf') if ZeroDivisionError else
 float('nan')

 The 'else' part handles any other exceptions (not necessarily a good idea!).

 or:

     result = foo(arg) except float('inf') if ZeroDivisionError else
 float('nan') if ValueError

 Handles a number of different exceptions.

 Option 2:

 Follow the lamba model and allow a colon inside this form of expression:

   x = float(string) except ValueError: float('nan')
   op(float(string) except ValueError: float('nan'))

 This has the virtue of closely matching the statement syntax, but
 embedding colons inside expressions is somewhat ugly. Yes, lambda
 already does it, but lambda can hardly be put forward as a paragon of
 beauty.

 A colon is also used in a dict literal.

 Option 3a/3b:

 Raymond's except-else suggestion:

   x = float(string) except ValueError else float('nan')
   op(float(string) except ValueError else float('nan'))

 [snip]
 -1
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/eric.pruitt%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Functionality in subprocess.Popen.terminate()

[Eric Pruitt]

On Tue, Aug 4, 2009 at 04:27, Nick Coghlanncogh...@gmail.com wrote:
 Eric Pruitt wrote:
 In my GSoC project, I have implemented asnychronous I/O in
 subprocess.Popen. Since the read/write operations are asynchronous, the
 program may have already exited by the time one calls the asyncread
 function I have implemented. While it returns the data just fine, I have
 come across an issue with the TerminateProcess function in Windows: if
 the program has already exited, when subprocess.Popen.Terminate calls
 the Windows built-in TerminateProcess function, an access denied
 error will occur. Should I just make it so that this exception is simply
 ignored or perform some kind of check to see if the process exists
 beforehand? If the latter, I have been unable to find a way to do so, to
 my liking at least. The solutions I saw would require code that seems a
 bit excessive to me.

 I'm pretty sure we already ignore some spurious error messages in cases
 like calling flush() in file.close(). I would suggest checking what the
 io module does in such cases and see what kind of precedent it sets.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ---


Sounds good enough to me but I was wondering if it might be a good
idea to add a function like pidinuse to subprocess as a whole that
would determine if a process ID was being used and return a simple
boolean value. I came across a number of people searching for a way to
determine if a PID was running (Google python check if pid exists)
so it seems like the implemented functionality would be of use to the
community as a whole, not just my wrapper class.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Functionality in subprocess.Popen.terminate()

[Eric Pruitt]

In my GSoC project, I have implemented asnychronous I/O in subprocess.Popen.
Since the read/write operations are asynchronous, the program may have
already exited by the time one calls the asyncread function I have
implemented. While it returns the data just fine, I have come across an
issue with the TerminateProcess function in Windows: if the program has
already exited, when subprocess.Popen.Terminate calls the Windows built-in
TerminateProcess function, an access denied error will occur. Should I
just make it so that this exception is simply ignored or perform some kind
of check to see if the process exists beforehand? If the latter, I have been
unable to find a way to do so, to my liking at least. The solutions I saw
would require code that seems a bit excessive to me.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

 On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote:
  I am implementing the file wrapper using changes to subprocess.Popen
  that also make it asynchronous and non-blocking so implementing r+
  should be trivial to do. How about handling stderr? I have the
  following ideas: leave out support for reading from stderr, make it
  so that there is an optional additional argument like outputstderr =
  False, create another function that toggles / sets whether stderr or
  stdout is returned or mix the two outputs.

 Leaving it out is always an option.

 As I see it, fundamentally you can either read from stdout and sterr as
 two different streams, or you can interleave (mix) them. To me, that
 suggests three functions:

 ProcessIOWrapper()  # read from stdout (or write to stdin etc.)
 ProcessIOWrapperStdErr()  # read/write from stderr
 ProcessIOWrapper2()  # read from mixed stdout and stderr

 I don't like a function to toggle between one and the other: that smacks
 of relying on a global setting in a bad way. I suppose you could add an
 optional argument to ProcessIOWrapper() to select between stdout,
 stderr, or both together.



 --
 Steven D'Aprano

How would having a toggle function rely on a global setting? Each class would
simply have its own member variable like self.readsstderr. It's a moot point
though as I've decided to use separate functions as you suggested. With
separate functions, the user doesn't have to worry about modifying the mode
keyword if stderr is needed and as an added bonus, it also makes the code
a lot more readable.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

My motivation came from an instance when I was using subprocess.Popen for a
Linux / Windows cross platform program. In part of the program, I was
writing and reading to a cron like object. On Windows, it was a text file
and on Linux it would be the crontab executable. Had I been able to
substitute the open() function with my wrapper, it would have been the
only change I had to make for cross platform compatibility; instead of
having to change numerous lines because Linux would need Popen and Windows
would need a regular file open(), I could simply make it so that if the
platform was Linux, my wrapper is used in place of that. Just another
example would be having an external program decrypt a file that can be in
plain text or encrypted that might go something like this:

if encryptionEnabled:
fileobj = subprocess.ProcessIOWrapper(gpg --decrypt
supersecret.html.gpg)
else:
fileobj = open(notsosecret.html)

From there, the functions would not have to be modified despite using a
running process versus a file object.

On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith g...@krypto.org wrote:

 On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitzgl...@twistedmatrix.com
 wrote:
  On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore p.f.mo...@gmail.com wrote:
 
  I like MRAB's idea of using a (non-standard) e flag to include
  stderr. So r reads from stdout, re reads from stdout+stderr.
 
  Anything more complicated probably should just use raw Popen
  objects. Don't overcomplicate the interface.
 
  In my opinion, mangling stderr and stdout together is already an
  overcomplication.  It shouldn't be implemented.
 
  It seems like a good idea, until you realize that subtle changes to your
 OS,
  environment, or buffering behavior may result in arbitrary, unparseable
  output.

 Agreed.  Leave stderr support out of this.  People who need stderr
 should use the full subprocess.Popen interface.  Quick hack unixy
 types will just run their process using a shell (which already seems
 to be the default based on the ls -l example) and add 21.  This
 functionality is basically the equivalent of adding the | symbol on
 either or both ends of a filename given to open() in perl.  (but
 without being so gross).

 I do wonder why you're trying to make it behave exactly like open()
 including the mode= syntax.

 Why not just define several names based on the behavior?

  ProcessReadWrapper()
  ProcessWriteWrapper()
  ProcessReadWriteWrapper()

 -gps

 
  For example, let's say you've got a program whose output is a list of
 lines,
  each one containing a number.  Sometimes it tries to import gtk, and
 fails
  to open its display.
 
  That's fine, and you can still deal with it, as long as the interleaved
  output looks like this:
 
  100
  200
  Gtk-WARNING **: cannot open display:
  300
  400
 
  but of course the output might (although unlikely with such small chunks
 of
  output) end up looking like this, instead:
 
  100
  2Gtk-WAR0NING0 **:
  can30not 0open display:
 
  400
 
  this is the sort of thing which is much more likely to happen once you
 start
  dealing with large volumes of data, where there are more page-boundaries
 for
  your buffers to get confused around, and you are playing with buffering
  options to improve performance.  In other words, it's something that
 fails
  only at scale or under load, and is therefore extremely difficult to
 debug.
 
  This option might be okay if it were allowed only on subprocesses opened
 in
  a text mode, and if the buffering logic involved forced stderr and stdout
 to
  be line-delimited, and interleave only lines, rather than arbitrary
 chunks
  of bytes.  Of course then if you use this flag with a program that
 outputs
  binary data with no newlines it will buffer forever and crash your
 program
  with a MemoryError, but at least that's easy to debug when it happens.
 
  ___
  Python-Dev mailing list
  Python-Dev@python.org
  http://mail.python.org/mailman/listinfo/python-dev
  Unsubscribe:
  http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
 
 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

Well, with a few changes to your code, that would indeed work (you are using
stdin as your pipe. Correct me if I'm wrong but if you intend to read from
it, you need to change it to stdout = subprocess.PIPE and the other lines
as well to reflect this change). My Google Summer of Code modifications to
subprocess.Popen provide non-blocking, asynchronous I/O support and my file
wrapper is built upon that augmented functionality. If I remember correctly,
when I was working on the program where I first thought a file wrapper for
subprocess.Popen would be rather handy, I also ran into blocking I/O as
well.

On Wed, Jul 29, 2009 at 20:20, Devin Cook devin.c.c...@gmail.com wrote:

 Hmm... can't you do this?

 if encryptionEnabled:
p = subprocess.Popen([gpg, --decrypt, supersecret.html.gpg],
 stdin = subprocess.PIPE)
fileobj = p.stdin
 else:
fileobj = open(notsosecret.html)

 I think that works. Is there something this way won't work for? You
 can also do the same thing to get stdout and stderr file objects. I
 guess a wrapper would simplify this process.

 -Devin

 On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitteric.pru...@gmail.com wrote:
  My motivation came from an instance when I was using subprocess.Popen for
 a
  Linux / Windows cross platform program. In part of the program, I was
  writing and reading to a cron like object. On Windows, it was a text file
  and on Linux it would be the crontab executable. Had I been able to
  substitute the open() function with my wrapper, it would have been the
  only change I had to make for cross platform compatibility; instead of
  having to change numerous lines because Linux would need Popen and
 Windows
  would need a regular file open(), I could simply make it so that if the
  platform was Linux, my wrapper is used in place of that. Just another
  example would be having an external program decrypt a file that can be in
  plain text or encrypted that might go something like this:
 
  if encryptionEnabled:
  fileobj = subprocess.ProcessIOWrapper(gpg --decrypt
  supersecret.html.gpg)
  else:
  fileobj = open(notsosecret.html)
 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Implementing File Modes

[Eric Pruitt]

Hello,

Since there was a bit of confusion last time, I'll start by saying I am
working on the subprocess.Popen module for Google Summer of Code. One of the
features I am implementing is a class so that a running process can stand in
in place of a file. For examples, instead of open( filelist, mode = 'r')
one would call ProcessIOWrapper( ls -l, mode = 'r'). I am trying to decide
if I should fully implement the mode argument. Right now, it essentially
ignores everything but a 'U' indicated universal newlines in the mode
argument. Should I leave that as is or make it so that things like r+,
w, a are handled the way they would be for an actual file?

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

I am implementing the file wrapper using changes to subprocess.Popen that
also make it asynchronous and non-blocking so implementing r+ should be
trivial to do. How about handling stderr? I have the following ideas: leave
out support for reading from stderr, make it so that there is an optional
additional argument like outputstderr = False, create another function
that toggles / sets whether stderr or stdout is returned or mix the two
outputs.

Thanks for the input,
Eric

On Mon, Jul 27, 2009 at 10:46, Paul Moore p.f.mo...@gmail.com wrote:

 2009/7/27 Eric Pruitt eric.pru...@gmail.com:
  Hello,
 
  Since there was a bit of confusion last time, I'll start by saying I am
  working on the subprocess.Popen module for Google Summer of Code. One of
 the
  features I am implementing is a class so that a running process can stand
 in
  in place of a file. For examples, instead of open( filelist, mode =
 'r')
  one would call ProcessIOWrapper( ls -l, mode = 'r'). I am trying to
 decide
  if I should fully implement the mode argument. Right now, it essentially
  ignores everything but a 'U' indicated universal newlines in the mode
  argument. Should I leave that as is or make it so that things like r+,
  w, a are handled the way they would be for an actual file?

 I would expect r to produce a pipe that reads from stdout of the
 subprocess, and w to produce a pipe that writes to stdin of the
 subprocess. a would be the same as w, and arguably r+ would be a
 bidirectional pipe - read from the subprocess stdout and write to its
 stdin.

 I'd be OK with r+ not being implemented (if it's too hard to avoid
 deadlocks) but r and w should be present.

 Paul.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile

[Eric Pruitt]

Hello,

I have written replacements for a few of Mark Hammond's PyWin32 functions
using ctypes to call upon the Windows kernel API. Code can be found here;
http://pastebin.com/m46de01f . When calling ReadFile, it appears that the
kernel API converts '/n' newlines '/r/n' newlines. I have not been able to
find any information about this but as far as I can tell, there is nothing
in my code that is causing the problem. Initially I suspected it related to
newline translation but the function in subprocess.Popen for translating
newlines only converts to '/n' newlines. When I replaced my ReadFile and
WriteFile functions with the PyWin32 functions I was imitating, the problem
still arose. I was hoping someone could confirm this problem for me from
experience or by testing out my code. If you would like the see the
functions used in full context, I have a Mercurial Google Code project that
can be checked out at
http://code.google.com/p/subprocdev/source/list(branch python3k). My
work entails modifications to subprocess.py so when
running the code, please update your PYTHONPATH variable so that my
subprocess.py will be imported.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

I will keep a grace period in mind when I am posting. In my defense,
however, I had been working on the problem for some time before posting this
and my messages in quick succession were in response to the suggestions
people offered. The problem has arisen again but I will work on it some more
and will also post a detailed post on my GSoC blog of the process I went
through when I encountered the issue.

On Mon, Jul 13, 2009 at 16:24, Martin v. Löwis mar...@v.loewis.de wrote:

  I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
  Microsoft Visual C++ Express Edition in order to test some modifications
  I made to the _subprocess.c file as part of my Google Summer of Code
  proposal.

 As a posting guideline, please be careful to not post too many messages
 in quick succession. If you run into a problem, try to solve it for
 yourself. If you fail, consult your mentor and/or wait a couple of
 hours. Then, when posting, summarize your findings and the alternatives
 that you tried unsuccessfully.

 If people learn that you post while being in the process of working
 on a problem, they will learn that it is better to wait a day or two
 before responding, in case you figure it out on your own.

 Regards,
 Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

The problem appears resolved again and I have two posts on the issue on my
blog located at http://subdev.blogspot.com/. I was missing an MSVC++
run-time DLL and re-installing Visual C++ Express fixed the problem. A bit
of a somewhat random note -- looking through some of the buildbot output for
Python 3.1, I noticed that at one point, the bot compilations had the same
issue; Google 
Cachehttp://74.125.95.132/search?q=cache:oadoPcYTjW8J:www.python.org/dev/buildbot/3.x/x86%2520XP-4%25203.x/builds/606/step-compile/0/text+LINK+:+fatal+error+LNK1181:+cannot+open+input+file+%27.%5Cpython31_d.lib%27cd=1hl=enct=clnkgl=usclient=firefox-a
.

On Mon, Jul 13, 2009 at 16:24, Martin v. Löwis mar...@v.loewis.de wrote:

  I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
  Microsoft Visual C++ Express Edition in order to test some modifications
  I made to the _subprocess.c file as part of my Google Summer of Code
  proposal.

 As a posting guideline, please be careful to not post too many messages
 in quick succession. If you run into a problem, try to solve it for
 yourself. If you fail, consult your mentor and/or wait a couple of
 hours. Then, when posting, summarize your findings and the alternatives
 that you tried unsuccessfully.

 If people learn that you post while being in the process of working
 on a problem, they will learn that it is better to wait a day or two
 before responding, in case you figure it out on your own.

 Regards,
 Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Windows Toolchain

[Eric Pruitt]

Hello,

I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
Visual C++ Express Edition in order to test some modifications I made to the
_subprocess.c file as part of my Google Summer of Code proposal. I cannot
seem to figure out how to compile Python on Windows and could use some
guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
work when I attempt to open and no error message is given when clicked. When
opened from the Visual C++, it says Solution folders are not supported in
this version of the application.

Any advice is greatly appreciated,
Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

Sorry, forgot to include my build log:

Command Lines:
http://pastebin.com/f25614b01

Output Window:
 Compiling...
 python.c
 Compiling resources...
 Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1
 Copyright (C) Microsoft Corporation.  All rights reserved.
 Linking...
 LINK : fatal error LNK1181: cannot open input file '.\python31_d.lib'
 Creating browse information file...
 Microsoft Browse Information Maintenance Utility Version 9.00.30729
 Copyright (C) Microsoft Corporation. All rights reserved.

On Mon, Jul 13, 2009 at 10:46, Eric Pruitt eric.pru...@gmail.com wrote:

 Hello,

 I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
 Visual C++ Express Edition in order to test some modifications I made to the
 _subprocess.c file as part of my Google Summer of Code proposal. I cannot
 seem to figure out how to compile Python on Windows and could use some
 guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
 work when I attempt to open and no error message is given when clicked. When
 opened from the Visual C++, it says Solution folders are not supported in
 this version of the application.

 Any advice is greatly appreciated,
 Eric

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

I opened the solution, hit Ctrl+F5 and it began compiling but it fails with
this error:
LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'

How do I specify that pythoncore be built? When I follow your
instructions, right clicking on the python project in the Solution
Explorer panel, I get the exact same link error as above. If I hit F7, the
build log changes completely. The last lines of output with F7 are as
follows:

 21Build log was saved at file://C:\Documents and
Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm
 21select - 1 error(s), 0 warning(s)
 == Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
==


On Mon, Jul 13, 2009 at 11:03, Amaury Forgeot d'Arc amaur...@gmail.comwrote:

 Hi,

 2009/7/13 Eric Pruitt eric.pru...@gmail.com:
  Hello,
 
  I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
 Microsoft
  Visual C++ Express Edition in order to test some modifications I made to
 the
  _subprocess.c file as part of my Google Summer of Code proposal. I cannot
  seem to figure out how to compile Python on Windows and could use some
  guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
  work when I attempt to open and no error message is given when clicked.

 Is Visual C++ correctly installed?

  When
  opened from the Visual C++, it says Solution folders are not supported
 in
  this version of the application.

 This is a warning from the Express Edition, which has absolutely no
 impact on the build. You can safely ignore it and continue.

 What steps did you perform exactly?  From your build log the
 pythoncore project was not built. Why?

 The following has always worked for me: In the Solution explorer
 panel, right-click on the python project, and click Build. This
 should build the project and all its dependencies.
 Or just hit the F7 key and watch the whole solution build.

 --
 Amaury Forgeot d'Arc

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
Unix style but after fixing them, I still have the same problem.

On Mon, Jul 13, 2009 at 11:43, Amaury Forgeot d'Arc amaur...@gmail.comwrote:

 2009/7/13 Eric Pruitt eric.pru...@gmail.com:
  I opened the solution, hit Ctrl+F5 and it began compiling but it fails
 with
  this error:
  LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'
 
  How do I specify that pythoncore be built? When I follow your
  instructions, right clicking on the python project in the Solution
  Explorer panel, I get the exact same link error as above. If I hit F7,
 the
  build log changes completely. The last lines of output with F7 are as
  follows:
 
  21Build log was saved at file://C:\Documents and
 
 Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm
  21select - 1 error(s), 0 warning(s)
  == Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
  ==
 

 Did you open the sub-project python.vcproj by mistake?
 You must open the solution file: pcbuild.sln.

 And if pcbuild.sln does not load correctly, check that it has correct
 line endings: it is supposed to have DOS line endings.

 --
 Amaury Forgeot d'Arc

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

The code I downloaded was the official Python 3.1 release tarball. I only
checked the line endings in the pcbuild.sln file so I am not sure about the
other files. I will attempt to build the Python 3000 SVN trunk shortly and
report back.

On Mon, Jul 13, 2009 at 13:09, Paul Moore p.f.mo...@gmail.com wrote:

 2009/7/13 Eric Pruitt eric.pru...@gmail.com:
  It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
 be
  Unix style but after fixing them, I still have the same problem.

 Where did you get your copy of the Python source from? If it's from
 Subversion, I'm surprised the line endings are wrong. If it's
 Mercurial, then this may be indicative of the still somewhat open
 question of how line endings will be handled in Mercurial (which
 doesn't have a server-side way of saying that a file should use
 platform-native line endings).

 If you still have the same problem after fixing the file, this may
 indicate further damage, or maybe the fix isn't doing the right thing
 as the file has already been opened (and mangled) by Visual Studio,

 I'd suggest trying to reproduce your issue with a clean checkout from
 Subversion.

 It would also help if you could describe precisely the steps you took
 to download the sources and build them (I appreciate that this may not
 be easy now, after you've tried various things).

 Paul.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

Well, I narrowed the culprit down to Windows DEP. I had been having trouble
with it for some time now, with it raising an error for nearly every program
I ran so I just disabled it completely and Python now compiles fine. Thank
you all for the assistance.

On Mon, Jul 13, 2009 at 13:42, Paul Moore p.f.mo...@gmail.com wrote:

 2009/7/13 Paul Moore p.f.mo...@gmail.com:
  2009/7/13 Eric Pruitt eric.pru...@gmail.com:
  It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
 be
  Unix style but after fixing them, I still have the same problem.
 [...]
  I'd suggest trying to reproduce your issue with a clean checkout from
  Subversion.

 FWIW, I did:

 svn co http://svn.python.org/projects/python/tags/r31/ py31

 Open pcbuild in Visual C++ 2008 Express. OK on the warning about
 solution folders.

 Right click python and choose Build.

 Works OK here. Also works if I download Python-3.1.tgz from python.org
 and unpack it using (command line) bsdtar. I wonder - did you use
 WinZip to unpack the archive? That has some sort of brain-damaged
 Smart CRLF conversion for tar files, which may have screwed up your
 files.

 Paul.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Bytes, Strings, Encoding

[Eric Pruitt]

Hello,

I am working on the subprocess.Popen module for Python 2.7 and am now moving
my changes over to Python 3.1 however I am having trouble with the whole
byte situation and I can't quite seem to understand how to go back and forth
between bytes and strings. I am also looking for the Python 3k equivalent
for the Python 2.X built-in buffer class.

One version of the file with my modifications can be found here  
http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9r=5b570f8cbfcaae859091eb01b21b183aa5221af9.
Lines 1 - 15 are me attempting to get around certain changes between
Python 3.0 and Python 2.7. Further down on line 916, we have the function
send and recv in which I am having the most trouble with bytes and
strings.

Any help is appreciated. Feel free to comment on my blog
http://subdev.blogspot.com/ or reply to the last.

Thanks in advance,
Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Popen asynchronous input for Windows

[Eric Pruitt]

Hello,

I am looking for alternatives to Josiah Carlson's asynchronous I/O patch for
subprocess.Popen. While his patch seems to work well, it relies on pywin32
which is not part of the standard Python library. If I cannot find an
alternative, I will be using cTypes with the parts of Mark Hammond's code
that I need, license permitting. Any suggestions are greatly appreciated.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Popen asynchronous input for Windows

[Eric Pruitt]

Thanks for the lead. I have the pywin32 source code and have found the files
that appear to contain the code I need inside of some *.i files. After a
bit of Googling and paying attention to the blatantly obvious *.cpp files, I
realized Hammond's code is written in C++ whereas Python uses C. I am
familiar with neither language very well but if I can't find a work-around
for Windows asynchronous I/O, I will be learning a bit of both of them.

Please feel free to suggest any other ideas that don't rely on my sparse
knowledge of C and C++.

Eric

On Thu, Jun 18, 2009 at 13:32, Christian Heimes li...@cheimes.de wrote:

 Eric Pruitt schrieb:
  Hello,
 
  I am looking for alternatives to Josiah Carlson's asynchronous I/O patch
 for
  subprocess.Popen. While his patch seems to work well, it relies on
 pywin32
  which is not part of the standard Python library. If I cannot find an
  alternative, I will be using cTypes with the parts of Mark Hammond's code
  that I need, license permitting. Any suggestions are greatly appreciated.

 The subprocess module several wrappers for win32 APIs in
 Modules/_subprocess.c. You could add the necessary functions.

 Christian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of 2.7 and 3.2

[Eric Pruitt]

I am in the process of implementing a number of often requested features and
proposed patches in the subprocess module for my Google Summer of Code 2009
project. For information on my progress, check out my blog located at *
http://subdev.blogspot.com/* http://subdev.blogspot.com/. Any comments and
suggestions are greatly appreciated.

 Just my 0.02 cents, but struggling with all warts of 2.5 subprocessing
  in Windows I would vote for more time for stabilizating things - not
  adding new features. Long awaited subprocess as replacement for
  os.popen() AFAIK is still incapable to asynchronously communicate with
  spawned processes on Windows. I would call this feature as critical
  even on 2.6  As a release testcase - try porting pyexpect module to
  this platform. Absence of native curses/console/readline module also
  makes Python one-way unix shell language while many users expect it to
  be crossplatform.

 I am not quite sure whether you are for new features or not. Your
 first sentence (vote for ... not adding new features) seems to
 suggest that you would not like to see new features, and your last
 sentence (absence of native curses/console/readline module)
 suggests that you *do* want to see new features (namely, a native
 curses module, and a native readline module).

 Which one is it?

 Regards,
 Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com