Re: os.popen and lengthy operations

[Dmitry Teslenko]

Hello!

On 24/09/2007, Tommy Nordgren <[EMAIL PROTECTED]> wrote:
>Your problem is that you are not reading the standard output and
> standard error streams in the correct way.
> You need to do the reading of standard out and standard err in
> parallell rather than sequentially.
> The called process can't proceed when it's output buffers are full.
> Pipes is limited by small buffers (for examples 512 bytes
> on Mac OS X)

Thanks to all for your comprehensive explanation!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen and lengthy operations

[Tommy Nordgren]

On 20 sep 2007, at 08.31, Dmitry Teslenko wrote:

> Hello!
> I'm using os.popen to perform lengthy operation such as building some
> project from source.
> It looks like this:
> def execute_and_save_output( command, out_file, err_file):
>
> import os
>
> def execute_and_save_output( command, out_file, err_file):
>   (i,o,e) = os.popen3( command )
>   try:
>   for line in o:
>   out_file.write( line )
>
>   for line in e:
>   err_file.write( line )
>   finally:
>   i.close()
>   o.close()
>   e.close()
>
> ...
> execute_and_save_output( '', out_file,  
> err_file)
>
> Problem is that script hangs on operations that take long to execute
> and have lots of output such as building scripts.
> -- 
Your problem is that you are not reading the standard output and  
standard error streams in the correct way.
You need to do the reading of standard out and standard err in  
parallell rather than sequentially.
The called process can't proceed when it's output buffers are full.  
Pipes is limited by small buffers (for examples 512 bytes
on Mac OS X)
-
This sig is dedicated to the advancement of Nuclear Power
Tommy Nordgren
[EMAIL PROTECTED]



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen and lengthy operations

[tokland]

On 20 sep, 08:31, "Dmitry Teslenko" <[EMAIL PROTECTED]> wrote:

> I'm using os.popen to perform lengthy operation such as building some
> project from source.

> def execute_and_save_output( command, out_file, err_file):
> (i,o,e) = os.popen3( command )

You should consider using the higher-level "subprocess" module:

import subprocess

def run(command, output, error, bufsize=None):
popen = subprocess.Popen(command.split(), bufsize=bufsize,
stdout=output, stderr=error)
popen.communicate()
return popen.returncode

example:

create = lambda path: open(path, "w")
run("ls /etc/services abc", create("/tmp/output"), create("/tmp/
error"))

Check how to use the "bufsize" parameter in the docs:

http://docs.python.org/lib/node529.html

arnau

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: os.popen and lengthy operations

[Marc 'BlackJack' Rintsch]

On Thu, 20 Sep 2007 10:31:43 +0400, Dmitry Teslenko wrote:

> I'm using os.popen to perform lengthy operation such as building some
> project from source.
> It looks like this:
> def execute_and_save_output( command, out_file, err_file):
> 
> import os
> 
> def execute_and_save_output( command, out_file, err_file):
>   (i,o,e) = os.popen3( command )
>   try:
>   for line in o:
>   out_file.write( line )
> 
>   for line in e:
>   err_file.write( line )
>   finally:
>   i.close()
>   o.close()
>   e.close()
> 
> ...
> execute_and_save_output( '', out_file, err_file)
> 
> Problem is that script hangs on operations that take long to execute
> and have lots of output such as building scripts.

Your code reads from the process' stdout until there is nothin to read
anymore and then from stderr.  The process might output something to both.
The output is buffered.  And when the stderr buffer is full the process
blocks until your application reads something from `e`.  That's where the
whole thing hangs.  You wait for something on `o` and the process waits
for you to read from `e` → deadlock.

You have to use threads to read both `o` and `e` or the `select` module to
look which file has something to read.

Ciao,
Marc 'BlackJack' Rintsch
-- 
http://mail.python.org/mailman/listinfo/python-list

os.popen and lengthy operations

[Dmitry Teslenko]

Hello!
I'm using os.popen to perform lengthy operation such as building some
project from source.
It looks like this:
def execute_and_save_output( command, out_file, err_file):

import os

def execute_and_save_output( command, out_file, err_file):
(i,o,e) = os.popen3( command )
try:
for line in o:
out_file.write( line )

for line in e:
err_file.write( line )
finally:
i.close()
o.close()
e.close()

...
execute_and_save_output( '', out_file, err_file)

Problem is that script hangs on operations that take long to execute
and have lots of output such as building scripts.
-- 
http://mail.python.org/mailman/listinfo/python-list