Re: tee-like behavior in Python
On Wed, May 9, 2012 at 8:35 AM, Florian Lindner wrote: > Hello, > > how can I achieve a behavior like tee in Python? > > * execute an application > * leave the output to stdout and stderr untouched > * but capture both and save it to a file (resp. file-like object) > > I have this code > > proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE, shlex.split() should just be used once, at "development time", to determine the form of the argument list. It shouldn't generally be used at runtime. Otherwise, you need to do the proper escaping/quoting yourself, which defeats the entire purpose of bypassing the shell. Cheers, Chris -- http://mail.python.org/mailman/listinfo/python-list
Re: tee-like behavior in Python
You've had some good responses already, but here're two more: 1) Easiest would be to use setvbuf in the child process, if you have access to its source. This allows you to force line-oriented buffering. 2) stdio likes to buffer to tty/pty's in a line-oriented manner, and other things in a block-oriented manner - by default, so users get pleasing output increments, and programs get efficiency. To trick stdio into buffering line-oriented by default when it's sending output to another program, you may have to talk to the child process using a pty, AKA a pseudo terminal. There are a few ways of doing this: A) Use CPython's pty module. B) Use something like pexpect. C) Check out the pty program in comp.sources.unix volume 25. This might be pretty easy too, assuming that pty still builds on a modern *ix. On Wed, May 9, 2012 at 8:35 AM, Florian Lindner wrote: > Hello, > > how can I achieve a behavior like tee in Python? > > * execute an application > * leave the output to stdout and stderr untouched > * but capture both and save it to a file (resp. file-like object) > > I have this code > > proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE, > stderr=subprocess.STDOUT) > while True: >out = proc.stdout.readline() >if out == '' and proc.poll() != None: > break >sys.stdout.write(out) >logfile.write(out) > > This works so far but always buffers a couple of lines and outputs > them en bloc. The final output is like it is desired but with a > significant delay. Is there a way around that? > > Thanks, > > Florian > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: tee-like behavior in Python
On May 9, 2012, at 11:35 AM, Florian Lindner wrote: > Hello, > > how can I achieve a behavior like tee in Python? > > * execute an application > * leave the output to stdout and stderr untouched > * but capture both and save it to a file (resp. file-like object) > > I have this code > > proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE, > stderr=subprocess.STDOUT) > while True: >out = proc.stdout.readline() >if out == '' and proc.poll() != None: > break >sys.stdout.write(out) >logfile.write(out) > > This works so far but always buffers a couple of lines and outputs > them en bloc. The final output is like it is desired but with a > significant delay. Is there a way around that? > > Thanks, > > Florian > -- > http://mail.python.org/mailman/listinfo/python-list Have you tried explicitly calling file.flush() on the log file? (The docs note that you may have to follow this with os.fsync() on some systems.) -Bill -- http://mail.python.org/mailman/listinfo/python-list
Re: tee-like behavior in Python
On Wed, May 9, 2012 at 11:35 AM, Florian Lindner wrote: > Hello, > > how can I achieve a behavior like tee in Python? > > * execute an application > * leave the output to stdout and stderr untouched > * but capture both and save it to a file (resp. file-like object) > > I have this code > > proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE, > stderr=subprocess.STDOUT) > while True: > out = proc.stdout.readline() > if out == '' and proc.poll() != None: > break > sys.stdout.write(out) > logfile.write(out) > > This works so far but always buffers a couple of lines and outputs > them en bloc. The final output is like it is desired but with a > significant delay. Is there a way around that? Perhaps this would help: http://docs.python.org/library/subprocess.html#popen-constructor specifically, the bits about setting bufsize to 0 indicating unbuffered behaviour. -- http://mail.python.org/mailman/listinfo/python-list
Re: tee-like behavior in Python
On 05/09/2012 11:35 AM, Florian Lindner wrote: > Hello, > > how can I achieve a behavior like tee in Python? > > * execute an application > * leave the output to stdout and stderr untouched > * but capture both and save it to a file (resp. file-like object) > > I have this code > > proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE, > stderr=subprocess.STDOUT) > while True: > out = proc.stdout.readline() > if out == '' and proc.poll() != None: >break > sys.stdout.write(out) > logfile.write(out) > > This works so far but always buffers a couple of lines and outputs > them en bloc. The final output is like it is desired but with a > significant delay. Is there a way around that? > > Thanks, > > Florian Chances are that other program is buffering its output. Many programs check if they're running on a tty, and turn off buffering for interactive use. But redirect them into a pipe, and they'll run as efficiently as possible, which includes buffering the output. -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
