Re: Open a command pipe for reading
Hi Rodrick, On 2010-08-17 18:40, Rodrick Brown wrote: I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line. I've tried using Popen to do this with no luck. I'm trying to simulate /bin/foo myfile.dat Is foo the 3rd-party conversion tool you've mentioned? It would be good to have a bit more context, e. g. an actual snippet from your code. And as the records are being printed do some calculations. pipe = Popen(exttool,shell=True,stdout=PIPE).stdout for data in pipe.readlines(): print data, This operation blocks forever I'm guessing it's trying to process the entire file at once. If you use `readlines` on a file object it reads the whole file at once and returns a list of the lines (including line end characters, by the way). What you probably want is for line in pipe: print line, which reads and prints the file contents line by line. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Open a command pipe for reading
In message mailman.2236.1282063313.1673.python-l...@python.org, Rodrick Brown wrote: Sent from my iPhone 4. Glad to hear you achieved it without losing the signal. :) -- http://mail.python.org/mailman/listinfo/python-list
Open a command pipe for reading
I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line. I've tried using Popen to do this with no luck. I'm trying to simulate /bin/foo myfile.dat And as the records are being printed do some calculations. pipe = Popen(exttool,shell=True,stdout=PIPE).stdout for data in pipe.readlines(): print data, This operation blocks forever I'm guessing it's trying to process the entire file at once. Sent from my iPhone 4. -- http://mail.python.org/mailman/listinfo/python-list
Re: Open a command pipe for reading
On Tue, Aug 17, 2010 at 9:40 AM, Rodrick Brown rodrick.br...@gmail.com wrote: I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line. I've tried using Popen to do this with no luck. I'm trying to simulate /bin/foo myfile.dat And as the records are being printed do some calculations. pipe = Popen(exttool,shell=True,stdout=PIPE).stdout I'd strongly suggest trying to avoid shell=True. for data in pipe.readlines(): for data in pipe: print data, This operation blocks forever I'm guessing it's trying to process the entire file at once. Indeed, that's how readlines() works, so don't use it. http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Open a command pipe for reading
On Tuesday 17 August 2010, it occurred to Rodrick Brown to exclaim: I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line. I've tried using Popen to do this with no luck. I'm trying to simulate /bin/foo myfile.dat And as the records are being printed do some calculations. pipe = Popen(exttool,shell=True,stdout=PIPE).stdout for data in pipe.readlines(): print data, This operation blocks forever I'm guessing it's trying to process the entire file at once. Yes. It is. That's what you're telling it to do: file.readline returns a list of all the lines in the file. What you want to do is iterate over the stream, as in: for line in pipe: process(line) Also, there's probably no need to use shell=True. Sent from my iPhone 4. Is that a fact? This is so interesting. Phones these days. Almost as annoyingly obnoxious as gmx and yahoo mail. -- http://mail.python.org/mailman/listinfo/python-list
