Rita wrote in news:AANLkTi=88dcpm_kqrs2g620obsnxz0majubfwpeme...@mail.gmail.com in gmane.comp.python.general:
[Top post relocated] > On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft <r...@rtw.me.uk> > wrote: > >> Rita wrote in >> news:AANLkTi=w95gxosc1tkt2bntgjqys1cbmdnojhokq4...@mail.gmail.com in >> gmane.comp.python.general: >> >> > >> > When using wait() it works a bit better but not consistent >> > def run(cmd): >> > p=subprocess.Popen(cmd,stdout=subprocess.PIPE) >> > rc=p.wait() >> > print rc >> > return p.stdout >> > >> >> > When the output of cmd is a small ascii file it works perfectly >> > fine, but when the file is large (more than 2MB) the process just >> > waits for ever (I am guessing its blocking?). >> >> Your OS has supplied a pipe buffer of 2MB, its full and the prossess >> is waiting until you read something from the pipe (i.e. >> p.stdout.read()). >> >> > When I use the communicate call >> > it works perfectly but my process is consuming way too much memory. >> > >> > Is there a better way to get my return code consistently >> > efficiently and not take up so much memory? >> >> If you don't need the processes output then don't use the PIPE >> argument. If you do you will need to read from p.stdout until the >> process is complete, then get the return code: >> >> while True: >> buf = p.stdout.read( 1024 ) >> # do somthing with buf >> if len( buf ) < 1024: >> break >> >> rc = p.wait() > > For extra points is there a way to speed up the p.stdout.read(bufsize) > ? Its unlikely that is the problem, most likely you are reading all of the 2MB OS pipe buffer and then having to wait for the programme you are calling to produce some more output. Before trying to speed things up be sure to test with real work being done to the output you recieve. If its still slow and you have multiple cores/processors then you will need to put you read loop in a thread to speed it up. -- Rob. -- http://mail.python.org/mailman/listinfo/python-list
