That's cool. The document is a very good starting point. To me this streams framework is a very useful component that should be merged into provisional packages of Python standard library. I still need to find time to read it thoroughly to adapt to what I know.
On Mon, Feb 24, 2014 at 9:57 PM, Kenny, Jason L <[email protected]> wrote: > Hi Anatoly, > > Attached is a word document with the overview of the deign I have with > thoughts on some good and bad of it. Hopefully I proofed the document enough, > document writing is not my strong suit. Anyways let me know if you have any > other questions, etc.. on the design. > > Eugene who has been helping with Parts work inside thinks he fixed a > Scons/Parts rebuild issues we have been dealing with. I testing it out, if it > looks good I will push this drop in a few days. > > Thanks for looking at this! > Jason > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of anatoly techtonik > Sent: Tuesday, February 18, 2014 10:35 AM > To: SCons developer list > Subject: Re: [Scons-dev] Fix async subprocess execution with proper handling > of std* streams > > Thanks. That would be great. ;) > > On Tue, Feb 18, 2014 at 5:21 PM, Kenny, Jason L <[email protected]> > wrote: >> I will get it pushed… life getting in the way. >> >> >> >> From: [email protected] [mailto:[email protected]] >> On Behalf Of Eugene Leskinen >> Sent: Tuesday, February 18, 2014 7:03 AM >> To: SCons developer list >> Subject: Re: [Scons-dev] Fix async subprocess execution with proper >> handling of std* streams >> >> >> >> Hi Anatoly, >> >> >> >> The code supporting I/O redirection in Parts is located in >> parts/part_logger.py file. Look at >> http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_log >> ger.py?revision=486&view=markup >> >> >> >> The code uses Python threading module to create threads for each >> stream and then runs a loop in every thready like your code does: >> >> >> >> line = ' ' >> >> try: >> >> while line: >> >> line = self.pipein.readline() >> >> if line: >> >> self.output.WriteStream(self.taskId, >> self.streamId, >> line) >> >> except: >> >> # There was an error... that shouldn't happen, but still it did. >> So we report it >> >> # to the caller and close our pipe end so that spawned >> program won't block >> >> self.error = traceback.format_exc() >> >> self.pipein.close() >> >> self.pipein = None >> >> >> >> Please note that public repo is a bit out-of-date. I hope Jason will >> find time to pull up-to-date version from Intel internal repository. >> >> >> >> 2014-02-18 16:30 GMT+04:00 anatoly techtonik <[email protected]>: >> >> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L >> <[email protected]> >> wrote: >>>> Fix async subprocess execution with proper handling of std* streams. >>> >>> So I fixed the issue with output in Parts. I not saying it could not >>> be clean up some more. However it solves the output issues, and adds >>> coloring ( which seems to need some fixing in certain cases.. ie mac >>> mostly) and logging support. It might not be too hard for someone to >>> move the code over to SCons and integrated its usage into SCons. >> >> Do you have, by any chance a description of output issues? >> It seems that I've lost my list during the last Windows partition crash. >> >> Also, can you pinpoint the parts in Parts that are relevant to the issues? >> I may have time to port them over to SCons. >> >> My approach >> https://bitbucket.org/techtonik/absub >> to make subprocess asynchronous by moving processing of input/output >> streams into separate threads. I don't remember why, but the work >> stalled. I remember there was some kind of error that I was not able >> to debug due to distributed nature of the process. I needed to write >> more tests for that, but it is either I run out of time, multithreaded >> tests were complicated or I just had an annual Windows filesystem >> crash. >> >> Now I am thinking about replacing concept of input/output streams with >> the concept of channel, which is a queue of strings with limited size. >> The queue can be rotated to become unlimited. Can be synchronous to >> wait (block) until messages are read if the queue is full. This may >> help to clear confusion around all that input output streams handling, >> pipes and descriptor inheritances. >> >> -- >> anatoly t. >> _______________________________________________ >> Scons-dev mailing list >> [email protected] >> http://two.pairlist.net/mailman/listinfo/scons-dev >> >> >> >> >> >> -- >> Regards, >> Eugene Leskinen >> >> >> _______________________________________________ >> Scons-dev mailing list >> [email protected] >> http://two.pairlist.net/mailman/listinfo/scons-dev >> > > > > -- > anatoly t. > _______________________________________________ > Scons-dev mailing list > [email protected] > http://two.pairlist.net/mailman/listinfo/scons-dev > > _______________________________________________ > Scons-dev mailing list > [email protected] > http://two.pairlist.net/mailman/listinfo/scons-dev > -- anatoly t. _______________________________________________ Scons-dev mailing list [email protected] http://two.pairlist.net/mailman/listinfo/scons-dev
