Hi, I have to say, that the ideas in the Multiplex_IO look great and would be good to incorporate into relax at some point in the future. The ideas could be evolved a little more and turned into a module for IO manipulation. This could implement filters (pre or post pending text, or any other operation) - i.e. the calling code could add a filter function to a queue of filters. And it could handle splitting and merging of streams. It could turn out to be very powerful. It's a real pity something like this does not exist in Python already! I would suggest tackling the slave IO capture problem first, and then this one later on.
Regards, Edward On 19 September 2011 15:22, Edward d'Auvergne <[email protected]> wrote: >>> The second problem is IO redirection. This occurs in a number of >>> places in relax. These include: >>> >>> 1) The --log command line flag which causes STDOUT and STDERR to be >>> sent to file. >>> >>> 2) The --tee command line flag which causes STDOUT and STDERR to be >>> sent both to file and to the terminal. >>> >>> 3) The test suite. The STDOUT and STDERR streams are caught and only >>> sent to STDERR if there is an error or failure in a test. >>> >>> 4) The relax controller (part of the GUI). This is a window to which >>> STDOUT and STDERR are directed. In the test-suite mode, the streams >>> also output to the terminal. >>> >>> 5) The multi-processor package. There are two parts. The first is >>> essentially a pre-filter which prepends certain tokens to the IO >>> stream (i.e. the 'M |', 'M E|', and 'S 1|' text). I cannot see how >>> we can do this as 4) is always set up after 5). So I am considering >>> removing this part. It will make it more difficult with debugging, >>> but I can see no other way. >>> >>> 6) The second part for the multi-processor package, which is >>> currently non-functional, is the catching of the IO streams of the >>> slave processes to send back to the master. I will try to mimic the >>> relax controller code here and store all slave text as a list with >>> flags specifying whether it is STDOUT or STDERR. Then the list can be >>> returned to the master at which point the text can be sent to the two >>> streams. >>> >>> The problem is that at each point here, sys.stdout and sys.stderr are >>> replaced and the order in which this happens is impossible to change. >>> Well 4) will always be last. >> >> I think the problem here is that the whole idea of replacing the std io >> streams multiple times is inflexible and painful. > > This is true. Python, like many programing languages, does not handle > IO streams very nicely. It would be good if you could set up some > processing pipe for the IO stream, but this looks very complicated to > implement. > > >> So I have come up with a >> strawman multiplexed io implimentation. The idea is that you replace stdio >> and stderr once and then insert IO elements to deal with the needs to block >> the output of io streams, record them and create Tees etc see what you >> think? Should we open a couple of new mail threads to discuss these things? > > Lets see how it goes. But we may need a few new threads. For your > code below, it looks like it could be a the start of a solution. The > question is, how do we design this so that the multi-processor package > is not dependent on relax? Should the Multiplex_IO object be part of > relax, or the multi package? I think that the main program should > have full control over the IO streams, and that the multi-processor > package should use what is available to it. One problem here is that > a program could change the IO streams in mid operation. For example > if I was to implement a function in the GUI which activates logging. > The Python program, as is with relax, could have many, many places > where IO streams are manipulated. So it would be more beneficial to > have an IO stream manipulation framework within the main program. The > multi-processor only needs to prepend text and capture slave IO > streams. > > Maybe an alternative would be to capture and store the IO streams only > on the slaves? This could be a fabric-specific implementation. For > example the uni-processor would not touch the streams. It would not > look as nice, but this would solve all the issues. The slave IO > streams are captured, the 'S 1 |', 'S 2 |', etc. text would be > prepended, the text sent back to the master via the memo objects, and > then the master can send it to what ever sys.stdout and sys.stderr are > currently set to by the calling Python program. As I mentioned > before, the slave's STDOUT and STDERR order can be preserved by > storing it all in a list whereby the IO stream is identified by a > flag, as is done in the relax controller GUI window. For example if > you have in a slave: > > sys.stdout.write('test\n') > sys.stderr.write('fail\n') > > Then the object passed to the master via the memo could look like: > > io =[ > ['S 1 | test\n', 0], > ['S 2 | fail\n', 1] > ] > > Then on the master: > > for line, stream in io: > if stream == 0: > sys.stdout.write(line) > else: > sys.stderr.write(line) > > This means that the two slave streams are merged, and then split again > by the master, preserving their order. What do you think the overhead > of such an operation would be? My guess would be only a few > milliseconds per slave process, as most of the work would be on the > master. I would guess that the overhead should be similar to the > current prependIO code. Only touching IO streams from the slaves > created by the multi-processor framework would not clash with any IO > manipulations the main program could ever imagine to do. > > Regards, > > Edward > _______________________________________________ relax (http://nmr-relax.com) This is the relax-devel mailing list [email protected] To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-devel
