Hi, I think it would be worth creating a temporary branch for this. I now have the new GUI running almost perfectly on GNU/Linux, Mac OS X, and MS Windows (that was a huge effort :S), and will hopefully make a new release very soon. The changing of the IO streaming will likely break relax for a little while, until all redirections are found and modified for a new system. The new release will have the mpi4py multi-processor fabric working in prompt/scripting mode, and the GUI running on the uni-processor fabric. A second release could have the mpi4py fabric running together with the GUI.
I would suggest placing the new code into the relax_io module, as this already has the code for logging, teeing, and splitting IO streams. What do you think about the concept of sinks? We could have the Multiplex_IO class handle multiple streams (more than 2, 3, etc), i.e. split the stdout stream into two, and then send it to two sinks - the terminal or test suite sink, and the relax controller. There could be filters applied to the streams, either prior to or after the split. And the filters could be ordered, i.e. we create an IO processing pipe, just like a piping in a shell. I don't know if we need the merger of streams though. This is currently used by the mpi4py fabric to send the IO from slaves to the terminal, but I would suggest we keep the IO streams separate (or interleaved to preserve order), and send this back to the master for post-processing. Cheers, Edward On 19 September 2011 18:01, Gary Thompson <[email protected]> wrote: > On 09/19/2011 02:52 PM, Edward d'Auvergne wrote: >> >> Hi, >> >> I have to say, that the ideas in the Multiplex_IO look great and would >> be good to incorporate into relax at some point in the future. The >> ideas could be evolved a little more and turned into a module for IO >> manipulation. This could implement filters (pre or post pending text, >> or any other operation) - i.e. the calling code could add a filter >> function to a queue of filters. And it could handle splitting and >> merging of streams. It could turn out to be very powerful. It's a >> real pity something like this does not exist in Python already! I >> would suggest tackling the slave IO capture problem first, and then >> this one later on. >> >> Regards, >> >> Edward > > Hi Edward > > I thinks both should be compatible as discussed in my previous mailing so I > will go ahead with the design. My Intention is to have a general io return > interface methodology and then implement a class to do it so ensuring > independence of implementation.. > > I will most probably try and implement a set of unit tests first... > > regards > gary > > >> >> >> On 19 September 2011 15:22, Edward d'Auvergne<[email protected]> >> wrote: >>>>> >>>>> The second problem is IO redirection. This occurs in a number of >>>>> places in relax. These include: >>>>> >>>>> 1) The --log command line flag which causes STDOUT and STDERR to be >>>>> sent to file. >>>>> >>>>> 2) The --tee command line flag which causes STDOUT and STDERR to be >>>>> sent both to file and to the terminal. >>>>> >>>>> 3) The test suite. The STDOUT and STDERR streams are caught and only >>>>> sent to STDERR if there is an error or failure in a test. >>>>> >>>>> 4) The relax controller (part of the GUI). This is a window to which >>>>> STDOUT and STDERR are directed. In the test-suite mode, the streams >>>>> also output to the terminal. >>>>> >>>>> 5) The multi-processor package. There are two parts. The first is >>>>> essentially a pre-filter which prepends certain tokens to the IO >>>>> stream (i.e. the 'M |', 'M E|', and 'S 1|' text). I cannot see how >>>>> we can do this as 4) is always set up after 5). So I am considering >>>>> removing this part. It will make it more difficult with debugging, >>>>> but I can see no other way. >>>>> >>>>> 6) The second part for the multi-processor package, which is >>>>> currently non-functional, is the catching of the IO streams of the >>>>> slave processes to send back to the master. I will try to mimic the >>>>> relax controller code here and store all slave text as a list with >>>>> flags specifying whether it is STDOUT or STDERR. Then the list can be >>>>> returned to the master at which point the text can be sent to the two >>>>> streams. >>>>> >>>>> The problem is that at each point here, sys.stdout and sys.stderr are >>>>> replaced and the order in which this happens is impossible to change. >>>>> Well 4) will always be last. >>>> >>>> I think the problem here is that the whole idea of replacing the std io >>>> streams multiple times is inflexible and painful. >>> >>> This is true. Python, like many programing languages, does not handle >>> IO streams very nicely. It would be good if you could set up some >>> processing pipe for the IO stream, but this looks very complicated to >>> implement. >>> >>> >>>> So I have come up with a >>>> strawman multiplexed io implimentation. The idea is that you replace >>>> stdio >>>> and stderr once and then insert IO elements to deal with the needs to >>>> block >>>> the output of io streams, record them and create Tees etc see what you >>>> think? Should we open a couple of new mail threads to discuss these >>>> things? >>> >>> Lets see how it goes. But we may need a few new threads. For your >>> code below, it looks like it could be a the start of a solution. The >>> question is, how do we design this so that the multi-processor package >>> is not dependent on relax? Should the Multiplex_IO object be part of >>> relax, or the multi package? I think that the main program should >>> have full control over the IO streams, and that the multi-processor >>> package should use what is available to it. One problem here is that >>> a program could change the IO streams in mid operation. For example >>> if I was to implement a function in the GUI which activates logging. >>> The Python program, as is with relax, could have many, many places >>> where IO streams are manipulated. So it would be more beneficial to >>> have an IO stream manipulation framework within the main program. The >>> multi-processor only needs to prepend text and capture slave IO >>> streams. >>> >>> Maybe an alternative would be to capture and store the IO streams only >>> on the slaves? This could be a fabric-specific implementation. For >>> example the uni-processor would not touch the streams. It would not >>> look as nice, but this would solve all the issues. The slave IO >>> streams are captured, the 'S 1 |', 'S 2 |', etc. text would be >>> prepended, the text sent back to the master via the memo objects, and >>> then the master can send it to what ever sys.stdout and sys.stderr are >>> currently set to by the calling Python program. As I mentioned >>> before, the slave's STDOUT and STDERR order can be preserved by >>> storing it all in a list whereby the IO stream is identified by a >>> flag, as is done in the relax controller GUI window. For example if >>> you have in a slave: >>> >>> sys.stdout.write('test\n') >>> sys.stderr.write('fail\n') >>> >>> Then the object passed to the master via the memo could look like: >>> >>> io =[ >>> ['S 1 | test\n', 0], >>> ['S 2 | fail\n', 1] >>> ] >>> >>> Then on the master: >>> >>> for line, stream in io: >>> if stream == 0: >>> sys.stdout.write(line) >>> else: >>> sys.stderr.write(line) >>> >>> This means that the two slave streams are merged, and then split again >>> by the master, preserving their order. What do you think the overhead >>> of such an operation would be? My guess would be only a few >>> milliseconds per slave process, as most of the work would be on the >>> master. I would guess that the overhead should be similar to the >>> current prependIO code. Only touching IO streams from the slaves >>> created by the multi-processor framework would not clash with any IO >>> manipulations the main program could ever imagine to do. >>> >>> Regards, >>> >>> Edward >>> > > > -- > ------------------------------------------------------------------- > Dr Gary Thompson [Homans Lab Research Coordinator] > > Astbury Centre for Structural Molecular Biology, > University of Leeds, > Leeds, LS2 9JT, West-Yorkshire, UK Tel. +44-113-3433024 > email: [email protected] Fax +44-113-3431935 > ------------------------------------------------------------------- > > > _______________________________________________ relax (http://nmr-relax.com) This is the relax-devel mailing list [email protected] To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-devel
