Hi Ed I have now had a bit more of a look at this so here are some comments
On Thu, Aug 25, 2011 at 2:21 PM, Edward d'Auvergne <[email protected]>wrote: > Hi, > > That's great that you have time, to have this working properly is > quite important. I didn't realise how much of an issue the merging of > the multi-processor code and GUI code would be. I have now merged the > multi-processor branch into the main 1.3 line, as well as the new GUI > branch, and that is where the problems are. I think there are at > least 2 problems currently occurring. The first has to do with the > GUI tests: > > $ relax --gui-test > > I looks like that there is a clash of the wxPython App.MainLoop() and > how the master processors of the multi package interact. There > appears to be race conditions, even in the uni-processor fabric. This > only occurs in the test-suite, I have just tested manual operation of > the GUI which works, so this makes things more difficult. The problem > was not existent in the gui_testing branch (svn co > svn+ssh://[email protected]/svn/relax/branches/gui_testing/@14200) > until after the multi-processor code was merged. > > I presume this problem is solved and it was due to either a deadlock or calling the wx event loop outside of its main thread. Looking at the source code there seems to be quite a bit of thread specific code(?) One of the aims of the multiprocessor frame-work was to avoid directly waiting on threads and locks etc. This is why the whole design is based round queues and callbacks as they are quite simple to deal with and don't directly expose any underlying threaded model. If you would like me to expand on this more please do say. If there are still multi threaded issues please do say and give some details of how to cause event loop problems or deadlocks > The second problem is IO redirection. This occurs in a number of > places in relax. These include: > > 1) The --log command line flag which causes STDOUT and STDERR to be > sent to file. > > 2) The --tee command line flag which causes STDOUT and STDERR to be > sent both to file and to the terminal. > > 3) The test suite. The STDOUT and STDERR streams are caught and only > sent to STDERR if there is an error or failure in a test. > > 4) The relax controller (part of the GUI). This is a window to which > STDOUT and STDERR are directed. In the test-suite mode, the streams > also output to the terminal. > > 5) The multi-processor package. There are two parts. The first is > essentially a pre-filter which prepends certain tokens to the IO > stream (i.e. the 'M |', 'M E|', and 'S 1|' text). I cannot see how > we can do this as 4) is always set up after 5). So I am considering > removing this part. It will make it more difficult with debugging, > but I can see no other way. > > 6) The second part for the multi-processor package, which is > currently non-functional, is the catching of the IO streams of the > slave processes to send back to the master. I will try to mimic the > relax controller code here and store all slave text as a list with > flags specifying whether it is STDOUT or STDERR. Then the list can be > returned to the master at which point the text can be sent to the two > streams. > > The problem is that at each point here, sys.stdout and sys.stderr are > replaced and the order in which this happens is impossible to change. > Well 4) will always be last. > I think the problem here is that the whole idea of replacing the std io streams multiple times is inflexible and painful. So I have come up with a strawman multiplexed io implimentation. The idea is that you replace stdio and stderr once and then insert IO elements to deal with the needs to block the output of io streams, record them and create Tees etc see what you think? Should we open a couple of new mail threads to discuss these things? regards gary below id the multiplexed io system ''' Created on 16 Sep 2011 @author: garyt ''' import unittest import sys class BaseIOElement: def __init__(self,name=""): self.name = name def get_name(self): return self.name class Multiplex_IO: def __init__(self,stream): self.stream = stream self.stack = [] def add_stream(self,stream): self.stack.append(stream) def write(self, string): for stream in self.stack: string = stream.write(string) if string == None: break if string != None: self.stream.write(string) def get_plexers(self): return list(self.stack) def set_plexers(self,stack): self.stack = list(stack) def flush(self): self.stream.flush() def __str__(self): names = [plexer.get_name() for plexer in self.stack] return " ".join(names) class PrependIO (BaseIOElement): def __init__(self, token="", name="prepend"): self.token = token # self.first_time =True def write(self,string): string = string.replace('\n', '\n' + self.token) # Handle the first line of output. # if self.first_time == True: # string = '\n' + self.token + string # self.first_time = False return string class CaptureIO(BaseIOElement): def __init__(self,name="capture"): self.buffer = [] def write(self,string): self.buffer.append(string) return string def get_value(self): return ''.join(self.buffer) def clear(self): self.buffer = [] class TeeIO(BaseIOElement): def __init__(self,stream, name="tee"): self.stream = stream def write(self,string): self.stream.write(string) return string class NullIO(BaseIOElement): def __init__(self): self.enabled = True def set_enaled(self, state): self.enabled = (state == True) def write(self,string): return None class Test: #class Test(unittest.TestCase): def setUp(self): pass def tearDown(self): pass def testPassThru(self): sys.stderr = Multiplex_IO(sys.stderr) sys.stderr.add_stream(PrependIO("wibble> ")) my_capture = CaptureIO() sys.stderr.add_stream(my_capture) print >> sys.stderr, "test" print >> sys.stderr, "test 2" print print "==capture start==", print my_capture.get_value() print "==capture end==" my_capture.clear() print "==clear capture start==", print my_capture.get_value() print "==clear capture end==" my_tee = TeeIO(sys.stdout) sys.stderr.add_stream(my_tee) print >> sys.stderr, "woggle" sys.stderr.add_stream(NullIO()) print >> sys.stderr, "you shouldn't see this" print sys.stderr if __name__ == "__main__": #import sys;sys.argv = ['', 'Test.testName'] # unittest.main() Test().testPassThru() > > Regards, > > Edward > > > On 25 August 2011 14:51, Gary Thompson <[email protected]> wrote: > > On 08/25/2011 11:40 AM, Edward d'Auvergne wrote: > >> > >> Hi, > >> > >> I'm working on this at the moment. Unfortunately the main 1.3 line is > >> severely broken at the moment! The new multi-processor package is > >> clashing with and causing relax to die hard on the test-suite and the > >> GUI. The main cause is IO redirection of sys.stdout and sys.stderr. > >> The multi-processor code (even in uni-processor mode) is hijacking the > >> streams and the test suite and GUI do not know what to do anymore. > >> Once I eliminate all the IO redirection of the multi package, apart > >> from the IO capture on the slave processes which is non-functional > >> anyway, then the test-suite should be back in order. I noticed you > >> performed an svnmerge, and this is likely causing the breakages in > >> your branch. > >> > >> Regards, > >> > >> Edward > >> > >> > > > > Hi Ed > > I do at last have time to work on this, would you like me to have a look > > at what is going on? > > > > which branch do I need to look at? and what are the specific problems > > with the io redirect that need to be addressed? > > > > regards > > gary > > > > > >> On 25 August 2011 12:23, Sébastien Morin<[email protected]> > >> wrote: > >>> > >>> Hi Ed, > >>> > >>> While working on "inversion-recovery" branch, I realized that the > system > >>> tests "Relax_fit.check_curve_fitting__exp_2param_neg" and > >>> "Relax_fit.check_curve_fitting__exp_3param_inv_neg" were failing. In > >>> order > >>> to pin point the problem, I checked on the main 1.3 branch (514439) and > >>> system test "Relax_fit.check_curve_fitting" also failed, with the same > >>> error: > >>> > >>> ========== > >>> ====================================================================== > >>> ERROR: check_curve_fitting > (test_suite.system_tests.relax_fit.Relax_fit) > >>> Check the results of the curve-fitting. > >>> ---------------------------------------------------------------------- > >>> > >>> relax> pipe.create(pipe_name='mf', pipe_type='mf') > >>> Traceback (most recent call last): > >>> File > >>> > >>> > "/Users/semor/Documents/pse-4/collaborations/relax/relax-1.3/test_suite/system_tests/relax_fit.py", > >>> line 60, in check_curve_fitting > >>> self.assertEqual(cdp.curve_type, 'exp') > >>> AttributeError: 'PipeContainer' object has no attribute 'curve_type' > >>> > >>> ---------------------------------------------------------------------- > >>> Ran 1 test in 0.001s > >>> > >>> FAILED (errors=1) > >>> ========== > >>> > >>> I tried changing the pipe type from "mf" to "relax_fit" (as it should > >>> be). > >>> This did not solve the issue... > >>> > >>> Any idea ? > >>> Thanks ! > >>> > >>> > >>> Séb :) > >>> > >>> -- > >>> Sébastien Morin, Ph.D. > >>> Postdoctoral Fellow, S. Grzesiek NMR Laboratory > >>> Department of Structural Biology > >>> Biozentrum, Universität Basel > >>> Klingelbergstrasse 70 > >>> 4056 Basel > >>> Switzerland > >>> > >>> > >>> _______________________________________________ > >>> relax (http://nmr-relax.com) > >>> > >>> This is the relax-devel mailing list > >>> [email protected] > >>> > >>> To unsubscribe from this list, get a password > >>> reminder, or change your subscription options, > >>> visit the list information page at > >>> https://mail.gna.org/listinfo/relax-devel > >>> > >> _______________________________________________ > >> relax (http://nmr-relax.com) > >> > >> This is the relax-devel mailing list > >> [email protected] > >> > >> To unsubscribe from this list, get a password > >> reminder, or change your subscription options, > >> visit the list information page at > >> https://mail.gna.org/listinfo/relax-devel > > > > > > -- > > ------------------------------------------------------------------- > > Dr Gary Thompson [Homans Lab Research Coordinator] > > > > Astbury Centre for Structural Molecular Biology, > > University of Leeds, > > Leeds, LS2 9JT, West-Yorkshire, UK Tel. +44-113-3433024 > > email: [email protected] Fax +44-113-3431935 > > ------------------------------------------------------------------- > > > > > > _______________________________________________ > relax (http://nmr-relax.com) > > This is the relax-devel mailing list > [email protected] > > To unsubscribe from this list, get a password > reminder, or change your subscription options, > visit the list information page at > https://mail.gna.org/listinfo/relax-devel >
_______________________________________________ relax (http://nmr-relax.com) This is the relax-devel mailing list [email protected] To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-devel
