Hi all, Just to confirm this. I have run the exact setup mentioned here, so multi-jobs on a cluster (lisa.sara.nl) and each job multi-threaded. I had to make some minor adjustments in the way jobs were setup (and especcialy how they wait for each other), specific to the cluster I used. It all runs fine, although the multi-job thing didn't buy me to much time, since much is spend on waiting for a job in the queue and then for loading models.
Cheers, Anne Schuth University of Amsterdam 2009/10/26 Barry Haddow <[email protected]> > Hi Joerg > > mert-moses.pl only sends one job to each instance of mosesmt. > > Suppose that you run mert-moses.pl in the non-clustered configuration > (single > host), but specify mosesmt as the decoder. Then mert-moses.pl will hand > the > whole tuning file (one job) to mosesmt. mosesmt would distribute the work > of > translating the file between its own threads, and write the translations > (and > nbest lists) to disk, in the correct order. > > If you run mert-moses.pl in the clustered configuration, but using > mosesmt, > then mert-moses.pl will split the tuning file up into segments, and send a > segment to each job. Since each of the jobs are themselves multi-threaded, > they would use a pool of threads to process the file, I should add that we > haven't this clustered configuration. > > hope that helps, > > regards > Barry > > > On Monday 26 October 2009 16:16, Jörg Tiedemann wrote: > > Hi, > > > > Thanks for your reply. I'm still a bit puzzled. Would mert-moses really > > send several jobs to the multi-threaded moses when running mert? > > Shouldn't there be something in the script that creates several requests > > simultaneously and collects the results when decoding is done? But > > maybe I misunderstand something here. > > Well, thanks again, > > > > Jörg > > > > Barry Haddow wrote: > > > Hi Joerg > > > > > > Multi-threaded moses should work fine with mert-moses.pl and > > > mert-moses-new.pl. You can set both the decoder and the decoder flags > on > > > the command line, so just change the decoder to mosesmt and add the > > > -threads n argument. moses-parallel should also work with mosesmt, if > you > > > specify the decoder correctly, although I don't think we've tried this. > > > So far we've only used mosesmt on single (multi-core) machines, but > > > there's no reason why it shouldn't work on a cluster, > > > > > > best regards > > > Barry > > > > > > On Monday 26 October 2009 07:41, Jörg Tiedemann wrote: > > >> Follow-up question: Is there already a mert-moses.pl script that > takes > > >> advantage of multi-threaded moses? What would be the best way of > > >> integrating this? Writing a new moses-parallel script that calls > > >> multi-threaded moses in parallel instead of creating cluster jobs? > Best > > >> would be of course to combine both features. Is there anything done in > > >> this direction already? > > >> Thanks in advance! > > >> > > >> Jörg > > >> > > >> Barry Haddow wrote: > > >>> Hi Joerg > > >>> > > >>> Yes, nbest lists work now - I should have updated the website! > > >>> > > >>> best regards > > >>> Barry > > >>> > > >>> On Saturday 24 October 2009 20:05:05 Joerg Tiedemann wrote: > > >>>> Some days ago I found the information about multi-threaded Moses on > > >>>> the website. Now I'm wondering if it is planned to make it possible > to > > >>>> use multi-threaded decoding together with MERT. That would be great! > > >>>> So far it says: > > >>>> "n-best lists are not available, although this should be fixed > > >>>> shortly" which sounds like you're working on this issue?! > > >>>> > > >>>> Let me know if there's any news about it! > > >>>> Thanks! > > >>>> > > >>>> Jörg > > >>>> _______________________________________________ > > >>>> Moses-support mailing list > > >>>> [email protected] > > >>>> http://mailman.mit.edu/mailman/listinfo/moses-support > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
