It's not exactly Moses, but one recent moses-support thread uncovered MGIZA++'s merge_alignment.py script fails with more than 10 CPU's because MGIZA++ part files are incremented by ascii codes (file names 0-9 continuing with ":", ";" "<", etc. but merge_alignment.py increments its counter in dec. This should be an easy candidate so update for mega-core multi-threading.
On Thu, 19 Apr 2012 09:57:43 +0700, Hieu Hoang <[email protected]> wrote: > The score program should be a good candidate for parallelisation. It > take an unnecessarily long time, especially for hiero grammar. Some > diddly stuff you have to do for good-Turing discount > > Adam's suufix array has also been partially integrated into Moses, it > needs to be finished. > > Disk space and io are also issues for hiero grammar > > Hieu > Sent from my flying horse > > On 18 Apr 2012, at 07:01 AM, Philipp Koehn <[email protected]> > wrote: > >> Hi everybody, >> >> the trend in computing is towards machines with more and more CPUs, >> and Moses can already take advantage of this during decoder. Just >> specify "-threads 10" and decoding is ten times faster (well, >> almost, >> and you actually have to have 10 CPUs on you machine). >> >> The training pipeline, however, is mostly single-threaded. >> >> Maybe you want to change that and learn something about threads >> in C++ in the process. >> >> We added some documentation about multi-threading in Moses: >> http://www.statmt.org/moses/?n=Moses.CodeGuideThreading >> which should get you started. >> >> There are several instant candidates: the phrase and rule extractors >> (only the hierarchical extract-rules is multi-threaded), rule >> scoring, >> word alignment symmetrization, etc. Maybe even some code that >> is currently in Perl or Python, such as the rule filtering, >> tokenization >> or truecasing. >> >> -phi >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
