Hi, very interesting indeed. After compiling with KenLM, instead of IRSTLM: What should the tuning command look like?
I ran the following (using IRSTLM): nohup nice ~/mosesdecoder/scripts/training/mert-moses.pl ~/corpora/Total1.sv-fr.clean.slutet_urval.sv ~/corpora/Total1.sv-fr.clean.slutet_urval.fr \ ~/mosesdecoder/bin/moses train/model/moses.ini --decoder-flags="-threads 4" -filtercmd '/home/per/mosesdecoder/scripts/training/filter-model-given-input.pl -Binarizer "~/mosesdecoder/bin/processPhraseTable"' --mertdir ~/mosesdecoder/bin/ &> mert.out & Should I just add --threads after mer-moses.pl ? Further "compile moses to use KenLM and configure the SMT model to use KenLM": 1) compile moses to use KenLM: "KenLM is compiled by default." Should I just remove the flag --with-irstlm=<root dir of the IRSTLM toolkit> ? And add 8 <factor> <size> filename.arpa to moses.ini? 2) looking at http://kheafield.com/code/kenlm/ I suppose I can build a KenLM 3-gram language model by: bin/lmplz -o 3 -S 80% -T /tmp <text >text.arpa Is there any more to it? Yours, Per Tunedal On Mon, Apr 29, 2013, at 17:49, Tom Hoar wrote: > When you said "it didn't work," what do you mean? How many cores were on > the tuning machine? You should also run mert-moses.pl with the --threads > option so the mert binary runs multithreaded. That's in addition to the > --decoder-flags "-threads all" option Ken mentioned, which tells the > moses binary to run multithreaded. > > You also have to compile moses to use KenLM and configure the SMT model > to use KenLM, not IRSTLM. IRSTLM is still single threaded. Most of the > tuning time is moses creating the translations. Moses will run single > threaded when configured IRSTLM. > > Tom > > > On 04/29/2013 10:33 PM, Arezki Sadoune wrote: > > Dear All, > > > > I'm currently working on a Phrase-based model from french to english. > > Assuming that the bitext corpora is very large, is there any way to > > use the multi-thread for the tuning purpose? > > > > I've already tried by the past to tune a similar system but it has > > taken 30 days on a single core. > > > > I've actually tried multithreaded tuning by adding the option -threads > > 16 to the mert script parameter ( > > /mosesdecoder/scripts/training/mert-moses.pl > > home/Moses/mosesdecoder/tunning1/tunning.true.fr > > /home/Moses/mosesdecoder/tunning1/tunning.true.en > > /home/Moses/mosesdecoder/bin/moses -threads 16 ...) > > > > but it didn't work. > > > > Thanks a lot > > > > Az > > > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
