you can delete all feature functions except for the UnknownWordPenalty. In the current moses, thats hardcoded into decoding. So be my guest and delete away!
Hieu Hoang http://www.hoang.co.uk/hieu On 13 January 2016 at 15:02, Read, James C <jcr...@essex.ac.uk> wrote: > OK, looks like it's running. Probably won't be able to see if it generates > useful weights until tomorrow. Thanks. > > > As a side line of thought. I was wondering how many lines of configuration > I could get away with deleting? How would Moses behave if I were to delete > the lexical reordering lines? The distortion? The word penalty? The phrase > penalty? > > > My goal is to get Moses to choose the best translation with reference to > the translation model only. > > > If I delete these other configuration lines will Moses use defaults for > these other options or completely disable their operation leaving just the > TM? > > > ------------------------------ > *From:* Hieu Hoang <hieuho...@gmail.com> > *Sent:* Wednesday, January 13, 2016 2:51 PM > > *To:* Read, James C > *Cc:* Moses Support > *Subject:* Re: [Moses-support] Tuning with no language model > > ok. The mert script create a temporary directory every time you run it. By > default it's named > mert-work > Since you ran mert with an incorrect moses.ini previously, it may have > polluted the temporary directory and cause problem now. > > You should find this temporary directory and delete it before running mert > again. > > ps ALL directories must be absolute, eg ../tuning_data/true.bg > > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 13 January 2016 at 14:45, Read, James C <jcr...@essex.ac.uk> wrote: > >> >> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$ >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/ >> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg >> ../tuning_data/true.en >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses >> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/moses-tm.ini >> --mertdir /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ >> >> with absolute path to moses-tm.ini does not resolve the problem. Still >> get the same error. >> >> ------------------------------ >> *From:* Read, James C >> *Sent:* Wednesday, January 13, 2016 2:40 PM >> *To:* Hieu Hoang >> >> *Cc:* Moses Support >> *Subject:* Re: [Moses-support] Tuning with no language model >> >> >> This is what I get when I run the same command as you: >> >> >> LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 >> Distortion0= 0.300000 >> UnknownWordPenalty0 UNTUNEABLE >> WordPenalty0= -1.000000 >> PhrasePenalty0= 0.200000 >> TranslationModel0= 0.200000 0.200000 0.200000 0.200000 >> >> Looks just like your output. However, this is what I get when I run the >> mert script with the following command: >> >> >> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$ >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/ >> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg >> ../tuning_data/true.en >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses >> binarised/moses-tm.ini --mertdir >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ >> >> >> Loading UnknownWordPenalty0 >> Loading WordPenalty0 >> Loading PhrasePenalty0 >> Loading LexicalReordering0 >> Loading Distortion0 >> Loading TranslationModel0 >> LM0 >> The following weights have no feature function. Maybe incorrectly spelt >> weights: LM0,Exit code: 1 >> The decoder died. CONFIG WAS -weight-overwrite 'PhrasePenalty0= 0.043478 >> WordPenalty0= -0.217391 TranslationModel0= 0.043478 0.043478 0.043478 >> 0.043478 Distortion0= 0.065217 LM0= 0.108696 LexicalReordering0= 0.065217 >> 0.065217 0.065217 0.065217 0.065217 0.065217' >> >> >> >> >> >> ------------------------------ >> *From:* Hieu Hoang <hieuho...@gmail.com> >> *Sent:* Wednesday, January 13, 2016 2:27 PM >> *To:* Read, James C >> *Cc:* Moses Support >> *Subject:* Re: [Moses-support] Tuning with no language model >> >> looks fine to me. This is what I get when I used it to run >> # $MOSES_DIR/bin/moses -f moses.ini -show-weights >> .... >> LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 >> Distortion0= 0.300000 >> UnknownWordPenalty0 UNTUNEABLE >> WordPenalty0= -1.000000 >> PhrasePenalty0= 0.200000 >> TranslationModel0= 0.200000 0.200000 0.200000 0.200000 >> >> Double check that the moses.ini file you sent me is truly what you're >> using. >> >> To be sure, you MUST use absolute paths for all file names, ie. not >> binarised/moses-tm.ini >> but >> /media/bigdata/whatever/binarised/moses-tm.ini >> >> On 13/01/16 14:16, Read, James C wrote: >> >> ######################### >> ### MOSES CONFIG FILE ### >> ######################### >> >> # input factors >> [input-factors] >> 0 >> >> # mapping steps >> [mapping] >> 0 T 0 >> >> [distortion-limit] >> 6 >> >> # feature functions >> [feature] >> UnknownWordPenalty >> WordPenalty >> PhrasePenalty >> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >> path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/phrase-table >> input-factor=0 output-factor=0 >> LexicalReordering name=LexicalReordering0 num-features=6 >> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >> path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/reordering-table >> Distortion >> >> # dense weights for feature functions >> [weight] >> # The default weights are NOT optimized for translation quality. You MUST >> tune the weights. >> # Documentation for tuning is here: >> http://www.statmt.org/moses/?n=FactoredTraining.Tuning >> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >> Moses - FactoredTraining/Tuning >> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >> www.statmt.org >> Tuning. Overview. During decoding, Moses scores translation hypotheses >> using a linear model. In the traditional approach, the features of the >> model are the ... >> >> UnknownWordPenalty0= 1 >> WordPenalty0= -1 >> PhrasePenalty0= 0.2 >> TranslationModel0= 0.2 0.2 0.2 0.2 >> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 >> Distortion0= 0.3 >> >> >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/ >> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg >> ../tuning_data/true.en >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses >> binarised/moses-tm.ini --mertdir >> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ >> >> >> ------------------------------ >> *From:* Hieu Hoang <hieuho...@gmail.com> <hieuho...@gmail.com> >> *Sent:* Wednesday, January 13, 2016 2:05 PM >> *To:* Read, James C >> *Cc:* Moses Support >> *Subject:* Re: [Moses-support] Tuning with no language model >> >> please show me the EXACT moses.ini and command that you use. >> >> >> >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 13 January 2016 at 14:04, Read, James C <jcr...@essex.ac.uk> wrote: >> >>> So now I get the following error: >>> >>> >>> >>> Loading UnknownWordPenalty0 >>> Loading WordPenalty0 >>> Loading PhrasePenalty0 >>> Loading LexicalReordering0 >>> Loading Distortion0 >>> Loading TranslationModel0 >>> LM0 >>> The following weights have no feature function. Maybe incorrectly spelt >>> weights: LM0,Exit code: 1 >>> >>> I didn't get this error without deleting LM0 line. >>> >>> ------------------------------ >>> *From:* Hieu Hoang <hieuho...@gmail.com> >>> *Sent:* Wednesday, January 13, 2016 1:58 PM >>> >>> *To:* Read, James C; Moses Support >>> *Subject:* Re: [Moses-support] Tuning with no language model >>> >>> there lies your problem. There's a weight for a non-existant feature >>> function >>> >>> On 13/01/16 13:57, Read, James C wrote: >>> >>> no >>> >>> >>> ------------------------------ >>> *From:* Hieu Hoang <hieuho...@gmail.com> <hieuho...@gmail.com> >>> *Sent:* Wednesday, January 13, 2016 1:56 PM >>> *To:* Read, James C; Moses Support >>> *Subject:* Re: [Moses-support] Tuning with no language model >>> >>> did you delete the line >>> LM0= 0.5 >>> too? >>> >>> On 13/01/16 13:53, Read, James C wrote: >>> >>> The following command works fine for when a language model is specified. >>> Deleting the line KENLM lazyken=0 name=LM0 factor=0 path=$PWD/blm.en order=3 >>> causes the script to fail with error: >>> >>> >>> ERROR: Failed to run '/pathtomoses/mosesdecoder/bin/moses -config >>> /pathtodata/binarised/moses.ini -show-weights'. at >>> /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl line 1744. >>> >>> >>> ######################### >>> ### MOSES CONFIG FILE ### >>> ######################### >>> >>> # input factors >>> [input-factors] >>> 0 >>> >>> # mapping steps >>> [mapping] >>> 0 T 0 >>> >>> [distortion-limit] >>> 6 >>> >>> # feature functions >>> [feature] >>> UnknownWordPenalty >>> WordPenalty >>> PhrasePenalty >>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>> path=/pathtodata/binarised/phrase-table input-factor=0 output-factor=0 >>> LexicalReordering name=LexicalReordering0 num-features=6 >>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>> path=/pathtodata/binarised/reordering-table >>> Distortion >>> KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3 >>> >>> # dense weights for feature functions >>> [weight] >>> # The default weights are NOT optimized for translation quality. You >>> MUST tune the weights. >>> # Documentation for tuning is here: >>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >>> http://www.statmt.org/moses/?n=FactoredTraining.Tuning >>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >>> Moses - FactoredTraining/Tuning >>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >>> <http://www.statmt.org>www.statmt.org >>> Tuning. Overview. During decoding, Moses scores translation hypotheses >>> using a linear model. In the traditional approach, the features of the >>> model are the ... >>> >>> UnknownWordPenalty0= 1 >>> WordPenalty0= -1 >>> PhrasePenalty0= 0.2 >>> TranslationModel0= 0.2 0.2 0.2 0.2 >>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 >>> Distortion0= 0.3 >>> LM0= 0.5 >>> >>> /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl >>> -no-filter-phrase-table true.fr true.en >>> /pathtomoses/mosesdecoder/bin/moses binarised/moses.ini --mertdir >>> /pathtomoses/mosesdecoder/bin/ >>> >>> >>> -- >>> Hieu Hoanghttp://www.hoang.co.uk/hieu >>> >>> >>> -- >>> Hieu Hoanghttp://www.hoang.co.uk/hieu >>> >>> >> >> -- >> Hieu Hoanghttp://www.hoang.co.uk/hieu >> >> >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
