Thanks,
and if I wanted Moses to use one feature of TM alone I'm guessing there's no way to do this other than zeroing out the undesired TM features? ________________________________ From: Hieu Hoang <[email protected]> Sent: Wednesday, January 13, 2016 3:05 PM To: Read, James C Cc: Moses Support Subject: Re: [Moses-support] Tuning with no language model you can delete all feature functions except for the UnknownWordPenalty. In the current moses, thats hardcoded into decoding. So be my guest and delete away! Hieu Hoang http://www.hoang.co.uk/hieu On 13 January 2016 at 15:02, Read, James C <[email protected]<mailto:[email protected]>> wrote: OK, looks like it's running. Probably won't be able to see if it generates useful weights until tomorrow. Thanks. As a side line of thought. I was wondering how many lines of configuration I could get away with deleting? How would Moses behave if I were to delete the lexical reordering lines? The distortion? The word penalty? The phrase penalty? My goal is to get Moses to choose the best translation with reference to the translation model only. If I delete these other configuration lines will Moses use defaults for these other options or completely disable their operation leaving just the TM? ________________________________ From: Hieu Hoang <[email protected]<mailto:[email protected]>> Sent: Wednesday, January 13, 2016 2:51 PM To: Read, James C Cc: Moses Support Subject: Re: [Moses-support] Tuning with no language model ok. The mert script create a temporary directory every time you run it. By default it's named mert-work Since you ran mert with an incorrect moses.ini previously, it may have polluted the temporary directory and cause problem now. You should find this temporary directory and delete it before running mert again. ps ALL directories must be absolute, eg ../tuning_data/true.bg<http://true.bg/> Hieu Hoang http://www.hoang.co.uk/hieu On 13 January 2016 at 14:45, Read, James C <[email protected]<mailto:[email protected]>> wrote: /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$ /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> -no-filter-phrase-table ../tuning_data/true.bg<http://true.bg> ../tuning_data/true.en /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/moses-tm.ini --mertdir /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ with absolute path to moses-tm.ini does not resolve the problem. Still get the same error. ________________________________ From: Read, James C Sent: Wednesday, January 13, 2016 2:40 PM To: Hieu Hoang Cc: Moses Support Subject: Re: [Moses-support] Tuning with no language model This is what I get when I run the same command as you: LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 Distortion0= 0.300000 UnknownWordPenalty0 UNTUNEABLE WordPenalty0= -1.000000 PhrasePenalty0= 0.200000 TranslationModel0= 0.200000 0.200000 0.200000 0.200000 Looks just like your output. However, this is what I get when I run the mert script with the following command: /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$ /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> -no-filter-phrase-table ../tuning_data/true.bg<http://true.bg> ../tuning_data/true.en /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses binarised/moses-tm.ini --mertdir /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading Distortion0 Loading TranslationModel0 LM0 The following weights have no feature function. Maybe incorrectly spelt weights: LM0,Exit code: 1 The decoder died. CONFIG WAS -weight-overwrite 'PhrasePenalty0= 0.043478 WordPenalty0= -0.217391 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 Distortion0= 0.065217 LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217' ________________________________ From: Hieu Hoang <[email protected]<mailto:[email protected]>> Sent: Wednesday, January 13, 2016 2:27 PM To: Read, James C Cc: Moses Support Subject: Re: [Moses-support] Tuning with no language model looks fine to me. This is what I get when I used it to run # $MOSES_DIR/bin/moses -f moses.ini -show-weights .... LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 Distortion0= 0.300000 UnknownWordPenalty0 UNTUNEABLE WordPenalty0= -1.000000 PhrasePenalty0= 0.200000 TranslationModel0= 0.200000 0.200000 0.200000 0.200000 Double check that the moses.ini file you sent me is truly what you're using. To be sure, you MUST use absolute paths for all file names, ie. not binarised/moses-tm.ini but /media/bigdata/whatever/binarised/moses-tm.ini On 13/01/16 14:16, Read, James C wrote: ######################### ### MOSES CONFIG FILE ### ######################### # input factors [input-factors] 0 # mapping steps [mapping] 0 T 0 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryCompact name=TranslationModel0 num-features=4 path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/phrase-table input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/reordering-table Distortion # dense weights for feature functions [weight] # The default weights are NOT optimized for translation quality. You MUST tune the weights. # Documentation for tuning is here: http://www.statmt.org/moses/?n=FactoredTraining.Tuning [http://www.statmt.org/moses/img/coin-tiny.png]<http://www.statmt.org/moses/?n=FactoredTraining.Tuning> Moses - FactoredTraining/Tuning<http://www.statmt.org/moses/?n=FactoredTraining.Tuning> www.statmt.org<http://www.statmt.org> Tuning. Overview. During decoding, Moses scores translation hypotheses using a linear model. In the traditional approach, the features of the model are the ... UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> -no-filter-phrase-table ../tuning_data/true.bg<http://true.bg> ../tuning_data/true.en /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses binarised/moses-tm.ini --mertdir /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/ ________________________________ From: Hieu Hoang <[email protected]><mailto:[email protected]> Sent: Wednesday, January 13, 2016 2:05 PM To: Read, James C Cc: Moses Support Subject: Re: [Moses-support] Tuning with no language model please show me the EXACT moses.ini and command that you use. Hieu Hoang http://www.hoang.co.uk/hieu On 13 January 2016 at 14:04, Read, James C <[email protected]<mailto:[email protected]>> wrote: So now I get the following error: Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading Distortion0 Loading TranslationModel0 LM0 The following weights have no feature function. Maybe incorrectly spelt weights: LM0,Exit code: 1 I didn't get this error without deleting LM0 line. ________________________________ From: Hieu Hoang <[email protected]<mailto:[email protected]>> Sent: Wednesday, January 13, 2016 1:58 PM To: Read, James C; Moses Support Subject: Re: [Moses-support] Tuning with no language model there lies your problem. There's a weight for a non-existant feature function On 13/01/16 13:57, Read, James C wrote: no ________________________________ From: Hieu Hoang <[email protected]><mailto:[email protected]> Sent: Wednesday, January 13, 2016 1:56 PM To: Read, James C; Moses Support Subject: Re: [Moses-support] Tuning with no language model did you delete the line LM0= 0.5 too? On 13/01/16 13:53, Read, James C wrote: The following command works fine for when a language model is specified. Deleting the line KENLM lazyken=0 name=LM0 factor=0 path=$PWD/blm.en order=3 causes the script to fail with error: ERROR: Failed to run '/pathtomoses/mosesdecoder/bin/moses -config /pathtodata/binarised/moses.ini -show-weights'. at /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> line 1744. ######################### ### MOSES CONFIG FILE ### ######################### # input factors [input-factors] 0 # mapping steps [mapping] 0 T 0 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryCompact name=TranslationModel0 num-features=4 path=/pathtodata/binarised/phrase-table input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/pathtodata/binarised/reordering-table Distortion KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3 # dense weights for feature functions [weight] # The default weights are NOT optimized for translation quality. You MUST tune the weights. # Documentation for tuning is here: <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> http://www.statmt.org/moses/?n=FactoredTraining.Tuning [http://www.statmt.org/moses/img/coin-tiny.png]<http://www.statmt.org/moses/?n=FactoredTraining.Tuning> Moses - FactoredTraining/Tuning<http://www.statmt.org/moses/?n=FactoredTraining.Tuning> <http://www.statmt.org>www.statmt.org<http://www.statmt.org> Tuning. Overview. During decoding, Moses scores translation hypotheses using a linear model. In the traditional approach, the features of the model are the ... UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> -no-filter-phrase-table true.fr<http://true.fr> true.en /pathtomoses/mosesdecoder/bin/moses binarised/moses.ini --mertdir /pathtomoses/mosesdecoder/bin/ -- Hieu Hoang http://www.hoang.co.uk/hieu -- Hieu Hoang http://www.hoang.co.uk/hieu -- Hieu Hoang http://www.hoang.co.uk/hieu
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
