Hi Hieu, OK, but then why does the filtering script have an option for using the binary "-Binarizer" Phrase Table?
Thanks, Steve *~/mosesdecoder/scripts/training/filter-model-given-input.pl <http://filter-model-given-input.pl/> filtered-newstest2011 mert-work/moses.ini ~/corpus/newstest2011.true.fr <http://newstest2011.true.fr/> -Binarizer ~/mosesdecoder/bin/processPhraseTableMin* On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <[email protected]> wrote: > ah. from the moses.ini file, it looks like you've binarized the > phrase-table (and the re-ordering model?) already. > > The filtering script only works with moses.ini which has the original text > pt and reordering files > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 16 September 2016 at 09:13, Steve Braich <[email protected]> wrote: > >> I included the moses.ini file in the original post. >> >> I checked in VIM using :set list. Everything looks good. I repost it >> here: >> >> # MERT optimized configuration >> # decoder /home/steve/mosesdecoder/bin/moses >> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >> # We were before running iteration 5 >> # finished Thu Sep 15 01:53:22 PDT 2016 >> ### MOSES CONFIG FILE ### >> ######################### >> >> # input factors >> [input-factors] >> 0 >> >> # mapping steps >> [mapping] >> 0 T 0 >> >> [distortion-limit] >> 6 >> >> # feature functions >> [feature] >> UnknownWordPenalty >> WordPenalty >> PhrasePenalty >> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >> path=/home/steve/working/binarised-model/phrase-table.minphr >> input-factor=0 output-factor=0 >> LexicalReordering name=LexicalReordering0 num-features=6 >> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >> path=/home/steve/working/binarised-model/reordering-table >> Distortion >> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >> order=3 >> >> # dense weights for feature functions >> >> [threads] >> 4 >> [weight] >> >> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 >> 0.0348721 >> Distortion0= 0.0721205 >> LM0= 0.0908638 >> WordPenalty0= -0.153776 >> PhrasePenalty0= 0.0874059 >> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >> UnknownWordPenalty0= 1 >> >> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <[email protected]> wrote: >> >>> look inside the file >>> mert-work/moses.ini >>> is there anything strange going on? >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 16 September 2016 at 09:02, Steve Braich <[email protected]> wrote: >>> >>>> Hi Hieu, >>>> I changed from relative path to absolute. Still same error. >>>> >>>> Thanks, >>>> Steve >>>> >>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <[email protected]> >>>> wrote: >>>> >>>>> don't use relative paths, ie. change >>>>> filtered-newstest2011 -> /whatever/filtered-newstest2011 >>>>> mert-work/moses.ini -> /whatever/mert-work/moses.ini >>>>> >>>>> >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> On 16 September 2016 at 05:42, Steve Braich <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I am going thru the steps described in the baseline system >>>>>> instructions. I have no problems until the very end when I try to filter >>>>>> out the test data from the binarised model. >>>>>> >>>>>> >>>>>> >>>>>> Here is the step in the instructions where the error appears: >>>>>> >>>>>> *The model that we've trained can then be filtered for this test set, >>>>>> meaning that we only retain the entries needed translate the test set. >>>>>> This >>>>>> will make the translation a lot faster.* >>>>>> >>>>>> >>>>>> >>>>>> *cd ~/working* >>>>>> >>>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>> <http://filter-model-given-input.pl> filtered-newstest2011 >>>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr >>>>>> <http://newstest2011.true.fr> -Binarizer >>>>>> ~/mosesdecoder/bin/processPhraseTableMin* >>>>>> >>>>>> >>>>>> >>>>>> Here is the error message: >>>>>> >>>>>> *Can't open '< /home/steve/working/binarised-model/reordering-table' >>>>>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>> <http://filter-model-given-input.pl> line 439.* >>>>>> >>>>>> >>>>>> >>>>>> Keep in mind the following: >>>>>> >>>>>> - I compiled moses with the following parameters. It is the same >>>>>> compilation. I did not recompile moses at any point during the >>>>>> process. >>>>>> ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0 >>>>>> --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0 >>>>>> NOTE: I did try recompiling with the absolute path for cmph. >>>>>> Same error. >>>>>> - I checked all paths first to make sure they exist. >>>>>> - I am able to translate segments using the binarised model with >>>>>> no problem >>>>>> - I provided my moses.ini file below. >>>>>> >>>>>> Questions: >>>>>> >>>>>> - Can I recompile moses after training and/or tuning, and expect >>>>>> that it should work for testing? I ask because I want to know if the >>>>>> solution is to modify the compile options, do I have to start from >>>>>> scratch >>>>>> again? (Can you filter a binarised model using moses with different >>>>>> compile options that were used to train the model?) >>>>>> >>>>>> Thanks, >>>>>> Steve >>>>>> >>>>>> >>>>>> Moses.ini file >>>>>> >>>>>> # MERT optimized configuration >>>>>> >>>>>> # decoder /home/steve/mosesdecoder/bin/moses >>>>>> >>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>>>>> >>>>>> # We were before running iteration 5 >>>>>> >>>>>> # finished Thu Sep 15 01:53:22 PDT 2016 >>>>>> >>>>>> ### MOSES CONFIG FILE ### >>>>>> >>>>>> ######################### >>>>>> >>>>>> >>>>>> # input factors >>>>>> >>>>>> [input-factors] >>>>>> >>>>>> 0 >>>>>> >>>>>> >>>>>> # mapping steps >>>>>> >>>>>> [mapping] >>>>>> >>>>>> 0 T 0 >>>>>> >>>>>> >>>>>> [distortion-limit] >>>>>> >>>>>> 6 >>>>>> >>>>>> >>>>>> # feature functions >>>>>> >>>>>> [feature] >>>>>> >>>>>> UnknownWordPenalty >>>>>> >>>>>> WordPenalty >>>>>> >>>>>> PhrasePenalty >>>>>> >>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr >>>>>> input-factor=0 output-factor=0 >>>>>> >>>>>> LexicalReordering name=LexicalReordering0 num-features=6 >>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>>>>> path=/home/steve/working/binarised-model/reordering-table >>>>>> >>>>>> Distortion >>>>>> >>>>>> KENLM name=LM0 factor=0 >>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>>>>> order=3 >>>>>> >>>>>> >>>>>> # dense weights for feature functions >>>>>> >>>>>> >>>>>> [threads] >>>>>> >>>>>> 4 >>>>>> >>>>>> [weight] >>>>>> >>>>>> >>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 >>>>>> 0.0348721 >>>>>> >>>>>> Distortion0= 0.0721205 >>>>>> >>>>>> LM0= 0.0908638 >>>>>> >>>>>> WordPenalty0= -0.153776 >>>>>> >>>>>> PhrasePenalty0= 0.0874059 >>>>>> >>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>>>>> >>>>>> UnknownWordPenalty0= 1 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Moses-support mailing list >>>>>> [email protected] >>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
