Ah, that makes sense. However, the instructions seem to lead you down the path I took of changing the moses.ini file to use the binary phrase table, to test how fast it is, THEN it does the filtering step.
Anyway, that resolved it. I am running the BLEU on it now. Thanks Hieu! Steve On Fri, Sep 16, 2016 at 2:17 AM, Hieu Hoang <hieuho...@gmail.com> wrote: > it will binarize AFTER it filters. It can't filter already binarized tables > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 16 September 2016 at 11:10, Steve Braich <steveb...@gmail.com> wrote: > >> Hi Hieu, >> >> OK, but then why does the filtering script have an option for using the >> binary "-Binarizer" Phrase Table? >> >> Thanks, >> Steve >> >> *~/mosesdecoder/scripts/training/filter-model-given-input.pl >> <http://filter-model-given-input.pl/> filtered-newstest2011 >> mert-work/moses.ini ~/corpus/newstest2011.true.fr >> <http://newstest2011.true.fr/> -Binarizer >> ~/mosesdecoder/bin/processPhraseTableMin* >> >> On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <hieuho...@gmail.com> wrote: >> >>> ah. from the moses.ini file, it looks like you've binarized the >>> phrase-table (and the re-ordering model?) already. >>> >>> The filtering script only works with moses.ini which has the original >>> text pt and reordering files >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 16 September 2016 at 09:13, Steve Braich <steveb...@gmail.com> wrote: >>> >>>> I included the moses.ini file in the original post. >>>> >>>> I checked in VIM using :set list. Everything looks good. I repost it >>>> here: >>>> >>>> # MERT optimized configuration >>>> # decoder /home/steve/mosesdecoder/bin/moses >>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>>> # We were before running iteration 5 >>>> # finished Thu Sep 15 01:53:22 PDT 2016 >>>> ### MOSES CONFIG FILE ### >>>> ######################### >>>> >>>> # input factors >>>> [input-factors] >>>> 0 >>>> >>>> # mapping steps >>>> [mapping] >>>> 0 T 0 >>>> >>>> [distortion-limit] >>>> 6 >>>> >>>> # feature functions >>>> [feature] >>>> UnknownWordPenalty >>>> WordPenalty >>>> PhrasePenalty >>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>>> path=/home/steve/working/binarised-model/phrase-table.minphr >>>> input-factor=0 output-factor=0 >>>> LexicalReordering name=LexicalReordering0 num-features=6 >>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>>> path=/home/steve/working/binarised-model/reordering-table >>>> Distortion >>>> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>>> order=3 >>>> >>>> # dense weights for feature functions >>>> >>>> [threads] >>>> 4 >>>> [weight] >>>> >>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 >>>> 0.0348721 >>>> Distortion0= 0.0721205 >>>> LM0= 0.0908638 >>>> WordPenalty0= -0.153776 >>>> PhrasePenalty0= 0.0874059 >>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>>> UnknownWordPenalty0= 1 >>>> >>>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <hieuho...@gmail.com> >>>> wrote: >>>> >>>>> look inside the file >>>>> mert-work/moses.ini >>>>> is there anything strange going on? >>>>> >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> On 16 September 2016 at 09:02, Steve Braich <steveb...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Hieu, >>>>>> I changed from relative path to absolute. Still same error. >>>>>> >>>>>> Thanks, >>>>>> Steve >>>>>> >>>>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <hieuho...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> don't use relative paths, ie. change >>>>>>> filtered-newstest2011 -> /whatever/filtered-newstest2011 >>>>>>> mert-work/moses.ini -> /whatever/mert-work/moses.ini >>>>>>> >>>>>>> >>>>>>> Hieu Hoang >>>>>>> http://www.hoang.co.uk/hieu >>>>>>> >>>>>>> On 16 September 2016 at 05:42, Steve Braich <steveb...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am going thru the steps described in the baseline system >>>>>>>> instructions. I have no problems until the very end when I try to >>>>>>>> filter >>>>>>>> out the test data from the binarised model. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Here is the step in the instructions where the error appears: >>>>>>>> >>>>>>>> *The model that we've trained can then be filtered for this test >>>>>>>> set, meaning that we only retain the entries needed translate the test >>>>>>>> set. >>>>>>>> This will make the translation a lot faster.* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *cd ~/working* >>>>>>>> >>>>>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>>>> <http://filter-model-given-input.pl> filtered-newstest2011 >>>>>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr >>>>>>>> <http://newstest2011.true.fr> -Binarizer >>>>>>>> ~/mosesdecoder/bin/processPhraseTableMin* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Here is the error message: >>>>>>>> >>>>>>>> *Can't open '< >>>>>>>> /home/steve/working/binarised-model/reordering-table' at >>>>>>>> /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>>>> <http://filter-model-given-input.pl> line 439.* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Keep in mind the following: >>>>>>>> >>>>>>>> - I compiled moses with the following parameters. It is the >>>>>>>> same compilation. I did not recompile moses at any point during the >>>>>>>> process. >>>>>>>> ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0 >>>>>>>> --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0 >>>>>>>> NOTE: I did try recompiling with the absolute path for cmph. >>>>>>>> Same error. >>>>>>>> - I checked all paths first to make sure they exist. >>>>>>>> - I am able to translate segments using the binarised model >>>>>>>> with no problem >>>>>>>> - I provided my moses.ini file below. >>>>>>>> >>>>>>>> Questions: >>>>>>>> >>>>>>>> - Can I recompile moses after training and/or tuning, and >>>>>>>> expect that it should work for testing? I ask because I want to >>>>>>>> know if >>>>>>>> the solution is to modify the compile options, do I have to start >>>>>>>> from >>>>>>>> scratch again? (Can you filter a binarised model using moses with >>>>>>>> different compile options that were used to train the model?) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Steve >>>>>>>> >>>>>>>> >>>>>>>> Moses.ini file >>>>>>>> >>>>>>>> # MERT optimized configuration >>>>>>>> >>>>>>>> # decoder /home/steve/mosesdecoder/bin/moses >>>>>>>> >>>>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>>>>>>> >>>>>>>> # We were before running iteration 5 >>>>>>>> >>>>>>>> # finished Thu Sep 15 01:53:22 PDT 2016 >>>>>>>> >>>>>>>> ### MOSES CONFIG FILE ### >>>>>>>> >>>>>>>> ######################### >>>>>>>> >>>>>>>> >>>>>>>> # input factors >>>>>>>> >>>>>>>> [input-factors] >>>>>>>> >>>>>>>> 0 >>>>>>>> >>>>>>>> >>>>>>>> # mapping steps >>>>>>>> >>>>>>>> [mapping] >>>>>>>> >>>>>>>> 0 T 0 >>>>>>>> >>>>>>>> >>>>>>>> [distortion-limit] >>>>>>>> >>>>>>>> 6 >>>>>>>> >>>>>>>> >>>>>>>> # feature functions >>>>>>>> >>>>>>>> [feature] >>>>>>>> >>>>>>>> UnknownWordPenalty >>>>>>>> >>>>>>>> WordPenalty >>>>>>>> >>>>>>>> PhrasePenalty >>>>>>>> >>>>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>>>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr >>>>>>>> input-factor=0 output-factor=0 >>>>>>>> >>>>>>>> LexicalReordering name=LexicalReordering0 num-features=6 >>>>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>>>>>>> path=/home/steve/working/binarised-model/reordering-table >>>>>>>> >>>>>>>> Distortion >>>>>>>> >>>>>>>> KENLM name=LM0 factor=0 >>>>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>>>>>>> order=3 >>>>>>>> >>>>>>>> >>>>>>>> # dense weights for feature functions >>>>>>>> >>>>>>>> >>>>>>>> [threads] >>>>>>>> >>>>>>>> 4 >>>>>>>> >>>>>>>> [weight] >>>>>>>> >>>>>>>> >>>>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 >>>>>>>> 0.0141898 0.0348721 >>>>>>>> >>>>>>>> Distortion0= 0.0721205 >>>>>>>> >>>>>>>> LM0= 0.0908638 >>>>>>>> >>>>>>>> WordPenalty0= -0.153776 >>>>>>>> >>>>>>>> PhrasePenalty0= 0.0874059 >>>>>>>> >>>>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>>>>>>> >>>>>>>> UnknownWordPenalty0= 1 >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Moses-support mailing list >>>>>>>> Moses-support@mit.edu >>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
