ah. from the moses.ini file, it looks like you've binarized the phrase-table (and the re-ordering model?) already.
The filtering script only works with moses.ini which has the original text pt and reordering files Hieu Hoang http://www.hoang.co.uk/hieu On 16 September 2016 at 09:13, Steve Braich <[email protected]> wrote: > I included the moses.ini file in the original post. > > I checked in VIM using :set list. Everything looks good. I repost it > here: > > # MERT optimized configuration > # decoder /home/steve/mosesdecoder/bin/moses > # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr > # We were before running iteration 5 > # finished Thu Sep 15 01:53:22 PDT 2016 > ### MOSES CONFIG FILE ### > ######################### > > # input factors > [input-factors] > 0 > > # mapping steps > [mapping] > 0 T 0 > > [distortion-limit] > 6 > > # feature functions > [feature] > UnknownWordPenalty > WordPenalty > PhrasePenalty > PhraseDictionaryCompact name=TranslationModel0 num-features=4 > path=/home/steve/working/binarised-model/phrase-table.minphr > input-factor=0 output-factor=0 > LexicalReordering name=LexicalReordering0 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/home/steve/working/binarised-model/reordering-table > Distortion > KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en > order=3 > > # dense weights for feature functions > > [threads] > 4 > [weight] > > LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 > 0.0348721 > Distortion0= 0.0721205 > LM0= 0.0908638 > WordPenalty0= -0.153776 > PhrasePenalty0= 0.0874059 > TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 > UnknownWordPenalty0= 1 > > On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <[email protected]> wrote: > >> look inside the file >> mert-work/moses.ini >> is there anything strange going on? >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 16 September 2016 at 09:02, Steve Braich <[email protected]> wrote: >> >>> Hi Hieu, >>> I changed from relative path to absolute. Still same error. >>> >>> Thanks, >>> Steve >>> >>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <[email protected]> wrote: >>> >>>> don't use relative paths, ie. change >>>> filtered-newstest2011 -> /whatever/filtered-newstest2011 >>>> mert-work/moses.ini -> /whatever/mert-work/moses.ini >>>> >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 16 September 2016 at 05:42, Steve Braich <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I am going thru the steps described in the baseline system >>>>> instructions. I have no problems until the very end when I try to filter >>>>> out the test data from the binarised model. >>>>> >>>>> >>>>> >>>>> Here is the step in the instructions where the error appears: >>>>> >>>>> *The model that we've trained can then be filtered for this test set, >>>>> meaning that we only retain the entries needed translate the test set. >>>>> This >>>>> will make the translation a lot faster.* >>>>> >>>>> >>>>> >>>>> *cd ~/working* >>>>> >>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>> <http://filter-model-given-input.pl> filtered-newstest2011 >>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr >>>>> <http://newstest2011.true.fr> -Binarizer >>>>> ~/mosesdecoder/bin/processPhraseTableMin* >>>>> >>>>> >>>>> >>>>> Here is the error message: >>>>> >>>>> *Can't open '< /home/steve/working/binarised-model/reordering-table' >>>>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>> <http://filter-model-given-input.pl> line 439.* >>>>> >>>>> >>>>> >>>>> Keep in mind the following: >>>>> >>>>> - I compiled moses with the following parameters. It is the same >>>>> compilation. I did not recompile moses at any point during the >>>>> process. >>>>> ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0 >>>>> --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0 >>>>> NOTE: I did try recompiling with the absolute path for cmph. Same >>>>> error. >>>>> - I checked all paths first to make sure they exist. >>>>> - I am able to translate segments using the binarised model with >>>>> no problem >>>>> - I provided my moses.ini file below. >>>>> >>>>> Questions: >>>>> >>>>> - Can I recompile moses after training and/or tuning, and expect >>>>> that it should work for testing? I ask because I want to know if the >>>>> solution is to modify the compile options, do I have to start from >>>>> scratch >>>>> again? (Can you filter a binarised model using moses with different >>>>> compile options that were used to train the model?) >>>>> >>>>> Thanks, >>>>> Steve >>>>> >>>>> >>>>> Moses.ini file >>>>> >>>>> # MERT optimized configuration >>>>> >>>>> # decoder /home/steve/mosesdecoder/bin/moses >>>>> >>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>>>> >>>>> # We were before running iteration 5 >>>>> >>>>> # finished Thu Sep 15 01:53:22 PDT 2016 >>>>> >>>>> ### MOSES CONFIG FILE ### >>>>> >>>>> ######################### >>>>> >>>>> >>>>> # input factors >>>>> >>>>> [input-factors] >>>>> >>>>> 0 >>>>> >>>>> >>>>> # mapping steps >>>>> >>>>> [mapping] >>>>> >>>>> 0 T 0 >>>>> >>>>> >>>>> [distortion-limit] >>>>> >>>>> 6 >>>>> >>>>> >>>>> # feature functions >>>>> >>>>> [feature] >>>>> >>>>> UnknownWordPenalty >>>>> >>>>> WordPenalty >>>>> >>>>> PhrasePenalty >>>>> >>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>>>> path=/home/steve/working/binarised-model/phrase-table.minphr >>>>> input-factor=0 output-factor=0 >>>>> >>>>> LexicalReordering name=LexicalReordering0 num-features=6 >>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>>>> path=/home/steve/working/binarised-model/reordering-table >>>>> >>>>> Distortion >>>>> >>>>> KENLM name=LM0 factor=0 >>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>>>> order=3 >>>>> >>>>> >>>>> # dense weights for feature functions >>>>> >>>>> >>>>> [threads] >>>>> >>>>> 4 >>>>> >>>>> [weight] >>>>> >>>>> >>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 >>>>> 0.0348721 >>>>> >>>>> Distortion0= 0.0721205 >>>>> >>>>> LM0= 0.0908638 >>>>> >>>>> WordPenalty0= -0.153776 >>>>> >>>>> PhrasePenalty0= 0.0874059 >>>>> >>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>>>> >>>>> UnknownWordPenalty0= 1 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>> >>>>> >>>> >>> >> >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
