Hi Hieu,

OK, but then why does the filtering script have an option for using the
binary "-Binarizer" Phrase Table?

Thanks,
Steve

*~/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl/> filtered-newstest2011
mert-work/moses.ini ~/corpus/newstest2011.true.fr
<http://newstest2011.true.fr/> -Binarizer
~/mosesdecoder/bin/processPhraseTableMin*

On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <hieuho...@gmail.com> wrote:

> ah. from the moses.ini file, it looks like you've binarized the
> phrase-table (and the re-ordering model?) already.
>
> The filtering script only works with moses.ini which has the original text
> pt and reordering files
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 16 September 2016 at 09:13, Steve Braich <steveb...@gmail.com> wrote:
>
>> I included the moses.ini file in the original post.
>>
>> I checked in VIM using :set list.  Everything looks good.  I repost it
>> here:
>>
>> # MERT optimized configuration
>> # decoder /home/steve/mosesdecoder/bin/moses
>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>> # We were before running iteration 5
>> # finished Thu Sep 15 01:53:22 PDT 2016
>> ### MOSES CONFIG FILE ###
>> #########################
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>>
>> [distortion-limit]
>> 6
>>
>> # feature functions
>> [feature]
>> UnknownWordPenalty
>> WordPenalty
>> PhrasePenalty
>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>> path=/home/steve/working/binarised-model/phrase-table.minphr
>> input-factor=0 output-factor=0
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/steve/working/binarised-model/reordering-table
>> Distortion
>> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>> order=3
>>
>> # dense weights for feature functions
>>
>> [threads]
>> 4
>> [weight]
>>
>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
>> 0.0348721
>> Distortion0= 0.0721205
>> LM0= 0.0908638
>> WordPenalty0= -0.153776
>> PhrasePenalty0= 0.0874059
>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>> UnknownWordPenalty0= 1
>>
>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <hieuho...@gmail.com> wrote:
>>
>>> look inside the file
>>>    mert-work/moses.ini
>>> is there anything strange going on?
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 16 September 2016 at 09:02, Steve Braich <steveb...@gmail.com> wrote:
>>>
>>>> Hi Hieu,
>>>> I changed from relative path to absolute.  Still same error.
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <hieuho...@gmail.com>
>>>> wrote:
>>>>
>>>>> don't use relative paths, ie. change
>>>>>    filtered-newstest2011 -> /whatever/filtered-newstest2011
>>>>>   mert-work/moses.ini -> /whatever/mert-work/moses.ini
>>>>>
>>>>>
>>>>> Hieu Hoang
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>> On 16 September 2016 at 05:42, Steve Braich <steveb...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am going thru the steps described in the baseline system
>>>>>> instructions.  I have no problems until the very end when I try to filter
>>>>>> out the test data from the binarised model.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is the step in the instructions where the error appears:
>>>>>>
>>>>>> *The model that we've trained can then be filtered for this test set,
>>>>>> meaning that we only retain the entries needed translate the test set. 
>>>>>> This
>>>>>> will make the translation a lot faster.*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *cd ~/working*
>>>>>>
>>>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>> <http://filter-model-given-input.pl> filtered-newstest2011
>>>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>>>>> <http://newstest2011.true.fr> -Binarizer
>>>>>> ~/mosesdecoder/bin/processPhraseTableMin*
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is the error message:
>>>>>>
>>>>>> *Can't open '< /home/steve/working/binarised-model/reordering-table'
>>>>>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>> <http://filter-model-given-input.pl> line 439.*
>>>>>>
>>>>>>
>>>>>>
>>>>>> Keep in mind the following:
>>>>>>
>>>>>>    - I compiled moses with the following parameters.  It is the same
>>>>>>    compilation.  I did not recompile moses at any point during the 
>>>>>> process.
>>>>>>    ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0
>>>>>>    --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
>>>>>>    NOTE: I did try recompiling with the absolute path for cmph.
>>>>>>    Same error.
>>>>>>    - I checked all paths first to make sure they exist.
>>>>>>    - I am able to translate segments using the binarised model with
>>>>>>    no problem
>>>>>>    - I provided my moses.ini file below.
>>>>>>
>>>>>> Questions:
>>>>>>
>>>>>>    - Can I recompile moses after training and/or tuning, and expect
>>>>>>    that it should work for testing?  I ask because I want to know if the
>>>>>>    solution is to modify the compile options, do I have to start from 
>>>>>> scratch
>>>>>>    again?  (Can you filter a binarised model using moses with different
>>>>>>    compile options that were used to train the model?)
>>>>>>
>>>>>> Thanks,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> Moses.ini file
>>>>>>
>>>>>> # MERT optimized configuration
>>>>>>
>>>>>> # decoder /home/steve/mosesdecoder/bin/moses
>>>>>>
>>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>>>>>>
>>>>>> # We were before running iteration 5
>>>>>>
>>>>>> # finished Thu Sep 15 01:53:22 PDT 2016
>>>>>>
>>>>>> ### MOSES CONFIG FILE ###
>>>>>>
>>>>>> #########################
>>>>>>
>>>>>>
>>>>>> # input factors
>>>>>>
>>>>>> [input-factors]
>>>>>>
>>>>>> 0
>>>>>>
>>>>>>
>>>>>> # mapping steps
>>>>>>
>>>>>> [mapping]
>>>>>>
>>>>>> 0 T 0
>>>>>>
>>>>>>
>>>>>> [distortion-limit]
>>>>>>
>>>>>> 6
>>>>>>
>>>>>>
>>>>>> # feature functions
>>>>>>
>>>>>> [feature]
>>>>>>
>>>>>> UnknownWordPenalty
>>>>>>
>>>>>> WordPenalty
>>>>>>
>>>>>> PhrasePenalty
>>>>>>
>>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr
>>>>>> input-factor=0 output-factor=0
>>>>>>
>>>>>> LexicalReordering name=LexicalReordering0 num-features=6
>>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>>>>> path=/home/steve/working/binarised-model/reordering-table
>>>>>>
>>>>>> Distortion
>>>>>>
>>>>>> KENLM name=LM0 factor=0 
>>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>>>>>> order=3
>>>>>>
>>>>>>
>>>>>> # dense weights for feature functions
>>>>>>
>>>>>>
>>>>>> [threads]
>>>>>>
>>>>>> 4
>>>>>>
>>>>>> [weight]
>>>>>>
>>>>>>
>>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
>>>>>> 0.0348721
>>>>>>
>>>>>> Distortion0= 0.0721205
>>>>>>
>>>>>> LM0= 0.0908638
>>>>>>
>>>>>> WordPenalty0= -0.153776
>>>>>>
>>>>>> PhrasePenalty0= 0.0874059
>>>>>>
>>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>>>>>>
>>>>>> UnknownWordPenalty0= 1
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Reply via email to