Everything checked out.  BLEU is 23.33, which is close enough to what the
instructions say (23.5).

Thank you!
Steve

On Fri, Sep 16, 2016 at 2:20 AM, Steve Braich <steveb...@gmail.com> wrote:

> Ah, that makes sense.  However, the instructions seem to lead you down the
> path I took of changing the moses.ini file to use the binary phrase table,
> to test how fast it is, THEN it does the filtering step.
>
> Anyway, that resolved it.  I am running the BLEU on it now.
>
> Thanks Hieu!
> Steve
>
> On Fri, Sep 16, 2016 at 2:17 AM, Hieu Hoang <hieuho...@gmail.com> wrote:
>
>> it will binarize AFTER it filters. It can't filter already binarized
>> tables
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 11:10, Steve Braich <steveb...@gmail.com> wrote:
>>
>>> Hi Hieu,
>>>
>>> OK, but then why does the filtering script have an option for using the
>>> binary "-Binarizer" Phrase Table?
>>>
>>> Thanks,
>>> Steve
>>>
>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>> <http://filter-model-given-input.pl/> filtered-newstest2011
>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>> <http://newstest2011.true.fr/> -Binarizer
>>> ~/mosesdecoder/bin/processPhraseTableMin*
>>>
>>> On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <hieuho...@gmail.com> wrote:
>>>
>>>> ah. from the moses.ini file, it looks like you've binarized the
>>>> phrase-table (and the re-ordering model?) already.
>>>>
>>>> The filtering script only works with moses.ini which has the original
>>>> text pt and reordering files
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 16 September 2016 at 09:13, Steve Braich <steveb...@gmail.com>
>>>> wrote:
>>>>
>>>>> I included the moses.ini file in the original post.
>>>>>
>>>>> I checked in VIM using :set list.  Everything looks good.  I repost it
>>>>> here:
>>>>>
>>>>> # MERT optimized configuration
>>>>> # decoder /home/steve/mosesdecoder/bin/moses
>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>>>>> # We were before running iteration 5
>>>>> # finished Thu Sep 15 01:53:22 PDT 2016
>>>>> ### MOSES CONFIG FILE ###
>>>>> #########################
>>>>>
>>>>> # input factors
>>>>> [input-factors]
>>>>> 0
>>>>>
>>>>> # mapping steps
>>>>> [mapping]
>>>>> 0 T 0
>>>>>
>>>>> [distortion-limit]
>>>>> 6
>>>>>
>>>>> # feature functions
>>>>> [feature]
>>>>> UnknownWordPenalty
>>>>> WordPenalty
>>>>> PhrasePenalty
>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr
>>>>> input-factor=0 output-factor=0
>>>>> LexicalReordering name=LexicalReordering0 num-features=6
>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>>>> path=/home/steve/working/binarised-model/reordering-table
>>>>> Distortion
>>>>> KENLM name=LM0 factor=0 
>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>>>>> order=3
>>>>>
>>>>> # dense weights for feature functions
>>>>>
>>>>> [threads]
>>>>> 4
>>>>> [weight]
>>>>>
>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
>>>>> 0.0348721
>>>>> Distortion0= 0.0721205
>>>>> LM0= 0.0908638
>>>>> WordPenalty0= -0.153776
>>>>> PhrasePenalty0= 0.0874059
>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>>>>> UnknownWordPenalty0= 1
>>>>>
>>>>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <hieuho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> look inside the file
>>>>>>    mert-work/moses.ini
>>>>>> is there anything strange going on?
>>>>>>
>>>>>> Hieu Hoang
>>>>>> http://www.hoang.co.uk/hieu
>>>>>>
>>>>>> On 16 September 2016 at 09:02, Steve Braich <steveb...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Hieu,
>>>>>>> I changed from relative path to absolute.  Still same error.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Steve
>>>>>>>
>>>>>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <hieuho...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> don't use relative paths, ie. change
>>>>>>>>    filtered-newstest2011 -> /whatever/filtered-newstest2011
>>>>>>>>   mert-work/moses.ini -> /whatever/mert-work/moses.ini
>>>>>>>>
>>>>>>>>
>>>>>>>> Hieu Hoang
>>>>>>>> http://www.hoang.co.uk/hieu
>>>>>>>>
>>>>>>>> On 16 September 2016 at 05:42, Steve Braich <steveb...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am going thru the steps described in the baseline system
>>>>>>>>> instructions.  I have no problems until the very end when I try to 
>>>>>>>>> filter
>>>>>>>>> out the test data from the binarised model.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is the step in the instructions where the error appears:
>>>>>>>>>
>>>>>>>>> *The model that we've trained can then be filtered for this test
>>>>>>>>> set, meaning that we only retain the entries needed translate the 
>>>>>>>>> test set.
>>>>>>>>> This will make the translation a lot faster.*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *cd ~/working*
>>>>>>>>>
>>>>>>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>>> <http://filter-model-given-input.pl> filtered-newstest2011
>>>>>>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>>>>>>>> <http://newstest2011.true.fr> -Binarizer
>>>>>>>>> ~/mosesdecoder/bin/processPhraseTableMin*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is the error message:
>>>>>>>>>
>>>>>>>>> *Can't open '<
>>>>>>>>> /home/steve/working/binarised-model/reordering-table' at
>>>>>>>>> /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>>> <http://filter-model-given-input.pl> line 439.*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Keep in mind the following:
>>>>>>>>>
>>>>>>>>>    - I compiled moses with the following parameters.  It is the
>>>>>>>>>    same compilation.  I did not recompile moses at any point during 
>>>>>>>>> the
>>>>>>>>>    process.
>>>>>>>>>    ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0
>>>>>>>>>    --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
>>>>>>>>>    NOTE: I did try recompiling with the absolute path for cmph.
>>>>>>>>>    Same error.
>>>>>>>>>    - I checked all paths first to make sure they exist.
>>>>>>>>>    - I am able to translate segments using the binarised model
>>>>>>>>>    with no problem
>>>>>>>>>    - I provided my moses.ini file below.
>>>>>>>>>
>>>>>>>>> Questions:
>>>>>>>>>
>>>>>>>>>    - Can I recompile moses after training and/or tuning, and
>>>>>>>>>    expect that it should work for testing?  I ask because I want to 
>>>>>>>>> know if
>>>>>>>>>    the solution is to modify the compile options, do I have to start 
>>>>>>>>> from
>>>>>>>>>    scratch again?  (Can you filter a binarised model using moses with
>>>>>>>>>    different compile options that were used to train the model?)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Moses.ini file
>>>>>>>>>
>>>>>>>>> # MERT optimized configuration
>>>>>>>>>
>>>>>>>>> # decoder /home/steve/mosesdecoder/bin/moses
>>>>>>>>>
>>>>>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>>>>>>>>>
>>>>>>>>> # We were before running iteration 5
>>>>>>>>>
>>>>>>>>> # finished Thu Sep 15 01:53:22 PDT 2016
>>>>>>>>>
>>>>>>>>> ### MOSES CONFIG FILE ###
>>>>>>>>>
>>>>>>>>> #########################
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # input factors
>>>>>>>>>
>>>>>>>>> [input-factors]
>>>>>>>>>
>>>>>>>>> 0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # mapping steps
>>>>>>>>>
>>>>>>>>> [mapping]
>>>>>>>>>
>>>>>>>>> 0 T 0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [distortion-limit]
>>>>>>>>>
>>>>>>>>> 6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # feature functions
>>>>>>>>>
>>>>>>>>> [feature]
>>>>>>>>>
>>>>>>>>> UnknownWordPenalty
>>>>>>>>>
>>>>>>>>> WordPenalty
>>>>>>>>>
>>>>>>>>> PhrasePenalty
>>>>>>>>>
>>>>>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>>>>>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr
>>>>>>>>> input-factor=0 output-factor=0
>>>>>>>>>
>>>>>>>>> LexicalReordering name=LexicalReordering0 num-features=6
>>>>>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0
>>>>>>>>> output-factor=0 path=/home/steve/working/binar
>>>>>>>>> ised-model/reordering-table
>>>>>>>>>
>>>>>>>>> Distortion
>>>>>>>>>
>>>>>>>>> KENLM name=LM0 factor=0 
>>>>>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>>>>>>>>> order=3
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # dense weights for feature functions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [threads]
>>>>>>>>>
>>>>>>>>> 4
>>>>>>>>>
>>>>>>>>> [weight]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621
>>>>>>>>> 0.0141898 0.0348721
>>>>>>>>>
>>>>>>>>> Distortion0= 0.0721205
>>>>>>>>>
>>>>>>>>> LM0= 0.0908638
>>>>>>>>>
>>>>>>>>> WordPenalty0= -0.153776
>>>>>>>>>
>>>>>>>>> PhrasePenalty0= 0.0874059
>>>>>>>>>
>>>>>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>>>>>>>>>
>>>>>>>>> UnknownWordPenalty0= 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Moses-support mailing list
>>>>>>>>> Moses-support@mit.edu
>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Reply via email to