The lexicalised distortion model should be turned off, it's only pertinent to phrase-based
Hieu Sent while bumping into things On 30 Dec 2012, at 01:19 AM, Ben Humphreys <[email protected]> wrote: > Hello all, > > I'm new to Moses and I've been trying to set up a baseline > hierarchical/syntax Moses system. > I got a regular system running using the "Baseline System" tutorial and now > I'm following the Syntax Tutorial > http://www.statmt.org/moses/?n=Moses.SyntaxTutorial > > > The error I get is when I try to tune my system using MERT: > Here's my tuning command: >> $ /home/ben/moses/mosesdecoder/scripts/training/mert-moses.pl \ >> /data/ben/hiertest/yomiuri_tuning.ja /data/ben/hiertest/yomiuri_tuning.en \ >> /home/ben/moses/mosesdecoder/bin/moses /data/ben/hiertest/model/moses.ini \ >> --mertdir /home/ben/moses/mosesdecoder/bin \ >> --maximum-iterations=10 > > The tail of the error log is: > >> exec: >> /mnt/home/ben/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> ./filtered /data/ben/hiertest/model/moses.ini >> /data/ben/hiertest/yomiuri_tuning.ja >> Executing: >> /mnt/home/ben/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> ./filtered /data/ben/hiertest/model/moses.ini >> /data/ben/hiertest/yomiuri_tuning.ja > filterphrases.out 2> filterphrases.err >> Asking moses for feature names and values from filtered/moses.ini >> Executing: /home/ben/moses/mosesdecoder/bin/moses -config >> filtered/moses.ini -inputtype 0 -show-weights > ./features.list >> Defined parameters (per moses.ini or switch): >> config: filtered/moses.ini >> cube-pruning-pop-limit: 1000 >> distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6 >> /mnt/home/ben/moses/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe >> input-factors: 0 >> inputtype: 0 >> lmodel-file: 8 0 5 /yew/ben/trans/nict.all.en.5.blm >> mapping: 0 T 0 1 T 1 >> max-chart-span: 20 1000 >> non-terminals: X >> search-algorithm: 3 >> show-weights: >> ttable-file: 6 0 0 5 >> /mnt/home/ben/moses/mert-work/filtered/phrase-table.0-0.1.1.gz 6 0 0 1 >> /data/ben/hiertest/model/glue-grammar >> ttable-limit: 20 >> weight-l: 0.5000 >> weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 >> weight-w: -1 >> /home/ben/moses/mosesdecoder/bin >> ScoreProducer: WordPenalty start: 0 end: 1 >> ScoreProducer: !UnknownWordPenalty start: 1 end: 2 >> Loading lexical distortion models...have 1 models >> ERROR:Lexicalized distortion model: Not enough weights, add to [weight-d] >> Exit code: 1 >> Failed to run moses with the config filtered/moses.ini at >> /home/ben/moses/mosesdecoder/scripts/training/mert-moses.pl line 1169. > > > > It says in the Syntax Tutorial that training a hierarchical model is just a > case of adding -hierarchical and a few other options to a normal Moses > training command. > > Here is my training command. Modified from my working non-hierarchical system > to add the one -hierarchical line: >> $ /home/ben/moses/mosesdecoder/scripts/training/train-model.perl \ >> -root-dir /data/ben/hiertest \ >> -corpus /data/ben/hiertest/hiertest.corpus -f ja -e en \ >> -hierarchical -glue-grammar -max-phrase-length 5 \ >> -alignment grow-diag-final-and -reordering msd-bidirectional-fe \ >> -lm 0:5:/yew/ben/trans/nict.all.en.5.blm:8 \ >> -external-bin-dir=/home/ben/moses/mosesdecoder/bin/ > > > I can add the -d flag with a bunch of weights to make the error go away, but > that seems like a hack, and I don't know what weights to use anyway. > > > Am I missing a step that sets up these weights in my moses.ini or something? > I didn't notice or get any issues like this with the non-hierarchical system. > > > Ben Humphreys > > PhD student > Kurohashi-Kawahara Lab > Kyoto University > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
