Hi Allan, Moses decoder get killed while loading phrase table. Probably you do not have enough RAM to load it. If that's the case then consider binarizing language model and phrase table, see advanced feature in moses page.
Best, Valters From: [email protected] [mailto:[email protected]] On Behalf Of Allan Jie Sent: Wednesday, March 12, 2014 7:34 AM To: [email protected] Subject: [Moses-support] Problem while running mert-moses.pl Dear all, I try to run tuning using this command. nohup nice /home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> \ /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh /home/wei_lu/experiment/data/dev/dev2010.lowercased.en \ /home/wei_lu/tools/mosesdecoder/bin/moses /home/wei_lu/pku_seg/model/moses.ini --working-dir /home/wei_lu/pku_seg/mert-work-index --mertdir /home/wei_lu/tools/mosesdecoder/bin/ -decoder-flags="-threads 4" But the result I got fail, here is the log. Using SCRIPTS_ROOTDIR: /home/wei_lu/tools/mosesdecoder/scripts filtering the phrase tables... Wed Mar 12 12:18:43 SGT 2014 exec: /home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl<http://filter-model-given-input.pl> ./filtered /home/wei_lu/pku_seg/model/moses.ini /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh Executing: /home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl<http://filter-model-given-input.pl> ./filtered /home/wei_lu/pku_seg/model/moses.ini /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh > filterphrases.out 2> filterphrases.err Asking moses for feature names and values from filtered/moses.ini Executing: /home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config filtered/moses.ini -inputtype 0 -show-weights > ./features.list Defined parameters (per moses.ini or switch): config: filtered/moses.ini distortion-limit: 6 feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=6 path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/wei_lu/experiment/lm/allan_lm.lm order=3 input-factors: 0 inputtype: 0 mapping: 0 T 0 show-weights: threads: 4 weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 /home/wei_lu/tools/mosesdecoder/bin line=UnknownWordPenalty FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 line=WordPenalty FeatureFunction: WordPenalty0 start: 1 end: 1 line=PhrasePenalty FeatureFunction: PhrasePenalty0 start: 2 end: 2 line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=6 path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 FeatureFunction: TranslationModel0 start: 3 end: 8 line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe FeatureFunction: LexicalReordering0 start: 9 end: 14 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 15 end: 15 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/wei_lu/experiment/lm/allan_lm.lm order=3 FeatureFunction: LM0 start: 16 end: 16 Loading the LM will be faster if you build a binary file. Reading /home/wei_lu/experiment/lm/allan_lm.lm ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 **************************************************************************************************** Loading table into memory...done. Start loading text SCFG phrase table. Moses format : [1507.000] seconds Reading /home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 ****sh: line 1: 49997 Killed /home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config filtered/moses.ini -inputtype 0 -show-weights > ./features.list Exit code: 137 Failed to run moses with the config filtered/moses.ini at /home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl<http://mert-moses.pl> line 1271. I have no idea why this happened. And then I try to run a small dataset(only one sentence in the tuning instance and relevant phrase-table), that works fine. I don't know what's this problem. Best Regards, Allan -- Research student, final-year undergraduate Extreme Scale Network Computing and service laboratory(http://www.cloud-uestc.cn/cloud-uestc-EN/index.html), School of Computer Science & Engineering, University of Electronic Science and Technology of China(http://www.oice.uestc.edu.cn/en/). Website: http://www.allanjie.ml/
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
