Hi, The phrase tables have data and also the paths to the tuning sets are correct. However, filtering process seems to produce errors:
The file mert-work/filterphrases.err has the content: Executing: mkdir -p /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered Considering factor 0 Done. filtering /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/rule-table.gz -> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1... binarizing...CreateOnDiskPt 1 1 5 100 2 /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1 /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.bin Can't exec "CreateOnDiskPt": No such file or directory at /usr/local/share/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 278, <PIPE> line 3721058. Kairit Hieu Hoang <[email protected]> kirjutas: > hi Kairit > > The phrase table > ---/filtered/phrase-table.0-0.1.1.gz > seems to have loaded in 0 seconds. > > This is the filtered phrase table. Check that the original phrase > table has data, and that the tuning set has data. > > hieu > > (ps. please subscribe to the mailing list to post emails) > > On 09/05/2012 09:23, [email protected] wrote: >> Hi, >> >> I'm trying to tune a hierarchical model and the process is aborted >> with Exit code 134 and the message that the decoder died. >> >> The command I'm running is: >> >> nohup nice >> /usr/local/share/moses/mosesdecoder/scripts/training/mert-moses.pl >> ../../parallel/europarl-v6.et-en.dev.et >> ../../parallel/europarl-v6.et-en.dev.en >> /usr/local/share/moses/mosesdecoder/dist/bin/moses >> train/model/moses.ini --mertdir >> /usr/local/share/moses/mosesdecoder/dist/bin/ --filtercmd >> '/usr/local/share/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> >> --Hierarchical'&> >> mert.out >> >> I have also tried this one and it doesn't work either: >> >> nohup nice >> /usr/local/share/moses/mosesdecoder/scripts/training/mert-moses.pl >> ../../parallel/europarl-v6.et-en.dev.et >> ../../parallel/europarl-v6.et-en.dev.en >> /usr/local/share/moses/mosesdecoder/dist/bin/moses >> train/model/moses.ini --mertdir >> /usr/local/share/moses/mosesdecoder/dist/bin/ --filtercmd >> '/usr/local/share/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> -Binarizer "CreateOnDiskPt 1 1 5 100 2" >> --Hierarchical'&> >> mert.out >> >> The content of moses.ini file is: >> >> ######################### >> ### MOSES CONFIG FILE ### >> ######################### >> >> # input factors >> [input-factors] >> >> >> # mapping steps >> [mapping] >> 0 T 0 >> 1 T 1 >> >> # translation tables: table type (hierarchical(0), textual (0), binary >> (1)), source-factors, target-factors, number of scores, file >> # OLD FORMAT is still handled for back-compatibility >> # OLD FORMAT translation tables: source-factors, target-factors, >> number of scores, file >> # OLD FORMAT a binary table type (1) is assumed >> [ttable-file] >> 6 0 0 5 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/rule-table.gz >> 6 0 0 1 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> >> # no generation models, no generation-file section >> >> # language models: type(srilm/irstlm), factors, order, file >> [lmodel-file] >> 8 0 3 /home/kairit/Documents/korpused/lm/europarl-v6.et-en.blm.en >> >> >> # limit on how many phrase translations e for each phrase f are loaded >> # 0 = all elements loaded >> [ttable-limit] >> 20 >> >> >> # language model weights >> [weight-l] >> 0.5000 >> >> >> # translation model weights >> [weight-t] >> 0.20 >> 0.20 >> 0.20 >> 0.20 >> 0.20 >> 1.0 >> >> # no generation models, no weight-generation section >> >> # word penalty >> [weight-w] >> -1 >> >> [cube-pruning-pop-limit] >> 1000 >> >> [non-terminals] >> X >> >> [search-algorithm] >> 3 >> >> [inputtype] >> 3 >> >> [max-chart-span] >> 20 >> 1000 >> >> The content of mert.out file is: >> >> nohup: ignoring input >> main::create_extractor_script() called too early to check prototype at >> /usr/local/share/moses/mosesdecoder/scripts/training/mert-moses.pl >> line 681. >> Using SCRIPTS_ROOTDIR: /usr/local/share/moses/mosesdecoder/scripts >> Use of uninitialized value $extractorargs in concatenation (.) or >> string at >> /usr/local/share/moses/mosesdecoder/scripts/training/mert-moses.pl >> line 370. >> filtering the phrase tables... Tue May 8 18:02:26 EEST 2012 >> exec: >> /usr/local/share/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> --Hierarchical ./filtered >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/moses.ini >> /home/kairit/Documents/korpused/parallel/europarl-v6.et-en.dev.et >> Executing: >> /usr/local/share/moses/mosesdecoder/scripts/training/filter-model-given-input.pl >> --Hierarchical ./filtered >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/moses.ini >> /home/kairit/Documents/korpused/parallel/europarl-v6.et-en.dev.et> >> filterphrases.out >> 2> >> filterphrases.err >> Asking moses for feature names and values from filtered/moses.ini >> Executing: /usr/local/share/moses/mosesdecoder/dist/bin/moses -config >> filtered/moses.ini -inputtype 0 -show-weights> ./features.list >> Defined parameters (per moses.ini or switch): >> config: filtered/moses.ini >> cube-pruning-pop-limit: 1000 >> input-factors: 0 >> inputtype: 0 >> lmodel-file: 8 0 3 >> /home/kairit/Documents/korpused/lm/europarl-v6.et-en.blm.en >> mapping: 0 T 0 1 T 1 >> max-chart-span: 20 1000 >> non-terminals: X >> search-algorithm: 3 >> show-weights: >> ttable-file: 6 0 0 5 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> 6 0 0 >> 1 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> ttable-limit: 20 >> weight-l: 0.5000 >> weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 >> weight-w: -1 >> Loading lexical distortion models...have 0 models >> Start loading LanguageModel >> /home/kairit/Documents/korpused/lm/europarl-v6.et-en.blm.en : [0.000] >> seconds >> Finished loading LanguageModels : [0.000] seconds >> Using uniform ttable-limit of 20 for all translation tables. >> Start loading PhraseTable >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> : >> [0.000] >> seconds >> filePath: >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> Start loading PhraseTable >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> : >> [0.000] >> seconds >> filePath: >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> Finished loading phrase tables : [0.000] seconds >> Start loading phrase table from >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> : >> [0.000] >> seconds >> Start loading new format pt model : [0.000] seconds >> Finished loading phrase tables : [72.000] seconds >> Start loading phrase table from >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> : >> [72.000] >> seconds >> Start loading new format pt model : [72.000] seconds >> Finished loading phrase tables : [72.000] seconds >> MERT starting values and ranges for random generation: >> lm = 0.500 ( 0.00 .. 1.00) >> w = -1.000 ( 0.00 .. 1.00) >> tm = 0.200 ( 0.00 .. 1.00) >> tm = 0.200 ( 0.00 .. 1.00) >> tm = 0.200 ( 0.00 .. 1.00) >> tm = 0.200 ( 0.00 .. 1.00) >> tm = 0.200 ( 0.00 .. 1.00) >> tm = 1.000 ( 0.00 .. 1.00) >> run 1 start at Tue May 8 18:05:59 EEST 2012 >> Parsing --decoder-flags: || >> Saving new config to: ./run1.moses.ini >> Saved: ./run1.moses.ini >> Normalizing lambdas: 0.500000 -1.000000 0.200000 0.200000 0.200000 >> 0.200000 0.200000 1.000000 >> DECODER_CFG = -w -0.285714 -lm 0.142857 -tm 0.057143 0.057143 0.057143 >> 0.057143 0.057143 0.285714 >> Executing: /usr/local/share/moses/mosesdecoder/dist/bin/moses >> -config filtered/moses.ini -inputtype 0 -w -0.285714 -lm 0.142857 -tm >> 0.057143 0.057143 0.057143 0.057143 0.057143 0.285714 -n-best-list >> run1.best100.out 100 -input-file >> /home/kairit/Documents/korpused/parallel/europarl-v6.et-en.dev.et> >> run1.out >> (1) run decoder to produce n-best lists >> params = >> decoder_config = -w -0.285714 -lm 0.142857 -tm 0.057143 0.057143 >> 0.057143 0.057143 0.057143 0.285714 >> Defined parameters (per moses.ini or switch): >> config: filtered/moses.ini >> cube-pruning-pop-limit: 1000 >> input-factors: 0 >> input-file: >> /home/kairit/Documents/korpused/parallel/europarl-v6.et-en.dev.et >> inputtype: 0 >> lmodel-file: 8 0 3 >> /home/kairit/Documents/korpused/lm/europarl-v6.et-en.blm.en >> mapping: 0 T 0 1 T 1 >> max-chart-span: 20 1000 >> n-best-list: run1.best100.out 100 >> non-terminals: X >> search-algorithm: 3 >> ttable-file: 6 0 0 5 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> 6 0 0 >> 1 >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> ttable-limit: 20 >> weight-l: 0.142857 >> weight-t: 0.057143 0.057143 0.057143 0.057143 0.057143 0.285714 >> weight-w: -0.285714 >> Loading lexical distortion models...have 0 models >> Start loading LanguageModel >> /home/kairit/Documents/korpused/lm/europarl-v6.et-en.blm.en : [0.000] >> seconds >> Finished loading LanguageModels : [0.000] seconds >> Using uniform ttable-limit of 20 for all translation tables. >> Start loading PhraseTable >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> : >> [0.000] >> seconds >> filePath: >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> Start loading PhraseTable >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> : >> [0.000] >> seconds >> filePath: >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> Finished loading phrase tables : [0.000] seconds >> Start loading phrase table from >> /home/kairit/Documents/korpused/working/hierarchical_et_en/mert-work/filtered/phrase-table.0-0.1.1.gz >> : >> [0.000] >> seconds >> Start loading new format pt model : [0.000] seconds >> Finished loading phrase tables : [73.000] seconds >> Start loading phrase table from >> /home/kairit/Documents/korpused/working/hierarchical_et_en/train/model/glue-grammar >> : >> [73.000] >> seconds >> Start loading new format pt model : [73.000] seconds >> Finished loading phrase tables : [73.000] seconds >> Created input-output object : [73.000] seconds >> Translating line 0 in thread id 0 >> ERROR:ERROR: search. Aborting >> >> Aborted >> Exit code: 134 >> The decoder died. CONFIG WAS -w -0.285714 -lm 0.142857 -tm 0.057143 >> 0.057143 0.057143 0.057143 0.057143 0.285714 >> >> >> Thanks if you can help me to find the problem. >> >> Kairit >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
