yes mert must be dying. Can you post the contents of the tune/ directory? and tail mert.log?
matt (from my phone) > Le 27 oct. 2016 à 00:49, John Hewitt <john...@seas.upenn.edu> a écrit : > > It seems like MERT isn't writing it's final config file (which is typical > of MERT, in my experience). I recall giving up and using kbmira. This final > config file is the one used in test, so I can see why skipping to test ends > up failing pretty quick. > > To answer your question, though, I haven't tried. Not in my bandwidth right > now. > > -John > > On Thu, Oct 27, 2016 at 12:44 AM, lewis john mcgibbney <lewi...@apache.org> > wrote: > >> Hi Folks, >> So I've been plodding away again and feel i am very close to generating my >> first language pack, however I've arrived at the following fankle!!! >> If I run a pipeline from start to finish it fails at the 'test-bundle-1' >> phase as below stating " [Errno 2] No such file or directory: >> '/usr/local/joshua_resources/russian_experiments/exp3/tune/ >> joshua.config.final'" >> >> lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ >> /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero >> --corpus >> /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en >> --tune >> /usr/local/joshua_resources/russian_experiments/data/ >> commoncrawl.ru-en.tune >> --test >> /usr/local/joshua_resources/russian_experiments/data/ >> commoncrawl.ru-en.test >> --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model >> training" --aligner berkeley --hadoop-mem 10g --tmp >> /usr/local/hadoop-2.5.2/hadoop_tmp_dir >> [train-copy-and-filter] cached, skipping... >> [train-tokenize-en] cached, skipping... >> [train-tokenize-ru] cached, skipping... >> [train-trim] cached, skipping... >> [train-lowercase-en] cached, skipping... >> [train-lowercase-ru] cached, skipping... >> [train-vocab-en] cached, skipping... >> [train-vocab-ru] cached, skipping... >> [tune-copy-and-filter] cached, skipping... >> [tune-tokenize-en] cached, skipping... >> [tune-tokenize-ru] cached, skipping... >> [tune-lowercase-en] cached, skipping... >> [tune-lowercase-ru] cached, skipping... >> [tune-vocab-en] cached, skipping... >> [tune-vocab-ru] cached, skipping... >> [test-copy-and-filter] cached, skipping... >> [test-tokenize-en] cached, skipping... >> [test-tokenize-ru] cached, skipping... >> [test-lowercase-en] cached, skipping... >> [test-lowercase-ru] cached, skipping... >> [test-vocab-en] cached, skipping... >> [test-vocab-ru] cached, skipping... >> [lm-sort-uniq] cached, skipping... >> [kenlm] cached, skipping... >> [compile-kenlm] cached, skipping... >> [glue-tune] cached, skipping... >> [tune-bundle] cached, skipping... >> [mert-1] rebuilding... >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/data/tune/corpus.en >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/tune/joshua.config >> [CHANGED] >> dep=tune/model/grammar.gz.packed/slice_00000.source >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/tune/joshua.config.final >> [NOT FOUND] >> cmd=/usr/local/incubator-joshua/scripts/training/run_tuner.py >> /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en >> /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.ru >> --tunedir /usr/local/joshua_resources/russian_experiments/exp3/tune >> --tuner >> mert --decoder >> /usr/local/joshua_resources/russian_experiments/exp3/tune/decoder_command >> --decoder-config >> /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config >> --decoder-output-file >> /usr/local/joshua_resources/russian_experiments/exp3/tune/output.nbest >> --decoder-log-file >> /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.log >> --iterations 10 --metric 'BLEU 4 closest' >> took 27 seconds (27s) >> [test-bundle-1] rebuilding... >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/tune/joshua.config.final >> [NOT FOUND] >> dep=grammar.gz >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/1/model/joshua.config >> cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force >> --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir >> /usr/local/joshua_resources/russian_experiments/exp3/tune/ >> joshua.config.final >> /usr/local/joshua_resources/russian_experiments/exp3/test/1/model >> --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s >> ||| %f ||| %c" -mark-oovs false' --pack-tm grammar.gz --tm >> /usr/local/joshua_resources/russian_experiments/exp3/data/ >> tune/grammar.glue >> JOB FAILED (return code 2) >> ERROR:root:ERROR: argument config: can't open >> '/usr/local/joshua_resources/russian_experiments/exp3/tune/ >> joshua.config.final': >> [Errno 2] No such file or directory: >> '/usr/local/joshua_resources/russian_experiments/exp3/tune/ >> joshua.config.final' >> >> However, if I run the pipeline with the --first-step test flag, then I get >> the following where the 'test-bundle-1' phase executes and completes >> flawlessly however the pipeline then goes on to die at the 'test-decode-1' >> phase!!! >> >> lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ >> /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero >> --corpus >> /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en >> --tune >> /usr/local/joshua_resources/russian_experiments/data/ >> commoncrawl.ru-en.tune >> --test >> /usr/local/joshua_resources/russian_experiments/data/ >> commoncrawl.ru-en.test >> --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model >> training" --aligner berkeley --hadoop-mem 10g --tmp >> /usr/local/hadoop-2.5.2/hadoop_tmp_dir --first-step test --grammar >> /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz >> --joshua-mem 10g >> [train-copy-and-filter] cached, skipping... >> [train-tokenize-en] cached, skipping... >> [train-tokenize-ru] cached, skipping... >> [train-trim] cached, skipping... >> [train-lowercase-en] cached, skipping... >> [train-lowercase-ru] cached, skipping... >> [train-vocab-en] cached, skipping... >> [train-vocab-ru] cached, skipping... >> [tune-copy-and-filter] cached, skipping... >> [tune-tokenize-en] cached, skipping... >> [tune-tokenize-ru] cached, skipping... >> [tune-lowercase-en] cached, skipping... >> [tune-lowercase-ru] cached, skipping... >> [tune-vocab-en] cached, skipping... >> [tune-vocab-ru] cached, skipping... >> [test-copy-and-filter] cached, skipping... >> [test-tokenize-en] cached, skipping... >> [test-tokenize-ru] cached, skipping... >> [test-lowercase-en] cached, skipping... >> [test-lowercase-ru] cached, skipping... >> [test-vocab-en] cached, skipping... >> [test-vocab-ru] cached, skipping... >> [glue-test] cached, skipping... >> [test-bundle-1] rebuilding... >> >> dep=/usr/local/incubator-joshua/scripts/training/ >> templates/tune/joshua.config >> dep=/usr/local/joshua_resources/russian_experiments/exp3/grammar.gz >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/1/model/joshua.config >> [CHANGED] >> cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force >> --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir >> /usr/local/incubator-joshua/scripts/training/templates/tune/joshua.config >> /usr/local/joshua_resources/russian_experiments/exp3/test/1/model >> --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s >> ||| %f ||| %c" -mark-oovs false' --pack-tm >> /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz --tm >> /usr/local/joshua_resources/russian_experiments/exp3/data/ >> test/grammar.glue >> took 5372 seconds (1h29m32s) >> [test-decode-1] rebuilding... >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/data/test/corpus.en >> [CHANGED] >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/decoder_command >> [CHANGED] >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/1/model/joshua.config >> [CHANGED] >> >> dep=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/1/model/grammar.gz.packed/slice_00000.source >> [CHANGED] >> dep=/usr/local/joshua_resources/russian_experiments/exp3/test/output >> [CHANGED] >> >> cmd=/usr/local/joshua_resources/russian_experiments/ >> exp3/test/decoder_command >> JOB FAILED (return code 1) >> >> I need to ask the question, using current (todays) master branch with >> Python 3, has anyone managed to build a language pack? I seem to have >> encountered several small niggling issues and this is the most recent of >> them. >> Thanks in advance for any guidance here. >> Lewis >> >> -- >> http://home.apache.org/~lewismc/ >> @hectorMcSpector >> http://www.linkedin.com/in/lmcgibbney >>
