It seems like MERT isn't writing it's final config file (which is typical of MERT, in my experience). I recall giving up and using kbmira. This final config file is the one used in test, so I can see why skipping to test ends up failing pretty quick.
To answer your question, though, I haven't tried. Not in my bandwidth right now. -John On Thu, Oct 27, 2016 at 12:44 AM, lewis john mcgibbney <[email protected]> wrote: > Hi Folks, > So I've been plodding away again and feel i am very close to generating my > first language pack, however I've arrived at the following fankle!!! > If I run a pipeline from start to finish it fails at the 'test-bundle-1' > phase as below stating " [Errno 2] No such file or directory: > '/usr/local/joshua_resources/russian_experiments/exp3/tune/ > joshua.config.final'" > > lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ > /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero > --corpus > /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en > --tune > /usr/local/joshua_resources/russian_experiments/data/ > commoncrawl.ru-en.tune > --test > /usr/local/joshua_resources/russian_experiments/data/ > commoncrawl.ru-en.test > --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model > training" --aligner berkeley --hadoop-mem 10g --tmp > /usr/local/hadoop-2.5.2/hadoop_tmp_dir > [train-copy-and-filter] cached, skipping... > [train-tokenize-en] cached, skipping... > [train-tokenize-ru] cached, skipping... > [train-trim] cached, skipping... > [train-lowercase-en] cached, skipping... > [train-lowercase-ru] cached, skipping... > [train-vocab-en] cached, skipping... > [train-vocab-ru] cached, skipping... > [tune-copy-and-filter] cached, skipping... > [tune-tokenize-en] cached, skipping... > [tune-tokenize-ru] cached, skipping... > [tune-lowercase-en] cached, skipping... > [tune-lowercase-ru] cached, skipping... > [tune-vocab-en] cached, skipping... > [tune-vocab-ru] cached, skipping... > [test-copy-and-filter] cached, skipping... > [test-tokenize-en] cached, skipping... > [test-tokenize-ru] cached, skipping... > [test-lowercase-en] cached, skipping... > [test-lowercase-ru] cached, skipping... > [test-vocab-en] cached, skipping... > [test-vocab-ru] cached, skipping... > [lm-sort-uniq] cached, skipping... > [kenlm] cached, skipping... > [compile-kenlm] cached, skipping... > [glue-tune] cached, skipping... > [tune-bundle] cached, skipping... > [mert-1] rebuilding... > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/data/tune/corpus.en > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/tune/joshua.config > [CHANGED] > dep=tune/model/grammar.gz.packed/slice_00000.source > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/tune/joshua.config.final > [NOT FOUND] > cmd=/usr/local/incubator-joshua/scripts/training/run_tuner.py > /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en > /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.ru > --tunedir /usr/local/joshua_resources/russian_experiments/exp3/tune > --tuner > mert --decoder > /usr/local/joshua_resources/russian_experiments/exp3/tune/decoder_command > --decoder-config > /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config > --decoder-output-file > /usr/local/joshua_resources/russian_experiments/exp3/tune/output.nbest > --decoder-log-file > /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.log > --iterations 10 --metric 'BLEU 4 closest' > took 27 seconds (27s) > [test-bundle-1] rebuilding... > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/tune/joshua.config.final > [NOT FOUND] > dep=grammar.gz > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/test/1/model/joshua.config > cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force > --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir > /usr/local/joshua_resources/russian_experiments/exp3/tune/ > joshua.config.final > /usr/local/joshua_resources/russian_experiments/exp3/test/1/model > --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s > ||| %f ||| %c" -mark-oovs false' --pack-tm grammar.gz --tm > /usr/local/joshua_resources/russian_experiments/exp3/data/ > tune/grammar.glue > JOB FAILED (return code 2) > ERROR:root:ERROR: argument config: can't open > '/usr/local/joshua_resources/russian_experiments/exp3/tune/ > joshua.config.final': > [Errno 2] No such file or directory: > '/usr/local/joshua_resources/russian_experiments/exp3/tune/ > joshua.config.final' > > However, if I run the pipeline with the --first-step test flag, then I get > the following where the 'test-bundle-1' phase executes and completes > flawlessly however the pipeline then goes on to die at the 'test-decode-1' > phase!!! > > lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ > /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero > --corpus > /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en > --tune > /usr/local/joshua_resources/russian_experiments/data/ > commoncrawl.ru-en.tune > --test > /usr/local/joshua_resources/russian_experiments/data/ > commoncrawl.ru-en.test > --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model > training" --aligner berkeley --hadoop-mem 10g --tmp > /usr/local/hadoop-2.5.2/hadoop_tmp_dir --first-step test --grammar > /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz > --joshua-mem 10g > [train-copy-and-filter] cached, skipping... > [train-tokenize-en] cached, skipping... > [train-tokenize-ru] cached, skipping... > [train-trim] cached, skipping... > [train-lowercase-en] cached, skipping... > [train-lowercase-ru] cached, skipping... > [train-vocab-en] cached, skipping... > [train-vocab-ru] cached, skipping... > [tune-copy-and-filter] cached, skipping... > [tune-tokenize-en] cached, skipping... > [tune-tokenize-ru] cached, skipping... > [tune-lowercase-en] cached, skipping... > [tune-lowercase-ru] cached, skipping... > [tune-vocab-en] cached, skipping... > [tune-vocab-ru] cached, skipping... > [test-copy-and-filter] cached, skipping... > [test-tokenize-en] cached, skipping... > [test-tokenize-ru] cached, skipping... > [test-lowercase-en] cached, skipping... > [test-lowercase-ru] cached, skipping... > [test-vocab-en] cached, skipping... > [test-vocab-ru] cached, skipping... > [glue-test] cached, skipping... > [test-bundle-1] rebuilding... > > dep=/usr/local/incubator-joshua/scripts/training/ > templates/tune/joshua.config > dep=/usr/local/joshua_resources/russian_experiments/exp3/grammar.gz > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/test/1/model/joshua.config > [CHANGED] > cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force > --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir > /usr/local/incubator-joshua/scripts/training/templates/tune/joshua.config > /usr/local/joshua_resources/russian_experiments/exp3/test/1/model > --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s > ||| %f ||| %c" -mark-oovs false' --pack-tm > /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz --tm > /usr/local/joshua_resources/russian_experiments/exp3/data/ > test/grammar.glue > took 5372 seconds (1h29m32s) > [test-decode-1] rebuilding... > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/data/test/corpus.en > [CHANGED] > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/test/decoder_command > [CHANGED] > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/test/1/model/joshua.config > [CHANGED] > > dep=/usr/local/joshua_resources/russian_experiments/ > exp3/test/1/model/grammar.gz.packed/slice_00000.source > [CHANGED] > dep=/usr/local/joshua_resources/russian_experiments/exp3/test/output > [CHANGED] > > cmd=/usr/local/joshua_resources/russian_experiments/ > exp3/test/decoder_command > JOB FAILED (return code 1) > > I need to ask the question, using current (todays) master branch with > Python 3, has anyone managed to build a language pack? I seem to have > encountered several small niggling issues and this is the most recent of > them. > Thanks in advance for any guidance here. > Lewis > > -- > http://home.apache.org/~lewismc/ > @hectorMcSpector > http://www.linkedin.com/in/lmcgibbney >
