Hi Folks, So I've been plodding away again and feel i am very close to generating my first language pack, however I've arrived at the following fankle!!! If I run a pipeline from start to finish it fails at the 'test-bundle-1' phase as below stating " [Errno 2] No such file or directory: '/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final'"
lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero --corpus /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en --tune /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en.tune --test /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en.test --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model training" --aligner berkeley --hadoop-mem 10g --tmp /usr/local/hadoop-2.5.2/hadoop_tmp_dir [train-copy-and-filter] cached, skipping... [train-tokenize-en] cached, skipping... [train-tokenize-ru] cached, skipping... [train-trim] cached, skipping... [train-lowercase-en] cached, skipping... [train-lowercase-ru] cached, skipping... [train-vocab-en] cached, skipping... [train-vocab-ru] cached, skipping... [tune-copy-and-filter] cached, skipping... [tune-tokenize-en] cached, skipping... [tune-tokenize-ru] cached, skipping... [tune-lowercase-en] cached, skipping... [tune-lowercase-ru] cached, skipping... [tune-vocab-en] cached, skipping... [tune-vocab-ru] cached, skipping... [test-copy-and-filter] cached, skipping... [test-tokenize-en] cached, skipping... [test-tokenize-ru] cached, skipping... [test-lowercase-en] cached, skipping... [test-lowercase-ru] cached, skipping... [test-vocab-en] cached, skipping... [test-vocab-ru] cached, skipping... [lm-sort-uniq] cached, skipping... [kenlm] cached, skipping... [compile-kenlm] cached, skipping... [glue-tune] cached, skipping... [tune-bundle] cached, skipping... [mert-1] rebuilding... dep=/usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config [CHANGED] dep=tune/model/grammar.gz.packed/slice_00000.source dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final [NOT FOUND] cmd=/usr/local/incubator-joshua/scripts/training/run_tuner.py /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.ru --tunedir /usr/local/joshua_resources/russian_experiments/exp3/tune --tuner mert --decoder /usr/local/joshua_resources/russian_experiments/exp3/tune/decoder_command --decoder-config /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config --decoder-output-file /usr/local/joshua_resources/russian_experiments/exp3/tune/output.nbest --decoder-log-file /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.log --iterations 10 --metric 'BLEU 4 closest' took 27 seconds (27s) [test-bundle-1] rebuilding... dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final [NOT FOUND] dep=grammar.gz dep=/usr/local/joshua_resources/russian_experiments/exp3/test/1/model/joshua.config cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final /usr/local/joshua_resources/russian_experiments/exp3/test/1/model --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s ||| %f ||| %c" -mark-oovs false' --pack-tm grammar.gz --tm /usr/local/joshua_resources/russian_experiments/exp3/data/tune/grammar.glue JOB FAILED (return code 2) ERROR:root:ERROR: argument config: can't open '/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final': [Errno 2] No such file or directory: '/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final' However, if I run the pipeline with the --first-step test flag, then I get the following where the 'test-bundle-1' phase executes and completes flawlessly however the pipeline then goes on to die at the 'test-decode-1' phase!!! lmcgibbn@LMC-056430 /usr/local/joshua_resources/russian_experiments/exp3 $ /usr/local/incubator-joshua/bin/pipeline.pl --rundir . --type hiero --corpus /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en --tune /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en.tune --test /usr/local/joshua_resources/russian_experiments/data/commoncrawl.ru-en.test --source en --target ru --readme "Experiment 3 Run 1 of ru --> en model training" --aligner berkeley --hadoop-mem 10g --tmp /usr/local/hadoop-2.5.2/hadoop_tmp_dir --first-step test --grammar /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz --joshua-mem 10g [train-copy-and-filter] cached, skipping... [train-tokenize-en] cached, skipping... [train-tokenize-ru] cached, skipping... [train-trim] cached, skipping... [train-lowercase-en] cached, skipping... [train-lowercase-ru] cached, skipping... [train-vocab-en] cached, skipping... [train-vocab-ru] cached, skipping... [tune-copy-and-filter] cached, skipping... [tune-tokenize-en] cached, skipping... [tune-tokenize-ru] cached, skipping... [tune-lowercase-en] cached, skipping... [tune-lowercase-ru] cached, skipping... [tune-vocab-en] cached, skipping... [tune-vocab-ru] cached, skipping... [test-copy-and-filter] cached, skipping... [test-tokenize-en] cached, skipping... [test-tokenize-ru] cached, skipping... [test-lowercase-en] cached, skipping... [test-lowercase-ru] cached, skipping... [test-vocab-en] cached, skipping... [test-vocab-ru] cached, skipping... [glue-test] cached, skipping... [test-bundle-1] rebuilding... dep=/usr/local/incubator-joshua/scripts/training/templates/tune/joshua.config dep=/usr/local/joshua_resources/russian_experiments/exp3/grammar.gz dep=/usr/local/joshua_resources/russian_experiments/exp3/test/1/model/joshua.config [CHANGED] cmd=/usr/local/incubator-joshua/scripts/support/run_bundler.py --force --symlink --absolute --verbose -T /usr/local/hadoop-2.5.2/hadoop_tmp_dir /usr/local/incubator-joshua/scripts/training/templates/tune/joshua.config /usr/local/joshua_resources/russian_experiments/exp3/test/1/model --copy-config-options '-top-n 300 -pop-limit 5000 -output-format "%i ||| %s ||| %f ||| %c" -mark-oovs false' --pack-tm /usr/local/joshua_resources/russian_experiments/exp3/grammar.gz --tm /usr/local/joshua_resources/russian_experiments/exp3/data/test/grammar.glue took 5372 seconds (1h29m32s) [test-decode-1] rebuilding... dep=/usr/local/joshua_resources/russian_experiments/exp3/data/test/corpus.en [CHANGED] dep=/usr/local/joshua_resources/russian_experiments/exp3/test/decoder_command [CHANGED] dep=/usr/local/joshua_resources/russian_experiments/exp3/test/1/model/joshua.config [CHANGED] dep=/usr/local/joshua_resources/russian_experiments/exp3/test/1/model/grammar.gz.packed/slice_00000.source [CHANGED] dep=/usr/local/joshua_resources/russian_experiments/exp3/test/output [CHANGED] cmd=/usr/local/joshua_resources/russian_experiments/exp3/test/decoder_command JOB FAILED (return code 1) I need to ask the question, using current (todays) master branch with Python 3, has anyone managed to build a language pack? I seem to have encountered several small niggling issues and this is the most recent of them. Thanks in advance for any guidance here. Lewis -- http://home.apache.org/~lewismc/ @hectorMcSpector http://www.linkedin.com/in/lmcgibbney
