[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137681#comment-16137681
]
Jeffrey Smith commented on JOSHUA-277:
--------------------------------------
There is a there a file lib/libken.s and bin/lmplz after I ran
download-deps.sh. Perhaps I should give whole output. It is below. it is in
response to the first part of the tutorial (get similar results with other
parts) - $JOSHUA/bin/pipeline.pl \
--rundir 1 \
--readme "Baseline Hiero run" \
--source es \
--target en \
--type hiero \
--corpus $FISHER/corpus/asr/fisher_train \
--tune $FISHER/corpus/asr/fisher_dev \
--test $FISHER/corpus/asr/fisher_dev2 \
--maxlen 11 \
--maxlen-tune 11 \
--maxlen-test 11 \
--tuner-iterations 1 \
--lm-order 3
_____________________________
source test1.sh
[train-copy-and-filter] cached, skipping...
[train-tokenize-es] cached, skipping...
[train-tokenize-en] cached, skipping...
[train-trim] cached, skipping...
[train-lowercase-es] cached, skipping...
[train-lowercase-en] cached, skipping...
[train-vocab-es] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-tokenize-es] cached, skipping...
[tune-tokenize-en.0] cached, skipping...
[tune-tokenize-en.1] cached, skipping...
[tune-tokenize-en.2] cached, skipping...
[tune-tokenize-en.3] cached, skipping...
[tune-trim] cached, skipping...
[tune-lowercase-es] cached, skipping...
[tune-lowercase-en.0] cached, skipping...
[tune-lowercase-en.1] cached, skipping...
[tune-lowercase-en.2] cached, skipping...
[tune-lowercase-en.3] cached, skipping...
[tune-vocab-es] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-tokenize-es] cached, skipping...
[test-tokenize-en.0] cached, skipping...
[test-tokenize-en.1] cached, skipping...
[test-tokenize-en.2] cached, skipping...
[test-tokenize-en.3] cached, skipping...
[test-trim] cached, skipping...
[test-lowercase-es] cached, skipping...
[test-lowercase-en.0] cached, skipping...
[test-lowercase-en.1] cached, skipping...
[test-lowercase-en.2] cached, skipping...
[test-lowercase-en.3] cached, skipping...
[test-vocab-es] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
dep=/data/joshua-tutorial/runs/1/data/tune/corpus.es [CHANGED]
dep=/data/joshua-tutorial/runs/1/tune/joshua.config [CHANGED]
dep=tune/model/grammar.gz.packed/slice_00000.source [CHANGED]
dep=/data/joshua-tutorial/runs/1/tune/joshua.config.final [NOT FOUND]
cmd=/data/joshua/scripts/training/run_tuner.py
/data/joshua-tutorial/runs/1/data/tune/corpus.es
/data/joshua-tutorial/runs/1/data/tune/corpus.en --tunedir
/data/joshua-tutorial/runs/1/tune --tuner mert --decoder
/data/joshua-tutorial/runs/1/tune/decoder_command --decoder-config
/data/joshua-tutorial/runs/1/tune/joshua.config --decoder-output-file
/data/joshua-tutorial/runs/1/tune/output.nbest --decoder-log-file
/data/joshua-tutorial/runs/1/tune/joshua.log --iterations 1 --metric 'BLEU 4
closest'
JOB FAILED (return code 1)
/data/joshua/ext/kenlm/util/file.cc:76 in int util::OpenReadOrThrow(const
char*) threw ErrnoException because `-1 == (ret = open(name, 00))'.
No such file or directory while opening
/home/ec2-user/data/joshua-tutorial/runs/1/tune/model/lm.kenlm
Traceback (most recent call last):
File "/data/joshua/scripts/training/run_tuner.py", line 554, in <module>
main(sys.argv)
File "/data/joshua/scripts/training/run_tuner.py", line 537, in main
run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder,
opts.decoder_config, opts.decoder_output_file, opts)
File "/data/joshua/scripts/training/run_tuner.py", line 418, in run_zmert
opts.metric, opts.iterations or 10)
File "/data/joshua/scripts/training/run_tuner.py", line 400, in setup_configs
for feature,weight in get_features(config):
File "/data/joshua/scripts/training/run_tuner.py", line 351, in get_features
output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" %
(JOSHUA, config_file), shell=True)
File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '/data/joshua/bin/joshua-decoder -c
/data/joshua-tutorial/runs/1/tune/joshua.config -show-weights -v 0' returned
non-zero exit status -6
> UnsatisfiedLinkError: no ken in java.library.path
> -------------------------------------------------
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
> Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result => 20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_00000.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
> ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
> at
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.<init>(LanguageModelFF.java:156)
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.<init>(StateMinimizingLanguageModel.java:50)
> ... 8 more
> Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
> in java.library.path
> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:52)
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:43)
> ... 11 more
> Traceback (most recent call last):
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 553, in <module>
> main(sys.argv)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 536, in main
> run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder,
> opts.decoder_config, opts.decoder_output_file, opts)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 417, in run_zmert
> opts.metric, opts.iterations or 10)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 399, in setup_configs
> for feature,weight in get_features(config):
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 351, in get_features
> output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" %
> (JOSHUA, config_file), shell=True)
> File
> "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 573, in check_output
> raise CalledProcessError(retcode, cmd, output=output)
> subprocess.CalledProcessError: Command
> '/Users/thammegr/work/projects/apache/incubator-joshua/bin/joshua-decoder -c
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> -show-weights -v 0' returned non-zero exit status 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
