[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thamme Gowda updated JOSHUA-277:
--------------------------------
Description:
I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
to test the latest build.
Assuming there few things are broken due to newer maven build system, I tried
to fix pipeline.pl to get the quick start guide working.
Which files from kenlm build should I add to JNI path? (I am unable to locate
the library file in the kenlm build output)
Here is the full log:
{code}
$JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
--no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
[train-copy-and-filter] cached, skipping...
[train-vocab-bn] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-vocab-bn] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-vocab-bn] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[source-numlines] cached, skipping...
[source-numlines] retrieved cached result => 20788
[berkeley-aligner-chunk-0] cached, skipping...
[aligner-combine] cached, skipping...
[pack-grammar] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
Error: Could not find or load main class
joshua.util.encoding.EncoderConfiguration
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
[CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
[CHANGED]
dep=tune/model/grammar.packed/slice_00000.source [CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
[NOT FOUND]
cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
--tunedir
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
mert --decoder
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
--decoder-config
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
--decoder-output-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
--decoder-log-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
--iterations 10 --metric 'BLEU 4 closest'
JOB FAILED (return code 1)
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:122)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
... 3 more
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
at
org.apache.joshua.decoder.ff.lm.LanguageModelFF.<init>(LanguageModelFF.java:156)
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.<init>(StateMinimizingLanguageModel.java:50)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
in java.library.path
at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:52)
... 11 more
Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:43)
... 11 more
Traceback (most recent call last):
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 553, in <module>
main(sys.argv)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 536, in main
run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder,
opts.decoder_config, opts.decoder_output_file, opts)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 417, in run_zmert
opts.metric, opts.iterations or 10)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 399, in setup_configs
for feature,weight in get_features(config):
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 351, in get_features
output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" %
(JOSHUA, config_file), shell=True)
File
"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command
'/Users/thammegr/work/projects/apache/incubator-joshua/bin/joshua-decoder -c
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
-show-weights -v 0' returned non-zero exit status 1
{code}
was:
{code}
$JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
--no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
[train-copy-and-filter] cached, skipping...
[train-vocab-bn] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-vocab-bn] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-vocab-bn] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[source-numlines] cached, skipping...
[source-numlines] retrieved cached result => 20788
[berkeley-aligner-chunk-0] cached, skipping...
[aligner-combine] cached, skipping...
[pack-grammar] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
Error: Could not find or load main class
joshua.util.encoding.EncoderConfiguration
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
[CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
[CHANGED]
dep=tune/model/grammar.packed/slice_00000.source [CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
[NOT FOUND]
cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
--tunedir
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
mert --decoder
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
--decoder-config
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
--decoder-output-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
--decoder-log-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
--iterations 10 --metric 'BLEU 4 closest'
JOB FAILED (return code 1)
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:122)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
... 3 more
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
at
org.apache.joshua.decoder.ff.lm.LanguageModelFF.<init>(LanguageModelFF.java:156)
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.<init>(StateMinimizingLanguageModel.java:50)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
in java.library.path
at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:52)
... 11 more
Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:43)
... 11 more
Traceback (most recent call last):
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 553, in <module>
main(sys.argv)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 536, in main
run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder,
opts.decoder_config, opts.decoder_output_file, opts)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 417, in run_zmert
opts.metric, opts.iterations or 10)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 399, in setup_configs
for feature,weight in get_features(config):
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 351, in get_features
output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" %
(JOSHUA, config_file), shell=True)
File
"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command
'/Users/thammegr/work/projects/apache/incubator-joshua/bin/joshua-decoder -c
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
-show-weights -v 0' returned non-zero exit status 1
{code}
> UnsatisfiedLinkError: no ken in java.library.path
> -------------------------------------------------
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
> Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result => 20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_00000.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
> ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
> at
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.<init>(LanguageModelFF.java:156)
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.<init>(StateMinimizingLanguageModel.java:50)
> ... 8 more
> Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
> in java.library.path
> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:52)
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at org.apache.joshua.decoder.ff.lm.KenLM.<clinit>(KenLM.java:43)
> ... 11 more
> Traceback (most recent call last):
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 553, in <module>
> main(sys.argv)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 536, in main
> run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder,
> opts.decoder_config, opts.decoder_output_file, opts)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 417, in run_zmert
> opts.metric, opts.iterations or 10)
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 399, in setup_configs
> for feature,weight in get_features(config):
> File
> "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
> line 351, in get_features
> output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" %
> (JOSHUA, config_file), shell=True)
> File
> "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 573, in check_output
> raise CalledProcessError(retcode, cmd, output=output)
> subprocess.CalledProcessError: Command
> '/Users/thammegr/work/projects/apache/incubator-joshua/bin/joshua-decoder -c
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> -show-weights -v 0' returned non-zero exit status 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
