[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140845#comment-16140845
]
Thamme Gowda commented on JOSHUA-277:
-
> Exception in thread "main" java.util.NoSuchElementException at
> org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
this usually happens when you are dealing with version of classes, libraries,
or JDK.
Suggestion:
verify that you have JDK 8 active in your environment
do a fresh build of joshua:
- go to master branch: git checkout master.
- get latest changes: git pull origin master
- Rebuild: mvn clean package
- export JOSHUA= correct directory, $PWD where you cloned Joshua
and repeat your pipeline.
> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137728#comment-16137728
]
Thamme Gowda commented on JOSHUA-277:
-
I (the guy who originally created this issue) had faced this problem while
trying to make Joshua run on OSX.
I can confirm that Joshua pipeline works without any hurdles on Linux. I had
previously built it on Ubuntu and a few days back I had built Joshua and
trained few models on Centos. it worked without any trouble. (I am not sure on
OSX yet)
[~jesmitty] Which is your OS?
Most likely, one of the dependencies (Kenlm) is not built or picked up by
joshua.
Hint: debug why "runs/1/tune/model/lm.kenlm" is not being created when you run
the pipeline.
> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
[jira] [Closed] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thamme Gowda closed JOSHUA-277.
---
Resolution: Won't Fix
> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
> ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
> at
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
> at
>
[jira] [Commented] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338604#comment-15338604
]
Thamme Gowda commented on JOSHUA-277:
-
Thanks for the link, [~post]. I will follow this tutorial.
> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
>Reporter: Thamme Gowda
>
> I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
> to test the latest build.
> Assuming there few things are broken due to newer maven build system, I tried
> to fix pipeline.pl to get the quick start guide working.
> Which files from kenlm build should I add to JNI path? (I am unable to locate
> the library file in the kenlm build output)
> Here is the full log:
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
> ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
> at
>
[jira] [Updated] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thamme Gowda updated JOSHUA-277:
Description:
I followed this guide http://joshua.incubator.apache.org/6.0/quick-start.html
to test the latest build.
Assuming there few things are broken due to newer maven build system, I tried
to fix pipeline.pl to get the quick start guide working.
Which files from kenlm build should I add to JNI path? (I am unable to locate
the library file in the kenlm build output)
Here is the full log:
{code}
$JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
--no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
[train-copy-and-filter] cached, skipping...
[train-vocab-bn] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-vocab-bn] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-vocab-bn] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[source-numlines] cached, skipping...
[source-numlines] retrieved cached result =>20788
[berkeley-aligner-chunk-0] cached, skipping...
[aligner-combine] cached, skipping...
[pack-grammar] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
Error: Could not find or load main class
joshua.util.encoding.EncoderConfiguration
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
[CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
[CHANGED]
dep=tune/model/grammar.packed/slice_0.source [CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
[NOT FOUND]
cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
--tunedir
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
mert --decoder
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
--decoder-config
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
--decoder-output-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
--decoder-log-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
--iterations 10 --metric 'BLEU 4 closest'
JOB FAILED (return code 1)
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
... 3 more
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
at
org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.(StateMinimizingLanguageModel.java:50)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
in java.library.path
at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52)
... 11 more
Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at
[jira] [Updated] (JOSHUA-277) UnsatisfiedLinkError: no ken in java.library.path
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thamme Gowda updated JOSHUA-277:
Summary: UnsatisfiedLinkError: no ken in java.library.path (was: Cant find
ken library )
> UnsatisfiedLinkError: no ken in java.library.path
> -
>
> Key: JOSHUA-277
> URL: https://issues.apache.org/jira/browse/JOSHUA-277
> Project: Joshua
> Issue Type: Bug
>Reporter: Thamme Gowda
>
> {code}
> $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
> --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
> --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
> [train-copy-and-filter] cached, skipping...
> [train-vocab-bn] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-vocab-bn] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-vocab-bn] cached, skipping...
> [test-vocab-en.0] cached, skipping...
> [test-vocab-en.1] cached, skipping...
> [test-vocab-en.2] cached, skipping...
> [test-vocab-en.3] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>20788
> [berkeley-aligner-chunk-0] cached, skipping...
> [aligner-combine] cached, skipping...
> [pack-grammar] cached, skipping...
> [lm-sort-uniq] cached, skipping...
> [kenlm] cached, skipping...
> [compile-kenlm] cached, skipping...
> [glue-tune] cached, skipping...
> Error: Could not find or load main class
> joshua.util.encoding.EncoderConfiguration
> [tune-bundle] cached, skipping...
> [mert-1] rebuilding...
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
> [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> [CHANGED]
> dep=tune/model/grammar.packed/slice_0.source [CHANGED]
>
> dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
> [NOT FOUND]
>
> cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
>
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
> --tunedir
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
> mert --decoder
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
> --decoder-config
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
> --decoder-output-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
> --decoder-log-file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
> --iterations 10 --metric 'BLEU 4 closest'
> JOB FAILED (return code 1)
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
> feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
> /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
> ... 3 more
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
> at
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
> at
> org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.(StateMinimizingLanguageModel.java:50)
> ... 8 more
> Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
> in java.library.path
> at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52)
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
>
[jira] [Updated] (JOSHUA-277) Cant find ken library
[
https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thamme Gowda updated JOSHUA-277:
Description:
{code}
$JOSHUA/bin/pipeline.pl --source bn --target en --type hiero
--no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en
--tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
[train-copy-and-filter] cached, skipping...
[train-vocab-bn] cached, skipping...
[train-vocab-en] cached, skipping...
[tune-copy-and-filter] cached, skipping...
[tune-vocab-bn] cached, skipping...
[tune-vocab-en.0] cached, skipping...
[tune-vocab-en.1] cached, skipping...
[tune-vocab-en.2] cached, skipping...
[tune-vocab-en.3] cached, skipping...
[test-copy-and-filter] cached, skipping...
[test-vocab-bn] cached, skipping...
[test-vocab-en.0] cached, skipping...
[test-vocab-en.1] cached, skipping...
[test-vocab-en.2] cached, skipping...
[test-vocab-en.3] cached, skipping...
[source-numlines] cached, skipping...
[source-numlines] retrieved cached result =>20788
[berkeley-aligner-chunk-0] cached, skipping...
[aligner-combine] cached, skipping...
[pack-grammar] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] cached, skipping...
[glue-tune] cached, skipping...
Error: Could not find or load main class
joshua.util.encoding.EncoderConfiguration
[tune-bundle] cached, skipping...
[mert-1] rebuilding...
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
[CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
[CHANGED]
dep=tune/model/grammar.packed/slice_0.source [CHANGED]
dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final
[NOT FOUND]
cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en
--tunedir
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner
mert --decoder
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command
--decoder-config
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config
--decoder-output-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest
--decoder-log-file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log
--iterations 10 --metric 'BLEU 4 closest'
JOB FAILED (return code 1)
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file
/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'!
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761)
at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
at org.apache.joshua.decoder.Decoder.(Decoder.java:122)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757)
... 3 more
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75)
at
org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156)
at
org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.(StateMinimizingLanguageModel.java:50)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken
in java.library.path
at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52)
... 11 more
Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:43)
... 11 more
Traceback (most recent call last):
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 553, in
main(sys.argv)
File
"/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py",
line 536, in
[jira] [Updated] (JOSHUA-277) Cant find ken library
[ https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thamme Gowda updated JOSHUA-277: Summary: Cant find ken library (was: ) > Cant find ken library > -- > > Key: JOSHUA-277 > URL: https://issues.apache.org/jira/browse/JOSHUA-277 > Project: Joshua > Issue Type: Bug >Reporter: Thamme Gowda > > $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero > --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en > --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en > [train-copy-and-filter] cached, skipping... > [train-vocab-bn] cached, skipping... > [train-vocab-en] cached, skipping... > [tune-copy-and-filter] cached, skipping... > [tune-vocab-bn] cached, skipping... > [tune-vocab-en.0] cached, skipping... > [tune-vocab-en.1] cached, skipping... > [tune-vocab-en.2] cached, skipping... > [tune-vocab-en.3] cached, skipping... > [test-copy-and-filter] cached, skipping... > [test-vocab-bn] cached, skipping... > [test-vocab-en.0] cached, skipping... > [test-vocab-en.1] cached, skipping... > [test-vocab-en.2] cached, skipping... > [test-vocab-en.3] cached, skipping... > [source-numlines] cached, skipping... > [source-numlines] retrieved cached result =>20788 > [berkeley-aligner-chunk-0] cached, skipping... > [aligner-combine] cached, skipping... > [pack-grammar] cached, skipping... > [lm-sort-uniq] cached, skipping... > [kenlm] cached, skipping... > [compile-kenlm] cached, skipping... > [glue-tune] cached, skipping... > Error: Could not find or load main class > joshua.util.encoding.EncoderConfiguration > [tune-bundle] cached, skipping... > [mert-1] rebuilding... > > dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn > [CHANGED] > > dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config > [CHANGED] > dep=tune/model/grammar.packed/slice_0.source [CHANGED] > > dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final > [NOT FOUND] > > cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py > > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn > > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en > --tunedir > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner > mert --decoder > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command > --decoder-config > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config > --decoder-output-file > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest > --decoder-log-file > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log > --iterations 10 --metric 'BLEU 4 closest' > JOB FAILED (return code 1) > Exception in thread "main" java.lang.RuntimeException: Unable to instantiate > feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file > /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'! > at > org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761) > at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514) > at org.apache.joshua.decoder.Decoder.(Decoder.java:122) > at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757) > ... 3 more > Caused by: java.lang.ExceptionInInitializerError > at > org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75) > at > org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156) > at > org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.(StateMinimizingLanguageModel.java:50) > ... 8 more > Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken > in java.library.path > at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52) > ... 11 more > Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) > at
[jira] [Updated] (JOSHUA-277)
[ https://issues.apache.org/jira/browse/JOSHUA-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thamme Gowda updated JOSHUA-277: Description: $JOSHUA/bin/pipeline.pl --source bn --target en --type hiero --no-prepare --aligner berkeley --corpus input/bn-en/tok/training.bn-en --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en [train-copy-and-filter] cached, skipping... [train-vocab-bn] cached, skipping... [train-vocab-en] cached, skipping... [tune-copy-and-filter] cached, skipping... [tune-vocab-bn] cached, skipping... [tune-vocab-en.0] cached, skipping... [tune-vocab-en.1] cached, skipping... [tune-vocab-en.2] cached, skipping... [tune-vocab-en.3] cached, skipping... [test-copy-and-filter] cached, skipping... [test-vocab-bn] cached, skipping... [test-vocab-en.0] cached, skipping... [test-vocab-en.1] cached, skipping... [test-vocab-en.2] cached, skipping... [test-vocab-en.3] cached, skipping... [source-numlines] cached, skipping... [source-numlines] retrieved cached result =>20788 [berkeley-aligner-chunk-0] cached, skipping... [aligner-combine] cached, skipping... [pack-grammar] cached, skipping... [lm-sort-uniq] cached, skipping... [kenlm] cached, skipping... [compile-kenlm] cached, skipping... [glue-tune] cached, skipping... Error: Could not find or load main class joshua.util.encoding.EncoderConfiguration [tune-bundle] cached, skipping... [mert-1] rebuilding... dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn [CHANGED] dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config [CHANGED] dep=tune/model/grammar.packed/slice_0.source [CHANGED] dep=/Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config.final [NOT FOUND] cmd=/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.bn /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/data/tune/corpus.en --tunedir /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune --tuner mert --decoder /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/decoder_command --decoder-config /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.config --decoder-output-file /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/output.nbest --decoder-log-file /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/joshua.log --iterations 10 --metric 'BLEU 4 closest' JOB FAILED (return code 1) Exception in thread "main" java.lang.RuntimeException: Unable to instantiate feature function 'StateMinimizingLanguageModel -lm_order 5 -lm_file /Users/thammegr/work/projects/apache/incubator-joshua/data/bn-en/tune/model/lm.kenlm'! at org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:761) at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514) at org.apache.joshua.decoder.Decoder.(Decoder.java:122) at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:757) ... 3 more Caused by: java.lang.ExceptionInInitializerError at org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.initializeLM(StateMinimizingLanguageModel.java:75) at org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:156) at org.apache.joshua.decoder.ff.lm.StateMinimizingLanguageModel.(StateMinimizingLanguageModel.java:50) ... 8 more Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: no ken in java.library.path at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:52) ... 11 more Caused by: java.lang.UnsatisfiedLinkError: no ken in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at org.apache.joshua.decoder.ff.lm.KenLM.(KenLM.java:43) ... 11 more Traceback (most recent call last): File "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", line 553, in main(sys.argv) File "/Users/thammegr/work/projects/apache/incubator-joshua/scripts/training/run_tuner.py", line 536, in main
[jira] [Created] (JOSHUA-277)
Thamme Gowda created JOSHUA-277: --- Summary: Key: JOSHUA-277 URL: https://issues.apache.org/jira/browse/JOSHUA-277 Project: Joshua Issue Type: Bug Reporter: Thamme Gowda -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JOSHUA-274) Use another HTTPServer other than Suns
[ https://issues.apache.org/jira/browse/JOSHUA-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313173#comment-15313173 ] Thamme Gowda commented on JOSHUA-274: - Kellen, yes lets keep the translation core library as minimal as possible. +1 for pointing out the issue. An option would be to setup submodule for this maven project to contain a Joshua REST service . 1. it will allow us to use Joshua from non-JVM languages. 2. since the language packs/models are huge, its not good idea to use JOSHUA from command line when we are making too many calls (since loading takes plenty of time for each CLI call) > Use another HTTPServer other than Suns > -- > > Key: JOSHUA-274 > URL: https://issues.apache.org/jira/browse/JOSHUA-274 > Project: Joshua > Issue Type: Improvement > Components: decoders >Affects Versions: 6.0.5 >Reporter: Lewis John McGibbney >Priority: Critical > Fix For: 6.1 > > > This issue concerns the use of the > [HttpServer|https://github.com/apache/incubator-joshua/blob/master/src/joshua/decoder/JoshuaDecoder.java#L31] > within JoshuaDecoder.java. > We should replace the com.sun.net.httpserver.HttpServer implementation and > other Sun classes with ones from the Java API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JOSHUA-262) Implement all logging as Slf4j over Log4j
[ https://issues.apache.org/jira/browse/JOSHUA-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313110#comment-15313110 ] Thamme Gowda commented on JOSHUA-262: - Yes, I think we can close this [~post] > Implement all logging as Slf4j over Log4j > - > > Key: JOSHUA-262 > URL: https://issues.apache.org/jira/browse/JOSHUA-262 > Project: Joshua > Issue Type: Improvement > Components: core >Affects Versions: 6.0.5 >Reporter: Lewis John McGibbney >Assignee: Thamme Gowda N > Fix For: 6.1 > > > [~hsaputra] suggested that we implement all logging as Slf4j over Log4j. If > we use [parameterized logging > notation|http://www.slf4j.org/faq.html#logging_performance] we can have good > logging in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JOSHUA-274) Use another HTTPServer other than Suns
[ https://issues.apache.org/jira/browse/JOSHUA-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304890#comment-15304890 ] Thamme Gowda commented on JOSHUA-274: - +1 for embedded Tomcat or jetty > Use another HTTPServer other than Suns > -- > > Key: JOSHUA-274 > URL: https://issues.apache.org/jira/browse/JOSHUA-274 > Project: Joshua > Issue Type: Improvement > Components: decoders >Affects Versions: 6.0.5 >Reporter: Lewis John McGibbney >Priority: Critical > Fix For: 6.1 > > > This issue concerns the use of the > [HttpServer|https://github.com/apache/incubator-joshua/blob/master/src/joshua/decoder/JoshuaDecoder.java#L31] > within JoshuaDecoder.java. > We should replace the com.sun.net.httpserver.HttpServer implementation and > other Sun classes with ones from the Java API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JOSHUA-270) pipeline.pl needs major refactoring
[
https://issues.apache.org/jira/browse/JOSHUA-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298820#comment-15298820
]
Thamme Gowda commented on JOSHUA-270:
-
Hi [~lewismc], I made a script to setup the environment for pipeline.pl script
without touching it .
May be helpful for testing and refactoring.
{code}
#!/usr/bin/env bash
echo "STEP: Going to get berkeleyaligner jar"
wget
https://github.com/apache/incubator-joshua/raw/e70677d2eab23daa7082173e6fe337d68aa12230/lib/berkeleyaligner.jar
\
-O $JOSHUA/lib/berkeleyaligner.jar
echo "STEP: Going to build GIZA"
cd $JOSHUA/ext/giza-pp/
make all
make install
echo "STEP: Going to build symal"
cd $JOSHUA/ext/symal/
make
cd $JOSHUA
echo "STEP: Going to get Hadoop distribution"
wget
http://apache.mirrors.tds.net/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz \
-O $JOSHUA/lib/hadoop-2.5.2.tar.gz
cd $JOSHUA
echo "STEP: Getting thrax"
mkdir -p thrax
wget -O /tmp/thrax-e6195e4a1f60edc58448e8922991fe6938c6daba.zip
https://github.com/joshua-decoder/thrax/archive/e6195e4a1f60edc58448e8922991fe6938c6daba.zip
unzip /tmp/thrax-e6195e4a1f60edc58448e8922991fe6938c6daba.zip
mv thrax-e6195e4a1f60edc58448e8922991fe6938c6daba $JOSHUA/thrax
echo "STEP: Building Thrax"
cd $JOSHUA/thrax
ant
cd $JOSHUA
{code}
> pipeline.pl needs major refactoring
> ---
>
> Key: JOSHUA-270
> URL: https://issues.apache.org/jira/browse/JOSHUA-270
> Project: Joshua
> Issue Type: Bug
> Components: pipeline
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
> Fix For: 6.1
>
>
> Right now
> [pipeline.pl|https://github.com/apache/incubator-joshua/blob/master/scripts/training/pipeline.pl]
> is well over 2000 lines long and extremely difficult to navigate.
> I propose the following
> * All ENV is refactored into an pipeline_environment file
> * All Command line parsing and definitions are refactored into a
> pipeline_cli file
> * Sanity checking is refactored into a pipeline_sanity_check file
> * Dependenct Variable Checking is refactored into
> pipeline_dependent_variable_setting file
> * filter and preprocess corpora is refactored into
> pipeline_filter_preprocess_corpora
> * pipeline_subsampling becomes a file
> * pipeline_alignment becomes a file
> * pipeline_parsing becomes a file
> * pipeline_thrax becomes a file
> * pipeline_tuning becomes a file
> * pipeline_testing becomes a file
> * pipeline_subreoutines becomes a file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JOSHUA-262) Implement all logging as Slf4j over Log4j
[ https://issues.apache.org/jira/browse/JOSHUA-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294664#comment-15294664 ] Thamme Gowda commented on JOSHUA-262: - that means: - 0 for DEBUG. When we set level to INFO, these messages don't show up. - 1 for INFO. No ambiguities here - 2+ It seems like DEBUG, although I am not sure Since log4j allows us to set different levels to specific classes and packages, we can tune them. I will send any System.err.print*s to ERROR and System.out.print*s to INFO. Please correct me if this is wrong. > Implement all logging as Slf4j over Log4j > - > > Key: JOSHUA-262 > URL: https://issues.apache.org/jira/browse/JOSHUA-262 > Project: Joshua > Issue Type: Improvement > Components: core >Affects Versions: 6.0.5 >Reporter: Lewis John McGibbney >Assignee: Thamme Gowda N > Fix For: 6.1 > > > [~hsaputra] suggested that we implement all logging as Slf4j over Log4j. If > we use [parameterized logging > notation|http://www.slf4j.org/faq.html#logging_performance] we can have good > logging in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
