[jira] [Commented] (JOSHUA-280) Existing Language packs not compatible with Joshua master
[
https://issues.apache.org/jira/browse/JOSHUA-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572431#comment-15572431
]
Matt Post commented on JOSHUA-280:
--
This is all fixed with the new language packer. Language packs will now include
the runtime and have no external dependencies (including on Joshua or $JOSHUA).
> Existing Language packs not compatible with Joshua master
> -
>
> Key: JOSHUA-280
> URL: https://issues.apache.org/jira/browse/JOSHUA-280
> Project: Joshua
> Issue Type: Bug
> Components: language packs
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Assignee: Matt Post
>Priority: Critical
> Fix For: 6.1
>
>
> When I work with the existing Spanish --> English language pack at
> http://cs.jhu.edu/~post/language-packs/language-pack-es-en-phrase-2015-03-06.tgz,
> I get the following error
> {code}
> lmcgibbn@LMC-032857
> /usr/local/Cellar/joshua/HEAD/libexec/language-pack-es-en-phrase-2015-03-06(NUTCH-2089)
> $ ./run-joshua-server.sh
> INFO - Parameters read from configuration file: joshua.config
> INFO - tm = 'moses -owner pt -maxspan 0 -path phrase-table.packed
> -max-source-len 5'
> INFO - defaultnonterminal = 'X'
> INFO - goalsymbol = 'GOAL'
> INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
> -lm_order 5 -lm_file lm.kenlm'
> INFO - markoovs = 'false'
> INFO - search = 'stack'
> INFO - pop-limit: 100
> INFO - poplimit = '100'
> INFO - topn = '0'
> INFO - useuniquenbest = 'true'
> INFO - outputformat = '%s'
> INFO - includealignindex = 'false'
> INFO - featurefunction = 'OOVPenalty'
> INFO - featurefunction = 'WordPenalty'
> INFO - featurefunction = 'Distortion'
> INFO - featurefunction = 'PhrasePenalty'
> INFO - c = 'joshua.config'
> INFO - server-port: 5674
> INFO - serverport = '5674'
> INFO - Read 9 weights (0 of them dense)
> INFO - Reading vocabulary: phrase-table.packed/vocabulary
> INFO - Read 191983 entries from the vocabulary
> INFO - Reading packed config: phrase-table.packed/config
> 102030405060708090.100%
> Exception in thread "main" java.lang.RuntimeException: The grammar at
> phrase-table.packed was packed with packer version 0, but the earliest
> supported version is 3
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.readConfig(PackedGrammar.java:1061)
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.(PackedGrammar.java:143)
> at
> org.apache.joshua.decoder.phrase.PhraseTable.(PhraseTable.java:65)
> at
> org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:603)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:126)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JOSHUA-280) Existing Language packs not compatible with Joshua master
[
https://issues.apache.org/jira/browse/JOSHUA-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359816#comment-15359816
]
Matt Post commented on JOSHUA-280:
--
The problem is with the phrase-based packed grammar. The decoder used to have
tons of special hooks to make phrase-based decoding work. A recent fix pushed
up enabled us to clean all of these out, but it means that we have to repack
the grammar for phrase-based systems. I'll work on this a minute now.
> Existing Language packs not compatible with Joshua master
> -
>
> Key: JOSHUA-280
> URL: https://issues.apache.org/jira/browse/JOSHUA-280
> Project: Joshua
> Issue Type: Bug
> Components: language packs
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.1
>
>
> When I work with the existing Spanish --> English language pack at
> http://cs.jhu.edu/~post/language-packs/language-pack-es-en-phrase-2015-03-06.tgz,
> I get the following error
> {code}
> lmcgibbn@LMC-032857
> /usr/local/Cellar/joshua/HEAD/libexec/language-pack-es-en-phrase-2015-03-06(NUTCH-2089)
> $ ./run-joshua-server.sh
> INFO - Parameters read from configuration file: joshua.config
> INFO - tm = 'moses -owner pt -maxspan 0 -path phrase-table.packed
> -max-source-len 5'
> INFO - defaultnonterminal = 'X'
> INFO - goalsymbol = 'GOAL'
> INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
> -lm_order 5 -lm_file lm.kenlm'
> INFO - markoovs = 'false'
> INFO - search = 'stack'
> INFO - pop-limit: 100
> INFO - poplimit = '100'
> INFO - topn = '0'
> INFO - useuniquenbest = 'true'
> INFO - outputformat = '%s'
> INFO - includealignindex = 'false'
> INFO - featurefunction = 'OOVPenalty'
> INFO - featurefunction = 'WordPenalty'
> INFO - featurefunction = 'Distortion'
> INFO - featurefunction = 'PhrasePenalty'
> INFO - c = 'joshua.config'
> INFO - server-port: 5674
> INFO - serverport = '5674'
> INFO - Read 9 weights (0 of them dense)
> INFO - Reading vocabulary: phrase-table.packed/vocabulary
> INFO - Read 191983 entries from the vocabulary
> INFO - Reading packed config: phrase-table.packed/config
> 102030405060708090.100%
> Exception in thread "main" java.lang.RuntimeException: The grammar at
> phrase-table.packed was packed with packer version 0, but the earliest
> supported version is 3
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.readConfig(PackedGrammar.java:1061)
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.(PackedGrammar.java:143)
> at
> org.apache.joshua.decoder.phrase.PhraseTable.(PhraseTable.java:65)
> at
> org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:603)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:126)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JOSHUA-280) Existing Language packs not compatible with Joshua master
[
https://issues.apache.org/jira/browse/JOSHUA-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359696#comment-15359696
]
Lewis John McGibbney commented on JOSHUA-280:
-
The existing chinese language pack works just fine
{code}
lmcgibbn@LMC-032857
/usr/local/Cellar/joshua/HEAD/libexec/zh-en-hiero-pack-2016-01(NUTCH-2089) $
./run-joshua-server.sh
Parameters read from configuration file:
tm = 'thrax -path grammar.packed -maxspan 20 -owner pt'
tm = 'thrax -path grammar.glue -maxspan -1 -owner glue'
defaultnonterminal = 'X'
goalsymbol = 'GOAL'
featurefunction = 'LanguageModel -lm_order 5 -lm_type berkeleylm -lm_file
lm.berkeleylm'
markoovs = 'false'
search = 'cky'
poplimit = '100'
topn = '0'
useuniquenbest = 'true'
outputformat = '%S'
includealignindex = 'false'
featurefunction = 'OOVPenalty'
featurefunction = 'WordPenalty'
Parameters overridden from the command line:
server-port: 5674
serverport = '5674'
c = 'joshua.config'
Read 10 weights (0 of them dense)
Reading vocabulary: grammar.packed/vocabulary
Read 300317 entries from the vocabulary
Reading packed config: grammar.packed/config
102030405060708090.100%
Reading encoder configuration: grammar.packed/encoding
Loaded 62685418 rules
Reading grammar from file grammar.glue...
MemoryBasedBatchGrammar: Read 4 rules with 4 distinct source sides from
'grammar.glue'
Memory used 3447.1 MB
Grammar loading took: 39 seconds.
Stateful object with state index 0
Loading Berkeley LM from binary lm.berkeleylm
FEATURE: tm_pt (weight 0.000)
FEATURE: tm_glue (weight 0.000)
FEATURE: lm_0, order 5 (weight 0.194)
FEATURE: OOVPenalty (weight 0.015)
FEATURE: WordPenalty (weight -0.460)
Grammar sorting happening lazily on-demand.
Model loading took 42 seconds
Memory used 4355.5 MB
** TCP Server running and listening on port 5674.
{code}
> Existing Language packs not compatible with Joshua master
> -
>
> Key: JOSHUA-280
> URL: https://issues.apache.org/jira/browse/JOSHUA-280
> Project: Joshua
> Issue Type: Bug
> Components: language packs
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.1
>
>
> When I work with the existing Spanish --> English language pack at
> http://cs.jhu.edu/~post/language-packs/language-pack-es-en-phrase-2015-03-06.tgz,
> I get the following error
> {code}
> lmcgibbn@LMC-032857
> /usr/local/Cellar/joshua/HEAD/libexec/language-pack-es-en-phrase-2015-03-06(NUTCH-2089)
> $ ./run-joshua-server.sh
> INFO - Parameters read from configuration file: joshua.config
> INFO - tm = 'moses -owner pt -maxspan 0 -path phrase-table.packed
> -max-source-len 5'
> INFO - defaultnonterminal = 'X'
> INFO - goalsymbol = 'GOAL'
> INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
> -lm_order 5 -lm_file lm.kenlm'
> INFO - markoovs = 'false'
> INFO - search = 'stack'
> INFO - pop-limit: 100
> INFO - poplimit = '100'
> INFO - topn = '0'
> INFO - useuniquenbest = 'true'
> INFO - outputformat = '%s'
> INFO - includealignindex = 'false'
> INFO - featurefunction = 'OOVPenalty'
> INFO - featurefunction = 'WordPenalty'
> INFO - featurefunction = 'Distortion'
> INFO - featurefunction = 'PhrasePenalty'
> INFO - c = 'joshua.config'
> INFO - server-port: 5674
> INFO - serverport = '5674'
> INFO - Read 9 weights (0 of them dense)
> INFO - Reading vocabulary: phrase-table.packed/vocabulary
> INFO - Read 191983 entries from the vocabulary
> INFO - Reading packed config: phrase-table.packed/config
> 102030405060708090.100%
> Exception in thread "main" java.lang.RuntimeException: The grammar at
> phrase-table.packed was packed with packer version 0, but the earliest
> supported version is 3
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.readConfig(PackedGrammar.java:1061)
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.(PackedGrammar.java:143)
> at
> org.apache.joshua.decoder.phrase.PhraseTable.(PhraseTable.java:65)
> at
> org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:603)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:126)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JOSHUA-280) Existing Language packs not compatible with Joshua master
[
https://issues.apache.org/jira/browse/JOSHUA-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359691#comment-15359691
]
Lewis John McGibbney commented on JOSHUA-280:
-
Same goes for the Arabic language pack
{code}
lmcgibbn@LMC-032857
/usr/local/Cellar/joshua/HEAD/libexec/language-pack-ar-en-phrase-2015-03-18(NUTCH-2089)
$ ./run-joshua-server.sh
INFO - Parameters read from configuration file: joshua.config
INFO - tm = 'moses -owner pt -maxspan 0 -max-source-len 5 -path
phrase-table.packed'
INFO - defaultnonterminal = 'X'
INFO - goalsymbol = 'GOAL'
INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
-lm_order 5 -lm_file gigaword.kenlm.v5'
INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
-lm_order 5 -lm_file lm.kenlm'
INFO - markoovs = 'false'
INFO - search = 'stack'
INFO - pop-limit: 100
INFO - poplimit = '100'
INFO - topn = '0'
INFO - useuniquenbest = 'true'
INFO - outputformat = '%s'
INFO - includealignindex = 'false'
INFO - featurefunction = 'OOVPenalty'
INFO - featurefunction = 'WordPenalty'
INFO - featurefunction = 'Distortion'
INFO - featurefunction = 'PhrasePenalty'
INFO - c = 'joshua.config'
INFO - server-port: 5674
INFO - serverport = '5674'
INFO - Read 10 weights (0 of them dense)
INFO - Reading vocabulary: phrase-table.packed/vocabulary
INFO - Read 297133 entries from the vocabulary
INFO - Reading packed config: phrase-table.packed/config
102030405060708090.100%
Exception in thread "main" java.lang.RuntimeException: The grammar at
phrase-table.packed was packed with packer version 0, but the earliest
supported version is 3
at
org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.readConfig(PackedGrammar.java:1061)
at
org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.(PackedGrammar.java:143)
at
org.apache.joshua.decoder.phrase.PhraseTable.(PhraseTable.java:65)
at
org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:603)
at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:514)
at org.apache.joshua.decoder.Decoder.(Decoder.java:126)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
{code}
> Existing Language packs not compatible with Joshua master
> -
>
> Key: JOSHUA-280
> URL: https://issues.apache.org/jira/browse/JOSHUA-280
> Project: Joshua
> Issue Type: Bug
> Components: language packs
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Critical
> Fix For: 6.1
>
>
> When I work with the existing Spanish --> English language pack at
> http://cs.jhu.edu/~post/language-packs/language-pack-es-en-phrase-2015-03-06.tgz,
> I get the following error
> {code}
> lmcgibbn@LMC-032857
> /usr/local/Cellar/joshua/HEAD/libexec/language-pack-es-en-phrase-2015-03-06(NUTCH-2089)
> $ ./run-joshua-server.sh
> INFO - Parameters read from configuration file: joshua.config
> INFO - tm = 'moses -owner pt -maxspan 0 -path phrase-table.packed
> -max-source-len 5'
> INFO - defaultnonterminal = 'X'
> INFO - goalsymbol = 'GOAL'
> INFO - featurefunction = 'StateMinimizingLanguageModel -lm_type kenlm
> -lm_order 5 -lm_file lm.kenlm'
> INFO - markoovs = 'false'
> INFO - search = 'stack'
> INFO - pop-limit: 100
> INFO - poplimit = '100'
> INFO - topn = '0'
> INFO - useuniquenbest = 'true'
> INFO - outputformat = '%s'
> INFO - includealignindex = 'false'
> INFO - featurefunction = 'OOVPenalty'
> INFO - featurefunction = 'WordPenalty'
> INFO - featurefunction = 'Distortion'
> INFO - featurefunction = 'PhrasePenalty'
> INFO - c = 'joshua.config'
> INFO - server-port: 5674
> INFO - serverport = '5674'
> INFO - Read 9 weights (0 of them dense)
> INFO - Reading vocabulary: phrase-table.packed/vocabulary
> INFO - Read 191983 entries from the vocabulary
> INFO - Reading packed config: phrase-table.packed/config
> 102030405060708090.100%
> Exception in thread "main" java.lang.RuntimeException: The grammar at
> phrase-table.packed was packed with packer version 0, but the earliest
> supported version is 3
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.readConfig(PackedGrammar.java:1061)
> at
> org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.(PackedGrammar.java:143)
> at
> org.apache.joshua.decoder.phrase.PhraseTable.(PhraseTable.java:65)
> at
> org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:603)
> at org.apache.joshua.decoder.De
