Hi,
I have a factored dataset. It involves 4 factors,
factor1|factor2|factor3|factor4. I have trained my model with such a
dataset.
Now when I want to tune my model, I encounter with the following error:
*Exception: moses/Word.cpp:159 in void
Moses::Word::CreateFromString(Moses::FactorDirection, const
std::vector<long unsigned int>&, const StringPiece&, bool, bool) threw
util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
factors in string '-|-|Punc|Punc*
The details of the error is in mert.txt file, which is attached to this
e-mail.
Thanks,
Kind Regards,
Hasan Sait Arslan
Using SCRIPTS_ROOTDIR: /export/students/sait/build/mosesdecoder/scripts
filtering the phrase tables... T dets 6 10:59:05 EET 2016
exec:
/export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl
./filtered
/export/students/sait/factored_translation/working/unfactored/model/moses.ini
/export/students/sait/factored_translation/dataset/dev.tr
Executing:
/export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl
./filtered
/export/students/sait/factored_translation/working/unfactored/model/moses.ini
/export/students/sait/factored_translation/dataset/dev.tr > filterphrases.out
2> filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all
-config filtered/moses.ini -show-weights
exec: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config
filtered/moses.ini -show-weights
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all
-config filtered/moses.ini -show-weights > ./features.list 2> /dev/null
MERT starting values and ranges for random generation:
Distortion0 = 0.300 ( 0.00 .. 1.00)
LM0 = 0.500 ( 0.00 .. 1.00)
WordPenalty0 = -1.000 ( 0.00 .. 1.00)
PhrasePenalty0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
featlist: Distortion0=0.300000
featlist: LM0=0.500000
featlist: WordPenalty0=-1.000000
featlist: PhrasePenalty0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
run 1 start at T dets 6 11:01:28 EET 2016
Parsing --decoder-flags: |-threads all|
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.300000 0.500000 -1.000000 0.200000 0.200000 0.200000
0.200000 0.200000
DECODER_CFG = -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429
0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143
WordPenalty0= -0.357143'
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all
-config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429
0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0=
0.107143 WordPenalty0= -0.357143' -n-best-list run1.best100.out 100 distinct
-input-file /export/students/sait/factored_translation/dataset/dev.tr >
run1.out
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all
-config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429
0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0=
0.107143 WordPenalty0= -0.357143' -n-best-list run1.best100.out 100 distinct
-input-file /export/students/sait/factored_translation/dataset/dev.tr > run1.out
1-10.20.2 0.2 0.2 0.20.30.5(1) run decoder to produce n-best lists
params = -threads all
decoder_config = -weight-overwrite 'TranslationModel0= 0.071429 0.071429
0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143
WordPenalty0= -0.357143'
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz
input-factor=0 output-factor=0 Distortion KENLM name=LM0 factor=0
path=/export/students/sait/factored_translation/lm/surface.lm order=3
input-factors: 0 1 2 3 4
input-file: /export/students/sait/factored_translation/dataset/dev.tr
mapping: 0 T 0
n-best-list: run1.best100.out 100 distinct
threads: all
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 Distortion0= 0.3 LM0= 0.5
weight-overwrite: TranslationModel0= 0.071429 0.071429 0.071429
0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143
WordPenalty0= -0.357143
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz
input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=Distortion
FeatureFunction: Distortion0 start: 7 end: 7
line=KENLM name=LM0 factor=0
path=/export/students/sait/factored_translation/lm/surface.lm order=3
Loading the LM will be faster if you build a binary file.
Reading /export/students/sait/factored_translation/lm/surface.lm
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
FeatureFunction: LM0 start: 8 end: 8
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading Distortion0
Loading LM0
Loading TranslationModel0
Start loading text phrase table. Moses format : [27.627] seconds
Reading
/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Created input-output object : [56.159] seconds
Exception: moses/Word.cpp:159 in void
Moses::Word::CreateFromString(Moses::FactorDirection, const std::vector<long
unsigned int>&, const StringPiece&, bool, bool) threw util::Exception because
`!isNonTerminal && i < factorOrder.size()'.
Too few factors in string '-|-|Punc|Punc'.
Exit code: 1
The decoder died. CONFIG WAS -weight-overwrite 'TranslationModel0= 0.071429
0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0=
0.107143 WordPenalty0= -0.357143'
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
