Hieu Hoang schreef op de 17e dag van de lentemaand van het jaar 2014:
> can you send me me the exact command you executed, the moses.ini file used,
> and a few lines of the input file
Here is the training command:
en_word=0
en_root=1
en_pos=2
en_rel=3
nl_word=0
nl_root=1
nl_pos=2
nl_postag=3
nl_rel=4
/net/aps/64/opt/moses/mosesdecoder/scripts/training/train-model.perl \
-cores 12 \
-root-dir train \
-corpus /net/aistaff/kleiweg/moses/europarl/factored/corpus/train-small.clean
\
-f en -e nl \
-alignment grow-diag-final-and \
-reordering msd-bidirectional-fe \
--lm $nl_word:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word:8
\
--lm $nl_root:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root:8
\
--lm $nl_pos:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos:8 \
--lm
$nl_postag:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag:8 \
--lm $nl_rel:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel:8 \
--alignment-factors $en_root-$nl_root \
--translation-factors
$en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel
\
--reordering-factors
$en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel
\
--generation-factors $nl_root,$nl_pos,$nl_postag,$nl_rel-$nl_word \
--decoding-steps t0:t1,t2,t3,t4,g0 \
-external-bin-dir /net/aps/64/opt/moses/mgizapp/bin \
-mgiza \
-mgiza-cpus=12 \
> training.out \
2> training.err
Here are three lines of training data, input:
I|I|prp|nsubj declare|declare|vbp|hd resumed|resume|vbd|hd the|the|dt|det
session|session|nn|hd of|of|in|hd the|the|dt|det European|European|nnp|nn
Parliament|Parliament|nnp|hd adjourned|adjourn|vbn|hd on|on|in|hd
Friday|Friday|nnp|hd 17|17|cd|num December|December|nnp|hd 1999|1999|cd|num
,|,|,|punct and|and|cc|cc I|I|prp|nsubj would|would|md|aux like|like|vb|hd
once|once|rb|advmod again|again|rb|hd to|to|to|aux wish|wish|vb|hd
you|you|prp|nsubj a|a|dt|det happy|happy|jj|amod new|new|jj|amod
year|year|nn|hd in|in|in|hd the|the|dt|det hope|hope|nn|hd that|that|in|mark
you|you|prp|nsubj enjoyed|enjoy|vbd|hd a|a|dt|det pleasant|pleasant|jj|amod
festive|festive|jj|amod period|period|nn|hd .|.|.|punct
although|although|in|mark ,|,|,|punct as|as|in|mark you|you|prp|nsubj
will|will|md|aux have|have|vb|aux seen|see|vbn|hd ,|,|,|punct the|the|dt|det
dreaded|dreaded|jj|amod '|'|"|punct millennium|millennium|nn|nn
bug|bug|nn|hd '|'|pos|punct failed|fail|vbn|hd to|to|to|hd
materialise|materialise|nn|pobj ,|,|,|punct still|still|rb|advmod
the|the|dt|det people|people|nns|hd in|in|in|hd a|a|dt|det number|number|nn|hd
of|of|in|hd countries|country|nns|pobj suffered|suffer|vbd|hd a|a|dt|det
series|series|nn|hd of|of|in|hd natural|natural|jj|amod
disasters|disaster|nns|hd that|that|wdt|nsubj truly|truly|rb|advmod
were|be|vbd|cop dreadful|dreadful|jj|hd .|.|.|punct
you|you|prp|nsubj have|have|vbp|aux requested|request|vbn|hd a|a|dt|det
debate|debate|nn|hd on|on|in|hd this|this|dt|det subject|subject|nn|hd
in|in|in|hd the|the|dt|det course|course|nn|hd of|of|in|hd the|the|dt|det
next|next|jj|amod few|few|jj|amod days|day|nns|hd ,|,|,|punct
during|during|in|hd this|this|dt|det part-session|part-session|nn|hd .|.|.|punct
And three lines of training data, target:
ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su
verklaar|verklaar|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det
zitting|zitting|noun|N(soort,ev,basis,zijd,stan)|hd van|van|prep|VZ(init)|hd
het|het|det|LID(bep,stan,evon)|det Europees|Europees|name|SPEC(deeleigen)|mwp
Parlement|Parlement|name|SPEC(deeleigen)|mwp ,|,|punct|LET()|--
die|die|pron|VNW(betr,pron,stan,vol,persoon,getal)|rhd op|op|prep|VZ(init)|hd
vrijdag|vrijdag|noun|N(eigen,ev,basis,zijd,stan)|hd
17|17|noun|TW(hoofd,vrij)|mwp december|december|noun|TW(hoofd,vrij)|mwp
werd|word|verb|WW(pv,verl,ev)|hd
onderbroken|onderbreek|verb|WW(vd,vrij,zonder)|hd ,|,|punct|LET()|--
te|te|comp|VZ(init)|cmp zijn|ben|verb|WW(inf,vrij,zonder)|hd
hervat|hervat|verb|WW(vd,vrij,zonder)|vc .|.|punct|LET()|--
ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su wens|wens|verb|WW(pv,tgw,ev)|hd
u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|hd allen|allen|adv|BW()|mod
een|een|det|LID(onbep,stan,agr)|det
gelukkig|gelukkig|adj|ADJ(prenom,basis,zonder!
)|mod nieuwjaar|nieuwjaar|noun|N(soort,ev,basis,onz,stan)|hd
en|en|vg|VG(neven)|crd hoop|hoop|verb|WW(pv,tgw,ev)|hd
dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su
een|een|det|LID(onbep,stan,agr)|det
goede|goed|adj|ADJ(prenom,basis,met-e,stan)|mod
vakantie|vakantie|noun|N(soort,ev,basis,zijd,stan)|hd
heeft|heb|verb|WW(pv,tgw,met-t)|hd gehad|heb|verb|WW(vd,vrij,zonder)|hd
.|.|punct|LET()|--
zoals|zoals|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su
heeft|heb|verb|WW(pv,tgw,met-t)|hd kunnen|kan|verb|WW(inf,vrij,zonder)|hd
constateren|constateer|verb|WW(inf,vrij,zonder)|hd ,|,|punct|LET()|--
is|ben|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det
grote|groot|adj|ADJ(prenom,basis,met-e,stan)|mod "|"|punct|LET()|--
millenniumbug|millenniumbug|name|N(eigen,ev,basis,zijd,stan)|hd
"|"|punct|LET()|-- uitgebleven|blijf_uit|verb|WW(vd,vrij,zonder)|hd
.|.|punct|LET()|-- de|de|det|LID(bep,stan,rest)|det
burgers|burger|noun|N(soort,mv,basis)|hd van|van|prep|VZ(init)|hd
een|een|det|LID(onbep,stan,agr)|det
aantal|aantal|noun|N(soort,ev,basis,onz,stan)|hd van|van|prep|VZ(init)|hd
onze|ons|det|VNW(bez,det,stan,vol,1,mv,prenom,met-e,rest)|det
lidstaten|lidstaat|noun|N(soort,mv,basis)|hd zijn|ben|verb|WW(pv,tgw,mv)|hd
daarentegen|daarentegen|adv|BW()|mod door|door|prep|VZ(init)|hd
verschrikkelijke|verschrikkelijk|adj|ADJ(prenom,basis,me!
t-e,stan)|mod natuurrampen|natuur_ramp|noun|N(soort,mv,basis)|hd
getroffen|tref|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|--
u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su
heeft|heb|verb|WW(pv,tgw,met-t)|hd
aangegeven|geef_aan|verb|WW(vd,vrij,zonder)|hd dat|dat|comp|VG(onder)|cmp
u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su
deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det
vergaderperiode|vergader_periode|noun|N(soort,ev,basis,zijd,stan)|hd
een|een|det|LID(onbep,stan,agr)|det
debat|debat|noun|N(soort,ev,basis,onz,stan)|hd
wilt|wil|verb|WW(pv,tgw,met-t)|hd over|over|prep|VZ(init)|hd
deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det
rampen|ramp|noun|N(soort,mv,basis)|hd .|.|punct|LET()|--
Here is the moses.ini created by the training:
#########################
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
1
2
3
# mapping steps
[mapping]
0 T 0
1 T 1
1 T 2
1 T 3
1 T 4
1 G 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz
input-factor=0 output-factor=0
PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz
input-factor=1 output-factor=1
PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz
input-factor=2 output-factor=2
PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz
input-factor=2 output-factor=3
PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz
input-factor=3 output-factor=4
Generation name=GenerationModel0 num-features=2
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz
input-factor=1,2,3,4 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering1 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering2 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering3 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=3
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering4 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz
Distortion
KENLM lazyken=0 name=LM0 factor=0
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3
KENLM lazyken=0 name=LM1 factor=1
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3
KENLM lazyken=0 name=LM2 factor=2
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3
KENLM lazyken=0 name=LM3 factor=3
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3
KENLM lazyken=0 name=LM4 factor=4
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3
# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
TranslationModel1= 0.2 0.2 0.2 0.2
TranslationModel2= 0.2 0.2 0.2 0.2
TranslationModel3= 0.2 0.2 0.2 0.2
TranslationModel4= 0.2 0.2 0.2 0.2
GenerationModel0= 0.3 0
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
LM1= 0.5
LM2= 0.5
LM3= 0.5
LM4= 0.5
This is the command I used:
moses -f train/model/moses.ini
And this is the output of that command, until it exits,
before any actual input:
Defined parameters (per moses.ini or switch):
config: train/model/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz
input-factor=0 output-factor=0 PhraseDictionaryMemory name=TranslationModel1
table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz
input-factor=1 output-factor=1 PhraseDictionaryMemory name=TranslationModel2
table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz
input-factor=2 output-factor=2 PhraseDictionaryMemory name=TranslationModel3
table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz
input-factor=2 output-factor=3 PhraseDictionaryMemory name=TranslationModel4
table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/trai!
n/model/phrase-table.3-4.gz input-factor=3 output-factor=4 Generation
name=GenerationModel0 num-features=2
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz
input-factor=1,2,3,4 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering1 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering2 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz
LexicalReordering!
name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe-
allff input-factor=2 output-factor=3
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering4 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz
Distortion KENLM lazyken=0 name=LM0 factor=0
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3 KENLM
lazyken=0 name=LM1 factor=1
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3 KENLM
lazyken=0 name=LM2 factor=2
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3 KENLM
lazyken=0 name=LM3 factor=3
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3
KENLM lazyken=0 name=LM4 factor=4
path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3
input-factors: 0 1 2 3
mapping: 0 T 0 1 T 1 1 T 2 1 T 3 1 T 4 1 G 0
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 TranslationModel1= 0.2 0.2 0.2 0.2
TranslationModel2= 0.2 0.2 0.2 0.2 TranslationModel3= 0.2 0.2 0.2 0.2
TranslationModel4= 0.2 0.2 0.2 0.2 GenerationModel0= 0.3 0 LexicalReordering0=
0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering3= 0.3 0.3 0.3 0.3
0.3 0.3 LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
LM1= 0.5 LM2= 0.5 LM3= 0.5 LM4= 0.5
/net/aps/64/opt/moses/mosesdecoder/bin
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20
num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz
input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz
input-factor=1 output-factor=1
FeatureFunction: TranslationModel1 start: 7 end: 10
line=PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz
input-factor=2 output-factor=2
FeatureFunction: TranslationModel2 start: 11 end: 14
line=PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz
input-factor=2 output-factor=3
FeatureFunction: TranslationModel3 start: 15 end: 18
line=PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4
path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz
input-factor=3 output-factor=4
FeatureFunction: TranslationModel4 start: 19 end: 22
Exception: bitset::set
>
> On 17 March 2014 17:21, Peter Kleiweg <[email protected]> wrote:
>
> >
> > I run train-model.perl, and it ends with "(9) create moses.ini"
> > without any error.
> >
> > When I try to start moses with "moses -f moses.ini" it fails
> > with "Exception: bitset::set"
> >
> > Is this a bug? What do I do?
> > What does it even mean, bitset::set?
--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
