Hi all,
For the last few weeks, I have been trying to train and tune a factored model.
I have found it difficult to implement and I now seek some assistance.
I am trying to build a simple factored model from English to Icelandic: T0-0,
T0,1-1. Where factor 0 is the surface and factor 1 is the POS. The training
data (source and target) I have has three factors `surface|pos|lemma`, the
lemma is ignored for now.
When tuning a factored model I run into problems. My first question is, what
factors should be in tuning data? It seems that I can have all factors as the
input/source but I'm unsure about the output/target.
I run the tuning like so (using 10 threads):
"$MOSESDECODER"/scripts/training/mert-moses.pl \
"$DEV_DATA_IN"."$LANG_FROM" \
"$DEV_DATA_OUT"."$LANG_TO" \
"$MOSESDECODER"/bin/moses "$BASE_MOSES_INI" \
--mertdir "$MOSESDECODER"/bin \
--working-dir "$TUNE_DIR" \
--decoder-flags="-threads $THREADS"
But then when starting to decode the decoder crashes.
Line Line 9: Initialize search took leave|VBP0.103 seconds total
soft|JJ hands|NNS ,|, Jess|NNP you|PRP very|RB weak|JJ ,|, including|VBG 4:
Initialize search took .|.
: Collecting options took 0.003severe|JJ ,|, his|PRP$I|PRP loved|VBD Sean|NNP
.|. you|PRP know|VBP .|.
seconds at moses/Manager.cpp Line 0.129 seconds total
joint|JJ examination|NN Line 2: Initialize search took 141
Segmentation fault
Exit code: 139
I am monitoring the memory usage and the decoder is only using about 4GB of
memory from the 32GB allocated when it crashes. Why the is the decoder
crashing? Are there some recommendations for settings when training a factored
model?
Haukur Páll Jónsson
Rannsóknarsérfræðingur | Tölvunarfræðideild
Research Specialist | School of Computer Science
Póstfang / E-mail: [email protected]
[1568888021669]
Háskólinn í Reykjavík | Reykjavik University
Menntavegur 1 | 101 Reykjavík | Iceland
Sími/Tel: +354 599 6200
www.hr.is
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
