Hieu Hoang http://statmt.org/hieu
On Mon, 9 Mar 2020 at 10:08, Haukur Páll Jónsson <hauku...@ru.is> wrote: > Hi all, > > > For the last few weeks, I have been trying to train and tune a factored > model. I have found it difficult to implement and I now seek some > assistance. > > > I am trying to build a simple factored model from English to Icelandic: > T0-0, T0,1-1. Where factor 0 is the surface and factor 1 is the POS. The > training data (source and target) I have has three factors > `surface|pos|lemma`, the lemma is ignored for now. > > > When tuning a factored model I run into problems. My first question is, > what factors should be in tuning data? It seems that I can have all factors > as the input/source but I'm unsure about the output/target. > The tuning data needs to also have factors 0 and 1. You should pre-process ttraining, tuning and test data in the same way I run the tuning like so (using 10 threads): > > > "$MOSESDECODER"/scripts/training/mert-moses.pl \ > "$DEV_DATA_IN"."$LANG_FROM" \ > "$DEV_DATA_OUT"."$LANG_TO" \ > "$MOSESDECODER"/bin/moses "$BASE_MOSES_INI" \ > --mertdir "$MOSESDECODER"/bin \ > --working-dir "$TUNE_DIR" \ > --decoder-flags="-threads $THREADS" > > > But then when starting to decode the decoder crashes. > > > Line Line 9: Initialize search took leave|VBP0.103 seconds total > soft|JJ hands|NNS ,|, Jess|NNP you|PRP very|RB weak|JJ ,|, including|VBG > 4: Initialize search took .|. > : Collecting options took 0.003severe|JJ ,|, his|PRP$I|PRP loved|VBD > Sean|NNP .|. you|PRP know|VBP .|. > seconds at moses/Manager.cpp Line 0.129 seconds total > joint|JJ examination|NN Line 2: Initialize search took 141 > Segmentation fault > Exit code: 139 > > > I am monitoring the memory usage and the decoder is only using about 4GB > of memory from the 32GB allocated when it crashes. Why the is the decoder > crashing? Are there some recommendations for settings when training a > factored model? > > > Haukur Páll Jónsson > > Rannsóknarsérfræðingur | Tölvunarfræðideild > > Research Specialist | School of Computer Science > > Póstfang / E-mail: hauku...@ru.is > > [1568888021669] > > Háskólinn í Reykjavík | Reykjavik University > > Menntavegur 1 | 101 Reykjavík | Iceland > > Sími/Tel: +354 599 6200 > > www.hr.is > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
