Hi Horia :) Could you please send me your training data ?
Marwen :) 2012/1/4 Horia Cucu <[email protected]> > Hi everyone, > > I'm trying to build a phrase table using 600k phrase pairs and I'm > encountering the following problem: > > $MOSES_SCRIPTS/training/train-model.perl -scripts-root-dir $MOSES_SCRIPTS > -corpus dict.train -f gr -e ph -lm > 0:3:/home/cucu/speechRoot/tools/wordsPhonetization/devel/phonetizer1/dict.train.ph.lm > Using SCRIPTS_ROOTDIR: /home/applications/moses/scripts > Using single-thread GIZA > (1) preparing corpus @ Wed Jan 4 11:49:51 EET 2012 > Executing: mkdir -p ./corpus > (1.0) selecting factors @ Wed Jan 4 11:49:51 EET 2012 > (1.1) running mkcls @ Wed Jan 4 11:49:51 EET 2012 > /home/applications/giza-pp/GIZA++-v2/mkcls -c50 -n2 > -pdict.train.gr-V./corpus/gr.vcb.classes opt > Executing: /home/applications/giza-pp/GIZA++-v2/mkcls -c50 -n2 - > pdict.train.gr -V./corpus/gr.vcb.classes opt > WARNING: StatVar.cc > > At this point the training freezes and I cannot do anything else. > > I've tried to localize this problem by selecting only some of the 600k > phrase pairs (*the first* 100k, the first 110k, etc.). Everything worked > fine with a dataset of up to 114993 phrase pairs, but failed for a dataset > of 114994 phrase-pairs. > I've also selected the last 100k phrases, the last 110k phrases, etc. > (thinking there could be a problem with the actual data). Here everything > worked fine with a dataset of up to 114996 phrase pairs, but failed for a > dataset of 114997 phrase-pairs. > > Could this be a memory problem? top shows a memory usage of 0% for the > mkcls process and tells me I have 15GB of free RAM (out of a total of > 16GB)... > > Do you have any ideas of what might be the problem?! What else should I > check? > > Thanks, > Horia > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
