Hi everyone, I'm trying to build a phrase table using 600k phrase pairs and I'm encountering the following problem:
$MOSES_SCRIPTS/training/train-model.perl -scripts-root-dir $MOSES_SCRIPTS -corpus dict.train -f gr -e ph -lm 0:3:/home/cucu/speechRoot/tools/wordsPhonetization/devel/phonetizer1/dict.train.ph.lm Using SCRIPTS_ROOTDIR: /home/applications/moses/scripts Using single-thread GIZA (1) preparing corpus @ Wed Jan 4 11:49:51 EET 2012 Executing: mkdir -p ./corpus (1.0) selecting factors @ Wed Jan 4 11:49:51 EET 2012 (1.1) running mkcls @ Wed Jan 4 11:49:51 EET 2012 /home/applications/giza-pp/GIZA++-v2/mkcls -c50 -n2 -pdict.train.gr-V./corpus/gr.vcb.classes opt Executing: /home/applications/giza-pp/GIZA++-v2/mkcls -c50 -n2 - pdict.train.gr -V./corpus/gr.vcb.classes opt WARNING: StatVar.cc At this point the training freezes and I cannot do anything else. I've tried to localize this problem by selecting only some of the 600k phrase pairs (*the first* 100k, the first 110k, etc.). Everything worked fine with a dataset of up to 114993 phrase pairs, but failed for a dataset of 114994 phrase-pairs. I've also selected the last 100k phrases, the last 110k phrases, etc. (thinking there could be a problem with the actual data). Here everything worked fine with a dataset of up to 114996 phrase pairs, but failed for a dataset of 114997 phrase-pairs. Could this be a memory problem? top shows a memory usage of 0% for the mkcls process and tells me I have 15GB of free RAM (out of a total of 16GB)... Do you have any ideas of what might be the problem?! What else should I check? Thanks, Horia
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
