hey alberto you should binarize the phrase table & LM so that they load only the necessary records when translating a sentence. This is critical for large models trained on large corpora like Europarl, it saves a lot of memory.
see http://www.statmt.org/moses/?n=Moses.AdvancedFeatures to binarize phrase table. To binarize LM, you must use IRSTLM http://sourceforge.net/projects/irstlm see their instructions on how to binarize LM 2008/9/9 Alberto Calderone <[EMAIL PROTECTED]> > Hello, > I am writing this email to ask something about moses and some memory issues > that we are experiencing in our tests. > We Have tried several time to build the LM and phrase table starting from > the Europarl corpus. > We tried with 5grams and 3grams and we have also tried to narrow down to > 1/2 the training text files. > Despite this series of tests we didn't manage to make moses start with our > generated LM and phrase-table. > Have you got any suggestion? what could we do to try and figure the problem > out, every time Moses, while is starting up, runs out of memory and crashes. > We are using a machine with 3+ Gb and thus this behaviours is suspicious to > us. No other processes are running on the test machine. > Most probably there is something in the step implying the usage of > "train-factored-phrase-model.perl" that we are mistaking. > > Thank you in advance for your kind help > > Alberto Calderone > > > > -- > -- > Alberto Calderonehttp://www.translated.net > Direct: (+39) 06 90 25 4 258 > Main: (+39) 06 90 25 4 001 > Fax: (+39) 06 233 200 102 > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
