Or just run kenlm/build_binary lm.arpa and it will spit out a memory usage estimate (covering the LM only).
On 08/26/11 09:24, Hieu Hoang wrote: > barry's right. > > Binarize the phrase table and the LM with irstlm or kenlm. Then just > look at the file sizes & add a few 100mb and that's your memory > requirement for adequate speed. > > You can run the phrase-based decoder in about 300mb if everything is > binarized. I run it on the iphone for fun :) > > (the chart decoder needs 1-2gb) > > On 26/08/2011 15:07, Barry Haddow wrote: >>> Ok, >>> i discovered that probably we can have a 64gb ram 8/12 cores >>> machine. >>> The requirements for translation are the same for the >>> training? >>> >>> I prepared two language models in binary format. And i >>> noticed that when the server is loading/translating it takes 89/90% of >>> ram (actually the test environment has 4gb of RAM), and 10% of cpu. >>> But >>> when there aren't pending translation the memory used is 0%. >>> So for >>> translation machine i still need a 8/12 cores, or i can have a >>> "smaller" machine? >>> For translation what is important? Memory or CPU? >>> >>> And for example with 64gb ram, approximatively how many models can i >>> load on the same machine (suppose we have models with ~400'000/800'000 >>> sentences)? >>> >> >> Hi Ivan >> >> As far as ram is concerned, you need enough to load your model, any more >> won't >> make much difference, and any less then it will run impossibly slow due to >> swapping. >> >> If your data is processed in batches then you can benefit from having more >> CPUs and running multi-threaded decoding. >> >> I'm afraid I've no figures mapping training sentences to model size. I'd >> suggest that you run some experiments in your setup. >> >> cheers - Barry >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
