Hi Hieu, Is there any way to limit the size of the cache or to empty it at the end of each sentence?
Cheers, David Quoting Hieu Hoang <[EMAIL PROTECTED]>: > Hey holger > > Yep, the decoder caches all used phrases. Put > [use-persistent-cache] > 0 > in your ini file or on commanad line to switch it off. > > The binary phrase table implementation also caches. I'm not sure how big of > a problem that is, and I'm not sure if the cache can be turned off. > > -----Original Message----- > From: Holger Schwenk [mailto:[EMAIL PROTECTED] > Sent: 23 April 2008 00:12 > To: Philipp Koehn; Hieu Hoang > Cc: Holger Schwenk > Subject: memory increase when translating very large corpora with binary > phrase tables > > Hello Philipp and Hieu, > > I want to translate very large corpora of >10M words. Therefore I use binary > phrase tables since filtering would probably result in a very large table. > > When Moses starts it needs about 9G of RAM (I've a big LM) which seems > reasonable to me. However, after several hours of processing (>1M > words) memory has increased to 20GB. > > Is this normal ? Does Moses implement a cache of all previously requested > phrases that continuously increases ? If yes, should I switch off > "-use-persistent-cache" or split all the data into smallre pieces ? > > Thanks for your help > > Holger > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
