Hey holger
Yep, the decoder caches all used phrases. Put
[use-persistent-cache]
0
in your ini file or on commanad line to switch it off.
The binary phrase table implementation also caches. I'm not sure how big of
a problem that is, and I'm not sure if the cache can be turned off.
-----Original Message-----
From: Holger Schwenk [mailto:[EMAIL PROTECTED]
Sent: 23 April 2008 00:12
To: Philipp Koehn; Hieu Hoang
Cc: Holger Schwenk
Subject: memory increase when translating very large corpora with binary
phrase tables
Hello Philipp and Hieu,
I want to translate very large corpora of >10M words. Therefore I use binary
phrase tables since filtering would probably result in a very large table.
When Moses starts it needs about 9G of RAM (I've a big LM) which seems
reasonable to me. However, after several hours of processing (>1M
words) memory has increased to 20GB.
Is this normal ? Does Moses implement a cache of all previously requested
phrases that continuously increases ? If yes, should I switch off
"-use-persistent-cache" or split all the data into smallre pieces ?
Thanks for your help
Holger
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
