Hi Tom If one of the moses caches was filling up, then I would expect that the process memory would increase, until the machine ground to a halt. The problem that Ivan had with the original version of his wrapper was slightly different, there was a fixed size i/o buffer that he wasn't emptying, which eventually deadlocked his process.
The lm cache that you mentioned below is, as far as I'm aware, specific to irstlm, so if you're not using irstlm, then the flag should have no effect. The translation option cache mainly helps by caching translation options for common phrases like 'the' or '.'. It is implemented as an LRU cache, and the decoder removes an entry when the cache reaches maximum size. I don't understand the quote from the manual about making sure this cache is frequently cleared - could you tell me where this quote comes from? Tuning the size of the translation options cache may help with performance, but it's unlikely to be the cause of the unexplained crashes. We frequently run multi-threaded decodes on mult-core machines and haven't witnessed any unexplained crashes. So I would quite like to eliminate the python/moses interaction as cause of error. Is it possible to run a similar experiment without the python wrapper, say by just passing moses your source sentences in a file? If it is moses that is crashing, then if you could allow it to generate a core file and make it available, then I'd have some chance of debugging it, cheers - Barry On Monday 29 August 2011 12:23, Tom Hoar wrote: > I've implemented a multi-threaded Python wrapper that loads moses > decoder and pipes strings through the moses binary. It's similar to Ivan > Uemlianin's code from May 04, 2010 on this listserv, but achieves a > throughput efficiency 398% CPU load on a quad-core host across multiple > documents processed in a queue. > > Here's the rub. The decoder & the > wrapper run great for about 2 hours. Then they halt with an unknown > error. It's difficult to trace because it takes hours to reproduce. I > can see that the Moses binary doesn't generate an error exit code. > There's no error message about a "broken" pipe. When I restart the > script on the file that was in-process at the time of hault, it runs > just fine and continues processing. Since the error occurs consistently > at the 2 hour mark, and it's not the file causing the halt, I suspect at > a cache or buffer somewhere is overloaded. I've checked my python code, > and don't believe there are any buffer overruns there. > > I'm hoping > someone can review my comments and give me some pointers about Moses' > caches and how to verify manage the caches. The Moses manual describes > three cache: > > * "-clean-lm-cache: clean language model caches after N > translations (default N=1)" : If -clean-lm-cache defaults to cleaning > the lm cache after each translation, I don't think this is a problem. > > * "-persistent-cache-size: maximum size of cache for translation > options (default 10,000 input phrases)" : Some of my files have my files > have 2,500 or more pages with 20-25 sentence lines each. This could > exceed the default 10,000 input phrase cache. Would it be better to bump > up the -persistent-cache-size value, or manage the number of phrase I > pass to the input? > * "-use-persistent-cache: cache translation > options across sentences (default true)" : Regarding cashing across > sentences (which presumably apples to -use-persistent-cache), the manual > says, "you should also make sure that the cache is frequently cleared." > How do I clear the cache? Does this require forcing moses itself to > unload, and then reload it? Also, the -use-persistent-cache value > defaults to "true". What is the effect of changing this to "false"? Does > it effectively disable this cache and eliminate the requirement to clear > it? > > Thanks, > Tom -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
