Thanks Barry. The quote comes from http://www.statmt.org/moses/manual/manual.pdf, page 185, section 5.4.4, "Caching across Sentences" paragraph.
I understood Ivan's problem. My script is only remotely similar. I also frequently use the multi-threading/multiprocessor configuration. However, this new multi-threading module provides non-blocking service of stderr and stdout output on separate threads. It's possible for my queue to piggyback input from multiple text file sources into the stream. So, moses never experiences a break with an EOF. This could conceivably be seen by moses as one huge file with multi-ten's of thousands of lines. My machine is a 4 GB Core2Quad. The "top" utility reports my memory stabilizes to about 100KB of free RAM during runtime. Swap file usage remains stable at only 68KB used out of 6 GB. I'm using a binarized/memmapped KenLM, trained with SRILM, configured with code 9 and binarized/memmapped tables. I like your idea of testing the moses core without the Python wrapper. First, I have a work-around and must finish processing this batch. Then, I'll save about 200 tokenized files. If I processing with the same moses configuration using a Bash script that redirects the input as discrete files moses will see an EOF and reload between files. This is different than my configuration. So, I'll concatenate the files to one huge file and then process over 10,000 lines in one pass. It'll take me about a week to get to this. I'll let you know. By the way, I use the Dash redirection for evaluation. During Evaluation and mert the "top" utility show peak CPU efficiency at "only" 385% on a quad-core. Is this also your experience with multi-threading on multiprocessors? Tom On Mon, 29 Aug 2011 12:58:26 +0100, Barry Haddow <[email protected]> wrote: > Hi Tom > > If one of the moses caches was filling up, then I would expect that > the > process memory would increase, until the machine ground to a halt. > The > problem that Ivan had with the original version of his wrapper was > slightly > different, there was a fixed size i/o buffer that he wasn't emptying, > which > eventually deadlocked his process. > > The lm cache that you mentioned below is, as far as I'm aware, > specific to > irstlm, so if you're not using irstlm, then the flag should have no > effect. > > The translation option cache mainly helps by caching translation > options for > common phrases like 'the' or '.'. It is implemented as an LRU cache, > and the decoder removes an entry when the cache reaches maximum size. > I don't understand the quote from the manual about making sure this > cache > is frequently cleared - could you tell me where this quote comes > from? > Tuning the size of the translation options cache may help with > performance, > but it's unlikely to be the cause of the unexplained crashes. > > We frequently run multi-threaded decodes on mult-core machines and > haven't > witnessed any unexplained crashes. So I would quite like to eliminate > the > python/moses interaction as cause of error. Is it possible to run a > similar > experiment without the python wrapper, say by just passing moses your > source > sentences in a file? If it is moses that is crashing, then if you > could allow it to generate a core file and make it available, then > I'd have > some chance of debugging it, > > cheers - Barry > > > > On Monday 29 August 2011 12:23, Tom Hoar wrote: >> I've implemented a multi-threaded Python wrapper that loads moses >> decoder and pipes strings through the moses binary. It's similar to >> Ivan >> Uemlianin's code from May 04, 2010 on this listserv, but achieves a >> throughput efficiency 398% CPU load on a quad-core host across >> multiple >> documents processed in a queue. >> >> Here's the rub. The decoder & the >> wrapper run great for about 2 hours. Then they halt with an unknown >> error. It's difficult to trace because it takes hours to reproduce. >> I >> can see that the Moses binary doesn't generate an error exit code. >> There's no error message about a "broken" pipe. When I restart the >> script on the file that was in-process at the time of hault, it runs >> just fine and continues processing. Since the error occurs >> consistently >> at the 2 hour mark, and it's not the file causing the halt, I >> suspect at >> a cache or buffer somewhere is overloaded. I've checked my python >> code, >> and don't believe there are any buffer overruns there. >> >> I'm hoping >> someone can review my comments and give me some pointers about >> Moses' >> caches and how to verify manage the caches. The Moses manual >> describes >> three cache: >> >> * "-clean-lm-cache: clean language model caches after N >> translations (default N=1)" : If -clean-lm-cache defaults to >> cleaning >> the lm cache after each translation, I don't think this is a >> problem. >> >> * "-persistent-cache-size: maximum size of cache for translation >> options (default 10,000 input phrases)" : Some of my files have my >> files >> have 2,500 or more pages with 20-25 sentence lines each. This could >> exceed the default 10,000 input phrase cache. Would it be better to >> bump >> up the -persistent-cache-size value, or manage the number of phrase >> I >> pass to the input? >> * "-use-persistent-cache: cache translation >> options across sentences (default true)" : Regarding cashing across >> sentences (which presumably apples to -use-persistent-cache), the >> manual >> says, "you should also make sure that the cache is frequently >> cleared." >> How do I clear the cache? Does this require forcing moses itself to >> unload, and then reload it? Also, the -use-persistent-cache value >> defaults to "true". What is the effect of changing this to "false"? >> Does >> it effectively disable this cache and eliminate the requirement to >> clear >> it? >> >> Thanks, >> Tom _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
