Hi, the strange thing is that I don't see the leak when I run moses in valgrind. Valgrind makes the process run much slower, which might explain the problem.
What I see in valgrind is a stable usage of 7.5G of memory up until sentence 130 (where I stopped the process, and valgrind reported a memory leak of only 256 MB). This took about 3 hours to run. When running the process without valgrind, I reach 11GB on sentence 70 after about 5-10 minutes. Best, Evgeny. P.S. The generally high memory usage is dominated by 5.9 GB large IRST LM. On Monday 16 November 2009 13:13:47 Hieu Hoang wrote: > Hi Evgeny > > I've tried to reproduce the mem leak with distinct n-best, I don't see a > problem. > > are you able to run valgrind and see where the leak is coming from? > > Evgeny Matusov wrote: > > Hi, > > > > I tried to run Moses to produce N-best lists with distinct hypotheses > > (setting -nbest-list <file> 500 distinct). Although I also > > set "-use-persistent-cache false", the memory usage continues to grow > > from sentence to sentence during translation. The memory consumption > > drops only 2-3 times in the first 100 sentences as opposed to after every > > sentence. After 100 to 150 sentences, the program dies with bad_alloc. > > > > Do you know what causes this problem? > > > > I run this on a task with about 700K sentence pairs (GALE-type data), > > setting stack size to 100, stack diversity to 10 and > > max-trans-opt-per-coverage to 15, and had no problems running translation > > with single-best output only. > > > > > > Thanks much, > > Evgeny. > > > > P.S. I just found that this issue had been discussed and presumably fixed > > sometime ago; however, I still have this problem with the latest svn > > commit of Moses. -- =================================== Dipl.-Inform. Evgeny Matusov Senior Speech and Language Engineer Applications Technology, Inc. E-mail: [email protected] Tel. +49-241-939-19815 Fax. +49-241-939-19816 =================================== _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
