Hi Marcin If you give Moses a file to translate, then it will read in the whole file in one thread, whilst translation begins in another thread. I guess this is a bug if you have very large files.
cheers - Barry Quoting Marcin Junczys-Dowmunt <[email protected]> on Tue, 14 Jan 2014 23:27:58 +0100: > Hi, > I think I have noticed some weird behaviour. If a huge input file > (millions of sentences) is processed by moses on stdin, memory usage > keeps growing and growing. At the beginning usage is about 2GB, after > roughly 20000 sentences it reaches around 8GB and it keeps growing. > > However, if I give moses only those 20000 sentences at stdin (cut off > with head -n 20000) memory usage stays below 3 GB. it seems the input is > somehow being buffered before translation? > > I came across this by retranslating the training corpus, looks buggy. > > Best, > Marcin > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
