Hi Bogdan Why do you set the maximum phrase length to 20? Such long phrases are unlikely to be useful, and could be the cause of the excessive resource usage.
Other than that, the system you describe should not be using up 192G ram. cheers - Barry On 01/08/16 20:40, Bogdan Vasilescu wrote: > Thanks Hieu, > > It runs out of memory around 3,000 sentences when n-best is the > default 100. It seems to do a little bit better if I set n-best to 10 > (5,000 sentences or so). The machine I'm running this on has 192 GB > RAM. I'm using the binary moses from > http://www.statmt.org/moses/RELEASE-3.0/binaries/linux-64bit/ > > My phrase table was built on 1,200,000 sentences (phrase length at > most 20). My language model is a 5-gram, built on close to 500,000,000 > sentences. > > Still, the question remains. Is there a way to perform tuning incrementally? > > I'm thinking: > - tune on a sample of my original tuning corpora; this generates an > updated moses.ini, with "better" weights > - use this moses.ini as input for a second tuning phase, on another > sample of my tuning corpora > - repeat until there is convergence in the weights > > Bogdan > > > On Mon, Aug 1, 2016 at 11:43 AM, Hieu Hoang <[email protected]> wrote: >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 29 July 2016 at 18:57, Bogdan Vasilescu <[email protected]> wrote: >>> Hi, >>> >>> I've trained a model and I'm trying to tune it using mert-moses.pl. >>> >>> I tried different size tuning corpora, and as soon as I exceed a >>> certain size (this seems to vary between consecutive runs, as well as >>> with other tuning parameters like --nbest), the process gets killed: >> it should work with any size tuning corpora. The only thin I can think of is >> if the tuning corpora is very large (1,000,000 sentences say) or the n-best >> list is very large (1,000,000 say) then the decoder or the mert script may >> use a lot of memory >>> >>> Killed >>> Exit code: 137 >>> The decoder died. CONFIG WAS -weight-overwrite ... >>> >>> Looking into the kernel logs in /var/log/kern.log suggests I'm running >>> out of memory: >>> >>> kernel: [98464.080899] Out of memory: Kill process 15848 (moses) score >>> 992 or sacrifice child >>> kernel: [98464.080920] Killed process 15848 (moses) >>> total-vm:414130312kB, anon-rss:194915316kB, file-rss:0kB >>> >>> Is there a way to perform tuning incrementally? >>> >>> I'm thinking: >>> - tune on a sample of my original tuning corpora; this generates an >>> updated moses.ini, with "better" weights >>> - use this moses.ini as input for a second tuning phase, on another >>> sample of my tuning corpora >>> - repeat until there is convergence in the weights >>> >>> Would this work? >>> >>> Many thanks in advance, >>> Bogdan >>> >>> -- >>> Bogdan (博格丹) Vasilescu >>> Postdoctoral Researcher >>> Davis Eclectic Computational Analytics Lab >>> University of California, Davis >>> http://bvasiles.github.io >>> http://decallab.cs.ucdavis.edu/ >>> @b_vasilescu >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
