Hi, I'm using IRSTML, so the EMS isn't an option, is it? http://www.statmt.org/moses/?n=Moses.Baseline : "The only wrinkle is that EMS currently (April 2012) does not support training with IRSTLM, so either you'll have to install and use SRILM (and refer to the main [EMS -> FactoredTraining.EMS]] documentation to see how to set up training) or see above for instructions on how to train the LM with IRSTLM (if you haven't already done so). "
I've counted to 2 million sentences in the corpus. My computer has 8 GB RAM and 4 cores. I'm happy I've reverted to building a phrase-based model, instead of a hierarchical one. Yours, Per Tunedal On Sat, Mar 9, 2013, at 15:53, Hieu Hoang wrote: > How much memory and how many cores does your computer have? How many > sentences are in your parallel corpus? > > The training shouldn't be overwhelming your computer, it should take the > same amount of memory & cpu whatever the data size or model. However, the > hierarchical model takes about 10 time longer to train than the > phrase-based, and the models are 10 times bigger. > > when tuning, imo, you need at least 4gb to run the hierarchical decoder, > but much more memory is better. With phrase-based, you can easily do it > with 2GB. > > if you're trying to create an online model, to run fast, you'll need a > computer that has enough memory as the size of the model. This could be > 100GB or 200GB or more for hiero models. > > Another way is to use suffix arrays, at the cost of slower overall > decoding > time. > > In summary, hiero models are a challenge to run. Phrase-based is easy. > > > > ps. if you're starting out with Moses, I recommend using the EMS. > http://www.statmt.org/moses/?n=FactoredTraining.EMS > you should take an example file and change it to your needs. The files > are > in the moses repository under > scripts/ems/example > The EMS saves you from have to learn all the arguments in every script. > However, it calls those scripts so you can see exactly how to call them > if > you want. > > > > On 9 March 2013 13:30, Per Tunedal <[email protected]> wrote: > > > > > Hi, > > the training seems to be an overwhelming task for my computer. If it > > ever succeeds, I will have to undertake the even more demanding task of > > tuning. Can anything be done to accelerate it? > > > > Specifically, I wonder if it's feasible to prune the translation table > > before doing the tuning. > > > > Yours, > > Per Tunedal > > > > PS I've abandoned the idea of building a Hierarchical phrase model, I'm > > now trying to make a phrase-based system. I suppose that would use less > > resources. > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > -- > Hieu Hoang > Research Associate > University of Edinburgh > http://www.hoang.co.uk _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
