The EMS has been updated since then to be able to train using IRSTLM. I'll change the webpage now
When using the EMS, set lm-training to lm-training = "$moses-script-dir/generic/trainlm-irst2.perl -cores $cores -irst-dir $irstlm-dir -temp-dir $working-dir/tmp" On 9 March 2013 15:11, Per Tunedal <[email protected]> wrote: > Hi, > I'm using IRSTML, so the EMS isn't an option, is it? > http://www.statmt.org/moses/?n=Moses.Baseline : > "The only wrinkle is that EMS currently (April 2012) does not support > training with IRSTLM, so either you'll have to install and use SRILM > (and refer to the main [EMS -> FactoredTraining.EMS]] documentation to > see how to set up training) or see above for instructions on how to > train the LM with IRSTLM (if you haven't already done so). " > > I've counted to 2 million sentences in the corpus. My computer has 8 GB > RAM and 4 cores. I'm happy I've reverted to building a phrase-based > model, instead of a hierarchical one. > > Yours, > Per Tunedal > > On Sat, Mar 9, 2013, at 15:53, Hieu Hoang wrote: > > How much memory and how many cores does your computer have? How many > > sentences are in your parallel corpus? > > > > The training shouldn't be overwhelming your computer, it should take the > > same amount of memory & cpu whatever the data size or model. However, the > > hierarchical model takes about 10 time longer to train than the > > phrase-based, and the models are 10 times bigger. > > > > when tuning, imo, you need at least 4gb to run the hierarchical decoder, > > but much more memory is better. With phrase-based, you can easily do it > > with 2GB. > > > > if you're trying to create an online model, to run fast, you'll need a > > computer that has enough memory as the size of the model. This could be > > 100GB or 200GB or more for hiero models. > > > > Another way is to use suffix arrays, at the cost of slower overall > > decoding > > time. > > > > In summary, hiero models are a challenge to run. Phrase-based is easy. > > > > > > > > ps. if you're starting out with Moses, I recommend using the EMS. > > http://www.statmt.org/moses/?n=FactoredTraining.EMS > > you should take an example file and change it to your needs. The files > > are > > in the moses repository under > > scripts/ems/example > > The EMS saves you from have to learn all the arguments in every script. > > However, it calls those scripts so you can see exactly how to call them > > if > > you want. > > > > > > > > On 9 March 2013 13:30, Per Tunedal <[email protected]> wrote: > > > > > > > > Hi, > > > the training seems to be an overwhelming task for my computer. If it > > > ever succeeds, I will have to undertake the even more demanding task of > > > tuning. Can anything be done to accelerate it? > > > > > > Specifically, I wonder if it's feasible to prune the translation table > > > before doing the tuning. > > > > > > Yours, > > > Per Tunedal > > > > > > PS I've abandoned the idea of building a Hierarchical phrase model, I'm > > > now trying to make a phrase-based system. I suppose that would use less > > > resources. > > > > > > _______________________________________________ > > > Moses-support mailing list > > > [email protected] > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > > > > > -- > > Hieu Hoang > > Research Associate > > University of Edinburgh > > http://www.hoang.co.uk > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
