Hi, if you are using experiment.perl, then the config parameter for phrase length is
max-phrase-length = 5 Similarly, if you call train-model.perl directly, then the switch is $MOSES/scripts/training/train-model.perl -max-phrase-length 5 [...] You can check the size of your model by looking at the size of the *minphr* *minlexr* and *binlm* files. These are binary data files that are loaded into RAM (see files referenced in moses.ini). Typically, the language model is the largest, because typically it is trained with large amounts of additional monolingual data. -phi On Fri, Dec 4, 2015 at 12:22 AM, Hegde, Sujay <[email protected]> wrote: > Hi Phillip, > > How do we limit phrase length during training .Is there a config > parameter in moses training config file? > > Is the phrase table the biggest model or the language model? ----> > We have 6-7 phrase tables that are combined in a log-linear fashion during > decoding. > > > > > Thanks and Regards, > Sujay, > Xerox Business Services, Bangalore, India > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Philipp Koehn > Sent: 03 December 2015 21:52 > To: Hegde, Sujay > Cc: [email protected]; MudaliarMudaliar, Preeti J > Subject: Re: [Moses-support] System requiremnts for Moses > > Hi, > > having such long sentences should cause all kinds of problems with word > alignment, so I am bit puzzled that they still show up when pruning the > phrase table. > > A good way to prune the phrase table is to limit the length of phrases (max 5 > does no harm, even max 4 is not a big deal), and reduce low probability > phrase pairs ($MOSES/scripts/training/threshold-filter.perl). > > Is the phrase table the biggest model or the language model? For the latter, > there are several compression options. > > -phi > > On Thu, Dec 3, 2015 at 12:32 AM, Hegde, Sujay <[email protected]> wrote: >> HI Philipp, >> >> >> >> Thanks a lot. >> >> >> >> Actually it’s a VIRTUAL machine. >> >> >> >> Also we have compressed the models into .minphr and >> .minlexr but we couldn’t prune it as while pruning we got an error >> saying some of the sentences in the Corpus are too long and it cannot be >> pruned. >> >> >> >> We used pruning using SALM and get the following error: >> >> >> >> /mnt/hd1/git/salm/Bin/Linux/Index/IndexSA.O64 >> opensub.train.it >> >> Initialize vocabulary file: opensub.train.it.id_voc >> >> Loading existing vocabulary file: opensub.train.it.id_voc >> >> Total 100 word types loaded >> >> Max VocID=100 >> >> Sentence 4152148 has more than 256 words. Can not handle such long sentence. >> Please cut it short first! >> >> >> >> Is there anything we could do about the above? >> >> >> >> >> >> >> >> Thanks and Regards, >> >> Sujay, >> >> Xerox Business Services, Bangalore, India >> >> >> >> From: [email protected] [mailto:[email protected]] On Behalf Of >> Philipp Koehn >> Sent: 03 December 2015 03:13 >> To: Hegde, Sujay >> Cc: [email protected] >> Subject: Re: [Moses-support] System requiremnts for Moses >> >> >> >> Hi, >> >> >> >> the machine you have is certainly sufficient even for large models. >> >> >> >> If you are running two language pairs in parallel and run into RAM >> problems, you may want to look into ways to compress the model files >> (phrase table, reordering table, language model) using either more >> efficient data structures (e.g., various KENLM options), or pruning the >> models. >> >> >> >> -phi >> >> >> >> >> >> On Tue, Dec 1, 2015 at 5:08 AM, Hegde, Sujay <[email protected]> wrote: >> >> Dear Moses Admin, >> >> >> >> We are using Moses decoder for commercial environment. >> >> >> >> We have 132GB RAM, 1TB disk and quadcore Virtual >> Machine with CentOs OS. >> >> >> >> We have 2 language pairs installed, and when running >> both the models together the Translation hangs(Takes a LONG time). >> >> It is fine when we run only one language model. >> >> >> >> Is there any Specific System requirements needed for moses? >> >> Please let me know >> >> >> >> Thanks and Regards, >> >> Sujay, >> >> Xerox Business Services, Bangalore, India >> >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
