you are again correct, binary LM is not supported under windows. it is usually handled by IRSTLM, which doesn't compile with VS05. u can extend IRSTLM to support windows, or write your own LM. all parts of the decoder is cross-platform, including the binary phrase table. however, we don't have control over external libraries like IRSTLM & SRILM. The training toolkit is unix-based because it is mostly legacy/external code. Hieu Hoang www.hoang.co.uk/hieu
-----Original Message----- From: Jie Wu [mailto:[EMAIL PROTECTED] Sent: 10 December 2007 16:33 To: Hieu Hoang Cc: [email protected] Subject: Re: [Moses-support] Language Model in Moses under VS 05 Hi, Hieu, Thanks for the reply. But I found out that if Moses of Windows version only supports the internal language model, that implies that loading language models in binary format is not supported. I am kind of confused, loading binary Language Models is handled by functions in which toolkit, SRILM, IRSTLM, or Moses. If I would like to have a Windows version of Moses that can load binary language models which could handle more than 3grams, what suggestion would you give when I make modifications? How about phrase tables? Does it have the same problems in terms of being loaded binary, windows vs. unix restrictions, etc? Thanks in advance Jie On Dec 6, 2007 6:19 AM, Hieu Hoang < [EMAIL PROTECTED]> wrote: hi jie, 1. you're correct. the only language model avaiable using vs05 is the internal LM, which only supports up to 3-gram. u're welcome to change the code to support more. IRSTLM is not aviable for VS05, and i'm not happy with the windows version of SRILM because it's difficult to compile & doesn't support some things we need, like compressed files etc. 2. the generation step is described in http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/emnlp2007-factored.pdf <http://www.iccs.inf.ed.ac.uk/%7Epkoehn/publications/emnlp2007-factored. pdf> the model can be created using the moses training scripts, described in http://www.statmt.org/moses/?n=FactoredTraining.HomePage Hieu Hoang www.hoang.co.uk/hieu -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jie Wu Sent: 05 December 2007 16:42 To: [email protected] Subject: [Moses-support] Language Model in Moses under VS 05 Hi, I have two questions: 1. I am studying Moses and found out that in VS05, Moses uses the internal language model rather than SRILM. And it turns out that the internal LM can only handle up to 3-grams. Does it mean in order to process n-grams with n>3, I have to use the SRILM or IRSTLM? BTW, what is Joint & Skip LM Model? 2. What is a generation table? As far as I know, the hypothesis is associated with a probability score multiplied with the translation, distortion and language model cost. The hypothesis with a highest probability score is the best possible translation. Where does a generation table come into play? and How to generate a generation table? Thanks in advance Jie -- ================================= Jie Wu Homepage: http://www.jiewu.info -- ================================= Jie Wu Homepage: http://www.jiewu.info
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
