Thanks Ken & Hieu, I’ll give KenLM a try. The reason for using Witten-Bell was because Kneser-Ney wasn’t able to cope up with the counts being generated for coarse language models. Sp, I’ll train my LM using SRILM with ngram order 8 and WB smoothing and use KenLM with order 8 in Moses.
Best, Jasneet > On Jan 23, 2016, at 3:38 AM, Kenneth Heafield <[email protected]> wrote: > > Hi, > > You can compile with --max-kenlm-order=8 or change the setting in the > Eclipse files. > > The ARPA file format is interchangeable. You can build an ARPA using > SRILM and Witten-Bell (though Bob Moore once called me out at a > conference for suggesting that as an alternative to Kneser-Ney) then > load with KenLM. > > Kenneth > > On 01/23/2016 05:39 AM, Jasneet Sabharwal wrote: >> Thanks Hieu. >> >> I’m using the eclipse project for development. I followed your video to >> set it up and I have linked the srilm and irstlm installations in the >> root directory of mosesdecoder. I first tried to compile the project, >> but neither the SRILM nor the IRSTLM LM cpp files get compiled. So, I >> added LM_IRST and included "${workspace_loc}/../../irstlm/include” path >> in the C/C++ Build settings of the project. But I still cannot compile >> IRST.cpp. >> >> The reason I’m not using the included KenLM is because my new feature >> function requires an 8-gram language model with witten bell smoothing, >> which is provided by SRILM. As, IRSTLM can use SRILM generated language >> models, so I decided to call IRSTLM code inside my feature function to >> get the score for a phrase. >> >> Any pointers on how can I debug the eclipse project with IRSTLM/SRILM? >> >> Best, >> Jasneet >> >> PS: When I compile the whole project using "./bjam -j4 >> —with-boost=<absolute path to boost> —with-cmph=<absolute path to cmph> >> —with-irstlm=<absolute path to irstlm>”, it successfully compiles >> without any errors. >> >> >>> On Jan 19, 2016, at 4:39 PM, Hieu Hoang <[email protected] >>> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>> >>> I believe Nadir Durrani's OSM uses KenLM inside it. You can look in >>> moses/FF/OSM-Feature >>> for tips >>> >>> On 20/01/16 00:31, Jasneet Sabharwal wrote: >>>> Thanks Hieu. >>>> >>>> One last question. What do you think is the best way to load the >>>> SRILM language model inside my custom feature function and to get a >>>> score for a string that my feature function created? >>>> >>>> Best,beli >>>> Jasneet >>>>> On Jan 17, 2016, at 3:45 AM, Hieu Hoang >>>>> <<mailto:[email protected] >>>>> <mailto:[email protected]>>[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> >>>>> >>>>> On 17/01/16 04:05, Jasneet Sabharwal wrote: >>>>>> Thanks Hieu, >>>>>> >>>>>> I had subscribed to the mailing list and I’m getting the digest, >>>>>> but not sure why my email went for your approval. When I get the >>>>>> alignments from GetAlignTerm(), the index of the source word is >>>>>> relative? To get the index in the source sentence, I’m assuming >>>>>> that I would need to get the starting position of the source words >>>>>> from CurrSourceWordsRange().GetStartPos() from current hypothesis >>>>>> and offset the source alignment index with that value? >>>>> yep. And to get the index in the target sentence, use >>>>> GetCurrTargetWordsRange().GetStartPos() >>>>>> >>>>>> Regards, >>>>>> Jasneet >>>>>>> On Jan 15, 2016, at 3:43 AM, Hieu Hoang <[email protected]> wrote: >>>>>>> >>>>>>> please subscribe to the Moses mailing list before posting to it. >>>>>>> You can subscribe here: >>>>>>> http://mailman.mit.edu/mailman/admin/moses-support >>>>>>> To answer you question - the target phrase has a method called >>>>>>> GetAlignTerm() >>>>>>> that contains the alignment for terminals. This comes from the >>>>>>> phrase-table, and ultimately from the word alignment. >>>>>>> >>>>>>> -------- Forwarded Message -------- >>>>>>> Subject: Moses-support post from [email protected] >>>>>>> requires >>>>>>> approval >>>>>>> Date: Wed, 13 Jan 2016 23:36:50 -0500 >>>>>>> From: [email protected] >>>>>>> To: [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> As list administrator, your authorization is requested for the >>>>>>> following mailing list posting: >>>>>>> >>>>>>> List: [email protected] >>>>>>> From: [email protected] >>>>>>> Subject: Getting alignments for current hypothesis in phrase based >>>>>>> model >>>>>>> Reason: Post by non-member to a members-only list >>>>>>> >>>>>>> At your convenience, visit: >>>>>>> >>>>>>> http://mailman.mit.edu/mailman/admindb/moses-support >>>>>>> >>>>>>> to approve or deny the request. >>>>>>> >>>>>>> >>>>>>> >>>>>>> <ForwardedMessage.eml><ForwardedMessage.eml> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Moses-support mailing list >>>>>> [email protected] >>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>> >>>>> -- >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>> >>> >>> -- >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] <mailto:[email protected]> >> http://mailman.mit.edu/mailman/listinfo/moses-support >> <http://mailman.mit.edu/mailman/listinfo/moses-support> >> > _______________________________________________ > Moses-support mailing list > [email protected] <mailto:[email protected]> > http://mailman.mit.edu/mailman/listinfo/moses-support > <http://mailman.mit.edu/mailman/listinfo/moses-support>
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
