Hi,
Read the comments in lm/model.hh and lm/left.hh. Short example:
#include "lm/model.hh"
#include <iostream>
#include <string>
int main() {
using namespace lm::ngram;
Model model("file.arpa");
State state(model.BeginSentenceState()), out_state;
const Vocabulary &vocab = model.GetVocabulary();
std::string word;
while (std::cin >> word) {
std::cout << model.Score(state, vocab.Index(word), out_state) << '\n';
state = out_state;
}
}
Kenneth
On 01/29/2016 04:10 PM, Jasneet Sabharwal wrote:
> Hi Ken,
>
> I have a language model trained on word classes. So, it is trained on
> sentences like “10 15 21 1 23”. I now have a feature function that
> generates a phrase “1 7 10 20”. I’ve been able to load my language model
> in my feature function. How can I get the score from my language model
> for this phrase?
>
> Regards,
> Jasneet
>> On Jan 23, 2016, at 7:11 AM, Jasneet Sabharwal
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>> Thanks Ken & Hieu,
>>
>> I’ll give KenLM a try. The reason for using Witten-Bell was because
>> Kneser-Ney wasn’t able to cope up with the counts being generated for
>> coarse language models. Sp, I’ll train my LM using SRILM with ngram
>> order 8 and WB smoothing and use KenLM with order 8 in Moses.
>>
>> Best,
>> Jasneet
>>> On Jan 23, 2016, at 3:38 AM, Kenneth Heafield <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hi,
>>>
>>> You can compile with --max-kenlm-order=8 or change the setting in the
>>> Eclipse files.
>>>
>>> The ARPA file format is interchangeable. You can build an ARPA using
>>> SRILM and Witten-Bell (though Bob Moore once called me out at a
>>> conference for suggesting that as an alternative to Kneser-Ney) then
>>> load with KenLM.
>>>
>>> Kenneth
>>>
>>> On 01/23/2016 05:39 AM, Jasneet Sabharwal wrote:
>>>> Thanks Hieu.
>>>>
>>>> I’m using the eclipse project for development. I followed your video to
>>>> set it up and I have linked the srilm and irstlm installations in the
>>>> root directory of mosesdecoder. I first tried to compile the project,
>>>> but neither the SRILM nor the IRSTLM LM cpp files get compiled. So, I
>>>> added LM_IRST and included "${workspace_loc}/../../irstlm/include” path
>>>> in the C/C++ Build settings of the project. But I still cannot compile
>>>> IRST.cpp.
>>>>
>>>> The reason I’m not using the included KenLM is because my new feature
>>>> function requires an 8-gram language model with witten bell smoothing,
>>>> which is provided by SRILM. As, IRSTLM can use SRILM generated language
>>>> models, so I decided to call IRSTLM code inside my feature function to
>>>> get the score for a phrase.
>>>>
>>>> Any pointers on how can I debug the eclipse project with IRSTLM/SRILM?
>>>>
>>>> Best,
>>>> Jasneet
>>>>
>>>> PS: When I compile the whole project using "./bjam -j4
>>>> —with-boost=<absolute path to boost> —with-cmph=<absolute path to cmph>
>>>> —with-irstlm=<absolute path to irstlm>”, it successfully compiles
>>>> without any errors.
>>>>
>>>>
>>>>> On Jan 19, 2016, at 4:39 PM, Hieu Hoang <[email protected]
>>>>> <mailto:[email protected]>
>>>>> <mailto:[email protected]>> wrote:
>>>>>
>>>>> I believe Nadir Durrani's OSM uses KenLM inside it. You can look in
>>>>> moses/FF/OSM-Feature
>>>>> for tips
>>>>>
>>>>> On 20/01/16 00:31, Jasneet Sabharwal wrote:
>>>>>> Thanks Hieu.
>>>>>>
>>>>>> One last question. What do you think is the best way to load the
>>>>>> SRILM language model inside my custom feature function and to get a
>>>>>> score for a string that my feature function created?
>>>>>>
>>>>>> Best,beli
>>>>>> Jasneet
>>>>>>> On Jan 17, 2016, at 3:45 AM, Hieu Hoang
>>>>>>> <<mailto:[email protected]>[email protected]
>>>>>>> <mailto:[email protected]>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17/01/16 04:05, Jasneet Sabharwal wrote:
>>>>>>>> Thanks Hieu,
>>>>>>>>
>>>>>>>> I had subscribed to the mailing list and I’m getting the digest,
>>>>>>>> but not sure why my email went for your approval. When I get the
>>>>>>>> alignments from GetAlignTerm(), the index of the source word is
>>>>>>>> relative? To get the index in the source sentence, I’m assuming
>>>>>>>> that I would need to get the starting position of the source words
>>>>>>>> from CurrSourceWordsRange().GetStartPos() from current hypothesis
>>>>>>>> and offset the source alignment index with that value?
>>>>>>> yep. And to get the index in the target sentence, use
>>>>>>> GetCurrTargetWordsRange().GetStartPos()
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Jasneet
>>>>>>>>> On Jan 15, 2016, at 3:43 AM, Hieu Hoang <[email protected]
>>>>>>>>> <mailto:[email protected]>> wrote:
>>>>>>>>>
>>>>>>>>> please subscribe to the Moses mailing list before posting to it.
>>>>>>>>> You can subscribe here:
>>>>>>>>> http://mailman.mit.edu/mailman/admin/moses-support
>>>>>>>>> To answer you question - the target phrase has a method called
>>>>>>>>> GetAlignTerm()
>>>>>>>>> that contains the alignment for terminals. This comes from the
>>>>>>>>> phrase-table, and ultimately from the word alignment.
>>>>>>>>>
>>>>>>>>> -------- Forwarded Message --------
>>>>>>>>> Subject:Moses-support post from [email protected]
>>>>>>>>> <mailto:[email protected]> requires
>>>>>>>>> approval
>>>>>>>>> Date:Wed, 13 Jan 2016 23:36:50 -0500
>>>>>>>>> From:[email protected]
>>>>>>>>> <mailto:[email protected]>
>>>>>>>>> To:[email protected] <mailto:[email protected]>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As list administrator, your authorization is requested for the
>>>>>>>>> following mailing list posting:
>>>>>>>>>
>>>>>>>>> List: [email protected]
>>>>>>>>> From: [email protected]
>>>>>>>>> Subject: Getting alignments for current hypothesis in phrase
>>>>>>>>> based model
>>>>>>>>> Reason: Post by non-member to a members-only list
>>>>>>>>>
>>>>>>>>> At your convenience, visit:
>>>>>>>>>
>>>>>>>>> http://mailman.mit.edu/mailman/admindb/moses-support
>>>>>>>>>
>>>>>>>>> to approve or deny the request.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <ForwardedMessage.eml><ForwardedMessage.eml>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Moses-support mailing list
>>>>>>>> [email protected] <mailto:[email protected]>
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>>> --
>>>>>>> Hieu Hoang
>>>>>>> http://www.hoang.co.uk/hieu
>>>>>>
>>>>>
>>>>> --
>>>>> Hieu Hoang
>>>>> http://www.hoang.co.uk/hieu
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> [email protected] <mailto:[email protected]>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> [email protected] <mailto:[email protected]>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing list
>> [email protected] <mailto:[email protected]>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
