Hi
Thanks for the feedback. I made the changes you suggested to avoid
heap allocation. However, I gave the example of unigram_score only
because it's the simplest one, but I mostly need 3 gram and 5 gram
scores, and unigram, bigram and 3-gram existence. For example:
float moses_score_bigram(lmi_data_t data, wid_t a, wid_t b) {
const lm::base::Model &model = *((LanguageModel*)data)->GetKen();
lm::ngram::State inState;
inState.length = 1; inState.words[0] = a.m_wid;
lm::ngram::State outState;
float score = model.Score(&inState, b.m_wid, &outState);
return pow(score,10.0);
}
Sylvain
On 14/02/12 17:19, Kenneth Heafield wrote:
> Hi,
>
> You could do this, avoiding heap allocation:
>
> float moses_score_unigram(lmi_data_t data, wid_t a) {
> const lm::base::Model&model = *((LanguageModel*)data)->GetKen();
> lm::ngram::State ignored;
> float score = model.Score(model.NullContextMemory(), a.m_wid,&ignored);
> // Probabilities are in log10 space
> return pow(score, 10.0);
> }
>
> Also, if you're only interested in unigram scores, I strongly suggest
> separate unigram language model because Kneser-Ney, modified or
> unmodified, assumes that you've backed off to a unigram and that
> assumption doesn't hold in this case.
>
> Kenneth
>
> On 02/14/2012 10:49 AM, Sylvain Raybaud wrote:
>> float moses_score_unigram(lmi_data_t data, wid_t a) {
>> const lm::base::Model* model = ((LanguageModel*)data)->GetKen();
>> lm::ngram::State * inState = new lm::ngram::State();
>> inState->length = 0;
>> lm::ngram::State * outState = new lm::ngram::State();
>>
>> float score = model->Score((const void*)&inState, a.m_wid,
>> (void*)&outState);
>>
>> delete inState;
>> delete outState;
>>
>> return exp(score);
>> }
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Sylvain Raybaud
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
