Hello, all
I think I've got it sorted out finally, so I'll give here so solution
for later reference, and in case someone would run across the same problem.
Context (you may skip this):
============================
I'm implementing a speech translation software based on Moses and
PocketSphinx. It has a confidence estimation (CE) module implementing
methods from the literature. Some of them are n-gram based. For the sake
of efficiency I want to use the decoder's LM object. Kenneth pointed out
that since LMs are mmaped, I could just reload them with no impact on
efficiency. Well, that's right, I have to investigate that even though
it wouldn't solve everything :)
Anyway, the CE module was implemented last summer and broken by changes
in the API. I wanted to fix it.
What I need:
============
* Get n-gram probability of every individual word in hypothesis (not
just the sentence n-gram likelihood)
* Get n-gram match length
It used to be possible using LMImplementation which doesn't seem to be
accessible anymore.
Proposed solution:
==================
I get a LanguageModel* from the decoder with the following code:
TranslationSystem ts =
StaticData::Instance().GetTranslationSystem(std::string("default"));
LMList list = ts.GetLanguageModels();
LanguageModel *lm = *(list.begin());
unfortunately, the methods I need are implemented in LMImplementation. I
used to get with lm->GetLMImplementation but this is no longer possible.
Kenneth mentionned that:
"LanguageModelKen has boost::shared_ptr<lm::ngram::Model> m_ngram
provides similar functionality with a different interface"
lm::ngram::Model::FullScoreReturn(const void *in_state, const WordIndex
new_word, void *out_state)
indeed provides n-gram probability and n-gram match length.
Now I needed to get my hands on m_ngram. Kenneth suggested the following
changes:
"Add a method
virtual const lm::base::Model *GetKen() const { return NULL; }
to LanguageModel in LM/Base.h . Then, edit Ken.cpp, overriding the method
const lm::base::Model *GetKen() const { return m_ngram.get(); }
"
which I did. I also had to add #include "lm/virtual_interface.hh" to
Base.h. I can send you the patches if you want. I must say, in order to
be able to develop and distribute a program using Moses API, I think
it's necessary to add some code to Moses so that it can be done without
patching Moses.
Back to my own code. I wrote a wrapper around KenLM methods to create an
appropriate state containing my context and call Score or FullScoreReturn:
float moses_score_unigram(lmi_data_t data, wid_t a) {
const lm::base::Model* model = ((LanguageModel*)data)->GetKen();
lm::ngram::State * inState = new lm::ngram::State();
inState->length = 0;
lm::ngram::State * outState = new lm::ngram::State();
float score = model->Score((const void*)&inState, a.m_wid,
(void*)&outState);
delete inState;
delete outState;
return exp(score);
}
there are some wild guesses in it, for example the exact type of the
"states" required by Score and FullScoreReturn, and I need to write many
more wrappers before I can compile and test it, so if you think it
cannot work, just tell me now :) I'll report back when it's done.
cheers, and thanks everybody for your help,
Sylvain
On 06/02/12 13:20, Sylvain Raybaud wrote:
> Hello
>
> I'm trying to update a software that makes use of Moses API. It uses (among
> others) Moses' LanguageModel and LanguageModelImplementation classes. It used
> to call method GetLMImplementation of an object of type LanguageModel, in
> order to use methods NewState and GetValueForgotState of class
> LanguageModelImplementation. That was in august. Unfortunatelly, now I've
> pulled a new version of Moses from git, and method GetLMImplementation seems
> to have vanished. Methods NewState and GetValueForgotState seems to still be
> implemented in class LanguageModelImplementation. Now I can get the LM using:
>
> TranslationSystem ts =
> StaticData::Instance().GetTranslationSystem(std::string("default"));
> LMList list = ts.GetLanguageModels();
> LanguageModel *lm = *(list.begin());
>
> But I could not find a way to get the LanguageModelImplementation object in
> order to call aforementioned method. Do you have an idea of what I should
> change in the code?
>
> By the way, documentation on http://www.statmt.org/moses/html/hierarchy.html
> seems to be obsolete, as it seems to be lacking many files, classes and
> functions.
>
> regards,
>
> Sylvain
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Sylvain Raybaud
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
