On Fri, Dec 19, 2008 at 17:15, Hongfei Jiang <[email protected]> wrote: > Hi, guys, > I heard from some guy that we do not need to modify the moses > decoder > for adding some new features to moses. > As I currently dig into the source code, I found some places we > should pay attentions to: > 1) phrase-extract/score.cpp specially, PhraseAlignment::create > and processPhrasePairs > 2) train-factored-phrase-model.perl as mentioned by Jason! > I'm trying these stuff now, not totally success yet. > Does my understanding right? Anything missing? > The key puzzle is that need we modify the Decoder code? > Can't the Moses decoder dynamically recognize how many features we > used when we > add the new features in the phrase table for each rule? > Best Regards, > -Hongfei Jiang
Hi Hongfei, I was talking about some general feature function that is some function of a hypothesis transition, not necessarily one that is based on a per-phrase feature. I don't know about the case of adding a new phrase table feature, but in general, you would definitely need to modify the decoder to add a new feature function. As it seemed to work for me, I'm fairly certain that my list is (was) exhaustive :-) Thanks, Jason > 2008/4/21 Jason Katz-Brown <[email protected]> >> >> Hi all, >> >> After snuffing out what I think is all of the steps required to add a >> new feature function to the Moses decoder and training system, I made >> a list of them. Perhaps it is helpful for others. Moses could >> certainly use more score producers to increase the count from anemic 5 >> :-) >> >> --Jason >> >> HOWTO: add a new feature function to Moses >> >> ==MOSES decoder libary (moses/src/)== >> * Write ScoreProducer subclass, say MyScoreProducer. For example, >> implement these methods: >> - MyScoreProducer(ScoreIndexManager &scoreIndexManager); >> + Calls scoreIndexManager.AddScoreProducer(this). >> - std::vector<float> CalcMyScore(Hypothesis* hypothesis) const; >> + Calculates the value of the feature function. >> - size_t GetNumScoreComponents() const; >> - std::string GetScoreProducerDescription() const; >> >> * Add MyScoreProducer private member m_myScoreProducer to StaticData >> and GetMyScoreProducer() method. >> * Add m_weightMyScore float private member to StaticData and >> GetWeightMyScore() method. >> * In Hypothesis::CalcScore(), calculate feature function score using >> MyScoreProducer::CalcMyScore() and incorporate it with >> m_scoreBreakdown.PlusEquals(). >> * In Parameter::Parameter(), AddParam("weight-m", "m", "weight for my >> score") where m is some letter. >> * In StaticData::LoadData(), load m_weightMyScore with >> m_parameter->GetParam(). >> * In StaticData::LoadData(), create MyScoreProducer, passing in >> m_scoreIndexManager, and add its weights to m_allWeights. These two >> steps must take place exactly right after each other. >> * In StaticData::~StaticData(), delete m_myScoreProducer. >> >> ==MOSES decoder command-line program (moses-cmd/src/)== >> * In moses-cmd/src/IOStream::OutputNBestList(), output "m: " followed >> by your weights, separated by spaces. >> >> ==MOSES training scripts (scripts/training/)== >> * In train-factored-phrase-model.perl, change create_ini() to output >> default weights for weight-m. >> * Add in mert-moses.pl something like >> "m" => [ [ 1.0, 0.0, 2.0 ] ], # my score >> to _one_ of $default_triples or $additional_triples. >> * Add in mert-moses.pl "m=weight-m" to $ABBR_FULL_MAP. >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
