Hi, if your new features are restricted to different scoring of phrase translations (e.g. dropped word detection, etc.), then yes, there is no need to the decoder, and it suffices to augment the phrase translation table.
-phi On Fri, Dec 19, 2008 at 8:15 AM, Hongfei Jiang <[email protected]> wrote: > Hi, guys, > I heard from some guy that we do not need to modify the moses > decoder > for adding some new features to moses. > As I currently dig into the source code, I found some places we > should pay attentions to: > 1) phrase-extract/score.cpp specially, PhraseAlignment::create > and processPhrasePairs > 2) train-factored-phrase-model.perl as mentioned by Jason! > I'm trying these stuff now, not totally success yet. > Does my understanding right? Anything missing? > The key puzzle is that need we modify the Decoder code? > Can't the Moses decoder dynamically recognize how many features we > used when we > add the new features in the phrase table for each rule? > Best Regards, > -Hongfei Jiang > > 2008/4/21 Jason Katz-Brown <[email protected]> >> >> Hi all, >> >> After snuffing out what I think is all of the steps required to add a >> new feature function to the Moses decoder and training system, I made >> a list of them. Perhaps it is helpful for others. Moses could >> certainly use more score producers to increase the count from anemic 5 >> :-) >> >> --Jason >> >> HOWTO: add a new feature function to Moses >> >> ==MOSES decoder libary (moses/src/)== >> * Write ScoreProducer subclass, say MyScoreProducer. For example, >> implement these methods: >> - MyScoreProducer(ScoreIndexManager &scoreIndexManager); >> + Calls scoreIndexManager.AddScoreProducer(this). >> - std::vector<float> CalcMyScore(Hypothesis* hypothesis) const; >> + Calculates the value of the feature function. >> - size_t GetNumScoreComponents() const; >> - std::string GetScoreProducerDescription() const; >> >> * Add MyScoreProducer private member m_myScoreProducer to StaticData >> and GetMyScoreProducer() method. >> * Add m_weightMyScore float private member to StaticData and >> GetWeightMyScore() method. >> * In Hypothesis::CalcScore(), calculate feature function score using >> MyScoreProducer::CalcMyScore() and incorporate it with >> m_scoreBreakdown.PlusEquals(). >> * In Parameter::Parameter(), AddParam("weight-m", "m", "weight for my >> score") where m is some letter. >> * In StaticData::LoadData(), load m_weightMyScore with >> m_parameter->GetParam(). >> * In StaticData::LoadData(), create MyScoreProducer, passing in >> m_scoreIndexManager, and add its weights to m_allWeights. These two >> steps must take place exactly right after each other. >> * In StaticData::~StaticData(), delete m_myScoreProducer. >> >> ==MOSES decoder command-line program (moses-cmd/src/)== >> * In moses-cmd/src/IOStream::OutputNBestList(), output "m: " followed >> by your weights, separated by spaces. >> >> ==MOSES training scripts (scripts/training/)== >> * In train-factored-phrase-model.perl, change create_ini() to output >> default weights for weight-m. >> * Add in mert-moses.pl something like >> "m" => [ [ 1.0, 0.0, 2.0 ] ], # my score >> to _one_ of $default_triples or $additional_triples. >> * Add in mert-moses.pl "m=weight-m" to $ABBR_FULL_MAP. >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
