Hi christian I can see nothing wrong with your suggestion so if u give me the change, I'll commit it to svn.
However, the decoder should be flooring the log score to -100, so something must be slipping thru the net. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christian Hardmeier Sent: 14 April 2008 12:25 To: [email protected] Subject: [Moses-support] Rounding defeats smoothing of lexical reorderingtables Hello, I've just ran across a problem with the estimation of lexical reordering tables. In train-factored-phrase-model.perl, the reordering probabilities are written out using a printf %.5f format specifier. Given a large enough corpus containing sufficiently small quantities of a certain phrase in a certain reordering condition, the probability estimate of this phrase/condition pair can be small enough to be rounded down to zero despite smoothing. In the decoder, the resulting zero probabilities probably get converted into a -Infinity logprob, which causes all kinds of havoc, including lots of phrases with infinite or NaN scores. Suggested fix: Use %g instead of %.5f everywhere in sub store_reordering_f and store_reordering_fe. Is there any place where zero probabilities actually could make sense or could they be filtered out on loading the tables? Best, Christian _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
