Moses uses log_e while Kenlm uses log_10 log_e(P) = log_e(10) log_10(P)
-17.715 = log_e(10)*-7.69363 Am I right? Nicola On Oct 25, 2012, at 4:09 PM, Adam Teichert wrote: > Hi all, > > After decoding, I get the output "I am ." from moses. I expected > the lm feature of the winning hypothesis to match the language model > score of the output (i.e. running kenlm on "I am .") but it doesn't > match. > > moses decoder says the lm feature should be: -17.7152 > kenlm says: -7.69363 > > Any ideas? (more verbose details below) > > Thanks, > > --Adam > > > moses: > 0 ||| I am . ||| d: 0 lm: -17.7152 0 w: -3 tm: -13.6791 -13.4943 > -7.13198 -2.7697 2.99969 ||| -17.7152 > > kenlm: > I=11 2 -0.960569 am=16 2 -3.58125 .=2 1 -2.85073 </s>=12 2 > -0.301086 Total: -7.69363 OOV: 0 > > lm.arpa: > > > \data\ > ngram 1=24 > ngram 2=61 > ngram 3=55 > ngram 4=37 > ngram 5=15 > > \1-grams: > -1.468523 <unk> > -1.847186 , -1.369804 > -1.949801 . -1.569853 > -1.963158 and -1.181599 > -2.323228 is -1.180351 > -2.361128 that -1.115681 > -2.58365 : -0.9018961 > -2.668149 it -0.9380397 > -2.68633 this -0.947085 > -2.810327 we -1.037582 > -2.848784 not -0.6793468 > -2.959159 I -1.039103 > -3.07579 </s> > -3.192658 my -0.771203 > -3.257855 no -0.6133907 > -99 <s> -2.155778 > -4.495705 am -0.2612615 > -3.793312 little -0.4288335 > -3.485892 me -0.7101038 > -4.420267 surprise -0.5936328 > -4.396488 surprised -0.7797279 > -4.735315 surprises -0.5008686 > -4.552582 surprising -0.5104726 > -4.420267 wonder -0.7249804 > > \2-grams: > -0.3010858 . </s> > -0.7091757 surprises me -0.2860005 > -0.8920909 surprises . -0.666689 > -0.9196912 surprised that -0.3720784 > -0.9494263 surprise , -0.2865064 > -0.9513271 surprising , -0.3125141 > -0.9605687 <s> I -2.082714 > -1.027536 surprises , -0.1418063 > -1.03712 surprising that -0.234587 > -1.113948 surprising . -0.818023 > -1.128903 wonder , -0.201676 > -1.129378 surprise that -0.2199759 > -1.144491 <s> we -1.811983 > -1.176047 surprised , -0.3082836 > -1.223108 me , -0.4326107 > -3.267176 , . -0.6048141 > -1.942681 , I -1.324145 > -1.624891 , and -1.041633 > -2.088251 , is -0.7182704 > -2.00548 , it -1.121896 > -2.659058 , no -0.426425 > -2.169326 , not -0.7878548 > -2.013858 , that -0.741353 > -4.511329 <s> am -1.611718 > -2.70271 <s> no -0.6921422 > -1.498533 I am -0.6396686 > -2.678573 I wonder -0.4021412 > -1.723108 am I -0.1898816 > -1.983081 am not -0.05170299 > -3.118301 and . -0.6289005 > -4.704317 and surprising -0.03819112 > -4.235544 and wonder -0.3394165 > -2.49379 is . -1.147667 > -2.878376 is it -0.3021726 > -2.286608 is no -0.2445829 > -1.610847 is not -0.7426848 > -4.088284 is surprising -0.2584699 > -1.43062 it . -1.05238 > -1.354294 it is -0.4715224 > -1.872162 little . -0.7615967 > -1.302207 me . -0.7159881 > -2.216844 no . -0.4275606 > -3.09122 no surprise -0.238984 > -3.29149 no surprises -0.1407324 > -2.364784 not . -0.5624467 > -3.183594 not it -0.0237771 > -3.857746 not me -0.1115348 > -3.5716 not surprise -0.333057 > -3.570876 not surprised -0.2835279 > -3.621675 not surprising -0.1414568 > -1.247403 surprise . -0.5275136 > -1.554344 surprise me -0.1131797 > -1.231814 surprised . -1.243741 > -1.353927 surprised me -0.2527946 > -2.455974 that . -0.8608306 > -1.859256 that is -0.5548451 > -1.854284 that it -0.8317314 > -2.064787 this . -0.8331819 > -2.12582 this is -0.4543184 > -2.882952 we . -0.9427112 > -1.525799 wonder . -1.009658 > > \3-grams: > -0.01054467 <s> am I -0.4033368 > -0.0125701 surprised . </s> > -0.02177685 wonder . </s> > -0.0255149 we . </s> > -0.02761195 is . </s> > -0.03434359 surprising . </s> > -0.03933057 little . </s> > -0.04950372 surprises . </s> > -0.05429693 and . </s> > -0.06596278 it . </s> > -0.06643473 , . </s> > -0.069778 surprise . </s> > -0.08091767 me . </s> > -0.0885259 that . </s> > -0.0999171 this . </s> > -0.1734623 no . </s> > -0.1389831 not . </s> > -2.214 , I wonder -0.9099113 > -1.619141 I am not -0.4131633 > -4.240031 , and . -0.2288982 > -3.244178 , is . -0.4621631 > -2.173454 , is no -0.5102387 > -2.217342 it is . -0.6322967 > -1.657225 it is not -0.3576518 > -2.566752 that is . -0.5047822 > -3.111846 that is it -0.1018341 > -2.304894 this is . -0.7973652 > -2.962754 this is it -0.08542722 > -0.6746925 , it is -0.9160917 > -1.739163 is it . -0.1419258 > -1.454397 not it . 0 > -0.8604823 that it is -0.6332519 > -0.5766525 not me . -0.3372244 > -0.9619833 surprise me . -0.1180146 > -0.8732141 surprised me . -0.5590731 > -2.067063 , no . -0.7058067 > -1.440352 <s> no . -1.143604 > -2.403745 is no . -0.5590731 > -2.466519 is no surprise -0.2155667 > -3.440862 , not me -0.5590731 > -2.300408 am not . -0.07168788 > -2.422223 am not surprised -0.08383179 > -2.473987 is not . -0.834681 > -3.623306 is not it -0.07617349 > -3.54658 is not surprising -0.1496836 > -1.219092 no surprise . -0.1016625 > -1.127067 not surprise me -0.066174 > -1.265616 not surprised . -0.228898 > -0.6443897 no surprises . -0.3372243 > -0.8107078 and surprising . -0.2288982 > -0.8876699 is surprising . -0.4621631 > -1.04663 not surprising . 0.004644036 > -0.9365889 , that is -0.8274354 > -1.870765 I wonder . -0.2288982 > -1.371585 and wonder . -0.2288982 > > \4-grams: > -0.004286873 this is . </s> > -0.00737656 not surprised . </s> > -0.008448888 that is . </s> > -0.009329482 , is . </s> > -0.0103978 <s> no . </s> > -0.01154472 is surprising . </s> > -0.01272331 I wonder . </s> > -0.01272331 and wonder . </s> > -0.01440236 it is . </s> > -0.017762 is not . </s> > -0.01994449 and surprising . </s> > -0.0208705 surprised me . </s> > -0.02207431 no surprises . </s> > -0.02910873 , no . </s> > -0.03122709 , and . </s> > -0.04654508 is it . </s> > -0.03536305 not me . </s> > -0.06025264 surprise me . </s> > -0.04138191 is no . </s> > -0.1147482 am not . </s> > -0.05426025 no surprise . </s> > -0.03472797 not surprising . </s> > -3.038089 , it is . -0.42836 > -2.386421 that it is . -0.759353 > -2.281374 , that is . -1.259269 > -1.008098 that is it . -0.2749732 > -0.9806256 this is it . -0.2749732 > -0.7204503 is not it . -0.6572088 > -0.09846061 , not me . -0.5780276 > -0.8568227 not surprise me . -0.3711458 > -1.815292 , is no . -0.4811176 > -2.135054 I am not . -0.6190532 > -1.894238 it is not . -1.09429 > -0.8054344 is no surprise . -0.2930098 > -0.9460168 am not surprised . -0.2210943 > -0.8717432 is not surprising . -0.4107019 > -2.35131 , I wonder . -0.2210943 > > \5-grams: > -0.0004608329 , that is . </s> > -0.001402984 it is not . </s> > -0.002472492 that it is . </s> > -0.004418593 am not surprised . </s> > -0.005315542 , it is . </s> > -0.007602483 , I wonder . </s> > -0.009067477 , not me . </s> > -0.01316113 is not surprising . </s> > -0.01323633 , is no . </s> > -0.01368839 is not it . </s> > -0.02408934 that is it . </s> > -0.02408934 this is it . </s> > -0.02462093 not surprise me . </s> > -0.0249456 I am not . </s> > -0.02678858 is no surprise . </s> > > \end\ > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
