kangaroo is less probable than snake. Which more than explains the difference you observed. Film at 11.
That p(<unk>) is pretty high. What happened when you used lmplz to build the model? Kenneth On 03/23/2016 09:28 AM, Bhat Irshad wrote: > I build a language model using IRSTLM on 20 million tokenized English > sentences and tested on the following two sentences: > > 1. Yesterday when I was walking towards home , I saw a kangaroo . > 2. smdnbs sadb jghsa sdabasd asasd tsados hasdb , I saw a snake . > > As we can the first portion of second sentence is completely trash while > first sentence is a proper grammatical one. I was surprised to see that > second sentence got higher probability score (-27.887135) than first one > (-28.91925). > > I guess this happened due to back-off, I am not sure though. > > echo 'Yesterday when I was walking towards home , I saw a kangaroo .' | > /usr/bin/query english-lcc-ilci-ukwac-tok-20M-n3.blm 2> /tmp/a > Yesterday=126222 2 -4.08843when=409 3 -2.51627I=260 3 -0.58336was=771 3 > -0.764257walking=1624 3 -2.58353towards=1335 3 -1.95033home=388 2 > -3.910977,=209 3 -1.15596I=260 3 -1.55485saw=4411 3 -2.31963a=131 3 > -0.886832kangaroo=106652 2 -5.3615108.=10 3 -1.24128</s>=11 3 > -0.00203508Total: -28.91925 OOV: 0 > Perplexity including OOVs:116.32170228822577 > Perplexity excluding OOVs:116.32170228822577 > OOVs:0 > Tokens:14 > > echo 'smdnbs sadb jghsa sdabasd asasd tsados hasdb , I saw a snake .' | > /usr/bin/query english-lcc-ilci-ukwac-tok-20M-n3.blm 2> /tmp/a > smdnbs=0 1 -4.0025997sadb=0 1 -2.23153jghsa=0 1 -2.23153sdabasd=0 1 > -2.23153asasd=0 1 -2.23153tsados=0 1 -2.23153hasdb=0 1 -2.23153,=209 1 > -1.42496I=260 2 -1.9045saw=4411 3 -2.31963a=131 3 -0.886832snake=3768 3 > -3.16116.=10 3 -0.793541</s>=11 3 -0.0047327Total: -27.887135 OOV: 7 > Perplexity including OOVs:98.16082104257269 > Perplexity excluding OOVs:31.57449745907425 > OOVs:7 > Tokens:14 > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
