Hi,
There's documentation at http://kheafield.com/code/kenlm/estimation/ on
how to build (mostly) comparable models. I just updated it to reflect
the new default behavior of interpolating unigrams. Please also make
sure you have Moses 36da8d1 or later, since the policy is that
documentation reflects the state of master.
Also keep in mind that SRILM's perplexity is comparable to "Perplexity
excluding OOVs" line from query. And ARPA files are compatible so
nothing prevents you from using the query program from the other toolkit.
Kenneth
On 10/08/14 05:56, Rico Sennrich wrote:
> oh, you're also using different smoothing, and possibly different
> handling of unknown words.
>
> lmplz defaults to SRILM's|| '-interpolate -kndiscount -unk -gt3min 1
> -gt4min 1 -gt5min 1'
>
> On 08/10/14 10:05, koormoosh wrote:
>> Thanks. Now it's 15 score closer to the KenLM, but still the
>> difference is significant, 22. compared to KenLM 9.
>> But still the difference is not close enough to be ignored.
>
>
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
