Just to get the word out more, trie is broken before 3847 for common pruning strategies as announced in "[Moses-support] kenlm updated in 3847". Admittedly the subject could have yelled more, but it's also easy to miss posts.
On 02/07/11 11:11, Kārlis Goba wrote: > Thanks, Kenneth, > > This was revision 3796, I will try with a newer version. > >> Hi, >> >> What revision of Moses are you using? Does this still happen >> after you >> run svn up and recompile Moses? >> >> Kenneth >> >> On 02/07/11 10:53, Kārlis Goba wrote: >>> Hi, >>> >>> My preferred way to build large LMs has been IRSTLM as it can handle >> large corpora nicely by splitting the task. The produced binary LMs >> work well with Moses. Then I decided to try the new and shiny KenLM. >> However, when converting the result to KenLM format, the converted LM >> gives a lot worse BLEU score. >>> >>> I am building the LMs in this way: >>> >>> 1) build-lm.sh - build iARPA LM >>> 2) prune-lm >>> 3) compile-lm - convert iARPA to binary IRSTLM (gives me 0.3346 BLEU) >>> 4) compile-lm --text=yes - convert iARPA to ARPA >>> 5) build_binary trie - convert ARPA to KenLM (gives me 0.2543 BLEU) >>> >>> The moses.ini for both cases is the same, differing only in the LM >> line (1 0 5 "/mnt/smt/lm/mt4-lv-lcase/lm-pruned.blm" vs "8 0 5 >> /mnt/smt/lm/mt4-lv-lcase/lm-pruned.mmap"). I have observed this in more >> than one case. >>> >>> I don't know who blame - either the conversion from iARPA to ARPA or >> KenLM. >>> >>> What is your best practice for estimating large LMs to be converted >> to KenLM? Is it SRILM? >>> >>> -- >>> Karlis >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
