Hi, I've not seen it in this list but what licenses is KenLM distributed under?
Kind regards, Lee Ball Infrastructure Manager lee.b...@appliedlanguage.com Skype ID: lee.ball_appliedlanguage Tel: +44 (0)844 854 8945 Applied Language Solutions High quality language solutions delivered on time...with a smile! www.appliedlanguage.com Tel (UK): +44 (0)845 367 7000 Tel (US): +1 (800) 579-5010 Riverside Court, Huddersfield Road, Delph, Oldham, OL3 5FZ. UK Registered in the UK 5122429 Pride in everything we do | Respect everyone like a friend [image: An Environmentally Friendly Company]Think of the environment; please don't print this e-mail unless you really need to. [image: Fast Track 100 2009][image: Queens Award for Business] On 19 October 2010 01:31, Kenneth Heafield <mo...@kheafield.com> wrote: > Hi Moses, > > Introducing kenlm in Moses trunk. You no longer need to download a > separate language model to use Moses; it's distributed with Moses and > compiled in by default on UNIX. This is threadsafe language model > inference code that returns the same probabilities as SRI (up to > floating point rounding). It loads APRA files in 2/3 the time SRI takes > and uses less memory too. Using kenlm is simple: in your [lmodel-file] > section, change the first digit to 8. For example, > > "0 0 2 foo.arpa" changes to "8 0 2 foo.arpa" > > For even faster loading, use the binary format: > > kenlm/build_binary foo.arpa foo.binary > > then simply provide the binary filename in your moses.ini e.g. > "8 0 2 foo.binary"; it auto detects binary files using magic bytes at > the beginning. > > The code is ready for use and provides correct results. Inference > is > slower than it should be due to inefficiencies in the Moses-side wrapper > code (it does a vocab lookup for all 5 words every time). I'm working > on it and once this is done I'll post some benchmarks against SRI and > IRST. The binary format is subject to change, but contains a version > number so on very rare occasions after, new versions will tell you to > rebuild your binary files. Windows is currently not supported (it uses > mmap) though I welcome contributions using #ifdef and CreateFileMapping. > > Have fun and let me know about your experiences with it. > > "Ken" > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
