[Moses-support] Using Moses language models

2011-07-13 Thread Marc LEGENDRE
Hello, I am trying to use the language models loaded by Moses ; I am using a 3-gram LM, and I need to know whether it contains a given N-gram or not. I tried to play around with LanguageModelImplementation::GetValueForgotState(...), but the boolean 'unknown' in the returned structure does not

Re: [Moses-support] Using Moses language models

2011-07-13 Thread Lee Ball (Applied Language)
What format is your language model in? What software did you use to create it? You can try opening the language model using more or cat and then grep for the particular ngram you're looking for. Kind regards, Lee Ball Infrastructure Manager lee.b...@appliedlanguage.com Applied Language

Re: [Moses-support] Using Moses language models

2011-07-13 Thread Philipp Koehn
Hi, But you're asking for a third piece of information. If you query for foo bar baz and I can tell you that it will never extend to * foo bar baz for any word * (due to pruning or filtering), then you need only remember foo bar (or even less). The trie knows this but because the pointers

Re: [Moses-support] Using Moses language models

2011-07-13 Thread Kenneth Heafield
On 07/13/11 15:53, Philipp Koehn wrote: Hi, But you're asking for a third piece of information. If you query for foo bar baz and I can tell you that it will never extend to * foo bar baz for any word * (due to pruning or filtering), then you need only remember foo bar (or

[Moses-support] More LM compression

2011-07-13 Thread Kenneth Heafield
Hi Moses, If trie uses too much memory, svn up to revision = 4074 then pass -a #bits to build_binary. It will minimize memory usage subject to the maximum number of bits you specify (so e.g. pass bits 40 to minimize memory usage). Compressing in this manner is lossless, but takes