Hi Nicola, I have had the same results with several different corpora, both large and small. I was filtering out 1-gram lines, but the problem occurs with/without the 1-gram lines.
I uploaded a very small, 26,806 line, lm corpus, with a file steps.txt that include the three command lines I used. It also includes the resulting files from each step. I'm using: IRSTLM 5.50.02 MOSES Decoder/KenLM SVN rev 3675 The error occurs when running 32-bit and 64-bit binaries. Filename: irstlm-error.tar.gz in the root of ftp link below. 66 MB file Thanks, Tom On Mon, 10 Jan 2011 08:52:40 +0100, Nicola Bertoldi <[email protected]> wrote: > Indeed this should not happen > > Tom, could you please upload the following data in our ftp area? > > - textual training data (if possible) > - LM in iARPA format > - LM in binary format > - list of commands you run > > For external uploading: > > ftp ftp.fbk.eu > login: ftphlt > password: houdine23# > > > > best > nicola > > On Jan 6, 2011, at 4:43 PM, Kenneth Heafield wrote: > >> Hey IRST, why are you generating positive log probabilities? >> >> I'll have to fix the error message to print the number 4 instead of >> ASCII value 4. >> >> On 01/06/11 04:13, [email protected] wrote: >>> I've been using IRSTLM's build-lm.sh to build an LM. Then >>> converted from >>> iARPA to ARPA with compile-lm --text yes. On a recent data set, I >>> encountered the following exception with KenLM's build_binary on >>> the >>> resulting ARPA file: >>> >>> >>> >>> Reading cased.irstlm.arpa >>> >>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---7 >>> 0---75---80---85---90---95--100 >>> ***********************terminate called after throwing an instance >>> of >>> 'lm::FormatLoadException' >>> what(): Positive probability 4.00956e-08 in the -gram at byte >>> 46366584 in file cased.irstlm.arpa >>> Aborted >>> >>> >>> >>> build_binary is successful with almost exactly the same data set in >>> lower case. >>> >>> Any ideas? >>> >>> Tom >>> >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
