Ah. I've changed the version number to force people to rebuild their binary files when the format changes. Old versions produced that error message when the version number changed. Quantization wasn't even implemented in kenlm until some time in 2011, so you're not going to get a Moses from 2010 to work with it.
As to the compiler errors on the tests, you're probably using an old version of Boost not supported by the tests. Kenneth On 10/10/11 17:57, marco turchi wrote: > Hi > what I have done is to download the last version of kenlm, built it > and run the build_binary. Then I call a 2010 version of moses using > the lm. > > I run the ./test.sh and I got these messages: > util/bit_packing_test.cc:14: error: expected constructor, destructor, > or type conversion before â(â token > util/bit_packing_test.cc:21: error: expected constructor, destructor, > or type conversion before â(â token > util/bit_packing_test.cc:59: error: expected `}' at end of input > util/bit_packing_test.cc:59: error: expected `}' at end of input > > I guess, I need to reinstall the full package moses+kenlm and biuld > everything together. > > Thanks a lot > Marco > > On Mon, Oct 10, 2011 at 6:46 PM, Kenneth Heafield <[email protected] > <mailto:[email protected]>> wrote: > > Hi, > > Number 8 means prefault and number 9 means lazy mmap. It's an > option and orthogonal to the data structure. > > Since the binary file is the in-memory representation, I do > paranoid checks to make sure your machine represents floats, > 64-bit integers, and such in the same way. For example a 32-bit > build will have different alignment than a 64-bit build. This > check is complaining. > > Please try build_binary and moses from the same build. If > that doesn't work, please send me the first kilobyte of your > binary file. > > Also, if you have Boost, can you cd kenlm && make clean && > ./test.sh and complain if there are any test failures? > > Kenneth > > > On 10/10/11 17:31, marco turchi wrote: >> Hi Kenneth, >> which number shall I use in the moses.ini 8 or 9 if i build my lm >> with these parameters -q 8 -b 8? >> >> I got this error when I run moses: >> In LanguageModelKen::Load: nGramOrder = 5 will be ignored. Using >> whatever the file has. >> terminate called after throwing an instance of >> 'lm::FormatLoadException' >> what(): File looks like it should be loaded with mmap, but the >> test values don't match. Was it built on a different machine or >> with a different compiler? >> >> I have the feeling that my moses version needs to be updated! >> >> Thanks a lot >> Marco >> >> On Sat, Oct 8, 2011 at 1:02 PM, marco turchi >> <[email protected] <mailto:[email protected]>> wrote: >> >> Thanks! >> I'm going to update my version. >> >> Cheers >> Marco >> >> >> On Sat, Oct 8, 2011 at 1:01 PM, Kenneth Heafield >> <[email protected] <mailto:[email protected]>> wrote: >> >> Fixed in revision 4314. There's still an issue with some >> SRILM models failing to build that I'll get to soon. >> >> On 10/08/11 11:52, marco turchi wrote: >>> Hi, >>> thanks a lot for the answer. >>> Great, so I can use -m 2048 to build it. Do you think it >>> is enough? >>> >>> Thanks again >>> Marco >>> >>> On Sat, Oct 8, 2011 at 12:46 PM, Kenneth Heafield >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> Hi, >>> >>> This looks like a bug in the trie implementation >>> due to some recent changes I made for left state >>> minimization. I'll fix it soon. A workaround is to >>> pass a large -m option to build_binary. >>> >>> Sorry, >>> >>> Kenneth >>> >>> >>> On 10/08/11 11:34, marco turchi wrote: >>>> Dear All, >>>> I'm trying to build a lm using a large dataset (> >>>> 11 M sentences). I have generated the Arpa format >>>> with irstlm and now I'd like to binarize it using >>>> kenlm. >>>> >>>> I have called the build_binary to estimate memory >>>> usage, and I got this >>>> >>>> Memory estimate: >>>> type MB >>>> probing 16129 assuming -p 1.5 >>>> trie 7462 without quantization >>>> trie 4361 assuming -q 8 -b 8 quantization >>>> trie 6440 assuming -a 22 array pointer compression >>>> trie 3339 assuming -a 22 -q 8 -b 8 array >>>> pointer compression and quantization >>>> >>>> then I run the binarization in this way: >>>> >>>> /nfs/staging/turchmo/moses/kenlmNew/build_binary -i >>>> -t /tmp/ -q 8 -b 8 trie irstLM.ARPA.txt >>>> irstLanguageModel.binary.lm >>>> >>>> but I got this error: >>>> >>>> lm/search_trie.cc:409 in void >>>> lm::ngram::trie::<unnamed>::SanityCheckCounts(const >>>> std::vector<long unsigned int, std::allocator<long >>>> unsigned int> >&, const std::vector<long unsigned >>>> int, std::allocator<long unsigned int> >&) threw >>>> util::Exception'. >>>> Longest count should be constant but it changed >>>> from 289546423 to 289546405 Byte: 37297517525 >>>> >>>> I have had a look into the mailing list, but I do >>>> not find any post with the same error. >>>> >>>> Any ideas? >>>> >>>> Thanks a lot >>>> Marco >>>> >>>> >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] <mailto:[email protected]> >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> >> >> > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
