Hieu, I am using phrase-based decoding, FYI: the binarizing process is : zcat MYMODEL/phrase-table.0,1-0,1.gz | sort | processPhraseTable -ttable 0 0 - -nscores 5 -out MYMODEL/phrase-table.0,1-0,1
(note that the decoder works fine with SRI lm) Bruno ________________________________________ From: [email protected] [[email protected]] On Behalf Of Hieu Hoang [[email protected]] Sent: Friday, February 18, 2011 1:26 PM To: [email protected] Subject: Re: [Moses-support] Bug when using KenLM in decoding are you using phrase-based or chart decoding? There's 2 versions of binary/on-demand phrase-tables: 1. The Binary phrase-table for phrase-based. 2. The OnDisk is for chart decoding. it's a bit of a confusing situation, apologies On 18/02/2011 19:14, Pouliquen, Bruno wrote: > I tried to install KenLM using the latest version of Moses (trunk Revision > 3884), the decoder loads the LM without problems but then fails when decoding > with following output: > (the decoding is from French to English, I use 5-grams Language model, 2 > factors, binarized phrase tables) > > ------------------------------------ > Loading lexical distortion models...have 1 models > Creating lexical reordering... > weights: 0.300 0.300 0.300 0.300 0.300 0.300 > binary file loaded, default OFF_T: -1 > Start loading LanguageModel MYMODEL-kenlm.bin : [1.000] seconds > Finished loading LanguageModels : [3.000] seconds > weight-generation: 0.300 0.000 > MYMODEL/generation.0,1-0.gz > Start loading PhraseTable MYMODEL/phrase-table.0,1-0,1 : [21.000] seconds > filePath: MYMODEL//phrase-table.0,1-0,1 > Start loading PhraseTable MYMODEL//phrase-table.0-0 : [21.000] seconds > filePath:MYMODEL/phrase-table.0-0 > Finished loading phrase tables : [21.000] seconds > IO from STDOUT/STDIN > Created input-output object : [21.000] seconds > Translating: test|U > > moses: OnDiskWrapper.cpp:59: bool OnDiskPt::OnDiskWrapper::OpenForLoad(const > std::string&): Assertion `m_fileSource.is_open()' failed. > Aborted > ------------------------------------ > the LM line in moses.ini is: > 8 0 5 MYMODEL-kenlm.bin > > FYI: I also tried without binarizing the language model, using the standard > ARPA format, it takes longer to load but the result is the same > > I have the impression that the binarized phrasetables may be the source of > the error... > > Does anybody has any idea? Did I missed something? > > thanks in advance, > Bruno Pouliquen > > > > World Intellectual Property Organization Disclaimer: > > This electronic message may contain privileged, confidential and > copyright protected information. If you have received this e-mail > by mistake, please immediately notify the sender and delete this > e-mail and all its attachments. Please ensure all e-mail attachments > are scanned for viruses prior to opening or using. > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support World Intellectual Property Organization Disclaimer: This electronic message may contain privileged, confidential and copyright protected information. If you have received this e-mail by mistake, please immediately notify the sender and delete this e-mail and all its attachments. Please ensure all e-mail attachments are scanned for viruses prior to opening or using. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
