Great - using kenlm seems to work. It looks like <unk> is responsible for the trouble. At least that's the only complaint I've seen when loading with kenlm. I forgot the '-unk' flag in ngram-count. Is that a big problem? I don't want to re-run the lm-training ...
One more thing: kenlm/build_binary crashes (because of the missing <unk>?) Reading /home/staff/joerg/projects/UUMT/wmt11/data/training-monolingual/news.shuffled.low.de.lm ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 Language model is missing <unk>. Substituting probability 0. *make: *** [/home/staff/joerg/projects/UUMT/wmt11/data/training-monolingual/news.shuffled.low.de.kenlm] Segmentation fault make: *** Deleting file `/home/staff/joerg/projects/UUMT/wmt11/data/training-monolingual/news.shuffled.low.de.kenlm' Thanks again, Jörg On Mon, Feb 7, 2011 at 12:08 AM, Kenneth Heafield <[email protected]> wrote: > The first error you report (body != 0) means malloc returned 0. That's > an out of memory condition (or a bug in SRI asking for 0 memory). Are > you you compiling 32-bit or running with any other hard limit on RAM? > > Don't know what your second error is. > > Try kenlm. It uses less memory and has more informative error messages. > > Kenneth > > On 02/06/11 17:57, Joerg Tiedemann wrote: >> Hi, >> >> I have a problem loading LMs generated from the news.shuffled data >> sets. The decoder dies with this message: >> >> Start loading LanguageModel >> /home/staff/joerg/projects/UUMT/wmt11/data/training-monolingual/news.shuffled.low.de.lm >> : [138.000] seconds >> moses: ../../include/LHash.cc:138: void LHash<KeyT, >> DataT>::alloc(unsigned int) [with KeyT = unsigned int, DataT = float]: >> Assertion `body != 0' failed. >> sh: line 1: 1692 Aborted >> /home/staff/joerg/projects/LetsMT/tools32/mosesdecoder/moses-cmd/src/moses >> -threads 4 -config filtered/moses.ini -inputtype 0 -w -0.217387 -lm >> 0.036238 0.036238 0.036238 -d 0.065216 0.065216 0.065216 0.065216 >> 0.065216 0.065216 0.065216 -tm 0.043477 0.043477 0.043477 0.043477 >> 0.043477 -n-best-list run1.best100.out 100 -input-file >> /home/staff/joerg/projects/UUMT/wmt11/data/dev/newstest2009-src.low.en >>> run1.out >> >> >> The LM is big but I don't think that memory is the problem. I have >> also a similar problem with a smaller Czech LM (but a different >> message): >> >> >> Start loading LanguageModel >> /home/staff/joerg/projects/UUMT/wmt11/data/training-monolingual/news.shuffled.low.cs.lm >> : [36.000] seconds >> Unexpected error. >> sh: line 1: 6737 Aborted >> /home/staff/joerg/projects/LetsMT/tools32/mosesdecoder/moses-cmd/src/moses >> -threads 4 -config filtered/moses.ini -inputtype 0 -w -0.217387 -lm >> 0.036238 0.036238 0.036238 -d 0.065216 0.065216 0.065216 0.065216 >> 0.065216 0.065216 0.065216 -tm 0.043477 0.043477 0.043477 0.043477 >> 0.043477 -n-best-list run1.best100.out 100 -input-file >> /home/staff/joerg/projects/UUMT/wmt11/data/dev/newstest2009-src.low.en >>> run1.out >> Exit code: 134 >> >> >> Any ideas? >> Thanks, >> >> Jörg >> >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- ********************************************************************************** Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/ _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
