Hi Nicola, Thanks for the reply.
I had already replied to Hiue, but forgot to mark the same to the mailing list. This is what i found..... I managed to find out what was irking the decoder. It seems it isn't happy if i provide a binarized language model. A normal language model works fine in the situation below. However it is strange that the mert should not be able to use the binarized IRSTLM language model. In order to upgrade to Irstlm 5.60.01, i guess i would have to upgrade my Moses version also. (I remember seeing some irstlm 5.60.01 specific changes in Moses revision list). So right now i will stick with my current version and complete set of experiments in hand. Later if I still find this issue, i will get back to you. Thanks and regards, Pratyush > Hi Pratyush > As Hieu said, the problem is probably due to a version incompatibility Hope the problem vanishes with the new version of IRSTLM > If not, please send me the smallest models (TM and LM) which cause the error so that I can debug > together with a snall test set and your configuration file > best regards, > Nicola > On Mar 16, 2011, at 9:14 AM, Hieu Hoang wrote: > hi pratyush > that seems a bit strange. I'm not sure what it could be. You may want to upgrade irstlm, it's now on version 5.60.01 > On 16/03/2011 01:43, Pratyush Banerjee wrote: > Hi All, > I have been using Moses Revision 3739 (from subversion trunk) for some time(about 3 months) now along with irstlm-5.50.01. > So far I never had any issues with the decoder. However yesterday onwards > am encountering a strange error during MERT. The decoder simply dies due to a segmentation fault. > This is the error shown in mert-err.log > Loading lexical distortion models...have 1 models > Creating lexical reordering... > weights: 0.065 0.065 0.065 0.065 0.065 0.065 > Loading table into memory...done. > Start loading LanguageModel > /home/pbanerjee/forum-translation/de2en/consumer/l_models/lm.irstlm.gz : [1.000] seconds > In LanguageModelIRST::Load: nGramOrder = 5 > Loading LM file (no MAP) > blmt > loadbin() > loading 252472 1-grams > loading 2695266 2-grams > loading 1998504 3-grams > loading 3828996 4-grams > loading 4506338 5-grams > done > OOV code is 252471 > IRST: m_unknownId=252471 > creating cache for storing prob, state and statesize of ngrams > Finished loading LanguageModels : [3.000] seconds > Start loading PhraseTable > /home/pbanerjee/forum-translation/de2en/consumer/mert-work/filtered/phrase-table.0-0.1.1 : [3.000] seconds > filePath: > /home/pbanerjee/forum-translation/de2en/consumer/mert-work/filtered/phrase-table.0-0.1.1 Finished loading phrase tables : [3.000] seconds > Start loading phrase table from > /home/pbanerjee/forum-translation/de2en/consumer/mert-work/filtered/phrase-table.0-0.1.1 : [3.000] seconds > sh: line 1: 18257 Segmentation fault > /home/pbanerjee/software/mosesdecoder-R3739/moses-cmd/src/moses -s 500 -config filtered/moses.ini -inputtype 0 -w -0.217391 -lm 0.108696 -d 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 -tm 0.043478 0.043478 0.043478 0.043478 0.043478 -n-best-list run1.best200.out > 200 -i /home/pbanerjee/forum-translation/de2en/consumer/orig/devset.de > run1.out > Exit code: 139 > The decoder died. CONFIG WAS -w -0.217391 -lm 0.108696 -d 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 -tm 0.043478 0.043478 0.043478 0.043478 0.043478 > This has been seen over multiple models, multiple language pairs even binarized language models. > I also ran moses with GDB and this was the output: > Loading lexical distortion models...have 1 models > Creating lexical reordering... > weights: 0.300 0.300 0.300 0.300 0.300 0.300 > Loading table into memory...done. > Start loading LanguageModel /data/Symantec/fr2en/l_models/lm.irstlm.gz : [0.000] seconds > In LanguageModelIRST::Load: nGramOrder = 5 > Loading LM file (no MAP) > blmt > loadbin() > loading 486152 1-grams > loading 4050539 2-grams > loading 10809711 3-grams > loading 15872277 4-grams > loading 17651596 5-grams > done > OOV code is 486151 > IRST: m_unknownId=486151 > creating cache for storing prob, state and statesize of ngrams > Finished loading LanguageModels : [6.000] seconds > Start loading PhraseTable > /data/Symantec/fr2en/mert-work/filtered/phrase-table.0-0.1.1 : [6.000] seconds > filePath: /data/Symantec/fr2en/mert-work/filtered/phrase-table.0-0.1.1 Finished loading phrase tables : [6.000] seconds > Start loading phrase table from > /data/Symantec/fr2en/mert-work/filtered/phrase-table.0-0.1.1 : [6.000] seconds > Program received signal SIGSEGV, Segmentation fault. > lmtable::mybsearch (size=<value optimized out>, offs=<value optimized out>, value=<value optimized out>, ptr=<value optimized out>, this=<value > optimized out>) at lmtable.cpp:1109 > 1109 lmtable.cpp: No such file or directory. > in lmtable.cpp > (gdb) bt > #0 0x00000000004f9a73 in lmtable::mybsearch (size=<value optimized out>, > offs=<value optimized out>, value=<value optimized out>, ptr=<value optimized out>, this=<value optimized out>) > #1 0x00000000004f9b72 in lmtable::search(int, table_entry_pos_t, table_entry_pos_t, int, int *, <anonymous enum>, char **) (this=0x77baa0, > lev=<value optimized out>, offs=<value optimized out>, n=78, > sz=78, ngp=<value optimized out>, action=LMT_FIND, > found=0x7fffffffb2b0) at lmtable.cpp:1062 > #2 0x00000000004fbde1 in lmtable::get (this=0x77baa0, ng=..., n=<value optimized out>, lev=<value optimized out>) at lmtable.cpp:1740 > #3 0x00000000004fc51d in lmtable::lprob (this=0x77baa0, ong=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. > ) at lmtable.cpp:2046 > #4 0x00000000004fcc42 in lmtable::clprob (this=0x77baa0, codes=<value optimized out>, sz=<value optimized out>, bow=0x0, bol=0x0, > state=0x7fffffffb940, statesize=0x7fffffffb948) at lmtable.cpp:2181 #5 0x00000000004ca4e6 in Moses::LanguageModelIRST::GetValue > (this=0x779dd0, contextFactor=<value optimized out>, finalState=<value optimized out>, len=0x7fffffffb94c) at LanguageModelIRST.cpp:212 #6 0x0000000000492806 in Moses::LanguageModel::CalcScore (this=0x1f43130, > phrase=..., fullScore=<value optimized out>, ngramScore=<value optimized out>) at LanguageModel.cpp:121 > #7 0x0000000000465931 in Moses::TargetPhrase::SetScore > (this=0x7fffffffbfb0, translationScoreProducer=<value optimized out>, scoreVector=<value optimized out>, weightT=<value optimized out>, > weightWP=<value optimized out>, languageModels=...) at > TargetPhrase.cpp:170 > #8 0x00000000004b2378 in Moses::PhraseDictionaryMemory::Load (this=<value > optimized out>, input=<value optimized out>, output=<value optimized out>, > filePath=<value optimized out>, > weight=<value optimized out>, tableLimit=<value optimized out>, > languageModels=..., weightWP=<value optimized out>) at > PhraseDictionaryMemory.cpp:125 > #9 0x00000000004af486 in Moses::PhraseDictionaryFeature::LoadPhraseTable > (this=0x87e9f60, system=<value optimized out>) at PhraseDictionary.cpp:96 > #10 0x00000000004afdfb in Moses::PhraseDictionaryFeature::InitDictionary (this=0x87e9f60, system=0x87ea268) at PhraseDictionary.cpp:194 > #11 0x000000000047161f in Moses::TranslationSystem::ConfigDictionaries (this=0x87ea268) at TranslationSystem.cpp:99 > #12 0x000000000045f8e8 in Moses::StaticData::LoadData (this=0x76e420, parameter=<value optimized out>) at StaticData.cpp:521 > #13 0x0000000000407bcd in LoadDataStatic (parameter=<value optimized out>) > at ../../moses/src/StaticData.h:250 > #14 main (parameter=<value optimized out>) at Main.cpp:334 > However this still could not provide me any details of the error. I saw a > similar error being pointed out in the Moses forums but that was attributed to memory issues. I am using considerable RAM machines (16Gb) and still this fails. > Could any one shed any light on this ? Is there anything wrong with the version of the moses I am using ? > Thanks and regards, > Pratyush > _______________________________________________ > Moses-support mailing list > [email protected]<mailto:[email protected]> > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ > Moses-support mailing list > [email protected]<mailto:[email protected]> > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
