Hi Hieu, It's actually KenLM I've just updated my moses.ini changing the 0 to 8 but I'm still getting the same error
Regards Massinissa 2014-03-22 11:40 GMT+01:00 Hieu Hoang <[email protected]>: > did you binarize the LM and what did you binarize it with? KenLM or IRSTLM? > > at the moment, it's set to SRILM, which doesn't have a binary format > > > > On 22 March 2014 10:10, Massinissa Ahmim < > [email protected]> wrote: > >> Hi Hieu, >> >> I used the same binarisation command as the one stated here-above >> >> I get this error when I try to translate something like : echo 'ceci est >> un test' | /mosesdecoder/bin/moses_chart -f moses.ini >> >> Here is my moses.ini : >> >> ######################### >> ### MOSES CONFIG FILE ### >> ######################### >> >> # input factors >> [input-factors] >> 0 >> >> # mapping steps >> [mapping] >> 0 T 0 >> 1 T 1 >> >> # translation tables: table type (hierarchical(0), textual (0), binary >> (1)), source-factors, target-factors, number of scores, file >> # OLD FORMAT is still handled for back-compatibility >> # OLD FORMAT translation tables: source-factors, target-factors, number >> of scores, file >> # OLD FORMAT a binary table type (1) is assumed >> [ttable-file] >> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table >> 6 0 0 1 >> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar >> >> # no generation models, no generation-file section >> >> # language models: type(srilm/irstlm), factors, order, file >> [lmodel-file] >> 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm >> >> >> # limit on how many phrase translations e for each phrase f are loaded >> # 0 = all elements loaded >> [ttable-limit] >> 20 >> >> >> # language model weights >> [weight-l] >> 0.5000 >> >> >> # translation model weights >> [weight-t] >> 0.20 >> 0.20 >> 0.20 >> 0.20 >> 0.20 >> 1.0 >> >> # no generation models, no weight-generation section >> >> # word penalty >> [weight-w] >> -1 >> >> [cube-pruning-pop-limit] >> 1000 >> >> [non-terminals] >> X >> >> [search-algorithm] >> 3 >> >> [inputtype] >> 3 >> >> [max-chart-span] >> 20 >> 1000 >> >> >> >> 2014-03-21 20:49 GMT+01:00 Hieu Hoang <[email protected]>: >> >> i've never seen that error before. Did you change anything else, other >>> than rebinarizing the phrase-table? What is the exact command you ran when >>> you got this error? Can I have a look at your moses.ini file >>> >>> >>> On 21 March 2014 18:32, Massinissa Ahmim < >>> [email protected]> wrote: >>> >>>> Thank you Hieu, >>>> >>>> I've binarised my rule-table again and it seems to be successfully >>>> loaded but I'm getting this message now : >>>> >>>> Reading /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table >>>> >>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 >>>> >>>> **************************************************************************************************** >>>> Exception: <fd:3>: Is a directory from zlib >>>> >>>> What can I do? >>>> >>>> Thanks ! >>>> >>>> Regards >>>> >>>> Massinissa >>>> >>>> >>>> 2014-03-21 11:18 GMT+01:00 Hieu Hoang <[email protected]>: >>>> >>>> everything seems to be ok. Are you sure the server you are running on >>>>> has access to the directory >>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table >>>>> >>>>> Also, you should consider updating your version of Moses to the >>>>> current github version, or version 2.1.1 >>>>> >>>>> >>>>> On 20 March 2014 11:20, Massinissa Ahmim < >>>>> [email protected]> wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> I've managed to train a hierarchical model using the following >>>>>> command : >>>>>> >>>>>> nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical >>>>>> --glue-grammar --score-options="--GoodTuring" -root-dir >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT -corpus >>>>>> /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm >>>>>> 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm >>>>>> -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 >& >>>>>> training.out & >>>>>> >>>>>> The resulting model works great but it is very slow. >>>>>> >>>>>> So I used CreatOnDiskPt to binarise the rule table as follows : >>>>>> >>>>>> /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2 >>>>>> model/rule-table.gz rules-table >>>>>> >>>>>> This outputs the following files >>>>>> >>>>>> -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat >>>>>> -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat >>>>>> -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat >>>>>> -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat >>>>>> -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat >>>>>> >>>>>> and I've updated my moses.ini as follows : >>>>>> >>>>>> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table >>>>>> 6 0 0 1 >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar >>>>>> >>>>>> But as I try to use it, I'm getting this : >>>>>> >>>>>> Defined parameters (per moses.ini or switch): >>>>>> config: moses.ini >>>>>> cube-pruning-pop-limit: 1000 >>>>>> input-factors: 0 >>>>>> inputtype: 3 >>>>>> lmodel-file: 0 0 5 >>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm >>>>>> mapping: 0 T 0 1 T 1 >>>>>> max-chart-span: 20 1000 >>>>>> non-terminals: X >>>>>> search-algorithm: 3 >>>>>> ttable-file: 6 0 0 1 >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar >>>>>> ttable-limit: 20 >>>>>> weight-l: 0.5000 >>>>>> weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 >>>>>> weight-w: -1 >>>>>> /mosesdecoder/bin >>>>>> ScoreProducer: WordPenalty start: 0 end: 1 >>>>>> ScoreProducer: !UnknownWordPenalty start: 1 end: 2 >>>>>> Loading lexical distortion models...have 0 models >>>>>> Start loading LanguageModel >>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] >>>>>> seconds >>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: line >>>>>> 80492: reached EOF before \end\ >>>>>> ScoreProducer: LM start: 2 end: 3 >>>>>> Finished loading LanguageModels : [0.056] seconds >>>>>> Using uniform ttable-limit of 20 for all translation tables. >>>>>> Start loading PhraseTable >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] >>>>>> seconds >>>>>> filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table >>>>>> ScoreProducer: PhraseModel start: 3 end: 4 >>>>>> Start loading PhraseTable >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar : >>>>>> [0.056] seconds >>>>>> filePath: >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar >>>>>> ScoreProducer: PhraseModel:2 start: 4 end: 5 >>>>>> Finished loading phrase tables : [0.056] seconds >>>>>> max-chart-span: 20 >>>>>> max-chart-span: 1000 >>>>>> Start loading phrase table from >>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] >>>>>> seconds >>>>>> Can't read /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table >>>>>> >>>>>> So I wondering if I did something wrong with my training >>>>>> command/binarisation or with the parameters in the moses.ini >>>>>> >>>>>> >>>>>> Many thanks >>>>>> >>>>>> Regads >>>>>> >>>>>> MA >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> [image: Description : Description : lingua_custodia_final full logo] >>>>>> >>>>>> *The Translation Trustee* >>>>>> >>>>>> *1, Place Charles de Gaulle* >>>>>> >>>>>> *78180 Montigny-le-Bretonneux* >>>>>> >>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* >>>>>> >>>>>> *Email :* *[email protected] >>>>>> <[email protected]>* >>>>>> >>>>>> *Website :* *www.linguacustodia.com >>>>>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com >>>>>> <http://www.thetranslationtrustee.com>* >>>>>> >>>>>> ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. >>>>>> >>>>>> Please do not print this email unless it is absolutely necessary. Spread >>>>>> environmental awareness. >>>>>> >>>>>> _______________________________________________ >>>>>> Moses-support mailing list >>>>>> [email protected] >>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Hieu Hoang >>>>> Research Associate >>>>> University of Edinburgh >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> [image: Description : Description : lingua_custodia_final full logo] >>>> >>>> *The Translation Trustee* >>>> >>>> *1, Place Charles de Gaulle* >>>> >>>> *78180 Montigny-le-Bretonneux* >>>> >>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* >>>> >>>> *Email :* *[email protected] >>>> <[email protected]>* >>>> >>>> *Website :* *www.linguacustodia.com >>>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com >>>> <http://www.thetranslationtrustee.com>* >>>> >>>> ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. >>>> >>>> Please do not print this email unless it is absolutely necessary. Spread >>>> environmental awareness. >>>> >>> >>> >>> >>> -- >>> Hieu Hoang >>> Research Associate >>> University of Edinburgh >>> http://www.hoang.co.uk/hieu >>> >>> >> >> >> -- >> >> [image: Description : Description : lingua_custodia_final full logo] >> >> *The Translation Trustee* >> >> *1, Place Charles de Gaulle* >> >> *78180 Montigny-le-Bretonneux* >> >> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* >> >> *Email :* *[email protected] >> <[email protected]>* >> >> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> - >> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>* >> >> ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. >> >> Please do not print this email unless it is absolutely necessary. Spread >> environmental awareness. >> > > > > -- > Hieu Hoang > Research Associate > University of Edinburgh > http://www.hoang.co.uk/hieu > > -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *[email protected] <[email protected]>* *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> - www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
<<inline: image001.jpg>>
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
