ah yes, change 6 to 2, but only for the rule-table, not glue rule
On 22 March 2014 18:35, Kenneth Heafield <[email protected]> wrote: > Hi, > > The error message said > > /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table > > is a directory. But IIRC phrase table 6 is "SCFG" and expects a file. > If you have binarized the phrase table specify a different magic number > in the moses.ini. > > Nothing to do with the language model but, for the record, SRILM > does > have a binary format. Just not a memory mapped binary format. > > Kenneth > > On 03/22/14 04:24, Massinissa Ahmim wrote: > > Hi Hieu, > > > > It's actually KenLM I've just updated my moses.ini changing the 0 to 8 > > but I'm still getting the same error > > > > Regards > > > > Massinissa > > > > > > 2014-03-22 11:40 GMT+01:00 Hieu Hoang <[email protected] > > <mailto:[email protected]>>: > > > > did you binarize the LM and what did you binarize it with? KenLM or > > IRSTLM? > > > > at the moment, it's set to SRILM, which doesn't have a binary format > > > > > > > > On 22 March 2014 10:10, Massinissa Ahmim > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Hieu, > > > > I used the same binarisation command as the one stated here-above > > > > I get this error when I try to translate something like : echo > > 'ceci est un test' | /mosesdecoder/bin/moses_chart -f moses.ini > > > > Here is my moses.ini : > > > > ######################### > > ### MOSES CONFIG FILE ### > > ######################### > > > > # input factors > > [input-factors] > > 0 > > > > # mapping steps > > [mapping] > > 0 T 0 > > 1 T 1 > > > > # translation tables: table type (hierarchical(0), textual (0), > > binary (1)), source-factors, target-factors, number of scores, > file > > # OLD FORMAT is still handled for back-compatibility > > # OLD FORMAT translation tables: source-factors, target-factors, > > number of scores, file > > # OLD FORMAT a binary table type (1) is assumed > > [ttable-file] > > 6 0 0 1 > /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table > > 6 0 0 1 > > > /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar > > > > # no generation models, no generation-file section > > > > # language models: type(srilm/irstlm), factors, order, file > > [lmodel-file] > > 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm > > > > > > # limit on how many phrase translations e for each phrase f are > > loaded > > # 0 = all elements loaded > > [ttable-limit] > > 20 > > > > > > # language model weights > > [weight-l] > > 0.5000 > > > > > > # translation model weights > > [weight-t] > > 0.20 > > 0.20 > > 0.20 > > 0.20 > > 0.20 > > 1.0 > > > > # no generation models, no weight-generation section > > > > # word penalty > > [weight-w] > > -1 > > > > [cube-pruning-pop-limit] > > 1000 > > > > [non-terminals] > > X > > > > [search-algorithm] > > 3 > > > > [inputtype] > > 3 > > > > [max-chart-span] > > 20 > > 1000 > > > > > > > > 2014-03-21 20:49 GMT+01:00 Hieu Hoang <[email protected] > > <mailto:[email protected]>>: > > > > i've never seen that error before. Did you change anything > > else, other than rebinarizing the phrase-table? What is the > > exact command you ran when you got this error? Can I have a > > look at your moses.ini file > > > > > > On 21 March 2014 18:32, Massinissa Ahmim > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Thank you Hieu, > > > > I've binarised my rule-table again and it seems to be > > successfully loaded but I'm getting this message now : > > > > Reading > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table > > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > > > **************************************************************************************************** > > Exception: <fd:3>: Is a directory from zlib > > > > What can I do? > > > > Thanks ! > > > > Regards > > > > Massinissa > > > > > > 2014-03-21 11:18 GMT+01:00 Hieu Hoang > > <[email protected] <mailto:[email protected]>>: > > > > everything seems to be ok. Are you sure the server > > you are running on has access to the directory > > > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > > > Also, you should consider updating your version of > > Moses to the current github version, or version 2.1.1 > > > > > > On 20 March 2014 11:20, Massinissa Ahmim > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Dear all, > > > > I've managed to train a hierarchical model using > > the following command : > > > > nohup > > /mosesdecoder/scripts/training/train-model.perl > > --hierarchical --glue-grammar > > --score-options="--GoodTuring" -root-dir > > /disque2/Preparation/syntactic/hierarchical_PSCT > > -corpus > > /disque2/Preparation/backoff/PSCT.tok.uni.low > > -f fr -e en -lm > > > 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm > > -external-bin-dir /root/external-bin-dir/ -mgiza > > -mgiza-cpus 30 >& training.out & > > > > The resulting model works great but it is very > > slow. > > > > So I used CreatOnDiskPt to binarise the rule > > table as follows : > > > > /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 > > 5 20 2 model/rule-table.gz rules-table > > > > This outputs the following files > > > > -rw-r--r-- 1 root root 85 20 mars 11:35 > > Misc.dat > > -rw-r--r-- 1 root root 781509322 20 mars 11:35 > > Source.dat > > -rw-r--r-- 1 root root 1758588137 20 mars 11:35 > > TargetColl.dat > > -rw-r--r-- 1 root root 1555353459 20 mars 11:35 > > TargetInd.dat > > -rw-r--r-- 1 root root 400714 20 mars 11:35 > > Vocab.dat > > > > and I've updated my moses.ini as follows : > > > > 6 0 0 1 > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > 6 0 0 1 > > > /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar > > > > But as I try to use it, I'm getting this : > > > > Defined parameters (per moses.ini or switch): > > config: moses.ini > > cube-pruning-pop-limit: 1000 > > input-factors: 0 > > inputtype: 3 > > lmodel-file: 0 0 5 > > > /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm > > mapping: 0 T 0 1 T 1 > > max-chart-span: 20 1000 > > non-terminals: X > > search-algorithm: 3 > > ttable-file: 6 0 0 1 > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > 6 0 0 1 > > > /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar > > ttable-limit: 20 > > weight-l: 0.5000 > > weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 > > weight-w: -1 > > /mosesdecoder/bin > > ScoreProducer: WordPenalty start: 0 end: 1 > > ScoreProducer: !UnknownWordPenalty start: 1 end: > 2 > > Loading lexical distortion models...have 0 models > > Start loading LanguageModel > > > /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm > > : [0.000] seconds > > > /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: > > line 80492: reached EOF before \end\ > > ScoreProducer: LM start: 2 end: 3 > > Finished loading LanguageModels : [0.056] seconds > > Using uniform ttable-limit of 20 for all > > translation tables. > > Start loading PhraseTable > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > : [0.056] seconds > > filePath: > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > ScoreProducer: PhraseModel start: 3 end: 4 > > Start loading PhraseTable > > > /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar > > : [0.056] seconds > > filePath: > > > /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar > > ScoreProducer: PhraseModel:2 start: 4 end: 5 > > Finished loading phrase tables : [0.056] seconds > > max-chart-span: 20 > > max-chart-span: 1000 > > Start loading phrase table from > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > : [0.056] seconds > > Can't read > > > /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table > > > > So I wondering if I did something wrong with my > > training command/binarisation or with the > > parameters in the moses.ini > > > > > > Many thanks > > > > Regads > > > > MA > > > > > > > > > > -- > > > > Description : Description : > > lingua_custodia_final full logo > > > > */The Translation Trustee/* > > > > *1, Place Charles de Gaulle* > > > > *78180 Montigny-le-Bretonneux* > > > > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 > > 40 84* > > > > *Email :** > > **[email protected] > > <mailto:[email protected]>*** > > > > *Website :** **www.linguacustodia.com > > <http://www.linguacustodia.com/> - > > www.thetranslationtrustee.com > > <http://www.thetranslationtrustee.com>* > > > > ü Pensez à l'environnement, n'imprimez ce > > courriel que si nécessaire. > > > > Please do not print this email unless it is > > absolutely necessary. Spread environmental > > awareness. > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] <mailto: > [email protected]> > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > > > > -- > > Hieu Hoang > > Research Associate > > University of Edinburgh > > http://www.hoang.co.uk/hieu > > > > > > > > > > -- > > > > Description : Description : lingua_custodia_final full > logo > > > > */The Translation Trustee/* > > > > *1, Place Charles de Gaulle* > > > > *78180 Montigny-le-Bretonneux* > > > > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* > > > > *Email :** **[email protected] > > <mailto:[email protected]>*** > > > > *Website :** **www.linguacustodia.com > > <http://www.linguacustodia.com/> - > > www.thetranslationtrustee.com > > <http://www.thetranslationtrustee.com>* > > > > ü Pensez à l'environnement, n'imprimez ce courriel que > > si nécessaire. > > > > Please do not print this email unless it is absolutely > > necessary. Spread environmental awareness. > > > > > > > > > > -- > > Hieu Hoang > > Research Associate > > University of Edinburgh > > http://www.hoang.co.uk/hieu > > > > > > > > > > -- > > > > Description : Description : lingua_custodia_final full logo > > > > */The Translation Trustee/* > > > > *1, Place Charles de Gaulle* > > > > *78180 Montigny-le-Bretonneux* > > > > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* > > > > *Email :** **[email protected] > > <mailto:[email protected]>*** > > > > *Website :** **www.linguacustodia.com > > <http://www.linguacustodia.com/> - > > www.thetranslationtrustee.com > > <http://www.thetranslationtrustee.com>* > > > > ü Pensez à l'environnement, n'imprimez ce courriel que si > > nécessaire. > > > > Please do not print this email unless it is absolutely > > necessary. Spread environmental awareness. > > > > > > > > > > -- > > Hieu Hoang > > Research Associate > > University of Edinburgh > > http://www.hoang.co.uk/hieu > > > > > > > > > > -- > > > > Description : Description : lingua_custodia_final full logo > > > > */The Translation Trustee/* > > > > *1, Place Charles de Gaulle* > > > > *78180 Montigny-le-Bretonneux* > > > > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* > > > > *Email :** **[email protected] > > <mailto:[email protected]>*** > > > > *Website :** **www.linguacustodia.com > > <http://www.linguacustodia.com/> - www.thetranslationtrustee.com > > <http://www.thetranslationtrustee.com>* > > > > ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. > > > > Please do not print this email unless it is absolutely necessary. Spread > > environmental awareness. > > > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
