Hey, I can only answer 5-7 5. The alignment file is the one that's usually called aligned.1.grow-diag-final-and and contains lines such as:
0-0 1-1 2-2 3-3 0-0 1-1 2-2 3-3 4-4 6. Yes. Basically prune vocab value of 16000 would take the 16000 most common words in the corpus and discard the rest (replace them with UNK) 7. Yes Cheers, Nick On Wed, Nov 26, 2014 at 3:44 PM, Tom Hoar < [email protected]> wrote: > Thanks again. It's very useful feedback. We're now preparing to move from > v1.0 to 3.x. We skipped Moses 2.x. So, I'm not familiar with the new > moses.ini syntax. > > Here are some more questions to help us get started playing with the > extract_training.py options: > > 1. I'm assuming corpus.e and corpus.f are the same prepared corpus > files as used in train-model.perl? > 2. Is it possible for corpus.e and corpus.f to be different from the > train-model.perl corpus, for example a smaller random sampling? > 3. The corpus files are tokenized and lower-cased and escaped the > same. > 4. Do the corpus files also need to enforce clean-corpus-n.perl max > tokens (100) and ratio (9:1) for src & tgt? These address (M)GIZA++ limits > and might not apply to BilingualLM. However, are there advantages to using > the limits or disadvantages to overriding them? I.e. can these corpus files > include lines that are filtered with clean-corpus-n.perl? > 5. What is the --align value? Is it the output of train-model.perl > step 3 or an file with word alignments for each line of the corpus.e and > corpus.f pair? > 6. Re --prune-source-vocab & --prune-target-vocab, do these thresholds > set the size of the vocabulary you reference in #4 below (i.e. 16K, 500K, > etc)? > 7. Re --source-context & --target-context, are these the BilingualLM > equivalents to a typical LM's order or ngrams for each? > 8. Re --tagged-corpus, is this for POS factored corpora? > > Thanks. > > > > On 11/26/2014 09:27 PM, Nikolay Bogoychev wrote: > > Hey, Tom > > 1) It's independent. You just add -with-oxlm and -with-nplm to the stack > 2) Yes, they are both thread safe, you can run the decoder with however > many threads you wish. > 3) It doesn't create a separate binary. The compilation flag adds a new > feature inside moses that is called BilingualNPLM and you have to add it to > your moses.ini with a weight. > 4) That depends on the vocabulary size used. With 16k source 16k target > about 100 megabytes. With 500000 about 1.5 gigabytes. > > Beware that the memory requirements during decoding are much larger, > because of premultiplication. If you have memory issues supply > "premultiply=false" to the BilingualNPLM line in moses.ini, but this is > likely going to slow down decoding by a lot. > > > Cheers, > > Nick > > On Wed, Nov 26, 2014 at 2:09 PM, Tom Hoar < > [email protected]> wrote: > >> Thanks Nikolay! This is a great start. I have a few clarification >> questions. >> >> 1) does this replace or run independently of traditional language models >> like KenLM? I.e. when compiling, we can use -with-kenlm, -with-irstlm, >> -with-randlm and -with-srilm together. Are -with-oxlm and -with-nplm added >> to the stack or are they exclusive? >> >> 2) It looks like your branch of nplm is thread-safe. Is oxlm also >> thread-safe? >> >> 3) You say, "To run it in moses as a feature function..." Does that mean >> compiling with your above option(s) creates a new runtime binary " >> BilingualNPLM" that replaces the moses binary, much like moseschart and >> mosesserver? Or, does BilingualNPLM run in a separate process that the >> Moses binary accesses during runtime? >> >> 4) How large do these LM files become? Are they comparable to traditional >> ARPA files, larger or smaller? Also, are they binarized with mmap reads or >> do they have to load into RAM? >> >> Thanks, >> Tom >> >> >> >> >> >> On 11/26/2014 08:04 PM, Nikolay Bogoychev wrote: >> >> Fix formatting... >> >> Hey, >> >> BilingualLM is implemented and as of last week resides within moses >> master: >> https://github.com/moses-smt/mosesdecoder/blob/master/moses/LM/BilingualLM.cpp >> >> To compile it you need a NeuralNetwork backend for it. Currently there >> are two supported: Oxlm and Nplm. Adding a new backend is relatively easy, >> you need to implement the interface as shown here: >> >> https://github.com/moses-smt/mosesdecoder/blob/master/moses/LM/bilingual-lm/BiLM_NPLM.h >> >> To compile with oxlm backend you need to compile moses with the switch >> -with-oxlm=/path/to/oxlm >> To compile with nplm backend you need to compile moses with the switch >> -with-nplm=/path/to/nplm (You need this fork of nplm >> https://github.com/rsennrich/nplm >> >> Unfortunately documentaiton is not yet available so here's a short >> summary how to train a model and use it using, the nplm backend: >> Use the extract training script to prepare aligned bilingual corpus: >> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/training/bilingual-lm/extract_training.py >> >> You need the following options: >> >> "-e", "--target-language", type="string", dest="target_language") >> //Mandatory, for example es >> "-f", "--source-language", type="string", dest="source_language") >> //Mandatory, for example en >> "-c", "--corpus", type="string", dest="corpus_stem") // path/to/corpus In >> the directory you have specified there should be files corpus.sourcelang >> and corpus.targetlang >> "-t", "--tagged-corpus", type="string", dest="tagged_stem") //Optional >> for backoff to pos tag >> "-a", "--align", type="string", dest="align_file") //Mandatory alignment >> file >> "-w", "--working-dir", type="string", dest="working_dir") //Output >> directory of the model >> "-n", "--target-context", type="int", dest="n") / >> "-m", "--source-context", type="int", dest="m") //The actual context size >> is 2*m + 1, this is the number of words on both left and right >> "-s", "--prune-source-vocab", type="int", dest="sprune") //cutoff >> vocabulary threshold >> "-p", "--prune-target-vocab", type="int", dest="tprune") //cutoff >> vocabulary threshold >> >> >> Then, use the training script to train the model: >> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/training/bilingual-lm/train_nplm.py >> >> Example execution is: >> >> train_nplm.py -w de-en-500250source/ -r de-en150nopos-source750 -n 16 >> -d 0 --nplm-home=/home/abmayne/code/deepathon/nplm_one_layer/ -c >> corpus.1.word -i 750 -o 750 >> >> where -i and -o are input and output embeddings >> -n is the total ngram size >> -d is the number of hidden layyers >> -w and -c are the same as the extract_training options >> -r is the output directory of the model >> >> Consult the python script for more detailed description of the options >> >> After you have done that in the output directory you should have a >> trained bilingual Neural Network language model >> >> To run it in moses as a feature function you need the following line: >> >> BilingualNPLM >> filepath=/mnt/gna0/nbogoych/new_nplm_german/de-en150nopos/train.10k.model.nplm.10 >> target_ngrams=4 source_ngrams=9 source_vocab=/mnt/gna0/ >> nbogoych/new_nplm_german/de-enIWSLTnopos/vocab.source >> target_vocab=/mnt/gna0/nbogoych/new_nplm_german/de- >> enIWSLTnopos/vocab.targe >> >> The source and target vocab is located in the working directory used to >> prepare the neural network language model. >> target_ngrams doesn't include the predicted word (so target_ngrams = 4, >> would mean 1 word predicted and 4 target context word) >> The total of the model would target_ngrams + source_ngrams + 1) >> >> I will write a proper documentation in the following weeks. If you have >> any problems runnning it, please consult me. >> >> Cheers, >> >> Nick >> >> >> On Wed, Nov 26, 2014 at 1:02 PM, Nikolay Bogoychev <[email protected]> >> wrote: >> >>> Hey, >>> >>> BilingualLM is implemented and as of last week resides within moses >>> master: >>> https://github.com/moses-smt/mosesdecoder/blob/master/moses/LM/BilingualLM.cpp >>> >>> To compile it you need a NeuralNetwork backend for it. Currently there >>> are two supported: Oxlm and Nplm. Adding a new backend is relatively easy, >>> you need to implement the interface as shown here: >>> >>> https://github.com/moses-smt/mosesdecoder/blob/master/moses/LM/bilingual-lm/BiLM_NPLM.h >>> >>> To compile with oxlm backend you need to compile moses with the switch >>> -with-oxlm=/path/to/oxlm >>> To compile with nplm backend you need to compile moses with the switch >>> -with-nplm=/path/to/nplm (You need this fork of nplm >>> https://github.com/rsennrich/nplm >>> >>> Unfortunately documentaiton is not yet available so here's a short >>> summary how to train a model and use it using, the nplm backend: >>> Use the extract training script to prepare aligned bilingual corpus: >>> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/training/bilingual-lm/extract_training.py >>> >>> You need the following options: >>> >>> "-e", "--target-language", type="string", dest="target_language") >>> //Mandatory, for example es "-f", "--source-language", type="string", >>> dest="source_language") //Mandatory, for example en "-c", "--corpus", >>> type="string", dest="corpus_stem") // path/to/corpus In the directory you >>> have specified there should be files corpus.sourcelang and >>> corpus.targetlang "-t", "--tagged-corpus", type="string", >>> dest="tagged_stem") //Optional for backoff to pos tag "-a", "--align", >>> type="string", dest="align_file") //Mandatory alignemtn file "-w", >>> "--working-dir", type="string", dest="working_dir") //Output directory of >>> the model "-n", "--target-context", type="int", dest="n") / "-m", >>> "--source-context", type="int", dest="m") //The actual context size is 2*m >>> + 1, this is the number of words on both left and right "-s", >>> "--prune-source-vocab", type="int", dest="sprune") //cutoff vocabulary >>> threshold "-p", "--prune-target-vocab", type="int", dest="tprune") //cutoff >>> vocabulary threshold >>> Then, use the training script to train the model: >>> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/training/bilingual-lm/train_nplm.py >>> >>> Example execution is: train_nplm.py -w de-en-500250source/ -r >>> de-en150nopos-source750 -n 16 -d 0 >>> --nplm-home=/home/abmayne/code/deepathon/nplm_one_layer/ -c corpus.1.word >>> -i 750 -o 750 >>> >>> where -i and -o are input and output embeddings >>> -n is the total ngram size >>> -d is the number of hidden layyers >>> -w and -c are the same as the extract_training options >>> -r is the output directory of the model >>> >>> Consult the python script for more detailed description of the options >>> >>> After you have done that in the output directory you should have a >>> trained bilingual Neural Network language model >>> >>> To run it in moses as a feature function you need the following line: >>> >>> BilingualNPLM >>> filepath=/mnt/gna0/nbogoych/new_nplm_german/de-en150nopos/train.10k.model.nplm.10 >>> target_ngrams=4 source_ngrams=9 >>> source_vocab=/mnt/gna0/nbogoych/new_nplm_german/de-enIWSLTnopos/vocab.sourcetarget_vocab=/mnt/gna0/nbogoych/new_nplm_german/de-enIWSLTnopos/vocab.targe >>> >>> The source and target vocab is located in the working directory used to >>> prepare the neural network language model. >>> target_ngrams doesn't include the predicted word (so target_ngrams = 4, >>> would mean 1 word predicted and 4 target context word) >>> The total of the model would target_ngrams + source_ngrams + 1) >>> >>> I will write a proper documentation in the following weeks. If you have >>> any problems runnning it, please consult me. >>> >>> Cheers, >>> >>> Nick >>> >>> >>> >>> >>> On Wed, Nov 26, 2014 at 11:53 AM, Tom Hoar < >>> [email protected]> wrote: >>> >>>> Hieu, >>>> >>>> Sorry I missed you in Vancouver. I just reviewed your slide deck from >>>> the MosesCore TAUS Round Table in Vancouver >>>> (taus-moses-industry-roundtable-2014-changes-in-moses-hieu-hoang-university-of-edinburgh). >>>> >>>> >>>> In particular, I'm interested in the "Bilingual Language Models" that >>>> "replicate Delvin et al, 2014". A search on statmt.org/moses doesn't >>>> show any hits searching for "delvin". So, A) is the code finished? If so B) >>>> are there any instructions how to enable/use this feature? If not, C) what >>>> kind of help do you need to test the code for release? >>>> >>>> -- >>>> >>>> Best regards, >>>> Tom Hoar >>>> Managing Director >>>> *Precision Translation Tools Co., Ltd.* >>>> Bangkok, Thailand >>>> Web: www.precisiontranslationtools.com >>>> Mobile: +66 87 345-1875 <%2B66%2087%20345-1875> >>>> Skype: tahoar >>>> >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> >>>> >>> >> >> >> _______________________________________________ >> Moses-support mailing >> [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > > > _______________________________________________ > Moses-support mailing > [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
