Hi Per Yes, the filtering script is "filter-model-given-input.pl" and the binariser (for phrase-based) is "processPhraseTable". So the following addition to the mert-moses.pl command should work:
-filtercmd 'my_path/filter-model-given-input.pl -Binarizer "~/mosesdecoder/bin/processPhraseTable"' (although I would use absolute paths) If you pass this binariser to the filter script, it will actually binarise both the phrase table and reordering table. cheers - Barry On 12/03/13 07:31, Per Tunedal wrote: > Hi Barry, > I'm not sure what's the "filter script" and what's the > "processPhraseTable". I've looked at the page: > http://www.statmt.org/moses/?n=Moses.SupportTools > Is this what you're driving at? > > "Filtering phrase tables for Moses > > Phrase tables easily get too big, but for the translation of a specific > set of text only a fraction of the table is needed. So, you may want to > filter the translation table, and this is possible with the script: > > filter-model-given-input.pl filter-dir config input-file > > This creates a filtered translation table with new configuration file in > the directory filter-dir from the model specified with the configuration > file config (typically named moses.ini), given the (tokenized) input > from the file input-file. > > In the advanced feature section, you find the additional option of > binarizing translation and reordering table, which allows these models > to be kept on disk and queried by the decoder. If you want to both > filter and binarize these tables, you can use the script: > > filter-model-given-input.pl filter-dir config input-file -Binarizer > binarizer > > The additional binarizer option points to the appropriate version of > processPhraseTable. " > > Is this what you mean? > > Should I add something like: > -filtercmd 'my_path/filter-model-given-input.pl' -Binarizer > "~/mosesdecoder/bin/processPhraseTable"' > > But what's the "processPhraseTable"? > > I'm a beginner. Although I've read much of the documentation, I don't > yet understand the details. > > Yours, > Per Tunedal > > On Mon, Mar 11, 2013, at 17:31, Barry Haddow wrote: >> Hi Per >> >> If you add the following to mert-moses.pl then it should work: >> >> -filtercmd '<path-to-filter-script> -Binarizer >> "<path-to-processPhraseTable>"' >> >> Obviously you replace the angle-bracket expressions with the paths, and >> you might have to fiddle with the quotes, >> >> cheers - Barry >> >> On 11/03/13 16:23, Per Tunedal wrote: >>> Hi Barry, >>> unfortunately, I don't fully understand your advice, as I'm not familiar >>> with how the scripts work etc. >>> At this stage I just copy some commands, and make some minor changes I >>> find appropriate. I'm at the script-kiddie stage! >>> >>> 1. What command shall I submit if I will let the mert-moses.pl script do >>> the job, as I first intended? >>> 2. What commands shall I submit if I choose to do some of the job before >>> running the mert-moses.pl script? >>> >>> Yours, >>> Per Tunedal >>> >>> >>> On Mon, Mar 11, 2013, at 16:40, Barry Haddow wrote: >>>> Hi Per >>>> >>>> It's probably as easy to filter and binarise the input yourself before >>>> tuning using the filter-model-given-input.pl script in Moses, and using >>>> the -Binarizer argument to pass it the path to the processPhraseTable >>>> binary. >>>> >>>> By default, mert-moses.pl will filter (so the command below will filter) >>>> and this may reduce the size of the phrase table sufficiently for you. >>>> You can add binarisation to mert-moses.pl by using the --filtercmd to >>>> give it the filter script with its binariser argument, >>>> >>>> cheers - Barry >>>> >>>> On 11/03/13 15:27, Per Tunedal wrote: >>>>> Hi, >>>>> thanks for your kind advice. I plan to use the tuning command displayed >>>>> at the Baseline page: >>>>> nohup nice ~/mosesdecoder/scripts/training/mert-moses.pl >>>>> ~/corpus/news-test2008.true.fr ~/corpus/news-test2008.true.en \ >>>>> ~/mosesdecoder/bin/moses train/model/moses.ini --mertdir >>>>> ~/mosesdecoder/bin/ &> mert.out & >>>>> with my own "small" tuning-corpus (10% of the training-corpus size). >>>>> >>>>> Will this "give [the tuning script] a binariser" ? If not, what shall I >>>>> add to the command? >>>>> >>>>> Yours, >>>>> Per Tunedal >>>>> >>>>> On Mon, Mar 11, 2013, at 16:00, Barry Haddow wrote: >>>>>> Hi Per >>>>>> >>>>>> The tuning script will filter the phrase table (leaving only the entries >>>>>> required for the tuning set) and then binarise it (if you give it a >>>>>> binariser) before running the actual tuning. So no, the whole table >>>>>> doesn't need to be loaded into memory during tuning. >>>>>> >>>>>> You could prune before tuning, and I don't know how this will compare to >>>>>> pruning after tuning. I'm not sure if anyone has tried it. The >>>>>> signifcance filtering (in advanced features) works (afaik), although >>>>>> there are a few steps involved in building the code. The relent >>>>>> filtering has bit-rotted a bit, but you can run it with Moses v0.91. >>>>>> >>>>>> cheers - Barry >>>>>> >>>>>> On 11/03/13 14:37, Per Tunedal wrote: >>>>>>> Hi Barry, >>>>>>> Binarise, yes. But before that I plan to prune the translation table >>>>>>> (Advanced features). Any hints? >>>>>>> >>>>>>> Back to my original question, can pruning be done before tuning? Is it >>>>>>> possible to binarise too, before tuning? (The base-line page suggests >>>>>>> binarisiation after tuning.) I fear that the tuning might be an >>>>>>> overwhelming task for my poor computer. >>>>>>> >>>>>>> Yours, >>>>>>> Per Tunedal >>>>>>> >>>>>>> BTW What is actually done when tuning? Has all the tables to be loaded >>>>>>> into memory? >>>>>>> >>>>>>> On Mon, Mar 11, 2013, at 10:42, Barry Haddow wrote: >>>>>>>> Hi Per >>>>>>>> >>>>>>>> You need to binarise the models (phrase table, reordering table and >>>>>>>> language model) before running Moses >>>>>>>> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc3 >>>>>>>> If you don't binarise then Moses will load all the tables into memory, >>>>>>>> so the memory requirement will be at least as large as the on-disk >>>>>>>> size, >>>>>>>> in fact a lot more since it doesn't store them efficiently. >>>>>>>> >>>>>>>> The complexity of training is not easy to calculate since there are a >>>>>>>> number of steps, but since one step involves sorting the list of >>>>>>>> extracted phrases the complexity must be at least as bad as that. >>>>>>>> >>>>>>>> cheers - Barry >>>>>>>> >>>>>>>> On 11/03/13 08:24, Per Tunedal wrote: >>>>>>>>> Hi Barry, >>>>>>>>> it turns out that me too have succeeded to build my model in one day. >>>>>>>>> I >>>>>>>>> forced a restart and checked the log and the working directory. All is >>>>>>>>> fine! The moses.ini file was created only 7 hours after submitting the >>>>>>>>> command to build the model. I don't understand why the computer didn't >>>>>>>>> respond, though. >>>>>>>>> >>>>>>>>> How does the time to build a model vary with the size of the corpus? >>>>>>>>> Linearly? Or quadratic? Or what? >>>>>>>>> >>>>>>>>> I've now tried to do a test translation, without any tuning. I soon >>>>>>>>> ran >>>>>>>>> out of memory: even the virtual memory was exhausted after a while. >>>>>>>>> Any >>>>>>>>> way to predict the memory needed? >>>>>>>>> >>>>>>>>> Yours, >>>>>>>>> Per Tunedal >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Mar 10, 2013, at 11:39, Barry Haddow wrote: >>>>>>>>>> Hi Per >>>>>>>>>> >>>>>>>>>> I would suggest starting from a smallish corpus, then building up to >>>>>>>>>> a >>>>>>>>>> larger one, to get experience with the process. Using the news >>>>>>>>>> commentary >>>>>>>>>> corpus described in the Moses baseline page, I was able to train and >>>>>>>>>> tune >>>>>>>>>> in an evening on my laptop. >>>>>>>>>> >>>>>>>>>> There have been papers on predicting quality given corpus size, but >>>>>>>>>> there's not an easy answer. Look for Marco Turchi at last year's >>>>>>>>>> EAMT, or >>>>>>>>>> (I think) one by Xerox Grenoble from last year. >>>>>>>>>> >>>>>>>>>> As regards europarl, yes there's noise, but the models are quite >>>>>>>>>> robust >>>>>>>>>> to it >>>>>>>>>> >>>>>>>>>> Cheers - Barry >>>>>>>>>> >>>>>>>>>> Per Tunedal <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Is there any way to predict the time for training and/or tuning, >>>>>>>>>>> given >>>>>>>>>>> the corpus size and the computer specifications? It would be nice to >>>>>>>>>>> know what would be a reasonable time for accomplishing the tasks. >>>>>>>>>>> Now >>>>>>>>>>> my computer has been running for 3 days and nights and doesn't >>>>>>>>>>> respond >>>>>>>>>>> any more: I cannot "wake it" to see what's going on. I don't know if >>>>>>>>>>> it's normal or if something has gone havoc. >>>>>>>>>>> >>>>>>>>>>> I agree with Ken Fasano, that it would be very useful to know how >>>>>>>>>>> big a >>>>>>>>>>> corpus is needed to get meaningful results. I would like to be able >>>>>>>>>>> to >>>>>>>>>>> judge the quality of the translation, to see if it would be useful >>>>>>>>>>> to >>>>>>>>>>> continue with Moses in some more serious manner. >>>>>>>>>>> >>>>>>>>>>> I'm a bit puzzled by the parameter limiting sentence length to, say >>>>>>>>>>> 80 >>>>>>>>>>> (characters?), giving that e.g. the Europarl corpus contains mainly >>>>>>>>>>> VERY >>>>>>>>>>> long sentences. Skipping long sentences probably implicates that >>>>>>>>>>> many >>>>>>>>>>> typical expressions are lost in the model. Wouldn't it be more >>>>>>>>>>> sensible >>>>>>>>>>> to skip short sentences? Or to make a representative sample of the >>>>>>>>>>> corpus, by doing a random sample of a sufficient size or something? >>>>>>>>>>> Yours, >>>>>>>>>>> Per Tunedal >>>>>>>>>>> >>>>>>>>>>> PS I've noticed that the Europarl corpus contains some very bad, >>>>>>>>>>> completely incomprehensible, translations. That makes me question >>>>>>>>>>> the >>>>>>>>>>> quality of that corpus. How are the translations actually done? By >>>>>>>>>>> humans relying heavily on machine translation? Sometimes letting >>>>>>>>>>> some >>>>>>>>>>> strange MT-translation pass? >>>>>>>>>>> >>>>>>>>>>> On Sat, Mar 9, 2013, at 18:43, Ken Fasano wrote: >>>>>>>>>>>> I'd like to respond to this thread. I, too, have limited resources >>>>>>>>>>>> (at >>>>>>>>>>>> work, at least) - a 3 GB RAM 32-bit Windows i5 machine with Linux >>>>>>>>>>>> running >>>>>>>>>>>> on VMWare with 2.5 GB RAM. Training and tuning take many hours; the >>>>>>>>>>>> machine is running BitParl over the weekend and may be done with >>>>>>>>>>>> NewsCommentary de-en (DE) on Monday when I get back to work. I'm >>>>>>>>>>>> afraid >>>>>>>>>>>> after all that I won't be able to subsequently run Collins on the >>>>>>>>>>>> English, train, tune, and decode all that and, even if it takes >>>>>>>>>>>> forever, >>>>>>>>>>>> expect it to run with the limited memory resources available. >>>>>>>>>>>> What I think I need to do is trim the corpora according to some >>>>>>>>>>>> criteria >>>>>>>>>>>> that isn't too complicated. Is it enough for learning purposes (we >>>>>>>>>>>> are a >>>>>>>>>>>> long way from any sort of real comparison of results, let alone >>>>>>>>>>>> production - this will receive proper hardware) to take the first n >>>>>>>>>>>> sentences, or every nth sentence? The result is simply to get a >>>>>>>>>>>> feel for >>>>>>>>>>>> the various modes of tree-based SMT, run hierarchical phrase, >>>>>>>>>>>> string-to-tree, tree-to-string and tree-to-tree without worrying >>>>>>>>>>>> which >>>>>>>>>>>> one is the best - the idea is just to get some experience with >>>>>>>>>>>> it.What >>>>>>>>>>>> would be a good number of sentences to take so that it runs >>>>>>>>>>>> relatively >>>>>>>>>>>> quickly, without killing RAM, but produces results that aren't >>>>>>>>>>>> useless? >>>>>>>>>>>> Thanks - and I'd like to thank everyone on the group for their >>>>>>>>>>>> eager >>>>>>>>>>>> helpfulness, and for discussing things that I as a newbie find very >>>>>>>>>>>> useful! >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Date: Sat, 9 Mar 2013 11:25:45 -0500 >>>>>>>>>>>>> From: [email protected] >>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>> Subject: Re: [Moses-support] Accelerate the tuning >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> It won't fix everything, but there is a long-term TODO to >>>>>>>>>>>>> rewrite >>>>>>>>>>>>> phrase table scoring to use binary files with vocabulary ids >>>>>>>>>>>>> instead of >>>>>>>>>>>>> text files. >>>>>>>>>>>>> >>>>>>>>>>>>> Kenneth >>>>>>>>>>>>> >>>>>>>>>>>>> On 03/09/13 08:30, Per Tunedal wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> the training seems to be an overwhelming task for my computer. >>>>>>>>>>>>>> If it >>>>>>>>>>>>>> ever succeeds, I will have to undertake the even more demanding >>>>>>>>>>>>>> task of >>>>>>>>>>>>>> tuning. Can anything be done to accelerate it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Specifically, I wonder if it's feasible to prune the translation >>>>>>>>>>>>>> table >>>>>>>>>>>>>> before doing the tuning. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yours, >>>>>>>>>>>>>> Per Tunedal >>>>>>>>>>>>>> >>>>>>>>>>>>>> PS I've abandoned the idea of building a Hierarchical phrase >>>>>>>>>>>>>> model, I'm >>>>>>>>>>>>>> now trying to make a phrase-based system. I suppose that would >>>>>>>>>>>>>> use less >>>>>>>>>>>>>> resources. >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Moses-support mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Moses-support mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Moses-support mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Moses-support mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> The University of Edinburgh is a charitable body, registered in >>>>>>>>>> Scotland, with registration number SC005336. >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Moses-support mailing list >>>>>>>>> [email protected] >>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>> >>>>>>>> -- >>>>>>>> The University of Edinburgh is a charitable body, registered in >>>>>>>> Scotland, with registration number SC005336. >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Moses-support mailing list >>>>>>> [email protected] >>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>> >>>>>> -- >>>>>> The University of Edinburgh is a charitable body, registered in >>>>>> Scotland, with registration number SC005336. >>>>>> >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>> >>>> -- >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
