Hi Per I would suggest starting from a smallish corpus, then building up to a larger one, to get experience with the process. Using the news commentary corpus described in the Moses baseline page, I was able to train and tune in an evening on my laptop.
There have been papers on predicting quality given corpus size, but there's not an easy answer. Look for Marco Turchi at last year's EAMT, or (I think) one by Xerox Grenoble from last year. As regards europarl, yes there's noise, but the models are quite robust to it Cheers - Barry Per Tunedal <[email protected]> wrote: >Hi, >Is there any way to predict the time for training and/or tuning, given >the corpus size and the computer specifications? It would be nice to >know what would be a reasonable time for accomplishing the tasks. Now >my computer has been running for 3 days and nights and doesn't respond >any more: I cannot "wake it" to see what's going on. I don't know if >it's normal or if something has gone havoc. > >I agree with Ken Fasano, that it would be very useful to know how big a >corpus is needed to get meaningful results. I would like to be able to >judge the quality of the translation, to see if it would be useful to >continue with Moses in some more serious manner. > >I'm a bit puzzled by the parameter limiting sentence length to, say 80 >(characters?), giving that e.g. the Europarl corpus contains mainly VERY >long sentences. Skipping long sentences probably implicates that many >typical expressions are lost in the model. Wouldn't it be more sensible >to skip short sentences? Or to make a representative sample of the >corpus, by doing a random sample of a sufficient size or something? >Yours, >Per Tunedal > >PS I've noticed that the Europarl corpus contains some very bad, >completely incomprehensible, translations. That makes me question the >quality of that corpus. How are the translations actually done? By >humans relying heavily on machine translation? Sometimes letting some >strange MT-translation pass? > >On Sat, Mar 9, 2013, at 18:43, Ken Fasano wrote: >> I'd like to respond to this thread. I, too, have limited resources (at >> work, at least) - a 3 GB RAM 32-bit Windows i5 machine with Linux running >> on VMWare with 2.5 GB RAM. Training and tuning take many hours; the >> machine is running BitParl over the weekend and may be done with >> NewsCommentary de-en (DE) on Monday when I get back to work. I'm afraid >> after all that I won't be able to subsequently run Collins on the >> English, train, tune, and decode all that and, even if it takes forever, >> expect it to run with the limited memory resources available. >> What I think I need to do is trim the corpora according to some criteria >> that isn't too complicated. Is it enough for learning purposes (we are a >> long way from any sort of real comparison of results, let alone >> production - this will receive proper hardware) to take the first n >> sentences, or every nth sentence? The result is simply to get a feel for >> the various modes of tree-based SMT, run hierarchical phrase, >> string-to-tree, tree-to-string and tree-to-tree without worrying which >> one is the best - the idea is just to get some experience with it.What >> would be a good number of sentences to take so that it runs relatively >> quickly, without killing RAM, but produces results that aren't useless? >> Thanks - and I'd like to thank everyone on the group for their eager >> helpfulness, and for discussing things that I as a newbie find very >> useful! >> >> >> >> >> > Date: Sat, 9 Mar 2013 11:25:45 -0500 >> > From: [email protected] >> > To: [email protected] >> > Subject: Re: [Moses-support] Accelerate the tuning >> > >> > Hi, >> > >> > It won't fix everything, but there is a long-term TODO to rewrite >> > phrase table scoring to use binary files with vocabulary ids instead of >> > text files. >> > >> > Kenneth >> > >> > On 03/09/13 08:30, Per Tunedal wrote: >> > > >> > > Hi, >> > > the training seems to be an overwhelming task for my computer. If it >> > > ever succeeds, I will have to undertake the even more demanding task of >> > > tuning. Can anything be done to accelerate it? >> > > >> > > Specifically, I wonder if it's feasible to prune the translation table >> > > before doing the tuning. >> > > >> > > Yours, >> > > Per Tunedal >> > > >> > > PS I've abandoned the idea of building a Hierarchical phrase model, I'm >> > > now trying to make a phrase-based system. I suppose that would use less >> > > resources. >> > > >> > > _______________________________________________ >> > > Moses-support mailing list >> > > [email protected] >> > > http://mailman.mit.edu/mailman/listinfo/moses-support >> > > >> > _______________________________________________ >> > Moses-support mailing list >> > [email protected] >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >_______________________________________________ >Moses-support mailing list >[email protected] >http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
