Hi Barry, it turns out that me too have succeeded to build my model in one day. I forced a restart and checked the log and the working directory. All is fine! The moses.ini file was created only 7 hours after submitting the command to build the model. I don't understand why the computer didn't respond, though.
How does the time to build a model vary with the size of the corpus? Linearly? Or quadratic? Or what? I've now tried to do a test translation, without any tuning. I soon ran out of memory: even the virtual memory was exhausted after a while. Any way to predict the memory needed? Yours, Per Tunedal On Sun, Mar 10, 2013, at 11:39, Barry Haddow wrote: > Hi Per > > I would suggest starting from a smallish corpus, then building up to a > larger one, to get experience with the process. Using the news commentary > corpus described in the Moses baseline page, I was able to train and tune > in an evening on my laptop. > > There have been papers on predicting quality given corpus size, but > there's not an easy answer. Look for Marco Turchi at last year's EAMT, or > (I think) one by Xerox Grenoble from last year. > > As regards europarl, yes there's noise, but the models are quite robust > to it > > Cheers - Barry > > Per Tunedal <[email protected]> wrote: > > >Hi, > >Is there any way to predict the time for training and/or tuning, given > >the corpus size and the computer specifications? It would be nice to > >know what would be a reasonable time for accomplishing the tasks. Now > >my computer has been running for 3 days and nights and doesn't respond > >any more: I cannot "wake it" to see what's going on. I don't know if > >it's normal or if something has gone havoc. > > > >I agree with Ken Fasano, that it would be very useful to know how big a > >corpus is needed to get meaningful results. I would like to be able to > >judge the quality of the translation, to see if it would be useful to > >continue with Moses in some more serious manner. > > > >I'm a bit puzzled by the parameter limiting sentence length to, say 80 > >(characters?), giving that e.g. the Europarl corpus contains mainly VERY > >long sentences. Skipping long sentences probably implicates that many > >typical expressions are lost in the model. Wouldn't it be more sensible > >to skip short sentences? Or to make a representative sample of the > >corpus, by doing a random sample of a sufficient size or something? > >Yours, > >Per Tunedal > > > >PS I've noticed that the Europarl corpus contains some very bad, > >completely incomprehensible, translations. That makes me question the > >quality of that corpus. How are the translations actually done? By > >humans relying heavily on machine translation? Sometimes letting some > >strange MT-translation pass? > > > >On Sat, Mar 9, 2013, at 18:43, Ken Fasano wrote: > >> I'd like to respond to this thread. I, too, have limited resources (at > >> work, at least) - a 3 GB RAM 32-bit Windows i5 machine with Linux running > >> on VMWare with 2.5 GB RAM. Training and tuning take many hours; the > >> machine is running BitParl over the weekend and may be done with > >> NewsCommentary de-en (DE) on Monday when I get back to work. I'm afraid > >> after all that I won't be able to subsequently run Collins on the > >> English, train, tune, and decode all that and, even if it takes forever, > >> expect it to run with the limited memory resources available. > >> What I think I need to do is trim the corpora according to some criteria > >> that isn't too complicated. Is it enough for learning purposes (we are a > >> long way from any sort of real comparison of results, let alone > >> production - this will receive proper hardware) to take the first n > >> sentences, or every nth sentence? The result is simply to get a feel for > >> the various modes of tree-based SMT, run hierarchical phrase, > >> string-to-tree, tree-to-string and tree-to-tree without worrying which > >> one is the best - the idea is just to get some experience with it.What > >> would be a good number of sentences to take so that it runs relatively > >> quickly, without killing RAM, but produces results that aren't useless? > >> Thanks - and I'd like to thank everyone on the group for their eager > >> helpfulness, and for discussing things that I as a newbie find very > >> useful! > >> > >> > >> > >> > >> > Date: Sat, 9 Mar 2013 11:25:45 -0500 > >> > From: [email protected] > >> > To: [email protected] > >> > Subject: Re: [Moses-support] Accelerate the tuning > >> > > >> > Hi, > >> > > >> > It won't fix everything, but there is a long-term TODO to rewrite > >> > phrase table scoring to use binary files with vocabulary ids instead of > >> > text files. > >> > > >> > Kenneth > >> > > >> > On 03/09/13 08:30, Per Tunedal wrote: > >> > > > >> > > Hi, > >> > > the training seems to be an overwhelming task for my computer. If it > >> > > ever succeeds, I will have to undertake the even more demanding task of > >> > > tuning. Can anything be done to accelerate it? > >> > > > >> > > Specifically, I wonder if it's feasible to prune the translation table > >> > > before doing the tuning. > >> > > > >> > > Yours, > >> > > Per Tunedal > >> > > > >> > > PS I've abandoned the idea of building a Hierarchical phrase model, I'm > >> > > now trying to make a phrase-based system. I suppose that would use less > >> > > resources. > >> > > > >> > > _______________________________________________ > >> > > Moses-support mailing list > >> > > [email protected] > >> > > http://mailman.mit.edu/mailman/listinfo/moses-support > >> > > > >> > _______________________________________________ > >> > Moses-support mailing list > >> > [email protected] > >> > http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> _______________________________________________ > >> Moses-support mailing list > >> [email protected] > >> http://mailman.mit.edu/mailman/listinfo/moses-support > >_______________________________________________ > >Moses-support mailing list > >[email protected] > >http://mailman.mit.edu/mailman/listinfo/moses-support > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
