I recall Barry working on the LM server at the Dublin MT Marathon. What is the current status of that?
On Wed, Feb 2, 2011 at 3:29 AM, Miles Osborne <[email protected]> wrote: > to add to Barry's excellent answer, we are currently working on a > client-server language model. this will mean that a cluster of > machines can be used, with a shared resource. it should also work > with multicore > > but in the short-term, you are probably better off with multicore > > Miles > > On 2 February 2011 06:06, Noubours, Sandra > <[email protected]> wrote: > > Hello Barry, hello Tom, > > > > thank you for your answers. I think I have a better idea about different > approaches to MOSES efficiency issues now. > > > > Best regards, > > Sandra > > > > -----Ursprüngliche Nachricht----- > > Von: Barry Haddow [mailto:[email protected]] > > Gesendet: Montag, 31. Januar 2011 10:52 > > An: [email protected] > > Cc: Noubours, Sandra; Tom Hoar > > Betreff: Re: [Moses-support] running moses on a cluster with sge > > > > Hi Sandra > > > > The short answer is that it really depends how big your models are. > Running on > > a cluster helps speed up tuning because most of the time in tuning is > spent > > decoding, which can be easily parallelised by splitting up the file into > > chunks. So each of the individual machines should be capable of loading > your > > models and running a decoder. > > > > The problem with using a cluster (as opposed to multicore) is that each > > machine has to have its own ram, and if you want to load large models > then > > you need a lot of ram. Whereas with multicore, each thread can access the > > same model. Sure, binarising saves a lot on ram usage, but it slows you > down > > and puts a lot of load on the filesystem which can cause problems on > > clusters. > > > > Our group's machines are a mixture of 8 and 16 core Xeon 2.67GHz, with > 36-72G > > ram, no sge. We also have access to the university cluster, but since the > > most ram you can get is 16G and sge hold jobs don't work at the moment we > > don't really use it for moses any more, > > > > hope that helps - regards - Barry > > > > On Monday 31 January 2011 07:42, Noubours, Sandra wrote: > >> Hello, > >> > >> > >> > >> thanks for the tips! When talking about using a Sun Grid Engine I was > >> referring tuning. Making use of a cluster is supposed to speed up the > >> tuning process (see http://www.statmt.org/moses/?n=Moses.FAQ#ntoc10). > In > >> this context I wondered what hardware exactly is needed for such a > cluster. > >> > >> > >> > >> Sandra > >> > >> > >> > >> > >> > >> > >> > >> Von: Tom Hoar [mailto:[email protected]] > >> Gesendet: Freitag, 28. Januar 2011 09:01 > >> An: Noubours, Sandra > >> Cc: [email protected] > >> Betreff: Re: [Moses-support] running moses on a cluster with sge > >> > >> > >> > >> Sandra, > >> > >> What kind of capacity do you need to support? I just finished > translating > >> 21,000 pages, over 1/2 million phrases, in 22 hours on an old Intel > >> Core2Quad, 2.4 Ghz with 4 GB RAM and a 4-disk RAID-0. Moses was > configured > >> with binarized phrase/reordering tables and kenlm binarized language > model. > >> The advances in Moses supporting efficient binarized tables/models are > >> great! > >> > >> We're planning tests for a 2-socket host with two Intel Xeon 5680 6-core > >> 3.33 Ghz CPU's, 48 GB RAM and 4 1-TB disks as RAID0. With 12 cores > >> (totaling 24 simultaneous threads according to Intel specs), we're > >> expecting to boot capacity to well over 15 million phrases per day on > one > >> host. > >> > >> What's the advantage of running Moses on a grid or cluster? > >> > >> Tom > >> > >> > >> > >> On Fri, 28 Jan 2011 08:40:22 +0100, "Noubours, Sandra" > >> <[email protected]> wrote: > >> > >> Hello, > >> > >> > >> > >> I would like to run Moses on a cluster. I am yet inexperienced in > using > >> Sun Grid as well as clusters in common. Could you give me any > instructions > >> or tips for implementing a Linux-Cluster with Sun Grid Engine for > running > >> Moses? > >> > >> a) What kind of cluster would you recommend, i.e. how many > machines, > >> how many cpus, what memory, etc.? > >> > >> b) When tuning is performed with the multicore option it does > not use > >> more than one cpu. Does the tuning step use more than one cpu when run > on a > >> cluster? > >> > >> c) Can Sun Grid implement a cluster virtually on one > computer, so > >> that jobs are spread locally to different cpus of one computer? > >> > >> > >> > >> Thank you and best regards! > >> > >> > >> > >> Sandra > > > > -- > > The University of Edinburgh is a charitable body, registered in > > Scotland, with registration number SC005336. > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, "Time Enough For Love"
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
