Thanks, Sandra. On 18 November 2011 07:03, Noubours, Sandra < [email protected]> wrote:
> Hello, > > I run moses (an October 2011 distribution) with SGE (using the ems). To > get moses running on my SGE cluster I had to do some changes. I am not sure > whether all of them have been necessary, but maybe you want to try editing > some of these lines: > > > [Correcting the order of qsub commands] > experiment.perl (somewhere around line 1030): > > #`qsub $qsub_args -e $step.STDERR $step -o $step.STDOUT`; > `qsub $qsub_args -e $step.STDERR -o $step.STDOUT $step`; # new > > > > [Correcting regular expression for splitting] > moses-parallel.perl (somewhere around line 457): > #split(/\s+/,$res); > #my $id=$_[2]; > my @IdArray = split(/[\s+]| /,$res); # new > my $id=$IdArray[2]; # new > > qsub-wrapper.perl (somewhere around line 151): > #split(/\s+/,$res); > #my $id=$_[2]; > my @IdArray = split(/[\s+]| /,$res); # new > my $id=$IdArray[2]; # new > > > I hope this is of some help. > > Best regards, > > Sandra > > > > > > -----Ursprüngliche Nachricht----- > Von: [email protected] [mailto:[email protected]] > Im Auftrag von Lane Schwartz > Gesendet: Mittwoch, 16. November 2011 16:11 > An: Guchun Zhang > Cc: Tom Hoar; [email protected] > Betreff: Re: [Moses-support] Tune on SGE II > > I definitely make use of SGE when I run my experiments. I had forgotten > Barry's post. I haven't updated my code in a while, but if SGE is currently > broken, I'll definitely want to help fix it. I won't be able to look at > things until at least December, though, and more likely January. > > Lane > > > On Wed, Nov 16, 2011 at 10:08 AM, Guchun Zhang <[email protected]> > wrote: > > Hi Tom, > > Thanks for Barry's post back in January. I wasn't a member at that > > time, so didn't receive such a post. My motives will be mostly because > > of cost. With SGE, I can reuse some of the old but still usable > > machines and expand the network when necessary. Plus, it is quite > > unlikely I will be able to get hold of machines as powerful as yours. > > So if I want to have a relatively faster Moses system, SGE is a pretty > good option for me. > > Hope this makes sense. > > Cheers, > > Guchun > > > > On 16 November 2011 12:27, Tom Hoar > > <[email protected]> > > wrote: > >> > >> Hi Guchun, > >> > >> May I ask what motivates you to work with SGE? Barry Haddow posted > >> this comment on the list in January this year. We have similar > experiences. > >> > >> Tom > >> > >> -------- Original Message -------- > >> Subject: Re: [Moses-support] running moses on a cluster with sge > >> Date: Mon, 31 Jan 2011 09:52:11 +0000 > >> From: Barry Haddow <[email protected]> > >> To: [email protected] > >> Cc: Tom Hoar <[email protected]>, "Noubours, Sandra" > >> <[email protected]> > >> > >> Hi Sandra > >> > >> The short answer is that it really depends how big your models are. > >> Running on a cluster helps speed up tuning because most of the time > >> in tuning is spent decoding, which can be easily parallelised by > >> splitting up the file into chunks. So each of the individual machines > >> should be capable of loading your models and running a decoder. > >> > >> The problem with using a cluster (as opposed to multicore) is that > >> each machine has to have its own ram, and if you want to load large > >> models then you need a lot of ram. Whereas with multicore, each > >> thread can access the same model. Sure, binarising saves a lot on ram > >> usage, but it slows you down and puts a lot of load on the filesystem > >> which can cause problems on clusters. > >> > >> Our group's machines are a mixture of 8 and 16 core Xeon 2.67GHz, > >> with 36-72G ram, no sge. We also have access to the university > >> cluster, but since the most ram you can get is 16G and sge hold jobs > >> don't work at the moment we don't really use it for moses any more, > >> > >> hope that helps - regards - Barry > >> > >> > >> > >> On Tue, 15 Nov 2011 22:02:48 +0000, Guchun Zhang > >> <[email protected]> > >> wrote: > >> > >> Sorry, forgot to attach the mert.out file. > >> > >> Guchun > > > > > > -- > > > > Guchun Zhang > > > > Localization Engineer > > Alpha CRC Ltd | Cambridge, UK > > Direct: +44 1223 431035 > > [email protected] > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- *Guchun Zhang* Localization Engineer Alpha CRC Ltd | Cambridge, UK Direct: +44 1223 431035 [email protected] <[email protected]>
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
