Hello, I run moses (an October 2011 distribution) with SGE (using the ems). To get moses running on my SGE cluster I had to do some changes. I am not sure whether all of them have been necessary, but maybe you want to try editing some of these lines:
[Correcting the order of qsub commands] experiment.perl (somewhere around line 1030): #`qsub $qsub_args -e $step.STDERR $step -o $step.STDOUT`; `qsub $qsub_args -e $step.STDERR -o $step.STDOUT $step`; # new [Correcting regular expression for splitting] moses-parallel.perl (somewhere around line 457): #split(/\s+/,$res); #my $id=$_[2]; my @IdArray = split(/[\s+]| /,$res); # new my $id=$IdArray[2]; # new qsub-wrapper.perl (somewhere around line 151): #split(/\s+/,$res); #my $id=$_[2]; my @IdArray = split(/[\s+]| /,$res); # new my $id=$IdArray[2]; # new I hope this is of some help. Best regards, Sandra -----Ursprüngliche Nachricht----- Von: [email protected] [mailto:[email protected]] Im Auftrag von Lane Schwartz Gesendet: Mittwoch, 16. November 2011 16:11 An: Guchun Zhang Cc: Tom Hoar; [email protected] Betreff: Re: [Moses-support] Tune on SGE II I definitely make use of SGE when I run my experiments. I had forgotten Barry's post. I haven't updated my code in a while, but if SGE is currently broken, I'll definitely want to help fix it. I won't be able to look at things until at least December, though, and more likely January. Lane On Wed, Nov 16, 2011 at 10:08 AM, Guchun Zhang <[email protected]> wrote: > Hi Tom, > Thanks for Barry's post back in January. I wasn't a member at that > time, so didn't receive such a post. My motives will be mostly because > of cost. With SGE, I can reuse some of the old but still usable > machines and expand the network when necessary. Plus, it is quite > unlikely I will be able to get hold of machines as powerful as yours. > So if I want to have a relatively faster Moses system, SGE is a pretty good > option for me. > Hope this makes sense. > Cheers, > Guchun > > On 16 November 2011 12:27, Tom Hoar > <[email protected]> > wrote: >> >> Hi Guchun, >> >> May I ask what motivates you to work with SGE? Barry Haddow posted >> this comment on the list in January this year. We have similar experiences. >> >> Tom >> >> -------- Original Message -------- >> Subject: Re: [Moses-support] running moses on a cluster with sge >> Date: Mon, 31 Jan 2011 09:52:11 +0000 >> From: Barry Haddow <[email protected]> >> To: [email protected] >> Cc: Tom Hoar <[email protected]>, "Noubours, Sandra" >> <[email protected]> >> >> Hi Sandra >> >> The short answer is that it really depends how big your models are. >> Running on a cluster helps speed up tuning because most of the time >> in tuning is spent decoding, which can be easily parallelised by >> splitting up the file into chunks. So each of the individual machines >> should be capable of loading your models and running a decoder. >> >> The problem with using a cluster (as opposed to multicore) is that >> each machine has to have its own ram, and if you want to load large >> models then you need a lot of ram. Whereas with multicore, each >> thread can access the same model. Sure, binarising saves a lot on ram >> usage, but it slows you down and puts a lot of load on the filesystem >> which can cause problems on clusters. >> >> Our group's machines are a mixture of 8 and 16 core Xeon 2.67GHz, >> with 36-72G ram, no sge. We also have access to the university >> cluster, but since the most ram you can get is 16G and sge hold jobs >> don't work at the moment we don't really use it for moses any more, >> >> hope that helps - regards - Barry >> >> >> >> On Tue, 15 Nov 2011 22:02:48 +0000, Guchun Zhang >> <[email protected]> >> wrote: >> >> Sorry, forgot to attach the mert.out file. >> >> Guchun > > > -- > > Guchun Zhang > > Localization Engineer > Alpha CRC Ltd | Cambridge, UK > Direct: +44 1223 431035 > [email protected] > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
