http://www.inf.ed.ac.uk/teaching/courses/mt/
2009/2/23 James Read <[email protected]>: > Thanks for that. > > I'm having difficulty finding the lecture notes on PK's homepage on > UoE site. Do you have a direct link to them? > > Quoting Barry Haddow <[email protected]>: > >> Hi James >> >> There was an investigation into different methods of parallelising model 1 in >> this paper: >> Fully Distributed EM for Very Large Datasets >> by: Jason Wolfe, Aria Haghighi, Dan Klein >> http://www.cs.berkeley.edu/~aria42/pubs/icml08-distributedem >> >> As for pseudo-code for the IBM models, check out Philipp Koehn's >> lecture notes >> on machine translation on the UoE site, >> >> cheers >> Barry >> >> >> On Friday 20 February 2009 13:51, Chris Dyer wrote: >>> Another architecture to consider is storing/distributing the ttable >>> from a single central repository. Most of the ttable is full of crap, >>> and for each sentence, you know exactly what parameters will be >>> required in advance of running your E step. However, by not >>> distributing stuff that you don't need, you'll save a lot of effort, >>> and the amount of memory used by each individual compute node will be >>> radically reduced. >>> >>> Chris >>> >>> On Fri, Feb 20, 2009 at 1:41 PM, Qin Gao <[email protected]> wrote: >>> > Hi, James, >>> > >>> > PGIZA++ is not using OpenMPI, and only use shared storage to transfer >>> > model files, that could be a bottleneck, MGIZA++ is just using >>> > multi-thread. So they are not quite complete and can be further improved, >>> > the advantage of PGIZA++ is it already decomposed every training step >>> > (E/M, model 1, hmm 3,4,5 etc), so it could be easy for you to understand >>> > the logic. >>> > >>> > The bottleneck is majorly the huge T-Table (translation table), which is >>> > larger during model 1 training and then becomes smaller as training goes >>> > on, every child has to get the table (typically several gigas for model 1 >>> > on large data). I think OpenMPI etc is a right way to go, and please let >>> > me know if you have any question on PGIZA++. >>> > >>> > Best, >>> > Qin >>> > >>> > On Thu, Feb 19, 2009 at 4:54 PM, James Read <[email protected]> wrote: >>> >> Wow! >>> >> >>> >> Thanks for that. That was great. I've had a quick read through your >>> >> paper. I'm guessing the basis of PGiza++ is OpenMPI calls and the basis >>> >> of MGiza++ is OpenMP calls right? >>> >> >>> >> Your paper was very fascinating. You mentioned I/O bottlenecks quite a >>> >> lot with reference to PGiza++ which is to be expected. Did you run any >>> >> experiments to find what those bottlenecks typically are? How many >>> >> processors did you hit before you started to lose speed up? Did this >>> >> number vary for different data sets? >>> >> >>> >> Also, you mention breaking up the files into chunks and working on them >>> >> on different processors. Obviously you're referring to some kind of data >>> >> decomposition plan. Does your algorithm have any kind of intelligent >>> >> data decomposition strategy for reducing communications? Or is it just a >>> >> case of cutting the file up into n bits and assigning each one to a >>> >> processor? >>> >> >>> >> The reason I ask is that our project would now have to come up with some >>> >> kind of superior data decomposition plan in order to justify proceeding >>> >> with the project. >>> >> >>> >> Thanks >>> >> >>> >> James >>> >> >>> >> Quoting Qin Gao <[email protected]>: >>> >>> Hi James, >>> >>> >>> >>> The GIZA++ is a very typical EM algorithm and probably you want to >>> >>> parallelize the e-step since it takes long time then M-Step. You may >>> >>> want to >>> >>> check out the PGIZA++ and MGIZA++ implementations which you can >>> >>> download in >>> >>> my homepage: >>> >>> >>> >>> http://www.cs.cmu.edu/~qing >>> >>> >>> >>> And you may also be interested in a paper describing the work: >>> >>> >>> >>> www.aclweb.org/anthology-new/W/W08/W08-0509.pdf >>> >>> >>> >>> Please let me know if there are anything I can help. >>> >>> >>> >>> Best, >>> >>> Qin >>> >>> >>> >>> On Thu, Feb 19, 2009 at 4:12 PM, James Read <[email protected]> >>> >>> >>> >>> wrote: >>> >>>> Hi all, >>> >>>> >>> >>>> as the title suggest I am involved in a project which may involve >>> >>>> parallelising the code of Giza++ so that it will run on supercomputers >>> >>>> scalably on n number of processors. This would have obvious benefits >>> >>>> to any researchers making regular use of Giza++ who would like it to >>> >>>> finish in minutes rather than hours. >>> >>>> >>> >>>> The first step of such a project was profiling Giza++ to see where the >>> >>>> executable spends most of its time on a typical run. Such profiling >>> >>>> indicated a number of candidate functions. One of which was >>> >>>> model1::em_loop found in the model1.cpp file. >>> >>>> >>> >>>> In order to parallelise such a function (using OpenMPI) it is >>> >>>> necessary to first come up with some kind of data decomposition >>> >>>> strategy which minimizes the latency of interprocessor communication >>> >>>> but ensures that the parallelisation has no side effects other than >>> >>>> running faster on a number of processors up to some optimal number of >>> >>>> processors where the latency of communication begins to outweigh the >>> >>>> benefits of throwing more processors at the job. >>> >>>> >>> >>>> In order to do this I am trying to gain an understanding of the logic >>> >>>> in the model1::em_loop function. However, intuitive comments are >>> >>>> lacking in the code. Does anyone on this list have a good internal >>> >>>> knowledge of this function? Enough to give a rough outline of the >>> >>>> logic it contains in some kind of readable pseudocode? >>> >>>> >>> >>>> Thanks >>> >>>> >>> >>>> P.S. Apologies to anybody to whom this email was not of interest. >>> >>>> >>> >>>> -- >>> >>>> The University of Edinburgh is a charitable body, registered in >>> >>>> Scotland, with registration number SC005336. >>> >>>> >>> >>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Moses-support mailing list >>> >>>> [email protected] >>> >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >>> >>> -- >>> >>> ========================================== >>> >>> Qin Gao >>> >>> Language Technology Institution >>> >>> Carnegie Mellon University >>> >>> http://geek.kyloo.net >>> >>> >>> >>> ----------------------------------------------------------------------- >>> >>>------------- Please help improving NLP articles on Wikipedia >>> >>> ========================================== >>> >> >>> >> -- >>> >> The University of Edinburgh is a charitable body, registered in >>> >> Scotland, with registration number SC005336. >>> > >>> > -- >>> > ========================================== >>> > Qin Gao >>> > Language Technology Institution >>> > Carnegie Mellon University >>> > http://geek.kyloo.net >>> > ------------------------------------------------------------------------- >>> >----------- Please help improving NLP articles on Wikipedia >>> > ========================================== >>> > >>> > >>> > _______________________________________________ >>> > Moses-support mailing list >>> > [email protected] >>> > http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
