For the memory issue, it's a problem if the max-phrase-length is too high. The decoder will generate a list of all possible sub-paths up to length max-phrase-length. If there are many possible paths, then it consumes lots of memory.
Another issue is distortion limits. I'm not sure how distortion calculated with lattice input. Hieu Hoang Research Associate (until March 2015) ** searching for interesting commercial MT position ** University of Edinburgh http://www.hoang.co.uk/hieu On 10 February 2015 at 12:19, Jorg Tiedemann <[email protected]> wrote: > > Just to make sure that I understand the consequences for decoding > correctly. Is it true that this only becomes a problem if the jump length > in the lattice is larger than the max-phrase-length flag, isn't it? > > Jörg > > > > > On Feb 6, 2015, at 5:03 PM, Hieu Hoang wrote: > > the decoder creates a list of all possible paths through the lattice up > the the max-phrase-length before it does the pt lookups > > So if the lattice is large and the max-phrase-length is high, it will > consume a lot of memory. > > in the old version of moses (v1), it only built the paths as it does the > lookup and only keep the path if it finds rules for it. This is more memory > efficient but the code was too convoluted > > On 06/02/15 12:43, Jorg Tiedemann wrote: > > Hi, > > I have a question about the -max-phrase-length flag. I have lattice > input that collects a lot of memory and eventually crashes when memory runs > out (over 90GB). Now I realized that setting max-phrase-length helps even > when setting it to length 7 which is the maximum length in my extracted > phrase table. I do't really see why this is the case but maybe I > misunderstand the flag and what it does. Is it not related to the max > length of translation options coming from the phrase table? > Thanks for clarifying this to me. > > cheers, > Jörg > > > > > > > _______________________________________________ > Moses-support mailing > [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support > > > -- > Hieu Hoang > Research Associate (until March 2015) > ** searching for interesting commercial MT position ** > University of Edinburghhttp://www.hoang.co.uk/hieu > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
