ok. I'm still getting segfault in Lex Reordering during loading. If you can provide the ro text file so I can binarize it myself, I can debug it. But not a priority, I can park the issue for another time
Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 4 September 2015 at 03:49, Michael Denkowski < [email protected]> wrote: > Hi Hieu, > > Yes, I have everything working together with the caveat about order in the > moses.ini file (https://github.com/moses-smt/mosesdecoder/pull/124). The > mmsapt files might be dependent on Boost version so I also included the > aligned bitext I used to build the model and rebuilt the tarball: > https://drive.google.com/file/d/0B6trVKD0-obBRHFjMGxRZTJvV1U/view?usp=sharing. > I made it a pull request instead of just merging it into master so you guys > could look over the changes since technically the default behavior could > change if the moses.ini file lists phrase tables before other features. > > Best, > Michael > > On Thu, Sep 3, 2015 at 8:44 PM, Hieu Hoang <[email protected]> wrote: > >> i saw your checkins, is it working for you now? >> >> Your test data doesn't seem to run for me, the lexical reordering file >> seems to be corrupt. >> >> (if you wanna share test data, can you do it via dropbox/google drive >> rather than the Moses github) >> >> >> On 03/09/2015 05:13, Michael Denkowski wrote: >> >> Sounds good. I added a small test model to my branch: >> <https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz> >> https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz. >> This translates a sample of fr-en news with a Mmsapt, surface LM, and >> 400-class LM. >> >> --Michael >> >> On Wed, Sep 2, 2015 at 2:56 AM, Hieu Hoang <[email protected]> wrote: >> >>> It should work. The function >>> EvaluateInIsolation() >>> in the LM is for optimisation reason. eg. if the target phrase is 'a b c >>> d' and the LM is a trigram, the trigrams 'a b c' and 'b c d' can be >>> precalculated in EvaluateInIsolation(). >>> >>> Implementing a pt for factors requires setting up some variables, which >>> may not have happen yet in mmsapt. if you can send me a small example >>> model, i'll see what i can do >>> >>> >>> On 01/09/2015 02:11, Ulrich Germann wrote: >>> >>> Hi Michael, >>> >>> I have no experience with factored models, so I'm speculating here to >>> some degree. The reason the phrase table calls EvaluateInIsolation is >>> because all "isolated" phrase scores are considered when pruning. In my >>> opinion pruning should not happen within the phrase tables (for exactly the >>> reason that it does not allow feature functions to be agnostic about other >>> feature functions) but by whatever object calls all the phrase tables and >>> does the generation. However, for software legacy reasons, that's the way >>> it is right now, and I'm not likely to address this issue any time soon >>> myself. The most reasonable fix for this in my opinion is to move pruning >>> where it belongs --- post all the factor generation stuff. >>> >>> Hieu is probably still the person with the best understanding of how >>> factored phrase table entry generation works, so maybe he can chime in on >>> this ... >>> >>> Cheers - Uli >>> >>> >>> On Mon, Aug 31, 2015 at 11:29 PM, Michael Denkowski < >>> <[email protected]>[email protected]> wrote: >>> >>>> Hi Ulrich, >>>> >>>> I was looking into using a class-based LM with your dynamic phrase >>>> table via generation models. I translate factor 0 to 0 with the Mmsapt, >>>> then generate target factor 1 (word class) with a GM. The class-based LM >>>> operates on factor 1. >>>> >>>> I'm hitting a segfault on what appears to be an order-of-operations >>>> issue with the PT and LM. In mmsapt.cpp:578, Mmsapt::mkTPhrase makes a >>>> call to tp->EvaluateInIsolation. This calls all of the models, including >>>> the LMs. The class LM tries to score factor 1, which doesn't exist yet >>>> (since generation happens after translation), and it dies. By nature, >>>> other phrase tables don't have this issue since they can just pull up >>>> pre-computed scores. >>>> >>>> Is scoring with all of the models here a strategic choice to get better >>>> performance or would it be sufficient to just score with the PT features? >>>> Thanks! >>>> >>>> --Michael >>>> >>> >>> >>> >>> -- >>> Ulrich Germann >>> Senior Researcher >>> School of Informatics >>> University of Edinburgh >>> >>> >>> _______________________________________________ >>> Moses-developers mailing >>> [email protected]http://mailman.mit.edu/mailman/listinfo/moses-developers >>> >>> >>> -- >>> Hieu Hoang >>> Researcher >>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu >>> >>> >> >> -- >> Hieu Hoang >> Researcher >> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu >> >> >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
