Bertoldi and Federico (2009) tackle a related problem when combining multiple phrase tables:
http://www.aclweb.org/anthology/W/W09/W09-0432.pdf They have to come up with phrase scores for entries that aren't in all of the base tables. They infer smoothed estimates using lexical probabilities. This may or may not be useful to you. - John D Burger MITRE On May 22, 2009, at 13:00, Sanne Korzec wrote: > Hi, > > Thanks for the previous replies. > > I am re-estimating the phrase pair table and enriching it with new > phrases. > The newly added phrases need values for prob, lw, inverse prob and > inverse > lw. > > Sometimes phrases are added in my systems whose lexical weights are > unknown. > For some the lw can be calculated, but for some it can not. For > reasons I > won't explain. I need to make a decision what to do with these unknown > values. > > I have considered setting them to a fixed number: e.g. 0.1 or 0.01 > or even > 0.0001. I have however no clue what the impact of these values are. > > I was hoping someone could point me in the right direction. I would > like to > make an educated guess on what this value should be, but I do not have > enough experience with MT to do this. > > I know from the previous replies that the values from the score > vector are > all multiplied together, after applying an exponential weight. But > it would > also help if someone could give me or point me towards the exact > formula. > > Thanks in advance, > Sanne > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > Philipp > Koehn > Sent: vrijdag 8 mei 2009 15:29 > To: [email protected] > Subject: Re: [Moses-support] lexical weighting and inverse > probabilities > > Hi, > > there should not be any zeros in this table, because that will, > as you write, lead to an overall zero probability. > > -phi > > On Fri, May 8, 2009 at 11:54 AM, Sanne Korzec <[email protected]> wrote: >> Ok thanks. >> >> Does this mean that if one of these values is zero in the table, >> one can >> leave the entry out? Multiplication gives a result of zero. Or does >> the >> exponential weight compensate for this? >> >> Sanne >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf Of >> Philipp >> Koehn >> Sent: donderdag 7 mei 2009 20:07 >> To: [email protected] >> Cc: [email protected] >> Subject: Re: [Moses-support] lexical weighting and inverse >> probabilities >> >> Hi, >> >> they are all multiplied together, after applying an exponential >> weight. >> >> -phi >> >> On Thu, May 7, 2009 at 4:51 PM, Sanne Korzec <[email protected]> wrote: >>> Hi, >>> >>> The final phrase pair table usually has a score vector of length 5: >>> >>> The components are: probability, lexical weights, inverse >>> probability, >>> inverde lex. Weights and a constant. >>> >>> How and why are the lexical weights, the inverse probabilities and >>> the >>> inverse lexical weighting exactly used during decoding? >>> >>> Sanne _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
