With a closed vocabulary LM, SRILM returns -inf on OOV and moses floors this to LOWEST_SCORE which is -100.0. If you want identical behavior from KenLM,
kenlm/build_binary -u -100.0 foo.arpa foo.binary Unless you passed -vocab to SRILM (and most people don't), <unk> never appears except as a unigram. Therefore, Chris is not getting any gain from additional conditioning. OOVs can be good: names of people who appear in the news, new product names etc. On 03/19/11 14:02, Alexander Fraser wrote: > Hi Folks -- > > An LM-OOV feature sounds like a good solution to me. Chris, have you > tried pegging the LM-OOV feature weight at an extremely high value? I > suspect the gains you are getting are due to the use of <unk> in LM > conditioning, i.e., p(word|... <unk> ...), rather than due to allowing > more LM-OOVs. > > If the LM-OOV feature were defaulted to an extremely high value, we > would get the behavior that Moses+SRILM has, but people who wanted to > could try training the weight. > > I think using an open-class LM without such a penalty is not a good > idea. I guess maybe the Moses+SRILM code defaults to a log probability > value of something like -20 for p(LM-OOV|any-context) regardless of > whether <unk> is present in the LM, so that is why it is OK to use an > open-class LM with SRILM. > > Cheers, Alex > > > On Sat, Mar 19, 2011 at 6:03 PM, Chris Dyer <[email protected]> wrote: >> I've started using an OOV feature (fires for each LM-OOV) together >> with an open-vocabulary LM, and found that this improves the BLEU >> score. Typically, the weight learned on the OOV feature (by MERT) is >> quite a bit more negative than the default amount estimated during LM >> training, but it is still far greater than the "avoid at all costs" >> moses/joshua OOV default behavior. As a result, there is a small >> increase in the number of OOVs in the output (I have not counted this >> number). However, the I find that the bleu score increases a bit for >> doing this (magnitude depends on a number of factors), and the "extra" >> OOVs typically occur in places where the possible English translation >> would have been completely nonsensical. >> -Chris >> >> On Sat, Mar 19, 2011 at 12:51 PM, Alexander Fraser >> <[email protected]> wrote: >>> Hi Folks, >>> >>> Is there some way to penalize LM-OOVs when using Moses+KenLM? I saw a >>> suggestion to create an open-vocab LM (I usually use closed-vocab) but >>> I think this means that in some context a LM-OOV could be produced in >>> preference to a non LM-OOV. This should not be the case in standard >>> phrase-based SMT (e.g., using the feature functions used in the Moses >>> baseline for the shared task for instance). Instead, Moses should >>> produce the minimal number of LM-OOVs possible. >>> >>> There are exceptions to this when using different feature functions. >>> For instance, we have a paper on trading off transliteration vs >>> semantic translation (for Hindi to Urdu translation), where the >>> transliterations are sometimes LM-OOV, but still a better choice than >>> available semantic translations (which are not LM-OOV). But the >>> overall SMT models we used supports this specific trade-off (and it >>> took work to make the models do this correctly, this is described in >>> the paper). >>> >>> I believe for the other three LM packages used with Moses the minimal >>> number of LM-OOVs is always produced. I've switched back to >>> Moses+SRILM for now due to this issue. I think it may be the case that >>> Moses+KenLM actually produces the maximal number of OOVs allowed by >>> the phrases loaded, which would be highly undesirable. Empirically, it >>> certainly produces more than Moses+SRILM in my experiments. >>> >>> Thanks and Cheers, Alex >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
