Hi andre
I don't entirely understand your idea but I think a lot more can be done
about unknown words.
At the moment, translating an unknown word is penalised by a score of -100
set by the
'UnknownWordPenaltyProducer' class in UnknownWordPenaltyProducer.cpp
This is done in
TranslationOption.cpp:line 55
Which is called from
TranslationOptionCollection.cpp line 226 & 232
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of André Lynum
Sent: 25 February 2008 10:56
To: [email protected]
Subject: [Moses-support] Penalizing unknown words during bilingual scoring
Hi, I'm working on modifying Moses to provide translation model scores for a
given source translation sentence pair.
I'm using the decoder, constraining the hypothesises it generates, and then
I examine the hypothesis stack which covers the most of the source input
(I'm wondering if I should look at all generated hypothesises). Here I look
for the highest scoring hypothesis, but I will need to account for the parts
of the source and translation that is not covered by the hypothesis.
I was thinking of adding a penalizing factor to the total score of the
hypothesises for each unknown word in the input pair. This is motivated by
the notion that each unknown word may be generated from a "null" phrase
pair. But I'm unsure about what factor to use and the scoring part of the
Moses code is a bit complicated and I would appreciate any insight in what
factor to use and how to apply it to the hypothesis score.
My initial notion was to use the same score as is used for translation
options generated for unknown word but I can't quite see where this is set
(as the negative word weight in TargetPhrase::SetScore() or as - inf in the
translationoption constructor ?). Any help that would help me understand
this part of the code would be greatly appreciated.
Regards
-andré lynum
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
