Hi, my understanding is that it is intentional that HTER can be optimized by the annotator to create a reference that has the lowest HTER score possible, while still being an acceptable (correct) translation of the source sentence.
-phi On Sun, Apr 7, 2013 at 5:51 PM, Baskaran Sankaran <[email protected]> wrote: > Hi all, > > I have some questions about HTER computation as explained in Snover et al., > 2006, AMTA. The paper states that the denominator term should not use the > length of the targeted reference (footnote 5) as this could be exploited by > the annotators to favour longer targeted references in order to get better > (lower) HTER scores. > > While this might be reasonable, I thunk such long targeted references will > also incur additional edit distance costs (say for deletion), increasing the > numerator and thereby resulting in poor (higher) HTER score. Isn't this > true? In that case isn't better to just use the length of the targeted > reference in denominator? > > Secondly they don't say what to use instead for the denominator. One > possibility would be to use the length of the untargeted reference, but not > sure if this is correct. And if there are multiple references, should we > just take the average reference length? > > Any clarifications/ comments would be appreciated. > > Thanks > - Baskaran > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
