Philipp Koehn <pkoehn@...> writes:
>
> Hi,
>
> since the BLEU score is based on the geometric mean between 1 to 4-gram
> precision, this often fails for single sentences. (no 4-gram match means
> multiplying with 0).
>
> One remedy is to add 1 to all precision counts, but this fix is not
implemented
> in any of the standard BLEU scorers.
>
> -phi
>
> On Mon, Jul 9, 2012 at 10:27 PM, Loki Cheng <moonloki@...> wrote:
> > Hi, all
> > I can't evaluate the BLEU score for single sentence pairs with
> > multi-bleu.perl. I don't know why? example just like below:
> >
> > Original Chinese sentence: 安理會 還 請 建設 和平 委員會 向 安理會 提供 咨詢
> 意見 。
> > English translation result: 安理會 還 請 建設 和平 委員會 提供 咨詢 意見 的
> 委員會 。
> >
> > error message: Use of uninitialized value in division (/) at
> > ./multi-bleu.perl line 116, <STDIN> line 1.
> >
> > So it's impossible to evaluate it?
> > Any suggestion will be appreciated.
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@...
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@...
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
There is also a simple bug with this error message. For single sentence pairs,
it doesn't make sense to calculate the confidence interval by looking at
samples.
my $ecart=0;
my $sample=20;
my $s=($#HYP+1);
if ($s >= $sample) { # Don't bother calculating ecart for small files.
for(my $n=0;$n<$sample;$n++) {
my $min = ($n*$s)/$sample;
my $max = (($n+1)*$s)/$sample;
my $sc=calculate_bleu($min, $max);
$sc=~/BLEU = (.*?),/;
if (abs($1-$BLEU)>$ecart) { $ecart=abs($1-$BLEU); }
}
$ecart=sprintf("%.2f",$ecart);
$SCORE=~s/(BLEU = .*?),/$1 +\/- $ecart,/;
}
For the two Chinese sentences,
BLEU = 45.65, 83.3/54.5/40.0/33.3 (BP=0.920, ratio=0.923, hyp_len=12,
ref_len=13)
John
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
