Hmm, looks like a limitation of the inital coders knowledge.
Nothing to worry about in an OpenSource project where lots
of coding youngsters did take part in.
The below coding:
if (mode->HDisplay * mode->VDisplay * scrp->fbFormat.bitsPerPixel
> scrp->videoRam * (1024 * 8))
return MODE_MEM;
is in my view a possible point of failure for a claculation overflow.
Mutilply three integers and you might get nicely across 2 GB and
wrap around to negative numbers.
1) For the current problem i do suggest a fix like this:
if ( ((long long)mode->HDisplay) * mode->VDisplay * scrp->fbFormat.bitsPerPixel
> ((long long)scrp->videoRam) * (1024 * 8))
return MODE_MEM;
2) A bit more paranoia would lead to something like this:
{
long long pixels, bits, sreen_bits;
pixels = (long long)mode->HDisplay * mode->VDisplay;
if( pixels < 0 ) return MODE_MEM;
bits = pixels * scrp->fbFormat.bitsPerPixel;
if( bits < 0 ) return MODE_MEM;
screen_bits = (long long)scrp->videoRam * (1024 * 8);
if( screen_bits < 0 ) return MODE_MEM;
if ( bits > screen_bits ) return MODE_MEM;
}
solved.
the second is most recommended. but even this might still might
need some revisiting when changing any of the types of the used
variables and therefore changing the condtions for what it was
designed.
-Alex.
-----Original Message-----
From: Luugi Marsan [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 05, 2002 00:17
To: [EMAIL PROTECTED]
Subject: Re: [Xpert]256 megs board fails Validation,
if (mode->HDisplay * mode->VDisplay * scrp->fbFormat.bitsPerPixel > scrp->videoRam * (1024 * 8))
return MODE_MEM;
--
where:
scrp ->videoRam = 256 * 1024 ( since videoRam is in Kbytes)
When calculating the right hand side of the comparison, the right hand side is equal to 256*1024*1024*8 = 2^31. The left hand side is also calculated in bits, since it multiplies the display by the number of bits per pixel.
Luugi
