---------- Forwarded message ----------
Date: Sat, 19 Jun 2004 18:21:20 -0400 (EDT)
From: Nilmoni Deb <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED], [EMAIL PROTECTED]
Subject: bug in rfb/Cursor.cxx causes vncviewer to crash in RealVNC 4.0
All,
The following bug is not listed in
http://www.realvnc.com/v4/known-bugs.html so I am reporting it.
When I run vncviewer (on unix) it crashes. Running within gdb gives this
backtrace:
Program received signal SIGSEGV, Segmentation fault.
0x8056ea7 in TXImage::lookup ()
(gdb) bt
#0 0x8056ea7 in TXImage::lookup ()
#1 0x80609e9 in rfb::PixelFormat::rgbFromPixel ()
#2 0x804bb7b in DesktopWindow::setCursor ()
#3 0x80529b5 in CConn::setCursor ()
#4 0x806f78c in rfb::CMsgReader::readSetCursor ()
#5 0x805a5e2 in rfb::CMsgReaderV3::readMsg ()
#6 0x8058e0f in rfb::CConnection::processMsg ()
#7 0x805475b in main ()
#8 0x401909ed in __libc_start_main () from /lib/libc.so.6
On further investigation, the bug appears to in the file rfb/Cursor.cxx,
in the function Cursor::getBitmap(Pixel* pix0, Pixel* pix1), within lines
100-106 as shown below:
101 if (!gotPix0 || pix == *pix0) {
102 gotPix0 = true;
103 *pix0 = pix;
104 } else if (!gotPix1 || pix == *pix1) {
105 gotPix1 = true;
106 *pix1 = pix;
Clearly, the == signs in lines 101 and 104 should be replaced by != signs
so that the corrected code looks like this:
101 if (!gotPix0 || pix != *pix0) {
102 gotPix0 = true;
103 *pix0 = pix;
104 } else if (!gotPix1 || pix != *pix1) {
105 gotPix1 = true;
106 *pix1 = pix;
After making these minor fixes, vncviewer does not crash anymore.
Also, this code can definitely be more optimized. For example,
- Why is getPF().bpp being called so many times ?
- y * maskBytesPerRow (line 92) can be computed outside the loop
for x
- Similarly, y * width() + x is computed too many times
unnecessarily
I am including a file which has this optimized version of the
Cursor::getBitmap function:
rdr::U8* Cursor::getBitmap(Pixel* pix0, Pixel* pix1)
{
bool gotPix0 = false;
bool gotPix1 = false;
rdr::U8Array source(maskLen());
memset(source.buf, 0, maskLen());
int bpp = getPF().bpp;
int wid = width();
int hei = height();
int maskBytesPerRow = (wid + 7) / 8;
for (int y = 0; y < hei; y++) {
int ym = y * maskBytesPerRow;
int ywid = y * wid;
for (int x = 0; x < wid; x++) {
int byte = ym + x / 8;
int bit = 7 - x % 8;
if (mask.buf[byte] & (1 << bit)) {
Pixel pix=0;
int position = ywid + x;
switch (bpp) {
case 8: pix = ((rdr::U8*) data)[position]; break;
case 16: pix = ((rdr::U16*)data)[position]; break;
case 32: pix = ((rdr::U32*)data)[position]; break;
}
if (!gotPix0 || pix != *pix0) {
gotPix0 = true;
*pix0 = pix;
cout << "\t pix0 = " << pix0[0] << endl;
} else if (!gotPix1 || pix != *pix1) {
gotPix1 = true;
*pix1 = pix;
cout << "\t pix1 = " << pix1[0] << endl;
source.buf[byte] |= (1 << bit);
} else {
// not a bitmap
return 0;
}
}
}
}
return source.takeBuf();
}
thanks
....
ps: Is there a bugzilla where bugs can be reported ?
_______________________________________________
VNC-List mailing list
[EMAIL PROTECTED]
To remove yourself from the list visit:
http://www.realvnc.com/mailman/listinfo/vnc-list
