Thanks for taking the time to post this! I'm seeing the same problem,
and got as far the stack trace, but hadn't identified a fix yet. The
fix helped (no more core dumps on Solaris), but I had to make one more
change to get it to work on Linux. I'm seeing cases where it get's all
the way through the code without setting pix1, returning an essentially
random value to caller. I don't completely understand what this code
is doing, but forcing the value to be zero if not otherwise set fixes my
core dumps.
}
+ // Don't return garbage if pix1 is not set
+ if ( !gotPix1 ) *pix1 = 0;
return source.takeBuf();
}
--
Peter Fales Lucent Technologies, Room 1C-436
N9IYJ 2000 N Naperville Rd PO Box 3033
internet: [EMAIL PROTECTED] Naperville, IL 60566-7033
work: (630) 979-8031
On Thu, Jun 24, 2004 at 08:19:07PM -0400, Nilmoni Deb wrote:
> ---------- Forwarded message ----------
> Date: Sat, 19 Jun 2004 18:21:20 -0400 (EDT)
> From: Nilmoni Deb <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED], [EMAIL PROTECTED]
> Subject: bug in rfb/Cursor.cxx causes vncviewer to crash in RealVNC 4.0
>
>
> All,
> The following bug is not listed in
> http://www.realvnc.com/v4/known-bugs.html so I am reporting it.
>
> When I run vncviewer (on unix) it crashes. Running within gdb gives this
> backtrace:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x8056ea7 in TXImage::lookup ()
> (gdb) bt
> #0 0x8056ea7 in TXImage::lookup ()
> #1 0x80609e9 in rfb::PixelFormat::rgbFromPixel ()
> #2 0x804bb7b in DesktopWindow::setCursor ()
> #3 0x80529b5 in CConn::setCursor ()
> #4 0x806f78c in rfb::CMsgReader::readSetCursor ()
> #5 0x805a5e2 in rfb::CMsgReaderV3::readMsg ()
> #6 0x8058e0f in rfb::CConnection::processMsg ()
> #7 0x805475b in main ()
> #8 0x401909ed in __libc_start_main () from /lib/libc.so.6
>
>
> On further investigation, the bug appears to in the file rfb/Cursor.cxx,
> in the function Cursor::getBitmap(Pixel* pix0, Pixel* pix1), within lines
> 100-106 as shown below:
>
> 101 if (!gotPix0 || pix == *pix0) {
> 102 gotPix0 = true;
> 103 *pix0 = pix;
> 104 } else if (!gotPix1 || pix == *pix1) {
> 105 gotPix1 = true;
> 106 *pix1 = pix;
>
>
> Clearly, the == signs in lines 101 and 104 should be replaced by != signs
> so that the corrected code looks like this:
>
>
> 101 if (!gotPix0 || pix != *pix0) {
> 102 gotPix0 = true;
> 103 *pix0 = pix;
> 104 } else if (!gotPix1 || pix != *pix1) {
> 105 gotPix1 = true;
> 106 *pix1 = pix;
>
> After making these minor fixes, vncviewer does not crash anymore.
>
>
>
> Also, this code can definitely be more optimized. For example,
> - Why is getPF().bpp being called so many times ?
> - y * maskBytesPerRow (line 92) can be computed outside the loop
> for x
> - Similarly, y * width() + x is computed too many times
> unnecessarily
>
> I am including a file which has this optimized version of the
> Cursor::getBitmap function:
>
> rdr::U8* Cursor::getBitmap(Pixel* pix0, Pixel* pix1)
> {
> bool gotPix0 = false;
> bool gotPix1 = false;
> rdr::U8Array source(maskLen());
> memset(source.buf, 0, maskLen());
>
> int bpp = getPF().bpp;
> int wid = width();
> int hei = height();
>
> int maskBytesPerRow = (wid + 7) / 8;
> for (int y = 0; y < hei; y++) {
> int ym = y * maskBytesPerRow;
> int ywid = y * wid;
> for (int x = 0; x < wid; x++) {
> int byte = ym + x / 8;
> int bit = 7 - x % 8;
> if (mask.buf[byte] & (1 << bit)) {
> Pixel pix=0;
> int position = ywid + x;
> switch (bpp) {
> case 8: pix = ((rdr::U8*) data)[position]; break;
> case 16: pix = ((rdr::U16*)data)[position]; break;
> case 32: pix = ((rdr::U32*)data)[position]; break;
> }
> if (!gotPix0 || pix != *pix0) {
> gotPix0 = true;
> *pix0 = pix;
> cout << "\t pix0 = " << pix0[0] << endl;
> } else if (!gotPix1 || pix != *pix1) {
> gotPix1 = true;
> *pix1 = pix;
> cout << "\t pix1 = " << pix1[0] << endl;
> source.buf[byte] |= (1 << bit);
> } else {
> // not a bitmap
> return 0;
> }
> }
> }
> }
> return source.takeBuf();
> }
>
>
>
>
>
>
> thanks
> ....
>
> ps: Is there a bugzilla where bugs can be reported ?
> _______________________________________________
> VNC-List mailing list
> [EMAIL PROTECTED]
> To remove yourself from the list visit:
> http://www.realvnc.com/mailman/listinfo/vnc-list
_______________________________________________
VNC-List mailing list
[EMAIL PROTECTED]
To remove yourself from the list visit:
http://www.realvnc.com/mailman/listinfo/vnc-list
