Hi Yonathan, you are right, this is the old code path with 24-bit XOR mask. In this case the patch is fine.
Thanks, Vitali. On 10/28/2014 6:27 PM, Yonathan wrote: > Hi Vitali, > > An extension that implements VRDEMousePtr will receive the AND mask > and 32-bit XOR image as you say. But an extension such as VNC that > only implements the old VRDEColorPointer will get a 24-bit color mask > with no alpha channel. (These callbacks correspond to RDP’s New > Pointer Update vs. Color Pointer Update.) The VNC extension cannot > tell whether there used to be an alpha channel, but if it wants to use > those pixels with AND=1 (patches 0001 and 0002) then it must ignore > the semitransparent pixels somehow. > > I have attached the cursors that I have been testing with. Without > patch 0003, the color on the transparent half gets inverted (becomes > pinkish white) in VNC. With patch 0003, they just become invisible. > Granted, it never looks good given the alpha threshold and blending > with black. But at least it should not do something too unexpected. > > Cheers, > Yonathan > > On Tue, Oct 28, 2014 at 4:14 AM, Vitali Pelenjow > <[email protected]> wrote: >> Hi Yonathan, >> >> the code in the third (ConsoleVRDPServer.cpp) patch should go into the VNC >> server. >> ConsoleVRDPServer.cpp must not modify the XOR mask for alpha cursors, >> because it will break them. >> It the the task of the VRDE (VNC or VRDP) to figure out what to do with the >> XOR mask >> if the client does not support the alpha channel. >> >> Thanks, >> Vitali. >> >> >> Yonathan wrote: >>> To Virtualbox maintainers, >>> >>> You may replace the third patch that I mentioned with the attached >>> one. This new patch does not modify the input ComSafeArrayIn >>> parameter. >>> >>> Please let me know what else I can do to rework any of these patches >>> so they can be merged more cleanly. >>> >>> Yonathan Randolph >>> Acuitus >>> >>> On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <[email protected]> wrote: >>>> Dear VBox, >>>> >>>> I hope you will enjoy these patches for the cursor in Windows. I have >>>> a question about the third patch (to ConsoleVRDPServer.cpp) though. >>>> >>>> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor. >>>> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE. >>>> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a >>>> function called by libvncserver. >>>> >>>> 2) The second patch fixes text selection cursors in Windows (which >>>> have AND=1, XOR=white). This is similar to the OSX and Linux >>>> implementations of UISession::setPointerShape within >>>> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead >>>> of always setting the pixel to black, I calculate the inversion. >>>> >>>> It also fixes color cursors because both RDP and our VNC extension >>>> settings use BGR, but the VNC extension was swapping R and B. >>>> >>>> 3) Explicitly zero the pixels of the color mask for semitransparent >>>> pixels that fall under the 50% threshold, so that we don’t confuse >>>> them for inverting pixels. I am not sure whether it is safe to modify >>>> the ComSafeArrayIn parameter here; do I have to copy it first? >>>> >>>> How to test these changes: >>>> 1. Compile vbox with VNC extension pack enabled. I use the following >>>> LocalConfig.kmk: >>>> VBOX_WITH_EXTPACK_VNC = 1 >>>> INCS += /path/to/libvncserver-git >>>> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs >>>> >>>> 2. Take a snapshot, then configure the VM to use the VNC extension: >>>> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty >>>> VNCPassword=12345 --vrdeport 5900 --vrde on >>>> >>>> 3. Start a Windows VM with VirtualBox guest additions installed. Make >>>> sure hardware acceleration is enabled, so that VNC will draw the >>>> cursor. >>>> >>>> 4. Connect to the VM with a VNC viewer. >>>> >>>> I would like to contribute this under the MIT license. >>>> >>>> Yonathan Randolph >>>> Acuitus >>>> >>>> >>>> _______________________________________________ >>>> vbox-dev mailing list >>>> [email protected] >>>> https://www.virtualbox.org/mailman/listinfo/vbox-dev >> _______________________________________________ vbox-dev mailing list [email protected] https://www.virtualbox.org/mailman/listinfo/vbox-dev
