Re: 30-bpp mode for dummy - exposes a bug somewhere else?

2016-09-19 Thread Antoine Martin
(snip)
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
>> numColors=, indices=, colors=0xd01a80,
>> pVisual=)
>> at dummy_driver.c:513
>> 513 dPtr->colors[index].red = colors[index].red << shift;
>> (gdb) bt
>> #0  0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
>> numColors=, indices=, colors=0xd01a80,
>> pVisual=)
>> at dummy_driver.c:513
>> #1  0x00480fb2 in CMapRefreshColors ()
>> #2  0x004816e8 in CMapReinstallMap ()
>> #3  0x004817ca in CMapSwitchMode ()
>> #4  0x0047407a in xf86SwitchMode ()
>> #5  0x00497a53 in xf86RandRSetMode ()
>> #6  0x0049808a in xf86RandRSetConfig ()
>> #7  0x004fa398 in RRCrtcSet ()
>> #8  0x00507a46 in ProcRRSetScreenConfig ()
>> #9  0x00436daf in Dispatch ()
>> #10 0x0043add3 in dix_main ()
>> #11 0x7fddc7fb8731 in __libc_start_main (main=0x424d20 ,
>> argc=19, argv=0x7ffc98ec5e18, init=, fini=> out>, rtld_fini=,
>> stack_end=0x7ffc98ec5e08) at ../csu/libc-start.c:289
>> #12 0x00424d59 in _start ()
>> (gdb)
>>
>> It's always crashing in palette or colormap functions, ie:
>> CMapDestroyColormap or DUMMYLoadPalette, etc.
>>
>> Could it be that there's a bug somewhere else that is only being
>> triggered with 30-bpp modes? Trying to use more colours / space than is
>> available in the current colourmap perhaps?
>> I had temporarily prevented crashes with only some (!) randr resizings
>> by clamping the "numColors" value to 256 in DUMMYLoadPalette.
> 
> I suspect it's related to DUMMYScreenInit calling xf86HandleColormaps
> with 256 for maxColors. That (and possibly other places) probably needs
> to be changed to allow for 1024 entries in colourmaps.
You're absolutely right, that fixes the problem.
I will send a patch shortly.

Just a thought: if a value lower than 1024 is likely to cause crashes
like this one, shouldn't there be some stronger validation of input
values in xf86HandleColormaps?

Thanks!
Antoine
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

Re: 30-bpp mode for dummy - exposes a bug somewhere else?

2016-09-19 Thread Michel Dänzer
On 16/09/16 06:58 PM, Antoine Martin wrote:
> Hi,
> 
> Adding support for 10 bits per pixel mode to the dummy driver:
> -- a/src/dummy_driver.c
> +++ b/src/dummy_driver.c
> @@ -313,6 +313,7 @@ DUMMYPreInit(ScrnInfoPtr pScrn, int flags)
> case 15:
> case 16:
> case 24:
> +case 30:
> break;
> default:
> xf86DrvMsg(pScrn->scrnIndex, X_ERROR,
> @@ -327,8 +328,8 @@ DUMMYPreInit(ScrnInfoPtr pScrn, int flags)
> pScrn->rgbBits = 8;
> 
>  /* Get the depth24 pixmap format */
> -if (pScrn->depth == 24 && pix24bpp == 0)
> -   pix24bpp = xf86GetBppFromDepth(pScrn, 24);
> +if (pScrn->depth >= 24 && pix24bpp == 0)
> +   pix24bpp = xf86GetBppFromDepth(pScrn, pScrn->depth);
> 
>  /*
>   * This must happen after pScrn->display has been set because
> 
> 
> This looks simple enough and works very well, stable. Unfortunately,
> this also causes any calls to RandR to crash the server:
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
> numColors=, indices=, colors=0xd01a80,
> pVisual=)
> at dummy_driver.c:513
> 513  dPtr->colors[index].red = colors[index].red << shift;
> (gdb) bt
> #0  0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
> numColors=, indices=, colors=0xd01a80,
> pVisual=)
> at dummy_driver.c:513
> #1  0x00480fb2 in CMapRefreshColors ()
> #2  0x004816e8 in CMapReinstallMap ()
> #3  0x004817ca in CMapSwitchMode ()
> #4  0x0047407a in xf86SwitchMode ()
> #5  0x00497a53 in xf86RandRSetMode ()
> #6  0x0049808a in xf86RandRSetConfig ()
> #7  0x004fa398 in RRCrtcSet ()
> #8  0x00507a46 in ProcRRSetScreenConfig ()
> #9  0x00436daf in Dispatch ()
> #10 0x0043add3 in dix_main ()
> #11 0x7fddc7fb8731 in __libc_start_main (main=0x424d20 ,
> argc=19, argv=0x7ffc98ec5e18, init=, fini= out>, rtld_fini=,
> stack_end=0x7ffc98ec5e08) at ../csu/libc-start.c:289
> #12 0x00424d59 in _start ()
> (gdb)
> 
> It's always crashing in palette or colormap functions, ie:
> CMapDestroyColormap or DUMMYLoadPalette, etc.
> 
> Could it be that there's a bug somewhere else that is only being
> triggered with 30-bpp modes? Trying to use more colours / space than is
> available in the current colourmap perhaps?
> I had temporarily prevented crashes with only some (!) randr resizings
> by clamping the "numColors" value to 256 in DUMMYLoadPalette.

I suspect it's related to DUMMYScreenInit calling xf86HandleColormaps
with 256 for maxColors. That (and possibly other places) probably needs
to be changed to allow for 1024 entries in colourmaps.


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel

30-bpp mode for dummy - exposes a bug somewhere else?

2016-09-16 Thread Antoine Martin
Hi,

Adding support for 10 bits per pixel mode to the dummy driver:
-- a/src/dummy_driver.c
+++ b/src/dummy_driver.c
@@ -313,6 +313,7 @@ DUMMYPreInit(ScrnInfoPtr pScrn, int flags)
case 15:
case 16:
case 24:
+case 30:
break;
default:
xf86DrvMsg(pScrn->scrnIndex, X_ERROR,
@@ -327,8 +328,8 @@ DUMMYPreInit(ScrnInfoPtr pScrn, int flags)
pScrn->rgbBits = 8;

 /* Get the depth24 pixmap format */
-if (pScrn->depth == 24 && pix24bpp == 0)
-   pix24bpp = xf86GetBppFromDepth(pScrn, 24);
+if (pScrn->depth >= 24 && pix24bpp == 0)
+   pix24bpp = xf86GetBppFromDepth(pScrn, pScrn->depth);

 /*
  * This must happen after pScrn->display has been set because


This looks simple enough and works very well, stable. Unfortunately,
this also causes any calls to RandR to crash the server:

Program received signal SIGSEGV, Segmentation fault.
0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
numColors=, indices=, colors=0xd01a80,
pVisual=)
at dummy_driver.c:513
513dPtr->colors[index].red = colors[index].red << shift;
(gdb) bt
#0  0x7fddc2a78b98 in DUMMYLoadPalette (pScrn=,
numColors=, indices=, colors=0xd01a80,
pVisual=)
at dummy_driver.c:513
#1  0x00480fb2 in CMapRefreshColors ()
#2  0x004816e8 in CMapReinstallMap ()
#3  0x004817ca in CMapSwitchMode ()
#4  0x0047407a in xf86SwitchMode ()
#5  0x00497a53 in xf86RandRSetMode ()
#6  0x0049808a in xf86RandRSetConfig ()
#7  0x004fa398 in RRCrtcSet ()
#8  0x00507a46 in ProcRRSetScreenConfig ()
#9  0x00436daf in Dispatch ()
#10 0x0043add3 in dix_main ()
#11 0x7fddc7fb8731 in __libc_start_main (main=0x424d20 ,
argc=19, argv=0x7ffc98ec5e18, init=, fini=, rtld_fini=,
stack_end=0x7ffc98ec5e08) at ../csu/libc-start.c:289
#12 0x00424d59 in _start ()
(gdb)

It's always crashing in palette or colormap functions, ie:
CMapDestroyColormap or DUMMYLoadPalette, etc.

Could it be that there's a bug somewhere else that is only being
triggered with 30-bpp modes? Trying to use more colours / space than is
available in the current colourmap perhaps?
I had temporarily prevented crashes with only some (!) randr resizings
by clamping the "numColors" value to 256 in DUMMYLoadPalette.

If you want to try it for yourself, just apply the 30-bpp patch at the
top of this file and run:
Xorg +extension RANDR  -config xorg.conf :100
With this config:
http://xpra.org/trac/raw-attachment/ticket/909/xorg.conf
Then you can crash the server reliably with:
DISPLAY=:100 xrandr -s 640x480

I think I am out of my depth (pun), how do I debug from here?

Cheers
Antoine
___
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: https://lists.x.org/mailman/listinfo/xorg-devel