Wasted a good chunk of time on that one.
>From f1a65814f119cb83d16e15093e81946f3adcde61 Mon Sep 17 00:00:00 2001
From: Pierre-Loup A. Griffais <pgriff...@nvidia.com>
Date: Tue, 20 Mar 2012 16:46:22 -0700
Subject: [PATCH] xrandr: move transform limit checking after scaling
This would trigger for legit scaled matrices, resulting in the wrong
extents getting computed.
Signed-off-by: Pierre-Loup A. Griffais <pgriff...@nvidia.com>
---
xrandr.c | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/xrandr.c b/xrandr.c
index 35dff3e..622f6a9 100644
--- a/xrandr.c
+++ b/xrandr.c
@@ -432,14 +432,15 @@ transform_point (XTransform *transform, double *xp, double *yp)
v = 0;
for (i = 0; i < 3; i++)
v += (XFixedToDouble (transform->matrix[j][i]) * vector[i]);
- if (v > 32767 || v < -32767)
- return False;
result[j] = v;
}
if (!result[2])
return False;
- for (j = 0; j < 2; j++)
+ for (j = 0; j < 2; j++) {
vector[j] = result[j] / result[2];
+ if (vector[j] > 32767 || vector[j] < -32767)
+ return False;
+ }
*xp = vector[0];
*yp = vector[1];
return True;
--
1.7.4.1
_______________________________________________
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: http://lists.x.org/mailman/listinfo/xorg-devel
