Bug#364024: gdImageAALine() can cause segfault
On Mon, 22 May 2006 08:41:32 +0800 Paul wrote: I'll try and get in touch with the upstream guys, and as for the debian-maintainer thing, lets do that if I find any more things to patch. So far those were the only problems I had. Good (both of the things)! - Jonas -- * Jonas Smedegaard - idealist og Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ - Enden er nær: http://www.shibumi.org/eoti.htm pgpDIDeHFFiew.pgp Description: PGP signature
Bug#364024: gdImageAALine() can cause segfault
Jonas Smedegaard wrote: On Fri, 21 Apr 2006 17:20:53 +0800 Paul wrote: I continued to work on the anti-aliasing, specifically the lack of aliasing when the line hits the edge of the image. Attached is a patch that is applied AFTER the last one I posted. It cleans up the lines, so tiled images now look awesome. Also attached is the build-script and the last diff, plus the little segfault test file, for completeness. Please let me know what you are going to do with these patches. I intend to get them applied upstream, I'm hoping you could be the one to do it since you are the maintainer of its package and are therefore a good guy. Thanks for the bugreport and patches. And sorry for the lack of response. I am applying the patches now, and they will be in next Debian package in Debian unstable (sid). For upstream inclusion I'd actually prefer if you would get in touch with them yourself. Both to properly get credited for your work, and also because you know better than me what you've done! As a sidenote: Are you interested in helping out maintaining the GD library for Debian? It does not rewuire you to become an official Debian developer (but is a plus if you choose to do so at a later time). I will then setup an SVN repository at alioth.debian.org and provide you write access. I will do the final uploading to Debian, but any and all improvements you make (as long as we agree on them - we shouldn't end up forking the code!) will be properly in your own name. - Jonas Hi Jonas, I'll try and get in touch with the upstream guys, and as for the debian-maintainer thing, lets do that if I find any more things to patch. So far those were the only problems I had. cheers! Paul -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#364024: gdImageAALine() can cause segfault
On Fri, 21 Apr 2006 17:20:53 +0800 Paul wrote: I continued to work on the anti-aliasing, specifically the lack of aliasing when the line hits the edge of the image. Attached is a patch that is applied AFTER the last one I posted. It cleans up the lines, so tiled images now look awesome. Also attached is the build-script and the last diff, plus the little segfault test file, for completeness. Please let me know what you are going to do with these patches. I intend to get them applied upstream, I'm hoping you could be the one to do it since you are the maintainer of its package and are therefore a good guy. Thanks for the bugreport and patches. And sorry for the lack of response. I am applying the patches now, and they will be in next Debian package in Debian unstable (sid). For upstream inclusion I'd actually prefer if you would get in touch with them yourself. Both to properly get credited for your work, and also because you know better than me what you've done! As a sidenote: Are you interested in helping out maintaining the GD library for Debian? It does not rewuire you to become an official Debian developer (but is a plus if you choose to do so at a later time). I will then setup an SVN repository at alioth.debian.org and provide you write access. I will do the final uploading to Debian, but any and all improvements you make (as long as we agree on them - we shouldn't end up forking the code!) will be properly in your own name. - Jonas -- * Jonas Smedegaard - idealist og Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ - Enden er nær: http://www.shibumi.org/eoti.htm pgpsufRU7Mh9T.pgp Description: PGP signature
Bug#364024: gdImageAALine() can cause segfault
Hi,
I continued to work on the anti-aliasing, specifically the lack of aliasing when
the line hits the edge of the image.
Attached is a patch that is applied AFTER the last one I posted. It cleans up
the lines, so tiled images now look awesome.
Also attached is the build-script and the last diff, plus the little segfault
test file, for completeness.
Please let me know what you are going to do with these patches. I intend to get
them applied upstream, I'm hoping you could be the one to do it since you are
the maintainer of its package and are therefore a good guy.
cheers,
Paul
--- libgd2-2.0.33/gd.c 2006-04-21 11:31:18.0 +0800
+++ libgd2-2.0.33/gd.c 2006-04-21 11:31:15.0 +0800
@@ -3095,7 +3095,8 @@
/* TBB: set the last pixel for consistency (=) */
while ((x 16) = x2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (y
8) 0xFF);
- gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col,
(~y 8) 0xFF);
+ if ((y 16) + 1 = im-cy2)
+gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col, (~y 8)
0xFF);
x += (1 16);
y += inc;
}
@@ -3116,7 +3117,8 @@
/* TBB: set the last pixel for consistency (=) */
while ((y16) = y2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (x
8) 0xFF);
- gdImageSetAAPixelColor(im, (x 16) + 1, (y
16),col, (~x 8) 0xFF);
+ if ((x 16) + 1 = im-cx2)
+gdImageSetAAPixelColor(im, (x 16) + 1, (y 16),col, (~x 8)
0xFF);
x += inc;
y += (116);
}
--- libgd2-2.0.33/gd.c 2006-04-21 16:37:55.0 +0800
+++ libgd2-2.0.33/gd.c 2006-04-21 16:39:07.0 +0800
@@ -3055,6 +3055,9 @@
im-tpixels[y][x]=gdTrueColorAlpha(dr, dg, db, gdAlphaOpaque);
}
+/* simple helper */
+inline int min_int(int a, int b) { return (a b ? a : b); }
+
static void gdImageAALine (gdImagePtr im, int x1, int y1, int x2, int y2, int
col)
{
/* keep them as 32bits */
@@ -3065,11 +3068,15 @@
gdImageLine(im, x1, y1, x2, y2, col);
return;
}
-/* TBB: use the clipping rectangle */
-if (clip_1d (x1, y1, x2, y2, im-cx1, im-cx2) == 0)
- return;
-if (clip_1d (y1, x1, y2, x2, im-cy1, im-cy2) == 0)
- return;
+
+ /* TBB: use the clipping rectangle */
+ /* use expanded image bounds to ensure we get all the AA pixels drawn on
the edges */
+ /* Note the +1 and -1 */
+ if (clip_1d (x1, y1, x2, y2, im-cx1-1, im-cx2+1) == 0)
+ return;
+ if (clip_1d (y1, x1, y2, x2, im-cy1-1, im-cy2+1) == 0)
+ return;
+
dx = x2 - x1;
dy = y2 - y1;
@@ -3089,15 +3096,25 @@
dx = x2 - x1;
dy = y2 - y1;
}
- x = x1 16;
+ x = x1;
y = y1 16;
inc = (dy * 65536) / dx;
+
/* TBB: set the last pixel for consistency (=) */
- while ((x 16) = x2) {
- gdImageSetAAPixelColor(im, x 16, y 16, col, (y
8) 0xFF);
- if ((y 16) + 1 = im-cy2)
-gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col, (~y 8)
0xFF);
- x += (1 16);
+ /* Constrain the loop to the image bounds */
+ if (x 0)
+ {
+ y -= inc * x;
+ x = 0;
+ }
+ x2 = min_int(x2,im-cx2);
+ while (x = x2) {
+ int py = y 16;
+ if (py = im-cy1 py = im-cy2)
+gdImageSetAAPixelColor(im, x, py, col, (y 8) 0xFF);
+ if (py + 1 = im-cy2)
+gdImageSetAAPixelColor(im, x, py + 1,col, (~y 8) 0xFF);
+ x += 1;
y += inc;
}
} else {
@@ -3112,15 +3129,26 @@
dy = y2 - y1;
}
x = x1 16;
- y = y1 16;
+ // y = y1 16;
+ y = y1;
inc = (dx * 65536) / dy;
+
/* TBB: set the last pixel for consistency (=) */
- while ((y16) = y2) {
- gdImageSetAAPixelColor(im, x 16, y 16, col, (x
8) 0xFF);
- if ((x 16) + 1 = im-cx2)
-gdImageSetAAPixelColor(im, (x 16) + 1, (y 16),col, (~x 8)
0xFF);
+ /* Constrain the loop to the image bounds */
+ if (y 0)
+ {
+ x -= inc * y;
+ y = 0;
+ }
+ y2 = min_int(y2,im-cy2);
+ while (y = y2) {
+ int px = x 16;
+ if (px = im-cx1 px = im-cx2)
+gdImageSetAAPixelColor(im, px, y, col, (x 8) 0xFF);
+ if (px + 1 = im-cx2)
+gdImageSetAAPixelColor(im, px + 1, y,col, (~x 8) 0xFF);
x += inc;
- y += (116);
+
Bug#364024: gdImageAALine() can cause segfault
Package: libgd2-xpm
Version: 2.0.33-3
Severity: important
I found an off-by-one bug that repeatedly causes segfaults:
$ cat test_segfault.c
#include gd.h
int main()
{
gdImagePtr im = gdImageCreateTrueColor(100,100);
int colour = gdTrueColorAlpha(100,100,100,100);
gdImageSetAntiAliased(im,colour);
gdImageLine(im, 95, 100, 100, 98, gdAntiAliased);
return 0;
}
$ gcc -o test_segfault test_segfault.c -lgd
$ ./test_segfault
Segmentation fault
$
I made a patch that solves the problem. Its attached.
thanks,
Paul
-- System Information:
Debian Release: testing/unstable
APT prefers testing
APT policy: (650, 'testing'), (600, 'unstable')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.15.20060419
Locale: LANG=en_AU, LC_CTYPE=en_AU (charmap=ISO-8859-1)
Versions of packages libgd2-xpm depends on:
ii libc6 2.3.6-7GNU C Library: Shared libraries
ii libfontconfig12.3.2-5.1 generic font configuration library
ii libfreetype6 2.1.10-1 FreeType 2 font engine, shared lib
ii libjpeg62 6b-12 The Independent JPEG Group's JPEG
ii libpng12-01.2.8rel-5 PNG library - runtime
ii libx11-6 6.9.0.dfsg.1-6 X Window System protocol client li
ii libxpm4 6.9.0.dfsg.1-6 X pixmap library
ii zlib1g1:1.2.3-11 compression library - runtime
libgd2-xpm recommends no packages.
-- no debconf information
--- gd.c2006-04-21 10:58:02.0 +0800
+++ gd.c2006-04-21 10:58:18.0 +0800
@@ -3095,7 +3095,9 @@
/* TBB: set the last pixel for consistency (=) */
while ((x 16) = x2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (y
8) 0xFF);
- gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col,
(~y 8) 0xFF);
+ // the +1 can push it past the image bounds
+ if ((y 16) + 1 = im-cy2)
+gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col, (~y 8)
0xFF);
x += (1 16);
y += inc;
}
@@ -3116,6 +3118,9 @@
/* TBB: set the last pixel for consistency (=) */
while ((y16) = y2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (x
8) 0xFF);
+ // the +1 can push it past the image bounds
+ if ((x 16) + 1 = im-cx2)
+gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col, (~y 8)
0xFF);
gdImageSetAAPixelColor(im, (x 16) + 1, (y
16),col, (~x 8) 0xFF);
x += inc;
y += (116);
Bug#364024: gdImageAALine() can cause segfault
Sorry, patch was slightly wrong.
I've also attached a script that downloads, patches and installs the
corrected -xpm and -xpm-dev packages.
thanks
Paul
#include gd.h
int main()
{
gdImagePtr im = gdImageCreateTrueColor(100,100);
int colour = gdTrueColorAlpha(100,100,100,100);
gdImageSetAntiAliased(im,colour);
gdImageLine(im, 95, 100, 100, 98, gdAntiAliased);
return 0;
}
make_libgd2.sh
Description: Bourne shell script
--- libgd2-2.0.33/gd.c 2006-04-21 11:31:18.0 +0800
+++ libgd2-2.0.33/gd.c 2006-04-21 11:31:15.0 +0800
@@ -3095,7 +3095,8 @@
/* TBB: set the last pixel for consistency (=) */
while ((x 16) = x2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (y
8) 0xFF);
- gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col,
(~y 8) 0xFF);
+ if ((y 16) + 1 = im-cy2)
+gdImageSetAAPixelColor(im, x 16, (y 16) + 1,col, (~y 8)
0xFF);
x += (1 16);
y += inc;
}
@@ -3116,7 +3117,8 @@
/* TBB: set the last pixel for consistency (=) */
while ((y16) = y2) {
gdImageSetAAPixelColor(im, x 16, y 16, col, (x
8) 0xFF);
- gdImageSetAAPixelColor(im, (x 16) + 1, (y
16),col, (~x 8) 0xFF);
+ if ((x 16) + 1 = im-cx2)
+gdImageSetAAPixelColor(im, (x 16) + 1, (y 16),col, (~x 8)
0xFF);
x += inc;
y += (116);
}
