Hi,
When the following program is compiled with GCC 6 compiler running on
Fedora 24 with "-O2" flag, the overflow check (rowstride / channels !=
width) gets eliminated presumably due to undefined behavior.
// gcc -c -O2 too-much-optimization.c
//
// objdump -d too-much-optimization.o
#include <stdio.h>
#include <stdlib.h>
void *gdk_pixbuf_new_reduced(int has_alpha, int width, int height)
{
void *buf = NULL;
int channels;
int rowstride;
if (width < 0)
return NULL;
if (height < 0)
return 0;
channels = has_alpha ? 4 : 3;
rowstride = width * channels;
if (rowstride / channels != width) /* overflow check */
return NULL;
buf = calloc(height, rowstride);
if (!buf)
return NULL;
return buf;
}
This code is present in the gdk-pixbuf software package. The
elimination of this overflow check leads to memory corruption problems
later on in the gdk-pixbuf library.
Is it possible to detect such code patterns (multiplication operation
followed by division to check for overflow) which invoke undefined
behavior using Coccinelle?
I have not written Coccinelle scripts before. Any help would be great.
Thanks,
Dhiru
_______________________________________________
Cocci mailing list
[email protected]
https://systeme.lip6.fr/mailman/listinfo/cocci
