[transcode-devel] Solution for segfault when using export_dv

[Kees Bakker]

Hi guys,

On my system transcode gives a segfault when export to dv.
After analyzing the problem I found to cause to be here:

static unsigned char *bufalloc(size_t size)
{

#ifdef HAVE_GETPAGESIZE
   long buffer_align=getpagesize();
#else
   long buffer_align=0;
#endif

   char *buf = malloc(size + buffer_align);

   long adjust;

   if (buf == NULL) {
       fprintf(stderr, "(%s) out of memory", __FILE__);
   }

   adjust = buffer_align - ((long) buf) % buffer_align;

   if (adjust == buffer_align)
      adjust = 0;

   return (unsigned char *) (buf + adjust);
}

Notice the "((long) buf) % buffer_align". On my system it
turns out that buf is in the high range, something like:
   0xb2c2c008
with the result that 'adjust' becomes 0x1ff8. And so the result
of bufalloc is misalligned AND too far into the buffer.
Next libdv will crash inside memset somewhere.

The solution is to use unsigned long. Here is a small patch
(hoping it survives the email).

Greetings, Kees Bakker

diff -ru transcode-1.0.4.orig/export/export_dv.c 
transcode-1.0.4/export/export_dv.c
--- transcode-1.0.4.orig/export/export_dv.c  2007-10-23 17:06:24.000000000 +0200
+++ transcode-1.0.4/export/export_dv.c  2005-07-04 09:23:00.000000000 +0200
@@ -61,13 +61,13 @@

    char *buf = malloc(size + buffer_align);

-   long adjust;
+   unsigned long adjust;

    if (buf == NULL) {
        fprintf(stderr, "(%s) out of memory", __FILE__);
    }

-   adjust = buffer_align - ((long) buf) % buffer_align;
+   adjust = buffer_align - ((unsigned long) buf) % buffer_align;

    if (adjust == buffer_align)
       adjust = 0;