On Wed, Jan 04, 2012 at 12:23:01AM -0700, John Brooks wrote: > On Tue, Jan 3, 2012 at 11:27 PM, John Brooks <[email protected]> > wrote: > > For small video dimensions, these calculations of the upper bound > > for pixel access may have a negative result. Using an unsigned > > comparison to bound a potentially negative value only works if > > the greater operand is non-negative. Fixed by doing edge emulation > > when the upper bound is probably negative, everywhere that this > > pattern appears. > > --- > > libavcodec/vc1dec.c | 8 ++++++-- > > 1 files changed, 6 insertions(+), 2 deletions(-) > > > > I guess compiling might be a good idea too.. > > Update with a fixed typo attached. I ran fate for the whole set as well. > > - John
> From 31702a01807255575b63b4451ac03cc49a461f4f Mon Sep 17 00:00:00 2001 > From: John Brooks <[email protected]> > Date: Sun, 11 Dec 2011 02:37:21 -0700 > Subject: [PATCH 1/1] vc1dec: fix invalid memory access for small video > dimensions > > For small video dimensions, these calculations of the upper bound > for pixel access may have a negative result. Using an unsigned > comparison to bound a potentially negative value only works if > the greater operand is non-negative. Fixed by doing edge emulation > when the upper bound is probably negative, everywhere that this > pattern appears. > --- > libavcodec/vc1dec.c | 8 ++++++-- > 1 files changed, 6 insertions(+), 2 deletions(-) > > diff --git a/libavcodec/vc1dec.c b/libavcodec/vc1dec.c > index 3cb7661..c2f679e 100644 > --- a/libavcodec/vc1dec.c > +++ b/libavcodec/vc1dec.c > @@ -568,6 +568,7 @@ static void vc1_mc_1mv(VC1Context *v, int dir) > } > > if (v->rangeredfrm || (v->mv_mode == MV_PMODE_INTENSITY_COMP) > + || s->h_edge_pos - s->mspel * 3 < 19 || v_edge_pos - s->mspel * 3 < > 19 > || (unsigned)(src_x - s->mspel) > s->h_edge_pos - (mx&3) - 16 - > s->mspel * 3 > || (unsigned)(src_y - s->mspel) > v_edge_pos - (my&3) - 16 - > s->mspel * 3) { > uint8_t *uvbuf = s->edge_emu_buffer + 19 * s->linesize; > @@ -799,8 +800,9 @@ static void vc1_mc_4mv_luma(VC1Context *v, int n, int dir) > if (fieldmv && (src_y & 1) && src_y < 4) > src_y--; > if (v->rangeredfrm || (v->mv_mode == MV_PMODE_INTENSITY_COMP) > + || s->h_edge_pos - s->mspel * 2 < 11 > || (unsigned)(src_x - s->mspel) > s->h_edge_pos - (mx & 3) - 8 - > s->mspel * 2 > - || (unsigned)(src_y - (s->mspel << fieldmv)) > v_edge_pos - (my & 3) > - ((8 + s->mspel * 2) << fieldmv)) { > + || (unsigned)(src_y - (s->mspel << fieldmv)) > FFMAX(v_edge_pos - > (my & 3) - ((8 + s->mspel * 2) << fieldmv), 0)) { this particular change should be not needed anymore or what? In general patch looks good. _______________________________________________ libav-devel mailing list [email protected] https://lists.libav.org/mailman/listinfo/libav-devel
