Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-20 Thread Michael Niedermayer
On Mon, Nov 20, 2017 at 01:31:06PM -0800, Dale Curtis wrote: > On Mon, Nov 20, 2017 at 12:34 PM, Michael Niedermayer < > mich...@niedermayer.cc> wrote: > > > On Mon, Nov 20, 2017 at 12:01:11PM -0800, Dale Curtis wrote: > > > Derp, that's the wrong comment for this; I meant that for another

Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-20 Thread Dale Curtis
On Mon, Nov 20, 2017 at 12:34 PM, Michael Niedermayer < mich...@niedermayer.cc> wrote: > On Mon, Nov 20, 2017 at 12:01:11PM -0800, Dale Curtis wrote: > > Derp, that's the wrong comment for this; I meant that for another thread. > > Returning an error for optional user data seems a bit harsh, but

Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-20 Thread Michael Niedermayer
On Mon, Nov 20, 2017 at 12:01:11PM -0800, Dale Curtis wrote: > Derp, that's the wrong comment for this; I meant that for another thread. > Returning an error for optional user data seems a bit harsh, but if that's > what you want, I'm happy to change it. Please let me know if this is the > route

Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-20 Thread Dale Curtis
Derp, that's the wrong comment for this; I meant that for another thread. Returning an error for optional user data seems a bit harsh, but if that's what you want, I'm happy to change it. Please let me know if this is the route you want to take. - dale On Mon, Nov 20, 2017 at 11:55 AM, Dale

Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-20 Thread Dale Curtis
On Fri, Nov 17, 2017 at 6:22 PM, Michael Niedermayer wrote: > > Not sure what is best but > throwing part of the version silently away is not correct > most likely erroring out and asking for a sample video to be uploaded > would make sense if such a file is encountered

Re: [FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-17 Thread Michael Niedermayer
On Fri, Nov 17, 2017 at 04:07:42PM -0800, Dale Curtis wrote: > decode_user_data() attempts to create an integer |build| > value with 8 bits of spacing for 3 components. However > each component is an int32_t, so shifting each component > is undefined for values outside of the 8 bit range. > >

[FFmpeg-devel] [mpeg4video] Fix undefined shift on assumed 8-bit input.

2017-11-17 Thread Dale Curtis
decode_user_data() attempts to create an integer |build| value with 8 bits of spacing for 3 components. However each component is an int32_t, so shifting each component is undefined for values outside of the 8 bit range. This patch simply clamps input to 8-bits per component. Signed-off-by: Dale