Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-08 Thread James Almer
On 7/8/2020 4:44 PM, Michael Niedermayer wrote: > On Tue, Jul 07, 2020 at 05:41:11PM -0300, James Almer wrote: >> On 7/7/2020 5:07 PM, Brian Kim wrote: >>> On Tue, Jul 7, 2020 at 6:34 AM James Almer wrote: If i understand this right, you could easily solve it with just the following

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-08 Thread Michael Niedermayer
On Tue, Jul 07, 2020 at 05:41:11PM -0300, James Almer wrote: > On 7/7/2020 5:07 PM, Brian Kim wrote: > > On Tue, Jul 7, 2020 at 6:34 AM James Almer wrote: > >> > >> If i understand this right, you could easily solve it with just the > >> following changes: > >> > >>> diff --git a/libavutil/imgutil

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-08 Thread Michael Niedermayer
On Tue, Jul 07, 2020 at 01:14:41PM -0700, Brian Kim wrote: > On Tue, Jul 7, 2020 at 4:35 AM Michael Niedermayer > wrote: > [...] > > I wonder if linesizes for newly added functions should be ptrdiff_t > > this would add some type converting loops though > > > > And size probably should be ptrdiff_

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-07 Thread James Almer
On 7/7/2020 5:07 PM, Brian Kim wrote: > On Tue, Jul 7, 2020 at 6:34 AM James Almer wrote: >> >> If i understand this right, you could easily solve it with just the >> following changes: >> >>> diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c >>> index 7f9c1b632c..48a373db01 100644 >>> ---

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-07 Thread Brian Kim
On Tue, Jul 7, 2020 at 4:35 AM Michael Niedermayer wrote: [...] > I wonder if linesizes for newly added functions should be ptrdiff_t > this would add some type converting loops though > > And size probably should be ptrdiff_t or int64_t to similarly be more future > proof Can these values be neg

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-07 Thread Brian Kim
On Tue, Jul 7, 2020 at 6:34 AM James Almer wrote: > > If i understand this right, you could easily solve it with just the > following changes: > > > diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c > > index 7f9c1b632c..48a373db01 100644 > > --- a/libavutil/imgutils.c > > +++ b/libavutil/i

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-07 Thread James Almer
On 7/1/2020 3:14 PM, Brian Kim wrote: > While running under Clang's UndefinedBehaviorSanitizer, I found a few > places where av_image_fill_pointers is called before buffers for the image > are allocated, so ptr is passed in as NULL. > > This leads to (currently harmless) UB when the plane sizes ar

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-07 Thread Michael Niedermayer
Hi On Wed, Jul 01, 2020 at 11:14:13AM -0700, Brian Kim wrote: > While running under Clang's UndefinedBehaviorSanitizer, I found a few > places where av_image_fill_pointers is called before buffers for the image > are allocated, so ptr is passed in as NULL. > > This leads to (currently harmless) U

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-06 Thread Brian Kim
I'm using clang 10.0.0 and configuring with with `configure --enable-gpl --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libopus --enable-libx264 --enable-libvpx --enable-no

Re: [FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-02 Thread Michael Niedermayer
On Wed, Jul 01, 2020 at 11:14:13AM -0700, Brian Kim wrote: > While running under Clang's UndefinedBehaviorSanitizer, I found a few > places where av_image_fill_pointers is called before buffers for the image > are allocated, so ptr is passed in as NULL. > > This leads to (currently harmless) UB wh

[FFmpeg-devel] libavutil/imgutils: UBSan nullptr-with-offset in av_image_fill_pointer

2020-07-01 Thread Brian Kim
While running under Clang's UndefinedBehaviorSanitizer, I found a few places where av_image_fill_pointers is called before buffers for the image are allocated, so ptr is passed in as NULL. This leads to (currently harmless) UB when the plane sizes are added to the null pointer, so I was wondering