Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On date Wednesday 2023-08-02 07:36:43 +0200, Stefano Sabatini wrote: > On date Friday 2023-07-28 09:44:37 +0200, Stefano Sabatini wrote: > > On date Wednesday 2023-07-26 10:52:57 +, Carotti, Elias wrote: [...] > > > Agreed. > > > For clarity I am attaching again the patch with Anton's changes just > > > rebased on the current master branch. > > > Anton, is this still ok to be merged? > > > > I'm fine with the current patch, the behavior can then be extended > > by extending side data with multiple-label for the same type. > > > > Anton, if you are fine with it I would merge this one (and maybe get > > included in the next release if we are not too late already). > > I'll push this in a few days unless there are more comments. Rebased and applied with a minor change (AV AV_VIDEO_HINT_* => AV_VIDEO_HINT_TYPE_*). Thanks Elias for the contribution! ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On date Friday 2023-07-28 09:44:37 +0200, Stefano Sabatini wrote: > On date Wednesday 2023-07-26 10:52:57 +, Carotti, Elias wrote: > > On Mon, 2023-07-24 at 01:27 +0200, Stefano Sabatini wrote: > > > CAUTION: This email originated from outside of the organization. Do > > > not click links or open attachments unless you can confirm the sender > > > and know the content is safe. > > > > > > > > > > > > > > > > > > sorry to nitpick, but if we have this information in the single > > > > rect > > > > should not we use that instead? > > > > > > > > Basically I see two options: > > > > 1. generic VideoHint, storing single hint type and rects > > > > > > > > for example we might have type=changed, and the rects storing only > > > > the > > > > changed rects > > > > > > > > If we want to extend this then we need to add a new type. The > > > > problem > > > > might be if we want to store more than one VideoHint in the side > > > > data > > > > (it it possible to store more than one AVVideoHint, each one with a > > > > different type, as side data?). Suppose for example we want to add > > > > a > > > > new hint type, e.g. ROI, then we can add a new AVHideoHint with > > > > type=roi - but I don't know if we can have several frame hint side > > > > data (each one with different individual types). > > > > > > > > 2. generic VideoHint, storing rects each one containing the hint > > > > type > > > > > > > > for example we might have a list of rects, each one with > > > > type=changed|constant|roi. This would allow one to store different > > > > kind of hints in the same AVVideoHint structure, but at the same > > > > time > > > > would enable inconsistent data (for example we might have changed > > > > and > > > > unchanged rects in the same list of rects) > > > > > > > > > I don't think it allowed such inconsistency the way it was made. > > Basically, the semantics was that the Hint type could be either > > CONSTANT or CHANGED and that was the basis to interpret the rects with > > type DAMAGE. > > Yes, but then there is no guarantee that you'll have all rects with > the same subtype, and you need to iterate and check each one to > check the subtype (so you might have mixed constant/changed and > unclear behavior to apply in this case). > > > > > In each case, storing the type in the generic AVVideHint and in > > > > each > > > > rect might lead to inconsistent states (e.g. you might have generic > > > > type=damage, and have type=roi and type=changed in the rects), in > > > > this > > > > case having the type in both the general structure and in each > > > > rects > > > > seems redundant and leads to possible data inconsistencies. > > > > > > > > ... > > > > > > > > > > > I would rather go to solution 1, but I'm not sure if having more > > > > than > > > > one hint in the side data (with different subtypes) is possible and > > > > wanted. If this is the case, probably the best solution would be to > > > > store more than one AVVideoHint (each one with an individual type > > > > and > > > > list of rects) in the single side data object. > > > > > > Elaborating on this. > > > > > > 1. Ideally we might want to extend the av_frame_get_side_data API to > > > be able to store more than one entry per side data, by adding a > > > unique > > > label to the side data, but this requires probably more effort and > > > extends the scope even further. > > > > > > 2. Alternatively, we might extend the hint API like this: > > > > > > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, > > > size_t nb_rects); > > > > > > AVVideoHint *av_video_hint_extend_side_data(AVFrame *frame, > > > size_t nb_rects); > > > > > > by marking the continuation on the previous hint (but this would > > > complicate deserialization as you need then to iterate to get all the > > > side data). > > > > > > Or we could make a variadic function like this: > > > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, size_t > > > nb_rects, ...); > > > > > > Alternatively, we might store in AVVideoHint more than one hint, but > > > this somehow conflict with the general design. > > > > > > ... > > > > > > As a low effort approach, I would keep the initial design of a single > > > hint type per side-data (dropping the possibility of having more than > > > one hint type in the same side data), which should be fixed > > > generically by implementing 1. > > > > Agreed. > > For clarity I am attaching again the patch with Anton's changes just > > rebased on the current master branch. > > Anton, is this still ok to be merged? > > I'm fine with the current patch, the behavior can then be extended > by extending side data with multiple-label for the same type. > > Anton, if you are fine with it I would merge this one (and maybe get > included in the next release if we are not too late already). I'll push this in a few days unless there are
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On date Wednesday 2023-07-26 10:52:57 +, Carotti, Elias wrote: > On Mon, 2023-07-24 at 01:27 +0200, Stefano Sabatini wrote: > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender > > and know the content is safe. > > > > > > > > > > > > sorry to nitpick, but if we have this information in the single > > > rect > > > should not we use that instead? > > > > > > Basically I see two options: > > > 1. generic VideoHint, storing single hint type and rects > > > > > > for example we might have type=changed, and the rects storing only > > > the > > > changed rects > > > > > > If we want to extend this then we need to add a new type. The > > > problem > > > might be if we want to store more than one VideoHint in the side > > > data > > > (it it possible to store more than one AVVideoHint, each one with a > > > different type, as side data?). Suppose for example we want to add > > > a > > > new hint type, e.g. ROI, then we can add a new AVHideoHint with > > > type=roi - but I don't know if we can have several frame hint side > > > data (each one with different individual types). > > > > > > 2. generic VideoHint, storing rects each one containing the hint > > > type > > > > > > for example we might have a list of rects, each one with > > > type=changed|constant|roi. This would allow one to store different > > > kind of hints in the same AVVideoHint structure, but at the same > > > time > > > would enable inconsistent data (for example we might have changed > > > and > > > unchanged rects in the same list of rects) > > > > > I don't think it allowed such inconsistency the way it was made. > Basically, the semantics was that the Hint type could be either > CONSTANT or CHANGED and that was the basis to interpret the rects with > type DAMAGE. Yes, but then there is no guarantee that you'll have all rects with the same subtype, and you need to iterate and check each one to check the subtype (so you might have mixed constant/changed and unclear behavior to apply in this case). > > > In each case, storing the type in the generic AVVideHint and in > > > each > > > rect might lead to inconsistent states (e.g. you might have generic > > > type=damage, and have type=roi and type=changed in the rects), in > > > this > > > case having the type in both the general structure and in each > > > rects > > > seems redundant and leads to possible data inconsistencies. > > > > > > ... > > > > > > > > I would rather go to solution 1, but I'm not sure if having more > > > than > > > one hint in the side data (with different subtypes) is possible and > > > wanted. If this is the case, probably the best solution would be to > > > store more than one AVVideoHint (each one with an individual type > > > and > > > list of rects) in the single side data object. > > > > Elaborating on this. > > > > 1. Ideally we might want to extend the av_frame_get_side_data API to > > be able to store more than one entry per side data, by adding a > > unique > > label to the side data, but this requires probably more effort and > > extends the scope even further. > > > > 2. Alternatively, we might extend the hint API like this: > > > > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, > > size_t nb_rects); > > > > AVVideoHint *av_video_hint_extend_side_data(AVFrame *frame, > > size_t nb_rects); > > > > by marking the continuation on the previous hint (but this would > > complicate deserialization as you need then to iterate to get all the > > side data). > > > > Or we could make a variadic function like this: > > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, size_t > > nb_rects, ...); > > > > Alternatively, we might store in AVVideoHint more than one hint, but > > this somehow conflict with the general design. > > > > ... > > > > As a low effort approach, I would keep the initial design of a single > > hint type per side-data (dropping the possibility of having more than > > one hint type in the same side data), which should be fixed > > generically by implementing 1. > > Agreed. > For clarity I am attaching again the patch with Anton's changes just > rebased on the current master branch. > Anton, is this still ok to be merged? I'm fine with the current patch, the behavior can then be extended by extending side data with multiple-label for the same type. Anton, if you are fine with it I would merge this one (and maybe get included in the next release if we are not too late already). Thanks. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On Mon, 2023-07-24 at 01:27 +0200, Stefano Sabatini wrote: > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender > and know the content is safe. > > > > > > sorry to nitpick, but if we have this information in the single > > rect > > should not we use that instead? > > > > Basically I see two options: > > 1. generic VideoHint, storing single hint type and rects > > > > for example we might have type=changed, and the rects storing only > > the > > changed rects > > > > If we want to extend this then we need to add a new type. The > > problem > > might be if we want to store more than one VideoHint in the side > > data > > (it it possible to store more than one AVVideoHint, each one with a > > different type, as side data?). Suppose for example we want to add > > a > > new hint type, e.g. ROI, then we can add a new AVHideoHint with > > type=roi - but I don't know if we can have several frame hint side > > data (each one with different individual types). > > > > 2. generic VideoHint, storing rects each one containing the hint > > type > > > > for example we might have a list of rects, each one with > > type=changed|constant|roi. This would allow one to store different > > kind of hints in the same AVVideoHint structure, but at the same > > time > > would enable inconsistent data (for example we might have changed > > and > > unchanged rects in the same list of rects) > > I don't think it allowed such inconsistency the way it was made. Basically, the semantics was that the Hint type could be either CONSTANT or CHANGED and that was the basis to interpret the rects with type DAMAGE. > > In each case, storing the type in the generic AVVideHint and in > > each > > rect might lead to inconsistent states (e.g. you might have generic > > type=damage, and have type=roi and type=changed in the rects), in > > this > > case having the type in both the general structure and in each > > rects > > seems redundant and leads to possible data inconsistencies. > > > > ... > > > > > I would rather go to solution 1, but I'm not sure if having more > > than > > one hint in the side data (with different subtypes) is possible and > > wanted. If this is the case, probably the best solution would be to > > store more than one AVVideoHint (each one with an individual type > > and > > list of rects) in the single side data object. > > Elaborating on this. > > 1. Ideally we might want to extend the av_frame_get_side_data API to > be able to store more than one entry per side data, by adding a > unique > label to the side data, but this requires probably more effort and > extends the scope even further. > > 2. Alternatively, we might extend the hint API like this: > > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, > size_t nb_rects); > > AVVideoHint *av_video_hint_extend_side_data(AVFrame *frame, > size_t nb_rects); > > by marking the continuation on the previous hint (but this would > complicate deserialization as you need then to iterate to get all the > side data). > > Or we could make a variadic function like this: > AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, size_t > nb_rects, ...); > > Alternatively, we might store in AVVideoHint more than one hint, but > this somehow conflict with the general design. > > ... > > As a low effort approach, I would keep the initial design of a single > hint type per side-data (dropping the possibility of having more than > one hint type in the same side data), which should be fixed > generically by implementing 1. Agreed. For clarity I am attaching again the patch with Anton's changes just rebased on the current master branch. Anton, is this still ok to be merged? Best, Elias NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico From fcd9051a9167f48083f0edc4bf44b3d09e568091 Mon Sep 17 00:00:00 2001 From: Elias Carotti Date: Mon, 10 Jul 2023 14:34:53 +0200 Subject: [PATCH] Add side data type to provide hint to the video encoders about unchanged portions of each frame. Signed-off-by: Anton Khirnov --- doc/APIchanges | 3 ++ libavutil/Makefile | 2 + libavutil/frame.h | 10 libavutil/version.h| 2 +- libavutil/video_hint.c | 82 libavutil/video_hint.h | 105 + 6 files changed, 203 insertions(+), 1 deletion(-) create mode 100644 libavutil/video_hint.c create mode 100644 libavutil/video_hint.h diff --git a/doc/APIchanges b/doc/APIchanges index 5afe8bcb75..2adc7b5225 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -2,6 +2,9 @@ The last version increases of all libraries were on
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On date Tuesday 2023-07-18 00:19:40 +0200, Stefano Sabatini wrote:
> On date Monday 2023-07-10 12:51:25 +, Carotti, Elias wrote:
> > On Mon, 2023-07-10 at 08:13 +, Carotti, Elias wrote:
> > >
> >
> > > AVVideoHint is a bad name for something like this.
> > > Could you borrow some wording from graphics and call it
> > > AVVideoDamagedHint or maybe AVVideoChangedAreaHint or a combination
> > > of both?
> > > I'd prefer the former, damage is standard language in graphics
> > > circles about what has changed since the last frame.
> > >
> > > Hi,
> > > I have no strong objections on this. Personally I also like the
> > > AVVideoDamagedHint name best, my only concern is that it is strictly
> > > related to the current use/implementation
> > > (it's true right now that's the only kind of hint) while it may turn
> > > out to be a bad naming decision should other forms of hinting for the
> > > encoder be added in the future.
> > > That said, I am fine with the change too.
>
> AVVideoDamagedHint would be too specific (what if we want to add
> "hinting" for different things?).
>
> > > Elias
> > >
> >
> > I added a type to the AVVideoRect struct. This should solve the naming
> > issue above while preserving the possibility to extend this to
> > different hinting types.
> > These are the only changes to Anton's version.
> > Best
> > Elias
> >
> >
> >
> >
> >
> >
> >
> > NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle
> > Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale:
> > 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico
> >
> >
>
> > From 8ef4f97410a6b78df048b71d9921a763da6255b3 Mon Sep 17 00:00:00 2001
> > From: Elias Carotti
> > Date: Mon, 10 Jul 2023 14:34:53 +0200
> > Subject: [PATCH] Add side data type to provide hint to the video encoders
> > about unchanged portions of each frame.
> >
> > Signed-off-by: Anton Khirnov
> > ---
> > doc/APIchanges | 3 ++
> > libavutil/Makefile | 2 +
> > libavutil/frame.h | 10
> > libavutil/version.h| 2 +-
> > libavutil/video_hint.c | 82 +
> > libavutil/video_hint.h | 117 +
> > 6 files changed, 215 insertions(+), 1 deletion(-)
> > create mode 100644 libavutil/video_hint.c
> > create mode 100644 libavutil/video_hint.h
> >
> > diff --git a/doc/APIchanges b/doc/APIchanges
> > index 27f835cfce..0cda51fdee 100644
> > --- a/doc/APIchanges
> > +++ b/doc/APIchanges
> > @@ -2,6 +2,9 @@ The last version increases of all libraries were on
> > 2023-02-09
> >
> > API changes, most recent first:
> >
> > +2023-07-xx - xx - lavu 58.15.100 - video_hint.h
> > + Add AVVideoHint API.
> > +
> > 2023-07-05 - xx - lavu 58.14.100 - random_seed.h
> >Add av_random_bytes()
> >
> > diff --git a/libavutil/Makefile b/libavutil/Makefile
> > index bd9c6f9e32..7828c94dc5 100644
> > --- a/libavutil/Makefile
> > +++ b/libavutil/Makefile
> > @@ -91,6 +91,7 @@ HEADERS = adler32.h
> > \
> >tea.h \
> >tx.h \
> >film_grain_params.h \
> > + video_hint.h
> >
> > ARCH_HEADERS = bswap.h \
> > intmath.h\
> > @@ -181,6 +182,7 @@ OBJS = adler32.o
> > \
> > uuid.o \
> > version.o\
> > video_enc_params.o \
> > + video_hint.o \
> > film_grain_params.o \
> >
> >
> > diff --git a/libavutil/frame.h b/libavutil/frame.h
> > index a491315f25..c0c1b23db7 100644
> > --- a/libavutil/frame.h
> > +++ b/libavutil/frame.h
> > @@ -214,6 +214,16 @@ enum AVFrameSideDataType {
> > * Ambient viewing environment metadata, as defined by H.274.
> > */
> > AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT,
> > +
> > +/**
> > + * Provide encoder-specific hinting information about changed/unchanged
> > + * portions of a frame. It can be used to pass information about which
> > + * macroblocks can be skipped because they didn't change from the
> > + * corresponding ones in the previous frame. This could be useful for
> > + * applications which know this information in advance to speed up
> > + * encoding.
> > + */
> > +AV_FRAME_DATA_VIDEO_HINT,
> > };
> >
> > enum AVActiveFormatDescription {
> > diff --git a/libavutil/version.h b/libavutil/version.h
> >
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On date Monday 2023-07-10 12:51:25 +, Carotti, Elias wrote:
> On Mon, 2023-07-10 at 08:13 +, Carotti, Elias wrote:
> >
>
> > AVVideoHint is a bad name for something like this.
> > Could you borrow some wording from graphics and call it
> > AVVideoDamagedHint or maybe AVVideoChangedAreaHint or a combination
> > of both?
> > I'd prefer the former, damage is standard language in graphics
> > circles about what has changed since the last frame.
> >
> > Hi,
> > I have no strong objections on this. Personally I also like the
> > AVVideoDamagedHint name best, my only concern is that it is strictly
> > related to the current use/implementation
> > (it's true right now that's the only kind of hint) while it may turn
> > out to be a bad naming decision should other forms of hinting for the
> > encoder be added in the future.
> > That said, I am fine with the change too.
AVVideoDamagedHint would be too specific (what if we want to add
"hinting" for different things?).
> > Elias
> >
>
> I added a type to the AVVideoRect struct. This should solve the naming
> issue above while preserving the possibility to extend this to
> different hinting types.
> These are the only changes to Anton's version.
> Best
> Elias
>
>
>
>
>
>
>
> NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle
> Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale:
> 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico
>
>
> From 8ef4f97410a6b78df048b71d9921a763da6255b3 Mon Sep 17 00:00:00 2001
> From: Elias Carotti
> Date: Mon, 10 Jul 2023 14:34:53 +0200
> Subject: [PATCH] Add side data type to provide hint to the video encoders
> about unchanged portions of each frame.
>
> Signed-off-by: Anton Khirnov
> ---
> doc/APIchanges | 3 ++
> libavutil/Makefile | 2 +
> libavutil/frame.h | 10
> libavutil/version.h| 2 +-
> libavutil/video_hint.c | 82 +
> libavutil/video_hint.h | 117 +
> 6 files changed, 215 insertions(+), 1 deletion(-)
> create mode 100644 libavutil/video_hint.c
> create mode 100644 libavutil/video_hint.h
>
> diff --git a/doc/APIchanges b/doc/APIchanges
> index 27f835cfce..0cda51fdee 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09
>
> API changes, most recent first:
>
> +2023-07-xx - xx - lavu 58.15.100 - video_hint.h
> + Add AVVideoHint API.
> +
> 2023-07-05 - xx - lavu 58.14.100 - random_seed.h
>Add av_random_bytes()
>
> diff --git a/libavutil/Makefile b/libavutil/Makefile
> index bd9c6f9e32..7828c94dc5 100644
> --- a/libavutil/Makefile
> +++ b/libavutil/Makefile
> @@ -91,6 +91,7 @@ HEADERS = adler32.h
> \
>tea.h \
>tx.h \
>film_grain_params.h \
> + video_hint.h
>
> ARCH_HEADERS = bswap.h \
> intmath.h\
> @@ -181,6 +182,7 @@ OBJS = adler32.o
> \
> uuid.o \
> version.o\
> video_enc_params.o \
> + video_hint.o \
> film_grain_params.o \
>
>
> diff --git a/libavutil/frame.h b/libavutil/frame.h
> index a491315f25..c0c1b23db7 100644
> --- a/libavutil/frame.h
> +++ b/libavutil/frame.h
> @@ -214,6 +214,16 @@ enum AVFrameSideDataType {
> * Ambient viewing environment metadata, as defined by H.274.
> */
> AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT,
> +
> +/**
> + * Provide encoder-specific hinting information about changed/unchanged
> + * portions of a frame. It can be used to pass information about which
> + * macroblocks can be skipped because they didn't change from the
> + * corresponding ones in the previous frame. This could be useful for
> + * applications which know this information in advance to speed up
> + * encoding.
> + */
> +AV_FRAME_DATA_VIDEO_HINT,
> };
>
> enum AVActiveFormatDescription {
> diff --git a/libavutil/version.h b/libavutil/version.h
> index 24af520e08..9e798b0e3f 100644
> --- a/libavutil/version.h
> +++ b/libavutil/version.h
> @@ -79,7 +79,7 @@
> */
>
> #define LIBAVUTIL_VERSION_MAJOR 58
> -#define LIBAVUTIL_VERSION_MINOR 14
> +#define LIBAVUTIL_VERSION_MINOR 15
> #define LIBAVUTIL_VERSION_MICRO 100
>
> #define LIBAVUTIL_VERSION_INT
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
On Mon, 2023-07-10 at 08:13 +, Carotti, Elias wrote:
>
> AVVideoHint is a bad name for something like this.
> Could you borrow some wording from graphics and call it
> AVVideoDamagedHint or maybe AVVideoChangedAreaHint or a combination
> of both?
> I'd prefer the former, damage is standard language in graphics
> circles about what has changed since the last frame.
>
> Hi,
> I have no strong objections on this. Personally I also like the
> AVVideoDamagedHint name best, my only concern is that it is strictly
> related to the current use/implementation
> (it's true right now that's the only kind of hint) while it may turn
> out to be a bad naming decision should other forms of hinting for the
> encoder be added in the future.
> That said, I am fine with the change too.
> Elias
>
I added a type to the AVVideoRect struct. This should solve the naming
issue above while preserving the possibility to extend this to
different hinting types.
These are the only changes to Anton's version.
Best
Elias
NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese
di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR
i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico
From 8ef4f97410a6b78df048b71d9921a763da6255b3 Mon Sep 17 00:00:00 2001
From: Elias Carotti
Date: Mon, 10 Jul 2023 14:34:53 +0200
Subject: [PATCH] Add side data type to provide hint to the video encoders
about unchanged portions of each frame.
Signed-off-by: Anton Khirnov
---
doc/APIchanges | 3 ++
libavutil/Makefile | 2 +
libavutil/frame.h | 10
libavutil/version.h| 2 +-
libavutil/video_hint.c | 82 +
libavutil/video_hint.h | 117 +
6 files changed, 215 insertions(+), 1 deletion(-)
create mode 100644 libavutil/video_hint.c
create mode 100644 libavutil/video_hint.h
diff --git a/doc/APIchanges b/doc/APIchanges
index 27f835cfce..0cda51fdee 100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09
API changes, most recent first:
+2023-07-xx - xx - lavu 58.15.100 - video_hint.h
+ Add AVVideoHint API.
+
2023-07-05 - xx - lavu 58.14.100 - random_seed.h
Add av_random_bytes()
diff --git a/libavutil/Makefile b/libavutil/Makefile
index bd9c6f9e32..7828c94dc5 100644
--- a/libavutil/Makefile
+++ b/libavutil/Makefile
@@ -91,6 +91,7 @@ HEADERS = adler32.h \
tea.h \
tx.h \
film_grain_params.h \
+ video_hint.h
ARCH_HEADERS = bswap.h \
intmath.h\
@@ -181,6 +182,7 @@ OBJS = adler32.o\
uuid.o \
version.o\
video_enc_params.o \
+ video_hint.o \
film_grain_params.o \
diff --git a/libavutil/frame.h b/libavutil/frame.h
index a491315f25..c0c1b23db7 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -214,6 +214,16 @@ enum AVFrameSideDataType {
* Ambient viewing environment metadata, as defined by H.274.
*/
AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT,
+
+/**
+ * Provide encoder-specific hinting information about changed/unchanged
+ * portions of a frame. It can be used to pass information about which
+ * macroblocks can be skipped because they didn't change from the
+ * corresponding ones in the previous frame. This could be useful for
+ * applications which know this information in advance to speed up
+ * encoding.
+ */
+AV_FRAME_DATA_VIDEO_HINT,
};
enum AVActiveFormatDescription {
diff --git a/libavutil/version.h b/libavutil/version.h
index 24af520e08..9e798b0e3f 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@
*/
#define LIBAVUTIL_VERSION_MAJOR 58
-#define LIBAVUTIL_VERSION_MINOR 14
+#define LIBAVUTIL_VERSION_MINOR 15
#define LIBAVUTIL_VERSION_MICRO 100
#define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
diff --git a/libavutil/video_hint.c b/libavutil/video_hint.c
new file mode 100644
index 00..5730ed6cfb
--- /dev/null
+++ b/libavutil/video_hint.c
@@ -0,0 +1,82 @@
+/*
+ * Copyright 2023 Elias Carotti
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ *
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
-Original Message- From: ffmpeg-devel On Behalf Of Lynne Sent: Sunday, July 9, 2023 3:11 PM To: FFmpeg development discussions and patches Subject: RE: [EXTERNAL][FFmpeg-devel] [PATCH] lavu: add AVVideoHint API CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Jul 9, 2023, 13:05 by an...@khirnov.net: > From: Elias Carotti > > Add side data type to provide hint to the video encoders about > unchanged portions of each frame. > > Signed-off-by: Anton Khirnov > --- > I've made couple small changes: > * rebased against current master > * consistently refer to rectangles or AVVideoRect, not blocks > * use size_t instead of int for AVVideoHint.nb_rects > * build unconditionally -- this is public API, it must always be > available regardless of what encoders are or are not available > * tweaked documentation > > If you have no objections to the changes, I'll push this in a few days. > --- > doc/APIchanges | 3 ++ > libavutil/Makefile | 2 + > libavutil/frame.h | 10 > libavutil/version.h| 2 +- > libavutil/video_hint.c | 82 +++ > libavutil/video_hint.h | 108 + > 6 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 > libavutil/video_hint.c create mode 100644 libavutil/video_hint.h > AVVideoHint is a bad name for something like this. Could you borrow some wording from graphics and call it AVVideoDamagedHint or maybe AVVideoChangedAreaHint or a combination of both? I'd prefer the former, damage is standard language in graphics circles about what has changed since the last frame. Hi, I have no strong objections on this. Personally I also like the AVVideoDamagedHint name best, my only concern is that it is strictly related to the current use/implementation (it's true right now that's the only kind of hint) while it may turn out to be a bad naming decision should other forms of hinting for the encoder be added in the future. That said, I am fine with the change too. Elias ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
-Original Message-
From: ffmpeg-devel On Behalf Of Anton Khirnov
Sent: Sunday, July 9, 2023 1:05 PM
To: ffmpeg-devel@ffmpeg.org
Subject: [EXTERNAL] [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
From: Elias Carotti
Add side data type to provide hint to the video encoders about unchanged
portions of each frame.
Signed-off-by: Anton Khirnov
---
I've made couple small changes:
* rebased against current master
* consistently refer to rectangles or AVVideoRect, not blocks
* use size_t instead of int for AVVideoHint.nb_rects
* build unconditionally -- this is public API, it must always be
available regardless of what encoders are or are not available
* tweaked documentation
If you have no objections to the changes, I'll push this in a few days.
Hi Anton,
I'm perfectly fine with all the changes, so feel free to proceed.
Best,
Elias
---
doc/APIchanges | 3 ++
libavutil/Makefile | 2 +
libavutil/frame.h | 10
libavutil/version.h| 2 +-
libavutil/video_hint.c | 82 +++
libavutil/video_hint.h | 108 +
6 files changed, 206 insertions(+), 1 deletion(-) create mode 100644
libavutil/video_hint.c create mode 100644 libavutil/video_hint.h
diff --git a/doc/APIchanges b/doc/APIchanges index 27f835cfce..0cda51fdee 100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09
API changes, most recent first:
+2023-07-xx - xx - lavu 58.15.100 - video_hint.h
+ Add AVVideoHint API.
+
2023-07-05 - xx - lavu 58.14.100 - random_seed.h
Add av_random_bytes()
diff --git a/libavutil/Makefile b/libavutil/Makefile index
bd9c6f9e32..7828c94dc5 100644
--- a/libavutil/Makefile
+++ b/libavutil/Makefile
@@ -91,6 +91,7 @@ HEADERS = adler32.h
\
tea.h \
tx.h \
film_grain_params.h \
+ video_hint.h
ARCH_HEADERS = bswap.h \
intmath.h\
@@ -181,6 +182,7 @@ OBJS = adler32.o
\
uuid.o \
version.o\
video_enc_params.o \
+ video_hint.o \
film_grain_params.o \
diff --git a/libavutil/frame.h b/libavutil/frame.h index a491315f25..c0c1b23db7
100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -214,6 +214,16 @@ enum AVFrameSideDataType {
* Ambient viewing environment metadata, as defined by H.274.
*/
AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT,
+
+/**
+ * Provide encoder-specific hinting information about changed/unchanged
+ * portions of a frame. It can be used to pass information about which
+ * macroblocks can be skipped because they didn't change from the
+ * corresponding ones in the previous frame. This could be useful for
+ * applications which know this information in advance to speed up
+ * encoding.
+ */
+AV_FRAME_DATA_VIDEO_HINT,
};
enum AVActiveFormatDescription {
diff --git a/libavutil/version.h b/libavutil/version.h index
24af520e08..9e798b0e3f 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@
*/
#define LIBAVUTIL_VERSION_MAJOR 58
-#define LIBAVUTIL_VERSION_MINOR 14
+#define LIBAVUTIL_VERSION_MINOR 15
#define LIBAVUTIL_VERSION_MICRO 100
#define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
diff --git a/libavutil/video_hint.c b/libavutil/video_hint.c new file mode
100644 index 00..5730ed6cfb
--- /dev/null
+++ b/libavutil/video_hint.c
@@ -0,0 +1,82 @@
+/*
+ * Copyright 2023 Elias Carotti
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along
Re: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
Jul 9, 2023, 13:05 by an...@khirnov.net: > From: Elias Carotti > > Add side data type to provide hint to the video encoders about unchanged > portions of each frame. > > Signed-off-by: Anton Khirnov > --- > I've made couple small changes: > * rebased against current master > * consistently refer to rectangles or AVVideoRect, not blocks > * use size_t instead of int for AVVideoHint.nb_rects > * build unconditionally -- this is public API, it must always be > available regardless of what encoders are or are not available > * tweaked documentation > > If you have no objections to the changes, I'll push this in a few days. > --- > doc/APIchanges | 3 ++ > libavutil/Makefile | 2 + > libavutil/frame.h | 10 > libavutil/version.h| 2 +- > libavutil/video_hint.c | 82 +++ > libavutil/video_hint.h | 108 + > 6 files changed, 206 insertions(+), 1 deletion(-) > create mode 100644 libavutil/video_hint.c > create mode 100644 libavutil/video_hint.h > AVVideoHint is a bad name for something like this. Could you borrow some wording from graphics and call it AVVideoDamagedHint or maybe AVVideoChangedAreaHint or a combination of both? I'd prefer the former, damage is standard language in graphics circles about what has changed since the last frame. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] lavu: add AVVideoHint API
From: Elias Carotti
Add side data type to provide hint to the video encoders about unchanged
portions of each frame.
Signed-off-by: Anton Khirnov
---
I've made couple small changes:
* rebased against current master
* consistently refer to rectangles or AVVideoRect, not blocks
* use size_t instead of int for AVVideoHint.nb_rects
* build unconditionally -- this is public API, it must always be
available regardless of what encoders are or are not available
* tweaked documentation
If you have no objections to the changes, I'll push this in a few days.
---
doc/APIchanges | 3 ++
libavutil/Makefile | 2 +
libavutil/frame.h | 10
libavutil/version.h| 2 +-
libavutil/video_hint.c | 82 +++
libavutil/video_hint.h | 108 +
6 files changed, 206 insertions(+), 1 deletion(-)
create mode 100644 libavutil/video_hint.c
create mode 100644 libavutil/video_hint.h
diff --git a/doc/APIchanges b/doc/APIchanges
index 27f835cfce..0cda51fdee 100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09
API changes, most recent first:
+2023-07-xx - xx - lavu 58.15.100 - video_hint.h
+ Add AVVideoHint API.
+
2023-07-05 - xx - lavu 58.14.100 - random_seed.h
Add av_random_bytes()
diff --git a/libavutil/Makefile b/libavutil/Makefile
index bd9c6f9e32..7828c94dc5 100644
--- a/libavutil/Makefile
+++ b/libavutil/Makefile
@@ -91,6 +91,7 @@ HEADERS = adler32.h
\
tea.h \
tx.h \
film_grain_params.h \
+ video_hint.h
ARCH_HEADERS = bswap.h \
intmath.h\
@@ -181,6 +182,7 @@ OBJS = adler32.o
\
uuid.o \
version.o\
video_enc_params.o \
+ video_hint.o \
film_grain_params.o \
diff --git a/libavutil/frame.h b/libavutil/frame.h
index a491315f25..c0c1b23db7 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -214,6 +214,16 @@ enum AVFrameSideDataType {
* Ambient viewing environment metadata, as defined by H.274.
*/
AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT,
+
+/**
+ * Provide encoder-specific hinting information about changed/unchanged
+ * portions of a frame. It can be used to pass information about which
+ * macroblocks can be skipped because they didn't change from the
+ * corresponding ones in the previous frame. This could be useful for
+ * applications which know this information in advance to speed up
+ * encoding.
+ */
+AV_FRAME_DATA_VIDEO_HINT,
};
enum AVActiveFormatDescription {
diff --git a/libavutil/version.h b/libavutil/version.h
index 24af520e08..9e798b0e3f 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@
*/
#define LIBAVUTIL_VERSION_MAJOR 58
-#define LIBAVUTIL_VERSION_MINOR 14
+#define LIBAVUTIL_VERSION_MINOR 15
#define LIBAVUTIL_VERSION_MICRO 100
#define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
diff --git a/libavutil/video_hint.c b/libavutil/video_hint.c
new file mode 100644
index 00..5730ed6cfb
--- /dev/null
+++ b/libavutil/video_hint.c
@@ -0,0 +1,82 @@
+/*
+ * Copyright 2023 Elias Carotti
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+
+#include "avstring.h"
+#include "frame.h"
+#include "macros.h"
+#include "mem.h"
+#include "video_hint.h"
+
+AVVideoHint *av_video_hint_alloc(size_t nb_rects,
+ size_t *out_size)
+{
+struct TestStruct {
+AVVideoHinthint;
+AVVideoRectrect;
+};
+const size_t
