Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Li, Zhong]

> On 31/01/18 08:16, Li, Zhong wrote:
> > As your comment, maybe we need to find a HEVC clip with CRA frames to
> test.
> 
> Try the conformance sample RAP_A_docomo_4.bit (you can find it in
> fate/hevc-conformance).  That contains CRA and RASL frames (and no IDR
> frames at all).
> 
> - Mark

Tested it. Yup, it is an issue just like your inference, the CRA frames aren't 
set as key frame for QSV. But software codec set as key frames.
I haven't find where MSDK expose this, maybe need file an issue to MSDK. 


___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Mark Thompson]

On 31/01/18 08:11, Li, Zhong wrote:
>>> @@ -416,6 +421,8 @@ FF_ENABLE_DEPRECATION_WARNINGS
> 
>>>  outsurf->Info.PicStruct & MFX_PICSTRUCT_FIELD_TFF;
> 
>>>  frame->interlaced_frame =
> 
>>>  !(outsurf->Info.PicStruct &
> 
>> MFX_PICSTRUCT_PROGRESSIVE);
> 
>>> +frame->pict_type =
> 
>> ff_qsv_map_pictype(out_frame->dec_info.FrameType);
> 
>>> +frame->key_frame = !!(out_frame->dec_info.FrameType &
> 
>>> + MFX_FRAMETYPE_IDR);
> 
>>>
> 
>>>  /* update the surface properties */
> 
>>>  if (avctx->pix_fmt == AV_PIX_FMT_QSV)
> 
>>>
> 
>>
> 
>> Does this do the right thing for H.265?  (Can't test right now.)
> 
> 
> 
> Testing should use a HEVC clips
> 
> 
> 
>> I'm asking that because I know that frame types are still a problem for H.265
> 
>> encoding, because not all IRAPs are IDR - when CRA frames are generated
> 
>> they aren't marked as key.  The FrameType field (as used here) is not
> 
>> sufficient in that case, but I never sorted out exactly what does contain the
> 
>> necessary information.
> 
>>
> 
>> - Mark
> 
> 
> 
> Exactly, only IDR type is exposed from FrameType here, and it is set by 
> parsing slice_header 
> (https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/shared/umc/codec/h265_dec/src/umc_h265_bitstream_headers.cpp#L1428
>  )
> 
> But the problem is (I am not familiar with HEVC as well as H264, so please 
> correct me if I am wrong), should CRA frames be treated as key frames?

Yes, they have to be.  IDR, CRA and BLA frames are all IRAP pictures, so they 
are key frames because seeking can always move to that point in the stream.

> See the description as blew (source 
> http://ieeexplore.ieee.org/document/6324417/?part=undefined%7Cfig4#fig4 )
> 
> [cid:image003.jpg@01D39AAE.23C1D950]
> 
> 
> 
> Here some leading pictures such as B30 can’t be decoded when random access 
> from I28.
> 
> As my understanding, all frames behand (in display order) key frame should be 
> decodable.

Yes, all frames after the IRAP in display order should be decodable.  However, 
B30 is not after I28 in display order - it be marked as RASL to indicate that 
it should be skipped when it is found in decode order.

> IDR it can make sure this, but CRA can’t. So at least not all of the CRA 
> frames can be treated as key frames.
> 
> Looks some people agree with me (refer Jeanlf’s comment on 
> https://github.com/gpac/gpac/issues/144)
I'm not familiar with the precise meaning of "sync sample" in this case, so I'm 
not sure.  If it means the same thing as IRAP (a point at which decoding of a 
stream can always start correctly without any pictures which precede it in 
decode order), then the commenter is wrong.


On 31/01/18 08:16, Li, Zhong wrote:
> As your comment, maybe we need to find a HEVC clip with CRA frames to test.

Try the conformance sample RAP_A_docomo_4.bit (you can find it in 
fate/hevc-conformance).  That contains CRA and RASL frames (and no IDR frames 
at all).

- Mark
___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Li, Zhong]

Sorry it is interrupted, I mean could you specify which hevc clip should be 
tested?

As your comment, maybe we need to find a HEVC clip with CRA frames to test.

From: Li, Zhong
Sent: Wednesday, January 31, 2018 4:11 PM
To: libav-devel@libav.org
Subject: RE: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and 
key_frame


> > @@ -416,6 +421,8 @@ FF_ENABLE_DEPRECATION_WARNINGS

> >  outsurf->Info.PicStruct & MFX_PICSTRUCT_FIELD_TFF;

> >  frame->interlaced_frame =

> >  !(outsurf->Info.PicStruct &

> MFX_PICSTRUCT_PROGRESSIVE);

> > +frame->pict_type =

> ff_qsv_map_pictype(out_frame->dec_info.FrameType);

> > +frame->key_frame = !!(out_frame->dec_info.FrameType &

> > + MFX_FRAMETYPE_IDR);

> >

> >  /* update the surface properties */

> >  if (avctx->pix_fmt == AV_PIX_FMT_QSV)

> >

>

> Does this do the right thing for H.265?  (Can't test right now.)



Testing should use a HEVC clips


___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Li, Zhong]

> > @@ -416,6 +421,8 @@ FF_ENABLE_DEPRECATION_WARNINGS

> >  outsurf->Info.PicStruct & MFX_PICSTRUCT_FIELD_TFF;

> >  frame->interlaced_frame =

> >  !(outsurf->Info.PicStruct &

> MFX_PICSTRUCT_PROGRESSIVE);

> > +frame->pict_type =

> ff_qsv_map_pictype(out_frame->dec_info.FrameType);

> > +frame->key_frame = !!(out_frame->dec_info.FrameType &

> > + MFX_FRAMETYPE_IDR);

> >

> >  /* update the surface properties */

> >  if (avctx->pix_fmt == AV_PIX_FMT_QSV)

> >

>

> Does this do the right thing for H.265?  (Can't test right now.)



Testing should use a HEVC clips



> I'm asking that because I know that frame types are still a problem for H.265

> encoding, because not all IRAPs are IDR - when CRA frames are generated

> they aren't marked as key.  The FrameType field (as used here) is not

> sufficient in that case, but I never sorted out exactly what does contain the

> necessary information.

>

> - Mark



Exactly, only IDR type is exposed from FrameType here, and it is set by parsing 
slice_header 
(https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/shared/umc/codec/h265_dec/src/umc_h265_bitstream_headers.cpp#L1428
 )

But the problem is (I am not familiar with HEVC as well as H264, so please 
correct me if I am wrong), should CRA frames be treated as key frames?

See the description as blew (source 
http://ieeexplore.ieee.org/document/6324417/?part=undefined%7Cfig4#fig4 )

[cid:image003.jpg@01D39AAE.23C1D950]



Here some leading pictures such as B30 can’t be decoded when random access from 
I28.

As my understanding, all frames behand (in display order) key frame should be 
decodable.

IDR it can make sure this, but CRA can’t. So at least not all of the CRA frames 
can be treated as key frames.

Looks some people agree with me (refer Jeanlf’s comment on 
https://github.com/gpac/gpac/issues/144)
___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Mark Thompson]

On 30/01/18 09:52, Zhong Li wrote:
> Currently key_frame and pict_type are unset.
> Add an extra param to fetch the picture type from qsv decoder
> 
> Signed-off-by: ChaoX A Liu 
> Signed-off-by: Zhong Li 
> ---
>  libavcodec/qsv.c  | 24 
>  libavcodec/qsv_internal.h |  3 +++
>  libavcodec/qsvdec.c   |  7 +++
>  3 files changed, 34 insertions(+)
> 
> diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
> index e78633d..ffe7e6c 100644
> --- a/libavcodec/qsv.c
> +++ b/libavcodec/qsv.c
> @@ -195,6 +195,30 @@ int ff_qsv_find_surface_idx(QSVFramesContext *ctx, 
> QSVFrame *frame)
>  return AVERROR_BUG;
>  }
>  
> +enum AVPictureType ff_qsv_map_pictype(int mfx_pic_type)
> +{
> +enum AVPictureType type;
> +switch (mfx_pic_type & 0x7) {
> +case MFX_FRAMETYPE_I:
> +if (mfx_pic_type & MFX_FRAMETYPE_S)
> +type = AV_PICTURE_TYPE_SI;
> +else
> +type = AV_PICTURE_TYPE_I;
> +break;
> +case MFX_FRAMETYPE_B:
> +type = AV_PICTURE_TYPE_B;
> +break;
> +case MFX_FRAMETYPE_P:
> + if (mfx_pic_type & MFX_FRAMETYPE_S)
> +type = AV_PICTURE_TYPE_SP;
> +else
> +type = AV_PICTURE_TYPE_P;
> +break;
> +}
> +
> +return type;
> +}
> +
>  static int qsv_load_plugins(mfxSession session, const char *load_plugins,
>  void *logctx)
>  {
> diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> index 975c8de..07ddc59 100644
> --- a/libavcodec/qsv_internal.h
> +++ b/libavcodec/qsv_internal.h
> @@ -48,6 +48,8 @@ typedef struct QSVMid {
>  typedef struct QSVFrame {
>  AVFrame *frame;
>  mfxFrameSurface1 surface;
> +mfxExtDecodedFrameInfo dec_info;
> +mfxExtBuffer *ext_param;
>  
>  int queued;
>  int used;
> @@ -83,6 +85,7 @@ int ff_qsv_codec_id_to_mfx(enum AVCodecID codec_id);
>  int ff_qsv_profile_to_mfx(enum AVCodecID codec_id, int profile);
>  
>  int ff_qsv_map_pixfmt(enum AVPixelFormat format, uint32_t *fourcc);
> +enum AVPictureType ff_qsv_map_pictype(int mfx_pic_type);
>  
>  int ff_qsv_init_internal_session(AVCodecContext *avctx, mfxSession *session,
>   const char *load_plugins);
> diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
> index f31172d..9ee4177 100644
> --- a/libavcodec/qsvdec.c
> +++ b/libavcodec/qsvdec.c
> @@ -232,6 +232,11 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext 
> *q, QSVFrame *frame)
>  
>  frame->surface.Data.MemId = >frames_ctx.mids[ret];
>  }
> +frame->surface.Data.ExtParam = >ext_param;
> +frame->surface.Data.NumExtParam = 1;
> +frame->ext_param = (mfxExtBuffer*)>dec_info;
> +frame->dec_info.Header.BufferId = MFX_EXTBUFF_DECODED_FRAME_INFO;
> +frame->dec_info.Header.BufferSz = sizeof(frame->dec_info);
>  
>  frame->used = 1;
>  
> @@ -416,6 +421,8 @@ FF_ENABLE_DEPRECATION_WARNINGS
>  outsurf->Info.PicStruct & MFX_PICSTRUCT_FIELD_TFF;
>  frame->interlaced_frame =
>  !(outsurf->Info.PicStruct & MFX_PICSTRUCT_PROGRESSIVE);
> +frame->pict_type = ff_qsv_map_pictype(out_frame->dec_info.FrameType);
> +frame->key_frame = !!(out_frame->dec_info.FrameType & 
> MFX_FRAMETYPE_IDR);
>  
>  /* update the surface properties */
>  if (avctx->pix_fmt == AV_PIX_FMT_QSV)
> 

Does this do the right thing for H.265?  (Can't test right now.)

I'm asking that because I know that frame types are still a problem for H.265 
encoding, because not all IRAPs are IDR - when CRA frames are generated they 
aren't marked as key.  The FrameType field (as used here) is not sufficient in 
that case, but I never sorted out exactly what does contain the necessary 
information.

- Mark
___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

Re: [libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Li, Zhong]

> From: Li, Zhong
> Sent: Tuesday, January 30, 2018 5:52 PM
> To: libav-devel@libav.org
> Cc: Li, Zhong ; Liu, ChaoX A 
> Subject: [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

AV_PICTURE_TYPE_NONE is not supported, so it will break the building.
Sorry for the carelessness. 
___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel

[libav-devel] [PATCH V2] lavc/qsvdec: expose frame pic_type and key_frame

[Zhong Li]

Currently key_frame and pict_type are unset.
Add an extra param to fetch the picture type from qsv decoder

Signed-off-by: ChaoX A Liu 
Signed-off-by: Zhong Li 
---
 libavcodec/qsv.c  | 24 
 libavcodec/qsv_internal.h |  3 +++
 libavcodec/qsvdec.c   |  7 +++
 3 files changed, 34 insertions(+)

diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
index e78633d..ffe7e6c 100644
--- a/libavcodec/qsv.c
+++ b/libavcodec/qsv.c
@@ -195,6 +195,30 @@ int ff_qsv_find_surface_idx(QSVFramesContext *ctx, 
QSVFrame *frame)
 return AVERROR_BUG;
 }
 
+enum AVPictureType ff_qsv_map_pictype(int mfx_pic_type)
+{
+enum AVPictureType type;
+switch (mfx_pic_type & 0x7) {
+case MFX_FRAMETYPE_I:
+if (mfx_pic_type & MFX_FRAMETYPE_S)
+type = AV_PICTURE_TYPE_SI;
+else
+type = AV_PICTURE_TYPE_I;
+break;
+case MFX_FRAMETYPE_B:
+type = AV_PICTURE_TYPE_B;
+break;
+case MFX_FRAMETYPE_P:
+ if (mfx_pic_type & MFX_FRAMETYPE_S)
+type = AV_PICTURE_TYPE_SP;
+else
+type = AV_PICTURE_TYPE_P;
+break;
+}
+
+return type;
+}
+
 static int qsv_load_plugins(mfxSession session, const char *load_plugins,
 void *logctx)
 {
diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
index 975c8de..07ddc59 100644
--- a/libavcodec/qsv_internal.h
+++ b/libavcodec/qsv_internal.h
@@ -48,6 +48,8 @@ typedef struct QSVMid {
 typedef struct QSVFrame {
 AVFrame *frame;
 mfxFrameSurface1 surface;
+mfxExtDecodedFrameInfo dec_info;
+mfxExtBuffer *ext_param;
 
 int queued;
 int used;
@@ -83,6 +85,7 @@ int ff_qsv_codec_id_to_mfx(enum AVCodecID codec_id);
 int ff_qsv_profile_to_mfx(enum AVCodecID codec_id, int profile);
 
 int ff_qsv_map_pixfmt(enum AVPixelFormat format, uint32_t *fourcc);
+enum AVPictureType ff_qsv_map_pictype(int mfx_pic_type);
 
 int ff_qsv_init_internal_session(AVCodecContext *avctx, mfxSession *session,
  const char *load_plugins);
diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
index f31172d..9ee4177 100644
--- a/libavcodec/qsvdec.c
+++ b/libavcodec/qsvdec.c
@@ -232,6 +232,11 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext 
*q, QSVFrame *frame)
 
 frame->surface.Data.MemId = >frames_ctx.mids[ret];
 }
+frame->surface.Data.ExtParam = >ext_param;
+frame->surface.Data.NumExtParam = 1;
+frame->ext_param = (mfxExtBuffer*)>dec_info;
+frame->dec_info.Header.BufferId = MFX_EXTBUFF_DECODED_FRAME_INFO;
+frame->dec_info.Header.BufferSz = sizeof(frame->dec_info);
 
 frame->used = 1;
 
@@ -416,6 +421,8 @@ FF_ENABLE_DEPRECATION_WARNINGS
 outsurf->Info.PicStruct & MFX_PICSTRUCT_FIELD_TFF;
 frame->interlaced_frame =
 !(outsurf->Info.PicStruct & MFX_PICSTRUCT_PROGRESSIVE);
+frame->pict_type = ff_qsv_map_pictype(out_frame->dec_info.FrameType);
+frame->key_frame = !!(out_frame->dec_info.FrameType & 
MFX_FRAMETYPE_IDR);
 
 /* update the surface properties */
 if (avctx->pix_fmt == AV_PIX_FMT_QSV)
-- 
1.8.3.1

___
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel