Hi,
On Mon, May 26, 2025 at 2:09 PM Timothée <
timothee.informati...@regaud-chapuy.fr> wrote:
> On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer
> <mich...@niedermayer.cc> wrote:
>
> > On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> >
> >> On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
> >> <mich...@niedermayer.cc> wrote:
> >>
> >>> On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer
> >>> wrote:
> >>>
> >>>> Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
> >>>> Bultje wrote:
> >>>>
> >>>>> Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>>>> timothee.informati...@regaud-chapuy.fr> wrote:
> >>>>>
> >>>>>> Hello, I am interested in expanding ffmpeg's capabilities
> >>>>>> to extract low-level data from video codecs. Specifically,
> >>>>>> I'd like to implement functionality that would allow
> >>>>>> exporting frame data, macroblock information, quantization
> >>>>>> tables, and similar codec-specific elements to binary
> >>>>>> files for further analysis. After searching through the
> >>>>>> documentation and existing features, I haven't found
> >>>>>> similar functionality, though I may have missed something.
> >>>>>> Has this been implemented before, or are there related
> >>>>>> features I should examine?
> >>>>>
> >>>>> Some older codecs implement minor variants for this, e.g.
> >>>>> grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a
> >>>>> frame's motion vectors to the picture data. I believe
> >>>>> there's an example app and possibly a filter to overlay MVs
> >>>>> on top of the video frame based on this concept. You could
> >>>>> extend this to cover other (macro)block info. There used to
> >>>>> be a variant of this for quant-tables also but I can't find
> >>>>> it, maybe it was removed.
> >>>>
> >>>> For motion vectors: ./ffplay -flags2 +export_mvs -i
> >>>> matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
> >>>> segmentation and type vissualization + also motion vectors:
> >>>> ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
> >>>> codecview=mv=pf+bf+bb For QP vissualization + also motion
> >>>> vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
> >>>> codecview=mv=pf+bf+bb For qp values dumped on the console
> >>>> ./ffplay -debug qp -i matrixbench_mpeg2.mpg
> >>>
> >>> And this can easily be extended to other codecs, ATM it should
> >>> work with all 16x16 MB based codecs like
> >>> msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization
> >>> need codecview to be extended or versions around 3.4 which
> >>> implemented it differently Implementing vissualization as done
> >>> currently with sidedata and codecview is simple and efficient.
> >>> It also would allow exporting the data to json by writing a
> >>> codec2json filter in place of codecview Also all decoders
> >>> already have all this data parsed and available so its simpler
> >>> than trying to do it in a decoder independant way I would thus
> >>> suggest implementations of this for modern codecs to follow the
> >>> same path as the existing code. thx
> >>
> >> Thanks for the helpful pointers! I will work on the codec2json
> >> filter. Looking at the code, I see where I can access sidedata but
> >> extracting qb table seems to fail. (in codecview.c l.233:
> >> ff_qp_table_extract() return 0 and qp_table is empty) (I use
> >> ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1
> >> output.mp4 -y) Is is qp extraction not implemented yet? Or is it
> >> because I have h264 video? If it's not implemented, I'm curious
> >> why there’s already code that appears to handle it.
> >
> > look at ff_print_debug_info2() theres probably something missing
> > The QP and MB type code was changed from being inside arrays of
> > pictures to sidedata. Something likely was lost/forgotten in the
> > process -debug qp works with h264 so likely teh export into sidedata
> > is not fully implemented
>
> Yes, this is exactly what is happening.
>
> After some research, I have found: `// FIXME qscale / qp ... stuff` in
> libavcodec/h264_slice.c, line 1871
>
> and also in libavfilter/qp_table.h:
> 34 /**
> 35 * Normalize the qscale factor
> 36 * FIXME Add support for other values of enum AVVideoEncParamsType
> 37 * besides AV_VIDEO_ENC_PARAMS_MPEG2.
> 38 */
> 39 static inline int ff_norm_qscale(int qscale, enum
> AVVideoEncParamsType type)
> 40 {
> 41 switch (type) {
> 42 case AV_VIDEO_ENC_PARAMS_MPEG2: return qscale >> 1;
> 43 }
> 44 return qscale;
> 45 }
>
> So if I understand correctly, QP tables are not attached to the frames
> except for MPEG2. This has been left as a FIXME for other codecs.
> Should I extend it to work for H.264, and maybe later for other
> codecs, thus removing the first FIXME and partially fixing the second?
> Or is this too specific and too complicated to maintain if it needs to
> be added for every codec?
I think it's fine to add it one codec at a time.
Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
