Hi Paul,
I tried to understand the per-frame side-data (and metadata)
mechanism. Adding my notes here in case they help a future reader.
Metadata and side-data seem like similar mechanisms to add auxiliary
information to each frame. Main difference seems to be that metadata
is simpler (a key/value dictionary), while side-data allows adding any
data struct.
Metadata operation is also simpler: There are filters that generate
metadata (e.g. "signalstats" generates key-values such as
`lavfi.signalstats.YMIN=3`, while "silencedetect" generates
audio-related key-values like `lavfi.silence_start=0`). There are also
2x filters that print metadata ("vf_metadata" and "af_ametadata"). So
e.g. to see what signalstats/silencedetect are generating, you can do:
```
$ ffmpeg -i in.264 -vf signalstats,metadata=mode=print -f null -
...
[Parsed_metadata_1 @ ...] frame:0 pts:0 pts_time:0
[Parsed_metadata_1 @ ...] lavfi.signalstats.YMIN=3
...
```
or:
```
$ ffmpeg -y -i in.wav -af "silencedetect=n=-10dB:d=1,ametadata=print"
/tmp/out.wav
...
[Parsed_ametadata_1 @ ...] frame:23 pts:47104 pts_time:0.981333
[Parsed_ametadata_1 @ ...] lavfi.silence_start=0
...
```
Side-data operation is more complicated. There is some side-data
information already generated (e.g. SEI_UNREGISTERED side-data). Some
information requires explicitly asking for it. For example, the
`MOTION_VECTORS` side-data, you need to enable
AV_CODEC_FLAG2_EXPORT_MVS (which means calling ffmpeg/ffplay/ffprobe
with "-flags2 +export_mvs"). The main filter to print side-data
information is showinfo (`vf_showinfo` and `af_ashowinfo`). Now, the
`vf_showinfo` filter only knows how to dump some of the side-data
structs. In particular, it does not know how to dump MOTION_VECTORS
side-data. So, if we add the motion vectors, and then ask showinfo to
print it, we see:
```
$ ffmpeg -hide_banner -flags2 +export_mvs -export_side_data +mvs
-export_side_data +prft -export_side_data +venc_params
-export_side_data +film_grain -i /tmp/in.264 -vf showinfo -f null
/dev/null
...
-- frame 0 is a key frame: We can see SEI_UNREGISTERED and VIDEO_ENC_PARAMS info
[Parsed_showinfo_0 @ 0x308fd40] config in time_base: 1/1200000, frame_rate: 25/1
[Parsed_showinfo_0 @ 0x308fd40] config out time_base: 0/0, frame_rate: 0/0
[Parsed_showinfo_0 @ 0x308fd40] n: 0 pts: 0 pts_time:0
duration: 48000 duration_time:0.04 pos: 0 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:1 type:I checksum:F6BBEA9F
plane_checksum:[AFB1432E 63F2F255 2887B50D] mean:[109 119 138]
stdev:[43.0 12.7 13.3]
[Parsed_showinfo_0 @ 0x308fd40] side data - User Data Unregistered:
[Parsed_showinfo_0 @ 0x308fd40] UUID=47564adc-5c4c-433f-94ef-c5113cd143a8
[Parsed_showinfo_0 @ 0x308fd40] User Data=01ffccccff0200e4dd42
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40] side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
-- frame 1 is a P-frame: we can see VIDEO_ENC_PARAMS info, and a
complain about "side-data type 8" (MOTION_VECTORS)
[Parsed_showinfo_0 @ 0x308fd40] n: 1 pts: 48000 pts_time:0.04
duration: 48000 duration_time:0.04 pos: 259304 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:0 type:B checksum:BC4E5C12
plane_checksum:[AEA8857A 34697DA4 805E58E5] mean:[109 119 138]
stdev:[43.0 12.6 13.3]
[Parsed_showinfo_0 @ 0x308fd40] side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
-- showinfo does not dump MOTION_VECTORS side-data
[Parsed_showinfo_0 @ 0x308fd40] side data - unknown side data type 8
(547280 bytes)
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
```
So the best way right now to see the MVs is to use
`doc/examples/extract_mvs`, which does exactly that:
```
$ make examples -j
...
$ doc/examples/extract_mvs in.264 | head -40 | \
csvcut -C framenum,source,flags |csvlook
| blockw | blockh | srcx | srcy | dstx | dsty | motion_x | motion_y
| motion_scale |
| ------ | ------ | ----- | ---- | ----- | ---- | -------- | --------
| ------------ |
| 16 | 16 | 20 | 26 | 8 | 8 | 49 | 72
| 4 |
| 16 | 16 | 152 | 15 | 136 | 8 | 65 | 28
| 4 |
| 16 | 8 | 360 | 3 | 360 | 4 | 1 | -6
| 4 |
| 16 | 8 | 360 | 13 | 360 | 12 | -1 | 4
| 4 |
| 16 | 16 | 440 | 10 | 440 | 8 | 3 | 10
| 4 |
| 8 | 16 | 829 | 7 | 836 | 8 | -31 | -6
| 4 |
| 8 | 16 | 844 | 7 | 844 | 8 | -1 | -4
| 4 |
```
> Yes, it's called codecview. We can help understand how it works if you ask
> more specific questions, but something like "git grep EXPORT_DATA_MVS
> ../libavcodec/mpeg*.c" and checking the complement code in codecview should
> explain the basics.
codecview will get the motion vectors (assuming you asked for them),
and will overlay arrows showing the MV on top of the original video.
```
$ ffmpeg -flags2 +export_mvs -i input -vf codecview=mv=pf+bf+bb output.%04d.png
```
Adding one of the resulting frames in the next email.
-Chema
On Sat, Sep 3, 2022 at 5:08 AM Ronald S. Bultje <rsbul...@gmail.com> wrote:
>
> Hi Chema,
>
> On Fri, Sep 2, 2022 at 11:12 AM Chema Gonzalez <ch...@berkeley.edu> wrote:
>>
>> So is there a filter that already dumps this information?
>
>
> Yes, it's called codecview. We can help understand how it works if you ask
> more specific questions, but something like "git grep EXPORT_DATA_MVS
> ../libavcodec/mpeg*.c" and checking the complement code in codecview should
> explain the basics.
>
> Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
