PR #25395 opened by Daniel Campos Ramos (danielcamposramos) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25395 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25395.patch
Closes #24531. This adds `frame_packing=pass|insert|remove|extract` to `h264_metadata` and `hevc_metadata`. A stream copy can add the stereo layout to the video bitstream without encoding it again, or extract an existing SEI description as packet Stereo 3D side data. Sparky Stereo OS carries this option in its FFmpeg package today; this series brings it to master. The action and layout are separate options, as @Forgejo_Fairy suggested on #24531. For example: ```sh ffmpeg -i input.mkv -map 0 -c copy -bsf:v h264_metadata=frame_packing=insert:frame_packing_type=side_by_side output.mkv ``` The default is pass. Insertion takes unset values from packet Stereo 3D side data, then stream side data. Explicit options take precedence. It writes before seek points, replaces later arrangements and cancellations, and applies packet descriptions even on non-key pictures. The filter describes an existing layout without rearranging its pixels. H.264 supports the eight arrangement types in D.2.26 Table D-9 of the June 2026 specification, numbered D-8 in the August 2024 specification. HEVC supports the three types in D.3.16 Table D.8. The filters reject forbidden quincunx combinations and nonzero content interpretation for H.264 type 6. Temporal layouts get a message on every packet. LEFT and RIGHT identify the physical eye, with the selected content interpretation determining whether it is constituent frame 0 or 1. Packed or unspecified views alternate in packet order and require display-order input. An explicit eye sets the phase for subsequent unspecified views. Flush restarts that sequence and makes the first H.264 access unit a seek point for display_orientation too. Extraction leaves the SEI intact and exports the last message with a non-reserved arrangement ID in the access unit when AVStereo3D can represent it. Cancellation, unknown content relationships and tile do not produce new side data. A conforming H.264 type 6 message produces an explicit 2D description. Extraction does not infer persistence on packets without a message. Both filters follow the existing top-level CBS handling without adding nested message parsing or layer selection. An API caller can copy the extracted packet description to the output stream before writing a container header. Matroska and MP4 use stream metadata for StereoMode and st3d; the CLI's output BSF does not automatically promote late packet descriptions to a track declaration. Two preceding CBS fixes place a new H.264 SEI before an IDR slice and delete the NAL unit when its last SEI message is removed. The regression compares independently parsed syntax and extracted side data against FATE references. It covers both eye orders, unspecified temporal views, option precedence, flush, cancellation, removal and extraction with unchanged coded bytes. The separate decoder corrections are outside this series. All 5,995 applicable FATE tests pass with GCC and clang, with no changed output among the 5,993 tests shared with master. The 34 stream-copy checks and 24 extraction-to-container checks pass too. AI partners Claude Sonnet and gpt-6-astra helped prepare this work. >From 23016a3d68c9488db1b1e5247b5aeeca04f00318 Mon Sep 17 00:00:00 2001 From: Daniel Campos Ramos <[email protected]> Date: Fri, 23 Oct 2026 05:25:16 +0000 Subject: [PATCH 1/4] avcodec/cbs_sei: put a new H.264 SEI unit in front of the IDR slice A new prefix SEI unit belongs before the first VCL NAL unit. The H.264 search excluded the IDR slice type, so an access unit containing an IDR slice and no SEI received its new SEI after the slice. A decoder then associates the message with the next picture. Signed-off-by: Daniel Campos Ramos <[email protected]> Assisted-by: Claude Sonnet via Claude Code Assisted-by: gpt-6-astra via Codex --- libavcodec/cbs_sei.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/cbs_sei.c b/libavcodec/cbs_sei.c index f354a100291..e3544d32491 100644 --- a/libavcodec/cbs_sei.c +++ b/libavcodec/cbs_sei.c @@ -311,7 +311,7 @@ static int cbs_sei_get_unit(CodedBitstreamContext *ctx, // (We can ignore auxiliary slices because we only have prefix // SEI in H.264 and an auxiliary picture must always follow a // primary picture.) - highest_vcl_type = H264_NAL_IDR_SLICE; + highest_vcl_type = H264_NAL_IDR_SLICE + 1; if (prefix) sei_type = H264_NAL_SEI; else -- 2.54.0 >From 4dd48e08dc96aeb532d59203d286351ca2262652 Mon Sep 17 00:00:00 2001 From: Daniel Campos Ramos <[email protected]> Date: Fri, 23 Oct 2026 05:25:17 +0000 Subject: [PATCH 2/4] avcodec/cbs_sei: delete the SEI unit when its last message is deleted Deleting the last message left a NAL unit containing only its header and trailing bits. CBS rejects that empty SEI, so a second bitstream filter could not read the output of h264_metadata=display_orientation=remove. Delete the unit when it has no messages left. Signed-off-by: Daniel Campos Ramos <[email protected]> Assisted-by: Claude Sonnet via Claude Code Assisted-by: gpt-6-astra via Codex --- libavcodec/cbs_sei.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/libavcodec/cbs_sei.c b/libavcodec/cbs_sei.c index e3544d32491..70c18b4ced4 100644 --- a/libavcodec/cbs_sei.c +++ b/libavcodec/cbs_sei.c @@ -582,6 +582,12 @@ void ff_cbs_sei_delete_message_type(CodedBitstreamContext *ctx, if (list->messages[j].payload_type == payload_type) cbs_sei_delete_message(list, j); } + + // An SEI unit needs at least one message. + if (!list->nb_messages) { + ff_cbs_delete_unit(au, i); + i--; + } } } -- 2.54.0 >From dbee03c4b3e4482a72f710de7d2793ed7b937b44 Mon Sep 17 00:00:00 2001 From: Daniel Campos Ramos <[email protected]> Date: Fri, 23 Oct 2026 05:25:17 +0000 Subject: [PATCH 3/4] avcodec/bsf/h264_metadata,hevc_metadata: add the frame_packing option Add pass, insert, remove and extract actions for frame packing arrangement SEI messages. Insertion takes the layout from explicit options, then packet Stereo 3D side data, then stream side data. Write before each seek point, on non-key packets carrying side data, and when replacing later arrangements or cancellations. Temporal arrangements are written on every packet, using LEFT and RIGHT as physical eyes and the selected content interpretation to determine the constituent frame number. Packed or unspecified temporal views alternate in packet order, which requires display-order input, and restart after a flush. An explicit physical eye sets the phase for subsequent unspecified views. Reject forbidden quincunx combinations and nonzero content interpretation for H.264 type 6. Extraction leaves the bitstream intact and exports the last non-reserved arrangement in the access unit as packet Stereo 3D side data when its layout is representable. It does not infer persistence on packets without a message. Flushing resets done_first_au as well as the temporal phase, so display_orientation also treats the first access unit after a flush as a seek point. This is intentional because a flush starts a new input segment whose first packet may have neither SPS nor a key flag. Signed-off-by: Daniel Campos Ramos <[email protected]> Assisted-by: Claude Sonnet via Claude Code Assisted-by: gpt-6-astra via Codex --- doc/bitstream_filters.texi | 136 ++++++++++++++++++++++++++ libavcodec/bsf/h264_metadata.c | 160 +++++++++++++++++++++++++++++++ libavcodec/bsf/h265_metadata.c | 168 ++++++++++++++++++++++++++++++++- libavcodec/cbs_sei.c | 153 ++++++++++++++++++++++++++++++ libavcodec/cbs_sei.h | 45 +++++++++ 5 files changed, 661 insertions(+), 1 deletion(-) diff --git a/doc/bitstream_filters.texi b/doc/bitstream_filters.texi index cc1cd453a73..65a161ce0c7 100644 --- a/doc/bitstream_filters.texi +++ b/doc/bitstream_filters.texi @@ -450,6 +450,77 @@ Insert mode works in conjunction with @code{rotate} and @code{flip} options. Any pre-existing Display orientation messages will be removed in insert or remove mode. Extract mode attaches the display matrix to the packet as side data. +@item frame_packing +Insert, remove or extract Frame packing arrangement SEI messages. +See H.264 section D.1.26 and D.2.26 for syntax and semantics. + +@table @samp +@item pass +@item insert +@item remove +@item extract +@end table + +Default is pass. + +Insert mode writes the message in front of every key frame, or in front of +every frame for temporal interleaving, and replaces a pre-existing message in +those access units. The layout comes from the @code{frame_packing_type}, +@code{frame_packing_view} and @code{frame_packing_quincunx} options, and what +they leave unset comes from the Stereo 3D side data of the stream or of the +packet, such as the one the Matroska demuxer reads from @code{StereoMode}. +Without a type and without side data the stream is passed unchanged. +This option describes an existing layout; it does not rearrange the pictures. +For temporal interleaving, a per-packet Stereo 3D LEFT or RIGHT view selects +the physical eye, with the content interpretation determining its constituent +frame number. Otherwise the frames alternate in packet order, starting with frame 0 +and restarting after a flush. In that case the input must be in display order. +An explicit view sets the phase for subsequent unspecified views. +Reordered input requires a per-packet view description. +Existing frame packing messages are replaced, including cancellation messages. +Packet side data is applied on the packet that carries it. + +Extract mode attaches a representable frame packing message from the access +unit to the packet as @code{AV_PKT_DATA_STEREO3D} side data, leaving the SEI +in the bitstream. If several messages are present, the last one with a +non-reserved arrangement ID is used. Cancellation messages, unknown content +interpretations and layouts without an @code{AVStereo3D} equivalent do not +produce side data. Extraction describes messages present in the packet; +it does not infer persistence on packets without a message. + +@item frame_packing_type +Set frame_packing_arrangement_type (H.264 table D-9). + +@table @samp +@item checkerboard +@item columns +@item rows +@item side_by_side +@item top_bottom +@item temporal +@item 2d +@item tile +@end table + +Default is to take it from the Stereo 3D side data. + +@item frame_packing_view +Set content_interpretation_type: which view the first frame of the arrangement +is. + +@table @samp +@item unspecified +@item left_first +@item right_first +@end table + +Default is to take it from the Stereo 3D side data, left first without any. + +@item frame_packing_quincunx +Set quincunx_sampling_flag. Default is to take it from the Stereo 3D side data; +it is always set for the checkerboard. +A value of 1 is invalid for temporal interleaving, 2D and tile layouts. + @item rotate Set rotation in display orientation SEI (anticlockwise angle in degrees). Range is -360 to +360. Default is NaN. @@ -568,6 +639,71 @@ representable if the chroma is subsampled (H.265 section 7.4.3.2.1). @item height Set width and height after crop. +@item frame_packing +Insert, remove or extract Frame packing arrangement SEI messages. +See H.265 section D.2.16 and D.3.16 for syntax and semantics. + +@table @samp +@item pass +@item insert +@item remove +@item extract +@end table + +Default is pass. + +Insert mode writes the message in front of every key frame, or in front of +every frame for temporal interleaving, and replaces a pre-existing message in +those access units. The layout comes from the @code{frame_packing_type}, +@code{frame_packing_view} and @code{frame_packing_quincunx} options, and what +they leave unset comes from the Stereo 3D side data of the stream or of the +packet, such as the one the Matroska demuxer reads from @code{StereoMode}. +Without a type and without side data the stream is passed unchanged. +This option describes an existing layout; it does not rearrange the pictures. +For temporal interleaving, a per-packet Stereo 3D LEFT or RIGHT view selects +the physical eye, with the content interpretation determining its constituent +frame number. Otherwise the frames alternate in packet order, starting with frame 0 +and restarting after a flush. In that case the input must be in display order. +An explicit view sets the phase for subsequent unspecified views. +Reordered input requires a per-packet view description. +Existing frame packing messages are replaced, including cancellation messages. +Packet side data is applied on the packet that carries it. + +Extract mode attaches a representable frame packing message from the access +unit to the packet as @code{AV_PKT_DATA_STEREO3D} side data, leaving the SEI +in the bitstream. If several messages are present, the last one with a +non-reserved arrangement ID is used. Cancellation messages, unknown content +interpretations and layouts without an @code{AVStereo3D} equivalent do not +produce side data. Extraction describes messages present in the packet; +it does not infer persistence on packets without a message. + +@item frame_packing_type +Set frame_packing_arrangement_type (H.265 table D.8). + +@table @samp +@item side_by_side +@item top_bottom +@item temporal +@end table + +Default is to take it from the Stereo 3D side data. + +@item frame_packing_view +Set content_interpretation_type: which view the first frame of the arrangement +is. + +@table @samp +@item unspecified +@item left_first +@item right_first +@end table + +Default is to take it from the Stereo 3D side data, left first without any. + +@item frame_packing_quincunx +Set quincunx_sampling_flag. Default is to take it from the Stereo 3D side data; +it must be zero for temporal interleaving. + @item level Set the level in the VPS and SPS. See H.265 section A.4 and tables A.6 and A.7. diff --git a/libavcodec/bsf/h264_metadata.c b/libavcodec/bsf/h264_metadata.c index b5d81e41e5e..e8c5808840d 100644 --- a/libavcodec/bsf/h264_metadata.c +++ b/libavcodec/bsf/h264_metadata.c @@ -21,6 +21,7 @@ #include "libavutil/common.h" #include "libavutil/mem.h" #include "libavutil/opt.h" +#include "libavutil/stereo3d.h" #include "libavcodec/bsf.h" #include "libavcodec/bsf_internal.h" @@ -82,6 +83,13 @@ typedef struct H264MetadataContext { int flip; H264RawSEIDisplayOrientation display_orientation_payload; + int frame_packing; + int frame_packing_type; + int frame_packing_view; + int frame_packing_quincunx; + int frame_packing_frame; + H264RawSEIFramePackingArrangement frame_packing_payload; + int level; } H264MetadataContext; @@ -465,6 +473,108 @@ static int h264_metadata_handle_display_orientation(AVBSFContext *bsf, return 0; } +static int h264_metadata_handle_frame_packing(AVBSFContext *bsf, + AVPacket *pkt, + CodedBitstreamFragment *au, + int seek_point) +{ + H264MetadataContext *ctx = bsf->priv_data; + H264RawSEIFramePackingArrangement *fp = &ctx->frame_packing_payload; + SEIFramePackingParams params; + SEIRawMessage *message = NULL; + const AVPacketSideData *sd; + const AVStereo3D *stereo = NULL; + int err, packet_stereo = 0; + + if (ctx->frame_packing == BSF_ELEMENT_EXTRACT) { + const H264RawSEIFramePackingArrangement *last = NULL; + + while (ff_cbs_sei_find_message(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + &message) == 0) { + const H264RawSEIFramePackingArrangement *payload = message->payload; + unsigned int id = payload->frame_packing_arrangement_id; + + if (id < 256 || (id >= 512 && id < (1U << 31))) + last = payload; + } + if (!last || last->frame_packing_arrangement_cancel_flag) + return 0; + + params = (SEIFramePackingParams) { + .type = last->frame_packing_arrangement_type, + .content_interpretation_type = last->content_interpretation_type, + .quincunx_sampling_flag = last->quincunx_sampling_flag, + .current_frame_is_frame0_flag = last->current_frame_is_frame0_flag, + }; + return ff_cbs_sei_frame_packing_to_packet(pkt, ¶ms); + } + + if (ctx->frame_packing == BSF_ELEMENT_REMOVE) { + ff_cbs_sei_delete_message_type(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT); + return 0; + } + + sd = av_packet_side_data_get(bsf->par_in->coded_side_data, + bsf->par_in->nb_coded_side_data, + AV_PKT_DATA_STEREO3D); + if (sd && sd->size >= sizeof(AVStereo3D)) + stereo = (const AVStereo3D *)sd->data; + if (pkt) { + size_t size; + uint8_t *data = av_packet_get_side_data(pkt, AV_PKT_DATA_STEREO3D, &size); + if (data && size >= sizeof(AVStereo3D)) { + stereo = (const AVStereo3D *)data; + packet_stereo = 1; + } + } + + err = ff_cbs_sei_frame_packing_params(bsf, AV_CODEC_ID_H264, + ctx->frame_packing_type, + ctx->frame_packing_view, + ctx->frame_packing_quincunx, stereo, + !ctx->frame_packing_frame, + ¶ms); + if (err == AVERROR(ENOENT)) + return 0; + if (err < 0) + return err; + + // The message of a frame sequence is needed on every frame, the others + // stay in force until the next one. + if (params.type != SEI_FPA_TYPE_INTERLEAVE_TEMPORAL && + !seek_point && !packet_stereo && + ff_cbs_sei_find_message(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + &message) < 0) + return 0; + + *fp = (H264RawSEIFramePackingArrangement) { + .frame_packing_arrangement_type = params.type, + .quincunx_sampling_flag = params.quincunx_sampling_flag, + .content_interpretation_type = params.content_interpretation_type, + .current_frame_is_frame0_flag = params.current_frame_is_frame0_flag, + .frame_packing_arrangement_repetition_period = params.persistence, + }; + + ff_cbs_sei_delete_message_type(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT); + err = ff_cbs_sei_add_message(ctx->common.output, au, 1, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + fp, NULL); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to add frame packing arrangement " + "SEI message to access unit.\n"); + return err; + } + + if (params.type == SEI_FPA_TYPE_INTERLEAVE_TEMPORAL) + ctx->frame_packing_frame = params.current_frame_is_frame0_flag; + + return 0; +} + static int h264_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, CodedBitstreamFragment *au) { @@ -536,6 +646,12 @@ static int h264_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, return err; } + if (pkt && ctx->frame_packing != BSF_ELEMENT_PASS) { + err = h264_metadata_handle_frame_packing(bsf, pkt, au, seek_point); + if (err < 0) + return err; + } + if (pkt) ctx->done_first_au = 1; @@ -656,6 +772,41 @@ static const AVOption h264_metadata_options[] = { "Display orientation SEI", display_orientation, FLAGS), + BSF_ELEMENT_OPTIONS_PIRE("frame_packing", + "Frame packing arrangement SEI", + frame_packing, FLAGS), + { "frame_packing_type", "Set frame_packing_arrangement_type (table D-9), default: from the stream's Stereo 3D side data", + OFFSET(frame_packing_type), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 7, FLAGS, .unit = "frame_packing_type" }, + { "checkerboard", "Checkerboard interleaving", + 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "columns", "Column interleaving", + 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "rows", "Row interleaving", + 0, AV_OPT_TYPE_CONST, { .i64 = 2 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "side_by_side", "Side by side", + 0, AV_OPT_TYPE_CONST, { .i64 = 3 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "top_bottom", "Top and bottom", + 0, AV_OPT_TYPE_CONST, { .i64 = 4 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "temporal", "Temporal interleaving (frame sequence)", + 0, AV_OPT_TYPE_CONST, { .i64 = 5 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "2d", "Two-dimensional picture", + 0, AV_OPT_TYPE_CONST, { .i64 = 6 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "tile", "Tile format", + 0, AV_OPT_TYPE_CONST, { .i64 = 7 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "frame_packing_view", "Set frame_packing_arrangement content_interpretation_type, default: from the stream's Stereo 3D side data", + OFFSET(frame_packing_view), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 2, FLAGS, .unit = "frame_packing_view" }, + { "unspecified", "No interpretation", + 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "left_first", "Frame 0 is the left view", + 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "right_first", "Frame 0 is the right view", + 0, AV_OPT_TYPE_CONST, { .i64 = 2 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "frame_packing_quincunx", "Set quincunx_sampling_flag", + OFFSET(frame_packing_quincunx), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 1, FLAGS }, + { "rotate", "Set rotation in display orientation SEI (anticlockwise angle in degrees)", OFFSET(rotate), AV_OPT_TYPE_DOUBLE, { .dbl = NAN }, -360.0, +360.0, FLAGS }, @@ -702,6 +853,14 @@ static const AVOption h264_metadata_options[] = { { NULL } }; +static void h264_metadata_flush(AVBSFContext *bsf) +{ + H264MetadataContext *ctx = bsf->priv_data; + + ctx->done_first_au = 0; + ctx->frame_packing_frame = 0; +} + static const AVClass h264_metadata_class = { .class_name = "h264_metadata_bsf", .item_name = av_default_item_name, @@ -721,4 +880,5 @@ const FFBitStreamFilter ff_h264_metadata_bsf = { .init = &h264_metadata_init, .close = &ff_cbs_bsf_generic_close, .filter = &ff_cbs_bsf_generic_filter, + .flush = &h264_metadata_flush, }; diff --git a/libavcodec/bsf/h265_metadata.c b/libavcodec/bsf/h265_metadata.c index 214ec4bd0f5..07810c1499d 100644 --- a/libavcodec/bsf/h265_metadata.c +++ b/libavcodec/bsf/h265_metadata.c @@ -18,12 +18,14 @@ #include "libavutil/common.h" #include "libavutil/opt.h" +#include "libavutil/stereo3d.h" #include "libavcodec/bsf.h" #include "libavcodec/bsf_internal.h" #include "libavcodec/cbs.h" #include "libavcodec/cbs_bsf.h" #include "libavcodec/cbs_h265.h" +#include "libavcodec/cbs_sei.h" #include "libavcodec/h2645data.h" #include "libavcodec/h265_profile_level.h" @@ -65,6 +67,15 @@ typedef struct H265MetadataContext { int level; int level_guess; int level_warned; + + int done_first_au; + + int frame_packing; + int frame_packing_type; + int frame_packing_view; + int frame_packing_quincunx; + int frame_packing_frame; + SEIRawFramePackingArrangement frame_packing_payload; } H265MetadataContext; @@ -412,11 +423,116 @@ static int h265_metadata_update_sps(AVBSFContext *bsf, return 0; } +static int h265_metadata_handle_frame_packing(AVBSFContext *bsf, + AVPacket *pkt, + CodedBitstreamFragment *au, + int seek_point) +{ + H265MetadataContext *ctx = bsf->priv_data; + SEIRawFramePackingArrangement *fp = &ctx->frame_packing_payload; + SEIFramePackingParams params; + SEIRawMessage *message = NULL; + const AVPacketSideData *sd; + const AVStereo3D *stereo = NULL; + int err, packet_stereo = 0; + + if (ctx->frame_packing == BSF_ELEMENT_EXTRACT) { + const SEIRawFramePackingArrangement *last = NULL; + + while (ff_cbs_sei_find_message(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + &message) == 0) { + const SEIRawFramePackingArrangement *payload = message->payload; + unsigned int id = payload->fp_arrangement_id; + + if (id < 256 || (id >= 512 && id < (1U << 31))) + last = payload; + } + if (!last || last->fp_arrangement_cancel_flag) + return 0; + if (last->fp_arrangement_type < SEI_FPA_TYPE_SIDE_BY_SIDE || + last->fp_arrangement_type > SEI_FPA_TYPE_INTERLEAVE_TEMPORAL) + return 0; + + params = (SEIFramePackingParams) { + .type = last->fp_arrangement_type, + .content_interpretation_type = last->fp_content_interpretation_type, + .quincunx_sampling_flag = last->fp_quincunx_sampling_flag, + .current_frame_is_frame0_flag = last->fp_current_frame_is_frame0_flag, + }; + return ff_cbs_sei_frame_packing_to_packet(pkt, ¶ms); + } + + if (ctx->frame_packing == BSF_ELEMENT_REMOVE) { + ff_cbs_sei_delete_message_type(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT); + return 0; + } + + sd = av_packet_side_data_get(bsf->par_in->coded_side_data, + bsf->par_in->nb_coded_side_data, + AV_PKT_DATA_STEREO3D); + if (sd && sd->size >= sizeof(AVStereo3D)) + stereo = (const AVStereo3D *)sd->data; + if (pkt) { + size_t size; + uint8_t *data = av_packet_get_side_data(pkt, AV_PKT_DATA_STEREO3D, &size); + if (data && size >= sizeof(AVStereo3D)) { + stereo = (const AVStereo3D *)data; + packet_stereo = 1; + } + } + + err = ff_cbs_sei_frame_packing_params(bsf, AV_CODEC_ID_HEVC, + ctx->frame_packing_type, + ctx->frame_packing_view, + ctx->frame_packing_quincunx, stereo, + !ctx->frame_packing_frame, + ¶ms); + if (err == AVERROR(ENOENT)) + return 0; + if (err < 0) + return err; + + // The message of a frame sequence is needed on every frame, the others + // stay in force until the next one. + if (params.type != SEI_FPA_TYPE_INTERLEAVE_TEMPORAL && + !seek_point && !packet_stereo && + ff_cbs_sei_find_message(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + &message) < 0) + return 0; + + *fp = (SEIRawFramePackingArrangement) { + .fp_arrangement_type = params.type, + .fp_quincunx_sampling_flag = params.quincunx_sampling_flag, + .fp_content_interpretation_type = params.content_interpretation_type, + .fp_current_frame_is_frame0_flag = params.current_frame_is_frame0_flag, + .fp_arrangement_persistence_flag = params.persistence, + }; + + ff_cbs_sei_delete_message_type(ctx->common.output, au, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT); + err = ff_cbs_sei_add_message(ctx->common.output, au, 1, + SEI_TYPE_FRAME_PACKING_ARRANGEMENT, + fp, NULL); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to add frame packing arrangement " + "SEI message to access unit.\n"); + return err; + } + + if (params.type == SEI_FPA_TYPE_INTERLEAVE_TEMPORAL) + ctx->frame_packing_frame = params.current_frame_is_frame0_flag; + + return 0; +} + static int h265_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, CodedBitstreamFragment *au) { H265MetadataContext *ctx = bsf->priv_data; - int err, i; + int err, i, has_vps; // If an AUD is present, it must be the first NAL unit. if (au->nb_units && au->units[0].type == HEVC_NAL_AUD) { @@ -464,11 +580,13 @@ static int h265_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, if (ctx->level == LEVEL_AUTO && !ctx->level_guess) h265_metadata_guess_level(bsf, au); + has_vps = 0; for (i = 0; i < au->nb_units; i++) { if (au->units[i].type == HEVC_NAL_VPS) { err = h265_metadata_update_vps(bsf, au->units[i].content); if (err < 0) return err; + has_vps = 1; } if (au->units[i].type == HEVC_NAL_SPS) { err = h265_metadata_update_sps(bsf, au->units[i].content); @@ -477,6 +595,20 @@ static int h265_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, } } + if (pkt && ctx->frame_packing != BSF_ELEMENT_PASS) { + // The current packet is a seek point for metadata insertion if it + // is the first one, has a VPS or is marked as containing a key frame. + int seek_point = !ctx->done_first_au || has_vps || + (pkt->flags & AV_PKT_FLAG_KEY); + + err = h265_metadata_handle_frame_packing(bsf, pkt, au, seek_point); + if (err < 0) + return err; + } + + if (pkt) + ctx->done_first_au = 1; + return 0; } @@ -531,6 +663,31 @@ static const AVOption h265_metadata_options[] = { OFFSET(num_ticks_poc_diff_one), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, FLAGS }, + BSF_ELEMENT_OPTIONS_PIRE("frame_packing", + "Frame packing arrangement SEI", + frame_packing, FLAGS), + { "frame_packing_type", "Set frame_packing_arrangement_type (table D.8, 3 to 5), default: from the stream's Stereo 3D side data", + OFFSET(frame_packing_type), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 5, FLAGS, .unit = "frame_packing_type" }, + { "side_by_side", "Side by side", + 0, AV_OPT_TYPE_CONST, { .i64 = 3 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "top_bottom", "Top and bottom", + 0, AV_OPT_TYPE_CONST, { .i64 = 4 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "temporal", "Temporal interleaving (frame sequence)", + 0, AV_OPT_TYPE_CONST, { .i64 = 5 }, .flags = FLAGS, .unit = "frame_packing_type" }, + { "frame_packing_view", "Set frame_packing_arrangement content_interpretation_type, default: from the stream's Stereo 3D side data", + OFFSET(frame_packing_view), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 2, FLAGS, .unit = "frame_packing_view" }, + { "unspecified", "No interpretation", + 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "left_first", "Frame 0 is the left view", + 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "right_first", "Frame 0 is the right view", + 0, AV_OPT_TYPE_CONST, { .i64 = 2 }, .flags = FLAGS, .unit = "frame_packing_view" }, + { "frame_packing_quincunx", "Set quincunx_sampling_flag", + OFFSET(frame_packing_quincunx), AV_OPT_TYPE_INT, + { .i64 = -1 }, -1, 1, FLAGS }, + { "crop_left", "Set left border crop offset", OFFSET(crop_left), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, HEVC_MAX_WIDTH, FLAGS }, @@ -577,6 +734,14 @@ static const AVOption h265_metadata_options[] = { { NULL } }; +static void h265_metadata_flush(AVBSFContext *bsf) +{ + H265MetadataContext *ctx = bsf->priv_data; + + ctx->done_first_au = 0; + ctx->frame_packing_frame = 0; +} + static const AVClass h265_metadata_class = { .class_name = "h265_metadata_bsf", .item_name = av_default_item_name, @@ -596,4 +761,5 @@ const FFBitStreamFilter ff_hevc_metadata_bsf = { .init = &h265_metadata_init, .close = &ff_cbs_bsf_generic_close, .filter = &ff_cbs_bsf_generic_filter, + .flush = &h265_metadata_flush, }; diff --git a/libavcodec/cbs_sei.c b/libavcodec/cbs_sei.c index 70c18b4ced4..c9058fd20ce 100644 --- a/libavcodec/cbs_sei.c +++ b/libavcodec/cbs_sei.c @@ -17,6 +17,7 @@ */ #include "libavutil/mem.h" +#include "libavutil/stereo3d.h" #include "cbs.h" #include "cbs_internal.h" #include "cbs_h264.h" @@ -710,3 +711,155 @@ const SEIMessageTypeDescriptor *ff_cbs_sei_find_type(CodedBitstreamContext *ctx, return NULL; } + +int ff_cbs_sei_frame_packing_params(void *log_ctx, enum AVCodecID codec_id, + int type, int content, int quincunx, + const AVStereo3D *stereo, int frame0, + SEIFramePackingParams *params) +{ + int explicit_type = type >= 0; + + if (type < 0) { + if (!stereo) + return AVERROR(ENOENT); + + switch (stereo->type) { + case AV_STEREO3D_CHECKERBOARD: + type = SEI_FPA_H264_TYPE_CHECKERBOARD; + break; + case AV_STEREO3D_COLUMNS: + type = SEI_FPA_H264_TYPE_INTERLEAVE_COLUMN; + break; + case AV_STEREO3D_LINES: + type = SEI_FPA_H264_TYPE_INTERLEAVE_ROW; + break; + case AV_STEREO3D_SIDEBYSIDE_QUINCUNX: + if (quincunx < 0) + quincunx = 1; + av_fallthrough; + case AV_STEREO3D_SIDEBYSIDE: + type = SEI_FPA_TYPE_SIDE_BY_SIDE; + break; + case AV_STEREO3D_TOPBOTTOM: + type = SEI_FPA_TYPE_TOP_BOTTOM; + break; + case AV_STEREO3D_FRAMESEQUENCE: + type = SEI_FPA_TYPE_INTERLEAVE_TEMPORAL; + break; + case AV_STEREO3D_2D: + type = SEI_FPA_H264_TYPE_2D; + break; + default: + av_log(log_ctx, AV_LOG_WARNING, "Stereo 3D type %s has no frame " + "packing arrangement.\n", av_stereo3d_type_name(stereo->type)); + return AVERROR(ENOENT); + } + } + + if (codec_id == AV_CODEC_ID_H264 ? type > 7 : + (type < SEI_FPA_TYPE_SIDE_BY_SIDE || type > SEI_FPA_TYPE_INTERLEAVE_TEMPORAL)) { + // A layout of the stream that the codec cannot signal is left alone, + // one that was asked for is an error. + av_log(log_ctx, explicit_type ? AV_LOG_ERROR : AV_LOG_WARNING, + "Frame packing arrangement type %d is not defined for %s.\n", + type, codec_id == AV_CODEC_ID_H264 ? "H.264" : "H.265"); + return explicit_type ? AVERROR(EINVAL) : AVERROR(ENOENT); + } + + if (content < 0) + content = type == SEI_FPA_H264_TYPE_2D ? 0 : + stereo && (stereo->flags & AV_STEREO3D_FLAG_INVERT) ? 2 : 1; + if (type == SEI_FPA_H264_TYPE_2D && content) { + av_log(log_ctx, AV_LOG_ERROR, + "A 2D frame packing arrangement requires content interpretation 0.\n"); + return AVERROR(EINVAL); + } + + if (type == SEI_FPA_TYPE_INTERLEAVE_TEMPORAL && stereo && + stereo->type == AV_STEREO3D_FRAMESEQUENCE && + (stereo->view == AV_STEREO3D_VIEW_LEFT || + stereo->view == AV_STEREO3D_VIEW_RIGHT)) + frame0 = (stereo->view == AV_STEREO3D_VIEW_LEFT) != (content == 2); + + // Table D-9: the checkerboard is always quincunx sampled. + if (type == SEI_FPA_H264_TYPE_CHECKERBOARD) + quincunx = 1; + else if (quincunx < 0) + quincunx = type == SEI_FPA_TYPE_SIDE_BY_SIDE && stereo && + stereo->type == AV_STEREO3D_SIDEBYSIDE_QUINCUNX; + if (quincunx && type >= SEI_FPA_TYPE_INTERLEAVE_TEMPORAL) { + av_log(log_ctx, AV_LOG_ERROR, + "Frame packing arrangement type %d cannot be quincunx sampled.\n", + type); + return AVERROR(EINVAL); + } + + *params = (SEIFramePackingParams) { + .type = type, + .content_interpretation_type = content, + .quincunx_sampling_flag = quincunx, + .current_frame_is_frame0_flag = type == SEI_FPA_TYPE_INTERLEAVE_TEMPORAL && + frame0, + // The message of a temporal interleaving applies to one frame only. + .persistence = type != SEI_FPA_TYPE_INTERLEAVE_TEMPORAL, + }; + + return 0; +} + +int ff_cbs_sei_frame_packing_to_packet(AVPacket *pkt, + const SEIFramePackingParams *params) +{ + AVStereo3D *stereo; + enum AVStereo3DType type; + size_t size; + int err; + + if (params->type != SEI_FPA_H264_TYPE_2D && + (params->content_interpretation_type < 1 || + params->content_interpretation_type > 2)) + return 0; + + switch (params->type) { + case SEI_FPA_H264_TYPE_CHECKERBOARD: + type = AV_STEREO3D_CHECKERBOARD; + break; + case SEI_FPA_H264_TYPE_INTERLEAVE_COLUMN: + type = AV_STEREO3D_COLUMNS; + break; + case SEI_FPA_H264_TYPE_INTERLEAVE_ROW: + type = AV_STEREO3D_LINES; + break; + case SEI_FPA_TYPE_SIDE_BY_SIDE: + type = params->quincunx_sampling_flag ? AV_STEREO3D_SIDEBYSIDE_QUINCUNX : + AV_STEREO3D_SIDEBYSIDE; + break; + case SEI_FPA_TYPE_TOP_BOTTOM: + type = AV_STEREO3D_TOPBOTTOM; + break; + case SEI_FPA_TYPE_INTERLEAVE_TEMPORAL: + type = AV_STEREO3D_FRAMESEQUENCE; + break; + case SEI_FPA_H264_TYPE_2D: + type = AV_STEREO3D_2D; + break; + default: + return 0; + } + + stereo = av_stereo3d_alloc_size(&size); + if (!stereo) + return AVERROR(ENOMEM); + stereo->type = type; + if (type != AV_STEREO3D_2D && params->content_interpretation_type == 2) + stereo->flags = AV_STEREO3D_FLAG_INVERT; + if (type == AV_STEREO3D_FRAMESEQUENCE) + stereo->view = params->current_frame_is_frame0_flag != + (params->content_interpretation_type == 2) ? + AV_STEREO3D_VIEW_LEFT : AV_STEREO3D_VIEW_RIGHT; + + err = av_packet_add_side_data(pkt, AV_PKT_DATA_STEREO3D, (uint8_t *)stereo, size); + if (err < 0) + av_free(stereo); + return err; +} diff --git a/libavcodec/cbs_sei.h b/libavcodec/cbs_sei.h index e3e30971040..1fff3d21e43 100644 --- a/libavcodec/cbs_sei.h +++ b/libavcodec/cbs_sei.h @@ -26,6 +26,8 @@ #include "sei.h" +struct AVStereo3D; + typedef struct SEIRawFillerPayload { uint32_t payload_size; } SEIRawFillerPayload; @@ -275,4 +277,47 @@ int ff_cbs_sei_read_message(CodedBitstreamContext *ctx, struct GetBitContext *rw int ff_cbs_sei_write_message(CodedBitstreamContext *ctx, struct PutBitContext *rw, SEIRawMessage *current); +/** + * What a bitstream filter needs to write a frame packing arrangement SEI + * message, whichever codec it is for. + */ +typedef struct SEIFramePackingParams { + int type; ///< frame_packing_arrangement_type + int content_interpretation_type; + int quincunx_sampling_flag; + int current_frame_is_frame0_flag; + int persistence; ///< 0 for the current frame only, 1 until the next message +} SEIFramePackingParams; + +/** + * Work out the frame packing arrangement to signal. + * + * What is not given is taken from the Stereo 3D description, the way the + * decoders map the message to it, but the other way round. + * + * @param log_ctx context for logging. + * @param codec_id AV_CODEC_ID_H264 or AV_CODEC_ID_HEVC. + * @param type frame_packing_arrangement_type (H.264 Table D-9, H.265 + * Table D.8), or a negative value to take it from stereo. + * @param content content_interpretation_type (0 unspecified, 1 left view + * first, 2 right view first), or a negative value to take + * it from stereo. + * @param quincunx quincunx_sampling_flag, or a negative value to take it + * from stereo. + * @param stereo Stereo 3D description, or NULL. + * @param frame0 current_frame_is_frame0_flag for temporal interleaving + * when stereo does not say which view the frame is. + * @return 0 on success, AVERROR(ENOENT) if there is nothing to signal (no + * type given and no usable stereo), another negative value if the + * parameters cannot be signalled in the codec. + */ +int ff_cbs_sei_frame_packing_params(void *log_ctx, enum AVCodecID codec_id, + int type, int content, int quincunx, + const struct AVStereo3D *stereo, int frame0, + SEIFramePackingParams *params); + +/** Attach a representable frame packing description to a packet. */ +int ff_cbs_sei_frame_packing_to_packet(AVPacket *pkt, + const SEIFramePackingParams *params); + #endif /* AVCODEC_CBS_SEI_H */ -- 2.54.0 >From 2235e2f0333818446fddf10a3c0058ad70282c6e Mon Sep 17 00:00:00 2001 From: Daniel Campos Ramos <[email protected]> Date: Fri, 23 Oct 2026 05:25:17 +0000 Subject: [PATCH 4/4] tests: check frame packing BSFs independently of decoder side data Read the emitted SEI syntax and NAL placement directly and compare the output with FATE references. Cover every layout and content interpretation, packet and option precedence, explicit physical eyes, unspecified temporal views, flush, replacement and removal. Check extracted packet side data, unchanged encoded bytes, cancellation, unsupported interpretations and 2D. Check that flushing also restores the first-access-unit seek point for display_orientation. Signed-off-by: Daniel Campos Ramos <[email protected]> Assisted-by: Claude Sonnet via Claude Code Assisted-by: gpt-6-astra via Codex --- tests/api/Makefile | 1 + tests/api/api-bsf-frame-packing-test.c | 349 +++++++++++++++++++++++++ tests/fate/cbs.mak | 7 + tests/ref/fate/cbs-h264-frame-packing | 114 ++++++++ tests/ref/fate/cbs-hevc-frame-packing | 82 ++++++ 5 files changed, 553 insertions(+) create mode 100644 tests/api/api-bsf-frame-packing-test.c create mode 100644 tests/ref/fate/cbs-h264-frame-packing create mode 100644 tests/ref/fate/cbs-hevc-frame-packing diff --git a/tests/api/Makefile b/tests/api/Makefile index e51dc6c5837..57ba2e7a027 100644 --- a/tests/api/Makefile +++ b/tests/api/Makefile @@ -1,3 +1,4 @@ +APITESTPROGS-$(call ALLYES, H264_METADATA_BSF HEVC_METADATA_BSF H264_DEMUXER HEVC_DEMUXER H264_PARSER HEVC_PARSER) += api-bsf-frame-packing APITESTPROGS-$(call ALLYES, LIBASTCENC_ENCODER LIBASTCENC_DECODER) += api-astc-profile APITESTPROGS-$(call ENCDEC, FLAC, FLAC) += api-flac APITESTPROGS-$(call DEMDEC, H264, H264) += api-h264 diff --git a/tests/api/api-bsf-frame-packing-test.c b/tests/api/api-bsf-frame-packing-test.c new file mode 100644 index 00000000000..785be2e1122 --- /dev/null +++ b/tests/api/api-bsf-frame-packing-test.c @@ -0,0 +1,349 @@ +/* + * Copyright (c) 2026 Daniel Campos Ramos + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "libavcodec/bsf.h" +#include "libavformat/avformat.h" +#include "libavutil/opt.h" +#include "libavutil/stereo3d.h" + +/* Read the SEI independently of CBS and the decoders. */ +static unsigned bits(const uint8_t *data, int *pos, int n) +{ + unsigned value = 0; + while (n--) { + value = (value << 1) | ((data[*pos >> 3] >> (7 - (*pos & 7))) & 1); + ++*pos; + } + return value; +} + +static unsigned ue(const uint8_t *data, int *pos) +{ + int n = 0; + while (!bits(data, pos, 1) && n < 31) + n++; + return (1U << n) - 1 + bits(data, pos, n); +} + +static void show(const AVPacket *pkt, int hevc) +{ + int start = -1, seen_vcl = 0, found = 0; + uint8_t *rbsp = av_malloc(pkt->size + AV_INPUT_BUFFER_PADDING_SIZE); + if (!rbsp) + exit(1); + for (int end = 0; end <= pkt->size; end++) { + int next = end + 3 <= pkt->size && !pkt->data[end] && + !pkt->data[end + 1] && pkt->data[end + 2] == 1; + if (!next && end != pkt->size) + continue; + if (start >= 0) { + int type = hevc ? (pkt->data[start] >> 1) & 63 : pkt->data[start] & 31; + int size = 0, zeros = 0; + if (hevc ? type < 32 : type >= 1 && type <= 5) + seen_vcl = 1; + if (hevc ? type == 39 || type == 40 : type == 6) { + for (int i = start + 1 + hevc; i < end; i++) { + int b = pkt->data[i]; + if (zeros == 2 && b == 3) { + zeros = 0; + continue; + } + rbsp[size++] = b; + zeros = b ? 0 : zeros + 1; + } + memset(rbsp + size, 0, AV_INPUT_BUFFER_PADDING_SIZE); + if (!size || rbsp[0] == 0x80) { + printf(" empty-sei"); + } else for (int i = 0; i < size && rbsp[i] != 0x80;) { + unsigned payload = 0, length = 0; + while (i < size && rbsp[i] == 255) + payload += rbsp[i++]; + if (i >= size) + exit(2); + payload += rbsp[i++]; + while (i < size && rbsp[i] == 255) + length += rbsp[i++]; + if (i >= size) + exit(2); + length += rbsp[i++]; + if (length > size - i) + exit(2); + if (payload == 45) { + int pos = 0, cancel, type, q, content, frame0, persistence; + const uint8_t *p = rbsp + i; + ue(p, &pos); + cancel = bits(p, &pos, 1); + if (cancel || length < 4) + exit(2); + type = bits(p, &pos, 7); + q = bits(p, &pos, 1); + content = bits(p, &pos, 6); + bits(p, &pos, 3); + frame0 = bits(p, &pos, 1); + bits(p, &pos, 2); + if (!q && type != 5) + bits(p, &pos, 16); + bits(p, &pos, 8); + persistence = hevc ? bits(p, &pos, 1) : ue(p, &pos); + printf(" %d,%d,%d,%d,%d,%d", type, q, content, frame0, persistence, !seen_vcl); + found++; + } + if (payload == 47) { + printf(" orientation"); + found++; + } + i += length; + } + } + } + start = end + 3; + } + if (!found) + printf(" none"); + putchar('\n'); + av_free(rbsp); +} + +static int add_stereo(AVPacketSideData **sd, int *nb_sd, int type, int invert, int view) +{ + size_t size; + AVStereo3D *st = av_stereo3d_alloc_size(&size); + if (!st) + return AVERROR(ENOMEM); + st->type = type; + st->flags = invert ? AV_STEREO3D_FLAG_INVERT : 0; + st->view = view; + if (!av_packet_side_data_add(sd, nb_sd, AV_PKT_DATA_STEREO3D, st, size, 0)) { + av_free(st); + return AVERROR(ENOMEM); + } + return 0; +} + +static void test(const char *label, const char *name, const AVCodecParameters *par, + AVPacket **input, int count, const char *options, + int stream_type, int packet_type, int invert, int view, int flush) +{ + AVBSFContext *ctx = NULL; + AVPacket *pkt = av_packet_alloc(); + int ret; + printf("%s:", label); + if (!pkt) + exit(1); + ret = av_bsf_alloc(av_bsf_get_by_name(name), &ctx); + if (ret < 0 || avcodec_parameters_copy(ctx->par_in, par) < 0) + exit(1); + av_packet_side_data_remove(ctx->par_in->coded_side_data, + &ctx->par_in->nb_coded_side_data, AV_PKT_DATA_STEREO3D); + if (stream_type >= 0 && add_stereo(&ctx->par_in->coded_side_data, + &ctx->par_in->nb_coded_side_data, + stream_type, invert, view) < 0) + exit(1); + ctx->time_base_in = (AVRational) { 1, 25 }; + ret = av_opt_set_from_string(ctx->priv_data, options, NULL, "=", ":"); + if (ret >= 0) + ret = av_bsf_init(ctx); + if (ret < 0) { + printf(" error\n"); + goto end; + } + for (int i = 0; i < count; i++) { + if (av_packet_ref(pkt, input[i]) < 0) + exit(1); + av_packet_side_data_remove(pkt->side_data, &pkt->side_data_elems, AV_PKT_DATA_STEREO3D); + if (packet_type >= 0 && add_stereo(&pkt->side_data, &pkt->side_data_elems, + packet_type, invert, + view < 0 ? (i ? AV_STEREO3D_VIEW_UNSPEC : + AV_STEREO3D_VIEW_RIGHT) : view) < 0) + exit(1); + if (flush) + av_bsf_flush(ctx); + ret = av_bsf_send_packet(ctx, pkt); + if (ret >= 0) + ret = av_bsf_receive_packet(ctx, pkt); + if (ret < 0) { + printf(" error\n"); + break; + } + show(pkt, par->codec_id == AV_CODEC_ID_HEVC); + av_packet_unref(pkt); + } +end: + av_packet_free(&pkt); + av_bsf_free(&ctx); +} + + +static void extract_test(const char *label, const char *name, + const AVCodecParameters *par, AVPacket **input, + int count, const char *prepare) +{ + AVBSFContext *ctx[2] = { NULL }; + AVPacket *pkt = av_packet_alloc(), *original = av_packet_alloc(); + int ret = 0; + + printf("%s:", label); + if (!pkt || !original) + exit(1); + for (int j = 0; j < 2; j++) { + if (av_bsf_alloc(av_bsf_get_by_name(name), &ctx[j]) < 0 || + avcodec_parameters_copy(ctx[j]->par_in, par) < 0) + exit(1); + ctx[j]->time_base_in = (AVRational) { 1, 25 }; + ret = av_opt_set_from_string(ctx[j]->priv_data, + j ? "frame_packing=extract" : prepare, + NULL, "=", ":"); + if (ret < 0 || (ret = av_bsf_init(ctx[j])) < 0) + goto end; + } + for (int i = 0; i < count; i++) { + const AVStereo3D *st; + size_t size; + if (av_packet_ref(pkt, input[i]) < 0) + exit(1); + for (int j = 0; j < 2; j++) { + ret = av_bsf_send_packet(ctx[j], pkt); + if (ret >= 0) + ret = av_bsf_receive_packet(ctx[j], pkt); + if (ret < 0) + goto end; + if (!j && av_packet_ref(original, pkt) < 0) + exit(1); + } + printf(" unchanged=%d", pkt->size == original->size && + !memcmp(pkt->data, original->data, pkt->size)); + st = (const AVStereo3D *)av_packet_get_side_data(pkt, AV_PKT_DATA_STEREO3D, &size); + if (st && size >= sizeof(*st)) + printf(" stereo=%d,%d,%d\n", st->type, st->flags, st->view); + else + printf(" none\n"); + av_packet_unref(original); + av_packet_unref(pkt); + } +end: + if (ret < 0) + printf(" error\n"); + av_bsf_free(&ctx[0]); + av_bsf_free(&ctx[1]); + av_packet_free(&pkt); + av_packet_free(&original); +} + +int main(int argc, char **argv) +{ + static const int types[] = { AV_STEREO3D_CHECKERBOARD, AV_STEREO3D_COLUMNS, + AV_STEREO3D_LINES, AV_STEREO3D_SIDEBYSIDE, AV_STEREO3D_TOPBOTTOM, + AV_STEREO3D_FRAMESEQUENCE, AV_STEREO3D_2D, AV_STEREO3D_SIDEBYSIDE_QUINCUNX }; + AVFormatContext *fmt = NULL; + AVPacket *input[3]; + const char *name; + int hevc; + if (argc != 3) + return 1; + hevc = !strcmp(argv[1], "hevc"); + name = hevc ? "hevc_metadata" : "h264_metadata"; + if (avformat_open_input(&fmt, argv[2], NULL, NULL) < 0 || + avformat_find_stream_info(fmt, NULL) < 0) + return 1; + for (int i = 0; i < 3; i++) { + input[i] = av_packet_alloc(); + if (!input[i] || av_read_frame(fmt, input[i]) < 0) + return 1; + } + for (int type = hevc ? 3 : 0; type <= (hevc ? 5 : 7); type++) { + for (int view = 0; view <= 2; view++) { + char label[64], options[128]; + snprintf(label, sizeof(label), "type%d-view%d", type, view); + snprintf(options, sizeof(options), "frame_packing=insert:frame_packing_type=%d:frame_packing_view=%d", type, view); + test(label, name, fmt->streams[0]->codecpar, input, 1, options, -1, -1, 0, 0, 0); + } + } + for (int i = 0; i < 8; i++) { + for (int invert = 0; invert < 2; invert++) { + char label[64]; + snprintf(label, sizeof(label), "stereo%d-invert%d", i, invert); + test(label, name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert", types[i], -1, invert, 0, 0); + } + } + test("remove", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=remove", -1, -1, 0, 0, 0); + test("packet-wins", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert", AV_STEREO3D_TOPBOTTOM, AV_STEREO3D_SIDEBYSIDE, 1, 0, 0); + test("option-wins", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert:frame_packing_type=4:frame_packing_view=1", AV_STEREO3D_SIDEBYSIDE, -1, 1, 0, 0); + test("quincunx", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert:frame_packing_type=3:frame_packing_quincunx=1", -1, -1, 0, 0, 0); + test("invalid-temporal-quincunx", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert:frame_packing_type=5:frame_packing_quincunx=1", -1, -1, 0, 0, 0); + for (int view = AV_STEREO3D_VIEW_LEFT; view <= AV_STEREO3D_VIEW_RIGHT; view++) { + test(view == AV_STEREO3D_VIEW_LEFT ? "packet-frame0" : "packet-frame1", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert:frame_packing_type=5", -1, AV_STEREO3D_FRAMESEQUENCE, 0, view, 0); + } + test("packet-nonkey", name, fmt->streams[0]->codecpar, input, 2, "frame_packing=insert", -1, AV_STEREO3D_TOPBOTTOM, 0, 0, 0); + test("replace-cancel", name, fmt->streams[0]->codecpar, input, 3, "frame_packing=insert:frame_packing_type=3", -1, -1, 0, 0, 0); + test("pass", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=pass", -1, -1, 0, 0, 0); + test("no-description", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert", -1, -1, 0, 0, 0); + test("inferred-quincunx", name, fmt->streams[0]->codecpar, input, 1, "frame_packing=insert:frame_packing_type=3", AV_STEREO3D_SIDEBYSIDE_QUINCUNX, -1, 0, 0, 0); + test("temporal", name, fmt->streams[0]->codecpar, input, 2, "frame_packing=insert:frame_packing_type=5", -1, -1, 0, 0, 0); + test("temporal-flush", name, fmt->streams[0]->codecpar, input, 2, "frame_packing=insert:frame_packing_type=5", -1, -1, 0, 0, 1); + + for (int invert = 0; invert < 2; invert++) { + for (int view = AV_STEREO3D_VIEW_PACKED; view <= AV_STEREO3D_VIEW_UNSPEC; view++) { + char label[64]; + snprintf(label, sizeof(label), "temporal-view%d-invert%d", view, invert); + test(label, name, fmt->streams[0]->codecpar, input, 2, + "frame_packing=insert:frame_packing_type=5", -1, + AV_STEREO3D_FRAMESEQUENCE, invert, view, 0); + } + } + test("temporal-override-content", name, fmt->streams[0]->codecpar, input, 1, + "frame_packing=insert:frame_packing_type=5:frame_packing_view=1", -1, + AV_STEREO3D_FRAMESEQUENCE, 1, AV_STEREO3D_VIEW_LEFT, 0); + for (int invert = 0; invert < 2; invert++) + test(invert ? "temporal-right-unspec-inverted" : "temporal-right-unspec", + name, fmt->streams[0]->codecpar, input, 2, + "frame_packing=insert:frame_packing_type=5", -1, + AV_STEREO3D_FRAMESEQUENCE, invert, -1, 0); + extract_test("extract-original", name, fmt->streams[0]->codecpar, input, 3, + "frame_packing=pass"); + extract_test("extract-absent", name, fmt->streams[0]->codecpar, input, 1, + "frame_packing=remove"); + for (int type = hevc ? 3 : 0; type <= (hevc ? 5 : 7); type++) { + for (int content = 0; content < 3; content++) { + char label[64], options[128]; + if (type == 6 && content) + continue; + snprintf(label, sizeof(label), "extract-type%d-content%d", type, content); + snprintf(options, sizeof(options), "frame_packing=insert:frame_packing_type=%d:frame_packing_view=%d", type, content); + extract_test(label, name, fmt->streams[0]->codecpar, input, type == 5 ? 2 : 1, options); + } + } + extract_test("extract-quincunx", name, fmt->streams[0]->codecpar, input, 1, + "frame_packing=insert:frame_packing_type=3:frame_packing_quincunx=1"); + if (!hevc) { + AVPacket *nonkey[] = { input[1], input[1] }; + test("orientation-no-flush", name, fmt->streams[0]->codecpar, nonkey, 2, + "display_orientation=insert:rotate=90", -1, -1, 0, 0, 0); + test("orientation-flush", name, fmt->streams[0]->codecpar, nonkey, 2, + "display_orientation=insert:rotate=90", -1, -1, 0, 0, 1); + } + for (int i = 0; i < 3; i++) + av_packet_free(&input[i]); + avformat_close_input(&fmt); + return 0; +} diff --git a/tests/fate/cbs.mak b/tests/fate/cbs.mak index d9200debc95..36b629d160d 100644 --- a/tests/fate/cbs.mak +++ b/tests/fate/cbs.mak @@ -219,3 +219,10 @@ $(foreach N,$(FATE_CBS_VP9_SAMPLES),$(eval $(call FATE_CBS_TEST,vp9,$(basename $ FATE_CBS_VP9-$(call FATE_CBS_DEPS, IVF, VP9, VP9, VP9, IVF) = $(FATE_CBS_vp9) FATE_SAMPLES_AVCONV += $(FATE_CBS_VP9-yes) fate-cbs-vp9: $(FATE_CBS_VP9-yes) + +FATE_CBS_FRAME_PACKING-$(call ALLYES, H264_METADATA_BSF HEVC_METADATA_BSF H264_DEMUXER HEVC_DEMUXER H264_PARSER HEVC_PARSER FILE_PROTOCOL) += fate-cbs-h264-frame-packing fate-cbs-hevc-frame-packing +fate-cbs-h264-frame-packing: CMD = run $(APITESTSDIR)/api-bsf-frame-packing-test$(EXESUF) h264 $(TARGET_SAMPLES)/h264/frame-packing-persistence.h264 +fate-cbs-hevc-frame-packing: CMD = run $(APITESTSDIR)/api-bsf-frame-packing-test$(EXESUF) hevc $(TARGET_SAMPLES)/hevc/frame-packing-persistence.hevc +$(FATE_CBS_FRAME_PACKING-yes): $(APITESTSDIR)/api-bsf-frame-packing-test$(EXESUF) +FATE_SAMPLES_FFMPEG += $(FATE_CBS_FRAME_PACKING-yes) +fate-cbs: $(FATE_CBS_FRAME_PACKING-yes) diff --git a/tests/ref/fate/cbs-h264-frame-packing b/tests/ref/fate/cbs-h264-frame-packing new file mode 100644 index 00000000000..73e385da356 --- /dev/null +++ b/tests/ref/fate/cbs-h264-frame-packing @@ -0,0 +1,114 @@ +type0-view0: 0,1,0,0,1,1 +type0-view1: 0,1,1,0,1,1 +type0-view2: 0,1,2,0,1,1 +type1-view0: 1,0,0,0,1,1 +type1-view1: 1,0,1,0,1,1 +type1-view2: 1,0,2,0,1,1 +type2-view0: 2,0,0,0,1,1 +type2-view1: 2,0,1,0,1,1 +type2-view2: 2,0,2,0,1,1 +type3-view0: 3,0,0,0,1,1 +type3-view1: 3,0,1,0,1,1 +type3-view2: 3,0,2,0,1,1 +type4-view0: 4,0,0,0,1,1 +type4-view1: 4,0,1,0,1,1 +type4-view2: 4,0,2,0,1,1 +type5-view0: 5,0,0,1,0,1 +type5-view1: 5,0,1,1,0,1 +type5-view2: 5,0,2,1,0,1 +type6-view0: 6,0,0,0,1,1 +type6-view1: error +type6-view2: error +type7-view0: 7,0,0,0,1,1 +type7-view1: 7,0,1,0,1,1 +type7-view2: 7,0,2,0,1,1 +stereo0-invert0: 0,1,1,0,1,1 +stereo0-invert1: 0,1,2,0,1,1 +stereo1-invert0: 1,0,1,0,1,1 +stereo1-invert1: 1,0,2,0,1,1 +stereo2-invert0: 2,0,1,0,1,1 +stereo2-invert1: 2,0,2,0,1,1 +stereo3-invert0: 3,0,1,0,1,1 +stereo3-invert1: 3,0,2,0,1,1 +stereo4-invert0: 4,0,1,0,1,1 +stereo4-invert1: 4,0,2,0,1,1 +stereo5-invert0: 5,0,1,1,0,1 +stereo5-invert1: 5,0,2,1,0,1 +stereo6-invert0: 6,0,0,0,1,1 +stereo6-invert1: 6,0,0,0,1,1 +stereo7-invert0: 3,1,1,0,1,1 +stereo7-invert1: 3,1,2,0,1,1 +remove: none +packet-wins: 3,0,2,0,1,1 +option-wins: 4,0,1,0,1,1 +quincunx: 3,1,1,0,1,1 +invalid-temporal-quincunx: error +packet-frame0: 5,0,1,1,0,1 +packet-frame1: 5,0,1,0,0,1 +packet-nonkey: 4,0,1,0,1,1 + 4,0,1,0,1,1 +replace-cancel: 3,0,1,0,1,1 + none + 3,0,1,0,1,1 +pass: 3,0,1,0,1,1 +no-description: 3,0,1,0,1,1 +inferred-quincunx: 3,1,1,0,1,1 +temporal: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-flush: 5,0,1,1,0,1 + 5,0,1,1,0,1 +temporal-view0-invert0: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-view1-invert0: 5,0,1,1,0,1 + 5,0,1,1,0,1 +temporal-view2-invert0: 5,0,1,0,0,1 + 5,0,1,0,0,1 +temporal-view3-invert0: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-view0-invert1: 5,0,2,1,0,1 + 5,0,2,0,0,1 +temporal-view1-invert1: 5,0,2,0,0,1 + 5,0,2,0,0,1 +temporal-view2-invert1: 5,0,2,1,0,1 + 5,0,2,1,0,1 +temporal-view3-invert1: 5,0,2,1,0,1 + 5,0,2,0,0,1 +temporal-override-content: 5,0,1,1,0,1 +temporal-right-unspec: 5,0,1,0,0,1 + 5,0,1,1,0,1 +temporal-right-unspec-inverted: 5,0,2,1,0,1 + 5,0,2,0,0,1 +extract-original: unchanged=1 stereo=1,0,0 + unchanged=1 none + unchanged=1 none +extract-absent: unchanged=1 none +extract-type0-content0: unchanged=1 none +extract-type0-content1: unchanged=1 stereo=4,0,0 +extract-type0-content2: unchanged=1 stereo=4,1,0 +extract-type1-content0: unchanged=1 none +extract-type1-content1: unchanged=1 stereo=7,0,0 +extract-type1-content2: unchanged=1 stereo=7,1,0 +extract-type2-content0: unchanged=1 none +extract-type2-content1: unchanged=1 stereo=6,0,0 +extract-type2-content2: unchanged=1 stereo=6,1,0 +extract-type3-content0: unchanged=1 none +extract-type3-content1: unchanged=1 stereo=1,0,0 +extract-type3-content2: unchanged=1 stereo=1,1,0 +extract-type4-content0: unchanged=1 none +extract-type4-content1: unchanged=1 stereo=2,0,0 +extract-type4-content2: unchanged=1 stereo=2,1,0 +extract-type5-content0: unchanged=1 none + unchanged=1 none +extract-type5-content1: unchanged=1 stereo=3,0,1 + unchanged=1 stereo=3,0,2 +extract-type5-content2: unchanged=1 stereo=3,1,2 + unchanged=1 stereo=3,1,1 +extract-type6-content0: unchanged=1 stereo=0,0,0 +extract-type7-content0: unchanged=1 none +extract-type7-content1: unchanged=1 none +extract-type7-content2: unchanged=1 none +extract-quincunx: unchanged=1 stereo=5,0,0 +orientation-no-flush: orientation + none +orientation-flush: orientation + orientation diff --git a/tests/ref/fate/cbs-hevc-frame-packing b/tests/ref/fate/cbs-hevc-frame-packing new file mode 100644 index 00000000000..ae22aae41dd --- /dev/null +++ b/tests/ref/fate/cbs-hevc-frame-packing @@ -0,0 +1,82 @@ +type3-view0: 3,0,0,0,1,1 +type3-view1: 3,0,1,0,1,1 +type3-view2: 3,0,2,0,1,1 +type4-view0: 4,0,0,0,1,1 +type4-view1: 4,0,1,0,1,1 +type4-view2: 4,0,2,0,1,1 +type5-view0: 5,0,0,1,0,1 +type5-view1: 5,0,1,1,0,1 +type5-view2: 5,0,2,1,0,1 +stereo0-invert0: 3,0,1,0,1,1 +stereo0-invert1: 3,0,1,0,1,1 +stereo1-invert0: 3,0,1,0,1,1 +stereo1-invert1: 3,0,1,0,1,1 +stereo2-invert0: 3,0,1,0,1,1 +stereo2-invert1: 3,0,1,0,1,1 +stereo3-invert0: 3,0,1,0,1,1 +stereo3-invert1: 3,0,2,0,1,1 +stereo4-invert0: 4,0,1,0,1,1 +stereo4-invert1: 4,0,2,0,1,1 +stereo5-invert0: 5,0,1,1,0,1 +stereo5-invert1: 5,0,2,1,0,1 +stereo6-invert0: 3,0,1,0,1,1 +stereo6-invert1: 3,0,1,0,1,1 +stereo7-invert0: 3,1,1,0,1,1 +stereo7-invert1: 3,1,2,0,1,1 +remove: none +packet-wins: 3,0,2,0,1,1 +option-wins: 4,0,1,0,1,1 +quincunx: 3,1,1,0,1,1 +invalid-temporal-quincunx: error +packet-frame0: 5,0,1,1,0,1 +packet-frame1: 5,0,1,0,0,1 +packet-nonkey: 4,0,1,0,1,1 + 4,0,1,0,1,1 +replace-cancel: 3,0,1,0,1,1 + none + 3,0,1,0,1,1 +pass: 3,0,1,0,1,1 +no-description: 3,0,1,0,1,1 +inferred-quincunx: 3,1,1,0,1,1 +temporal: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-flush: 5,0,1,1,0,1 + 5,0,1,1,0,1 +temporal-view0-invert0: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-view1-invert0: 5,0,1,1,0,1 + 5,0,1,1,0,1 +temporal-view2-invert0: 5,0,1,0,0,1 + 5,0,1,0,0,1 +temporal-view3-invert0: 5,0,1,1,0,1 + 5,0,1,0,0,1 +temporal-view0-invert1: 5,0,2,1,0,1 + 5,0,2,0,0,1 +temporal-view1-invert1: 5,0,2,0,0,1 + 5,0,2,0,0,1 +temporal-view2-invert1: 5,0,2,1,0,1 + 5,0,2,1,0,1 +temporal-view3-invert1: 5,0,2,1,0,1 + 5,0,2,0,0,1 +temporal-override-content: 5,0,1,1,0,1 +temporal-right-unspec: 5,0,1,0,0,1 + 5,0,1,1,0,1 +temporal-right-unspec-inverted: 5,0,2,1,0,1 + 5,0,2,0,0,1 +extract-original: unchanged=1 stereo=1,0,0 + unchanged=1 none + unchanged=1 none +extract-absent: unchanged=1 none +extract-type3-content0: unchanged=1 none +extract-type3-content1: unchanged=1 stereo=1,0,0 +extract-type3-content2: unchanged=1 stereo=1,1,0 +extract-type4-content0: unchanged=1 none +extract-type4-content1: unchanged=1 stereo=2,0,0 +extract-type4-content2: unchanged=1 stereo=2,1,0 +extract-type5-content0: unchanged=1 none + unchanged=1 none +extract-type5-content1: unchanged=1 stereo=3,0,1 + unchanged=1 stereo=3,0,2 +extract-type5-content2: unchanged=1 stereo=3,1,2 + unchanged=1 stereo=3,1,1 +extract-quincunx: unchanged=1 stereo=5,0,0 -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
