PR #24726 opened by rnissl URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24726 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24726.patch
This is 5/9 in the GoPro Max review order. It applies on current master by itself. Inject APP0..APP15 from a length-prefixed sidecar into each MJPEG packet, after SOI. Length 0 leaves the packet unchanged. An APP1-only payload keeps encoder APP0 (typical JFIF). Preferred review order, each request on its own: 1/9 avutil/log: flush stderr and match Windows warning yellow 2/9 avfilter: add yuvrgb_cuda 3/9 avfilter: add gopromax2_cuda GoPro Max 2 EAC stitch 4/9 avcodec: add mjpeg_nvjpeg NVIDIA JPEG encoder 5/9 avcodec/bsf: add jpegapp to inject JPEG APP segments 6/9 avfilter: add gopromax2 GoPro Max 2 EAC stitch 7/9 avfilter: add gopromax2_opencl GoPro Max 2 EAC stitch 8/9 avfilter: add gopromax_opencl GoPro Max 1 stitch 9/9 avcodec/mjpegenc: encode packed RGB via YUV conversion After 1-5, mapillary_tools processes GoPro Max 2 files GPU-only in my setup: scale_cuda=yuv444p10, yuvrgb_cuda=rgb0, gopromax2_cuda, mjpeg_nvjpeg. jpegapp inserts the APP1 segment while encoding, so the JPEG ffmpeg writes needs no postprocessing. After 6, the Max 2 CPU fallback works. After 7, Max 2 via OpenCL works. After 8, original GoPro Max (Max 1) via OpenCL works. 9/9 is needed when 4/9 cannot be used, for example because no nvjpeg library is present, and encoding falls back to the software mjpeg encoder. That encoder then receives packed RGB (rgb24). Upstream mjpegenc does not accept it; this change converts it to YUV first. The six patches I sent to ffmpeg-devel on 20 September are obsolete. Once the filter requests are merged, I will send one short documentation request that only adds the cross-references between these sections. From 57992cb45fcfb73bfd367ab8f96f5e8cc09093a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Reinhard=20Ni=C3=9Fl?= <[email protected]> Date: Sun, 20 Sep 2026 21:15:19 +0200 Subject: [PATCH] avcodec/bsf: add jpegapp to inject JPEG APP segments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Inject APP0..APP15 from a length-prefixed sidecar into each MJPEG packet, after SOI. Length 0 leaves the packet unchanged. An APP1-only payload keeps encoder APP0 (typical JFIF). Co-authored-by: Cursor <[email protected]> Signed-off-by: Reinhard Nißl <[email protected]> --- Changelog | 1 + doc/bitstream_filters.texi | 33 ++++ libavcodec/bitstream_filters.c | 1 + libavcodec/bsf/Makefile | 1 + libavcodec/bsf/jpegapp.c | 269 +++++++++++++++++++++++++++++++++ 5 files changed, 305 insertions(+) create mode 100644 libavcodec/bsf/jpegapp.c diff --git a/Changelog b/Changelog index 1cfa9db1b6..ed98fe8580 100644 --- a/Changelog +++ b/Changelog @@ -15,6 +15,7 @@ version <next>: - H.264 data partitioning support - DSD (dsd_msbf) encoder +- jpegapp bitstream filter version 9.0: - Extend AMF Color Converter (vf_vpp_amf) HDR capabilities diff --git a/doc/bitstream_filters.texi b/doc/bitstream_filters.texi index a00757d315..15e3479786 100644 --- a/doc/bitstream_filters.texi +++ b/doc/bitstream_filters.texi @@ -612,6 +612,39 @@ For example, to remux 30 MB/sec NTSC IMX to MOV: ffmpeg -i input.mxf -c copy -bsf:v imxdump -tag:v mx3n output.mov @end example +@section jpegapp + +Inject JPEG APP0..APP15 segments from a length-prefixed sidecar into each +MJPEG packet, after SOI. + +The sidecar is a sequence of records, one per encoded JPEG, in encode +order: + +@example +u32be length +u8 payload[length] +@end example + +@var{length} does not include the 4-byte prefix. Payload @code{0} leaves +that packet unchanged. A non-empty payload must be a concatenation of +complete APP segments (@code{FF En} plus the 16-bit length that includes +those two length bytes). + +If the payload starts with APP0 and the packet already has an APP0 after +SOI (typical JFIF), that encoder APP0 is dropped so the sidecar can +replace it. An APP1-only payload keeps the encoder APP0: +SOI, APP1, APP0, DQT, ... + +The filter requires @option{filename}. Extra unused sidecar bytes are +ignored; a packet without a matching record is an error. + +@example +ffmpeg -i in.mp4 -c:v mjpeg -bsf:v jpegapp=filename=app.bin frame_%d.jpg +@end example + +On Windows, escape the drive colon in the BSF option: +@code{filename='E\:/path/app.bin'}. + @section mjpeg2jpeg Convert MJPEG/AVI1 packets to full JPEG/JFIF packets. diff --git a/libavcodec/bitstream_filters.c b/libavcodec/bitstream_filters.c index f57c2632a0..ed57db5281 100644 --- a/libavcodec/bitstream_filters.c +++ b/libavcodec/bitstream_filters.c @@ -49,6 +49,7 @@ extern const FFBitStreamFilter ff_hapqa_extract_bsf; extern const FFBitStreamFilter ff_hevc_metadata_bsf; extern const FFBitStreamFilter ff_hevc_mp4toannexb_bsf; extern const FFBitStreamFilter ff_imx_dump_header_bsf; +extern const FFBitStreamFilter ff_jpegapp_bsf; extern const FFBitStreamFilter ff_lcevc_merge_bsf; extern const FFBitStreamFilter ff_lcevc_metadata_bsf; extern const FFBitStreamFilter ff_media100_to_mjpegb_bsf; diff --git a/libavcodec/bsf/Makefile b/libavcodec/bsf/Makefile index ec224e8774..c0badf5ed9 100644 --- a/libavcodec/bsf/Makefile +++ b/libavcodec/bsf/Makefile @@ -30,6 +30,7 @@ OBJS-$(CONFIG_DOVI_RPU_BSF) += bsf/dovi_rpu.o OBJS-$(CONFIG_DOVI_SPLIT_BSF) += bsf/dovi_split.o OBJS-$(CONFIG_HEVC_MP4TOANNEXB_BSF) += bsf/hevc_mp4toannexb.o OBJS-$(CONFIG_IMX_DUMP_HEADER_BSF) += bsf/imx_dump_header.o +OBJS-$(CONFIG_JPEGAPP_BSF) += bsf/jpegapp.o OBJS-$(CONFIG_LCEVC_MERGE_BSF) += bsf/lcevc_merge.o OBJS-$(CONFIG_LCEVC_METADATA_BSF) += bsf/lcevc_metadata.o OBJS-$(CONFIG_MEDIA100_TO_MJPEGB_BSF) += bsf/media100_to_mjpegb.o diff --git a/libavcodec/bsf/jpegapp.c b/libavcodec/bsf/jpegapp.c new file mode 100644 index 0000000000..119b3f761a --- /dev/null +++ b/libavcodec/bsf/jpegapp.c @@ -0,0 +1,269 @@ +/* + * Inject per-frame JPEG APP segments from a length-prefixed sidecar. + * + * Copyright (c) 2026 Reinhard Nißl + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * Sidecar format (big-endian), one record per encoded JPEG packet: + * + * u32be length + * u8 payload[length] + * + * Payload is zero or more complete JPEG APP segments (APP0..APP15), + * each ``FF En`` + 16-bit length (length includes those two bytes). + * Length 0 leaves the packet unchanged. + * + * After SOI the payload is inserted. If it starts with APP0 and the + * packet already has an APP0 (typical JFIF), that encoder APP0 is + * dropped so the sidecar can replace it. An APP1-only payload keeps + * the encoder APP0: SOI, APP1, APP0, DQT, ... + */ + +#include <errno.h> +#include <inttypes.h> +#include <limits.h> +#include <stdint.h> +#include <stdio.h> +#include <string.h> + +#include "libavutil/error.h" +#include "libavutil/file_open.h" +#include "libavutil/intreadwrite.h" +#include "libavutil/log.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" + +#include "libavcodec/bsf.h" +#include "libavcodec/bsf_internal.h" +#include "libavcodec/mjpeg.h" + +#define JPEGAPP_MAX_RECORD (16 * 1024 * 1024) + +typedef struct JPEGAppContext { + const AVClass *class; + char *filename; + FILE *file; + uint8_t *buf; + int buf_size; + int64_t packet_index; +} JPEGAppContext; + +static int jpeg_app_segment_len(const uint8_t *data, int size) +{ + int len; + + if (size < 4 || data[0] != 0xff || data[1] < APP0 || data[1] > APP15) + return AVERROR_INVALIDDATA; + len = AV_RB16(data + 2); + if (len < 2 || len > size - 2) + return AVERROR_INVALIDDATA; + return 2 + len; +} + +static int jpegapp_validate_payload(const uint8_t *data, int size) +{ + int pos = 0, n; + + while (pos < size) { + n = jpeg_app_segment_len(data + pos, size - pos); + if (n < 0) + return n; + pos += n; + } + return 0; +} + +static int jpegapp_init(AVBSFContext *ctx) +{ + JPEGAppContext *s = ctx->priv_data; + + if (!s->filename || !s->filename[0]) { + av_log(ctx, AV_LOG_ERROR, "Option filename is required\n"); + return AVERROR(EINVAL); + } + + s->file = avpriv_fopen_utf8(s->filename, "rb"); + if (!s->file) { + int err = errno ? AVERROR(errno) : AVERROR(EIO); + av_log(ctx, AV_LOG_ERROR, "Could not open '%s'\n", s->filename); + return err; + } + return 0; +} + +static void jpegapp_close(AVBSFContext *ctx) +{ + JPEGAppContext *s = ctx->priv_data; + + if (s->file) { + if (getc(s->file) != EOF) + av_log(ctx, AV_LOG_VERBOSE, + "jpegapp: unused sidecar data after %"PRId64" packets\n", + s->packet_index); + fclose(s->file); + s->file = NULL; + } + av_freep(&s->buf); + s->buf_size = 0; +} + +static int jpegapp_read_record(AVBSFContext *ctx, int *out_size) +{ + JPEGAppContext *s = ctx->priv_data; + uint8_t hdr[4]; + uint32_t n; + size_t got; + int ret; + + got = fread(hdr, 1, 4, s->file); + if (got == 0 && feof(s->file)) { + av_log(ctx, AV_LOG_ERROR, + "jpegapp: no sidecar record for packet %"PRId64"\n", + s->packet_index); + return AVERROR_EOF; + } + if (got != 4) { + av_log(ctx, AV_LOG_ERROR, "jpegapp: truncated length prefix\n"); + return AVERROR_INVALIDDATA; + } + n = AV_RB32(hdr); + if (n > JPEGAPP_MAX_RECORD) { + av_log(ctx, AV_LOG_ERROR, "jpegapp: record length %u too large\n", n); + return AVERROR_INVALIDDATA; + } + if (n > (uint32_t)s->buf_size) { + ret = av_reallocp(&s->buf, n); + if (ret < 0) { + s->buf_size = 0; + return ret; + } + s->buf_size = n; + } + if (n) { + got = fread(s->buf, 1, n, s->file); + if (got != n) { + av_log(ctx, AV_LOG_ERROR, "jpegapp: truncated record payload\n"); + return AVERROR_INVALIDDATA; + } + ret = jpegapp_validate_payload(s->buf, n); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "jpegapp: payload is not a sequence of APP0..APP15 segments\n"); + return ret; + } + } + *out_size = n; + return 0; +} + +static int jpegapp_filter(AVBSFContext *ctx, AVPacket *out) +{ + JPEGAppContext *s = ctx->priv_data; + AVPacket *in; + int rec_size = 0, src_off, dst_size, ret; + + ret = ff_bsf_get_packet(ctx, &in); + if (ret < 0) + return ret; + + ret = jpegapp_read_record(ctx, &rec_size); + if (ret < 0) + goto fail; + + if (in->size < 2 || AV_RB16(in->data) != 0xffd8) { + av_log(ctx, AV_LOG_ERROR, "jpegapp: packet is not JPEG (missing SOI)\n"); + ret = AVERROR_INVALIDDATA; + goto fail; + } + + src_off = 2; + if (rec_size >= 2 && s->buf[0] == 0xff && s->buf[1] == APP0 && + in->size >= 4 && in->data[2] == 0xff && in->data[3] == APP0) { + int skip = jpeg_app_segment_len(in->data + 2, in->size - 2); + if (skip < 0) { + ret = skip; + goto fail; + } + src_off = 2 + skip; + } + + if (!rec_size) { + ret = av_packet_ref(out, in); + goto fail; + } + + if (in->size - src_off > INT_MAX - 2 - rec_size) { + ret = AVERROR(ERANGE); + goto fail; + } + dst_size = 2 + rec_size + (in->size - src_off); + ret = av_new_packet(out, dst_size); + if (ret < 0) + goto fail; + ret = av_packet_copy_props(out, in); + if (ret < 0) + goto fail; + + memcpy(out->data, in->data, 2); + memcpy(out->data + 2, s->buf, rec_size); + memcpy(out->data + 2 + rec_size, in->data + src_off, in->size - src_off); + s->packet_index++; + av_packet_free(&in); + return 0; + +fail: + if (ret >= 0) + s->packet_index++; + if (ret < 0) + av_packet_unref(out); + av_packet_free(&in); + return ret; +} + +#define OFFSET(x) offsetof(JPEGAppContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_BSF_PARAM) +static const AVOption jpegapp_options[] = { + { "filename", "sidecar of u32be-length + APP-segment records", OFFSET(filename), + AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, + { "file", "sidecar of u32be-length + APP-segment records", OFFSET(filename), + AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, + { NULL }, +}; + +static const AVClass jpegapp_class = { + .class_name = "jpegapp", + .item_name = av_default_item_name, + .option = jpegapp_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +static const enum AVCodecID jpegapp_codec_ids[] = { + AV_CODEC_ID_MJPEG, AV_CODEC_ID_NONE, +}; + +const FFBitStreamFilter ff_jpegapp_bsf = { + .p.name = "jpegapp", + .p.codec_ids = jpegapp_codec_ids, + .p.priv_class = &jpegapp_class, + .priv_data_size = sizeof(JPEGAppContext), + .init = jpegapp_init, + .close = jpegapp_close, + .filter = jpegapp_filter, +}; -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
