PR #24760 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24760
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24760.patch
**Depends on #24702.** This is one commit on top of #24702's approved head
0d0ba85092 (branch `dnn-postproc-error`). It follows up on the non-blocking
note in Forgejo_Fairy's review 8947 on #24702, so #24702 stays unchanged.
Please merge it after #24702. Until then the diff shown here includes #24702's
eight commits, and only the last commit is new.
**Problem.** `ff_dnn_start_inference_async()` in
`libavfilter/dnn/dnn_backend_common.c` returned the result of a failed
`pthread_create()` as is. That is a positive error number such as `EAGAIN`,
`EINVAL` or `EPERM`, not an `AVERROR` code. #24702's f2a854508d ("return the
ff_dnn_flush() error at EOF") makes `dnn_processing`, `dnn_detect` and
`dnn_classify` return the `ff_dnn_flush()` result from `activate()` after they
set EOF on the output. The filter graph and the buffersink treat only negative
values as errors, so a positive value there is dropped and the sink returns a
plain `AVERROR_EOF`. The route review 8947 identifies is the Torch backend in
async mode with `batch_size > 1`: the partial last batch is left for
`dnn_flush_th()`, which returns `execute_model_th()`'s result, and that is the
`ff_dnn_start_inference_async()` result unchanged. If the thread for that batch
cannot be created, the failed inference ends as a normal EOF.
**Fix.** The change is the one proposed in review 8947:
if (ret != 0) {
av_log(ctx, AV_LOG_ERROR, "Unable to start async inference: %s\n",
av_err2str(AVERROR(ret)));
return ret == EAGAIN ? AVERROR_EXTERNAL : AVERROR(ret);
}
`EAGAIN` is mapped to `AVERROR_EXTERNAL` because `AVERROR(EAGAIN)` is not an
error to the filter graph. It means "no output yet, try again", while the flush
has already given up the batch and set EOF. The test below shows this: a flush
that returns `AVERROR(EAGAIN)` at EOF is dropped just like a positive value,
and the sink reports EOF. The log line still names the original cause
("Resource temporarily unavailable").
**Callers.** In libavfilter/dnn, `pthread_create()` is called only in
`ff_dnn_start_inference_async()`, and only the TensorFlow and Torch backends
call that function. ONNX Runtime and OpenVINO do not use it.
- `dnn_backend_tf.c`, `execute_model_tf()`: checks `!= 0` and ignores the
value. It jumps to `err` with `ret` still 0 (from `fill_model_input_tf()`), so
it returns 0 whatever this function returns. This is not changed here (see
below).
- `dnn_backend_tf.c`, `dnn_flush_tf()`: returns the value unchanged to
`ff_dnn_flush()`.
- `dnn_backend_torch.cpp`, `execute_model_th()`: `ret != 0` goes to `err`,
which returns `ret`. From there `dnn_execute_model_th()` returns it if it is
nonzero, `dnn_flush_th()` returns it unchanged and `get_output_th()` returns
it. `get_output_th()` builds its task synchronously, so it does not reach the
async start.
- The filters (`dnn_processing`, `dnn_detect`, `dnn_classify`, `derain`, `sr`)
test the `ff_dnn_execute_model*()` and `ff_dnn_flush()` results with `!= 0`. On
`execute_model` failure, `dnn_processing`, `dnn_detect` and `dnn_classify`
return `AVERROR(EIO)`, and `derain` and `sr` return the value.
No caller tests for a positive value or `> 0`, so a negative code still fails
in every caller.
**Testing.** Darwin 27.0.0 arm64 (Mac Studio). ONNX Runtime 1.30.0, OpenVINO
2026.4.0 and TensorFlow 2.18.0 (C API), all enabled in one configure. Two
builds: "before" = 0d0ba85092 (#24702 head) and "after" = 52e469dcbd (this
commit; built and tested as 52e469dcbd, the same tree before re-signing). Both
built with no warnings in `dnn_backend_common.c`. LibTorch is not installed on
this machine, so the Torch backend was not built or run.
Fault injection. A test program links the libraries of each build and builds a
real `buffer -> format=rgb24,dnn_processing -> buffersink` graph with the
selected backend (`proc_valid.onnx`, `proc_valid_64.onnx` on OpenVINO,
`tf_rgb_valid.pb`). Two frames go through real inference and are drained. Then,
to reproduce the Torch partial-batch flush, the program replaces the backend
module's `flush` with a function that does what `dnn_flush_th()` ->
`execute_model_th()` does there: it returns the result of
`ff_dnn_start_inference_async()`. A `DYLD_INSERT_LIBRARIES` interposer makes
`pthread_create()` fail with the given error for that one thread only. The
program then sends EOF and logs `dnn_processing`'s `activate()` return (at EOF
this is `flush_frame()`'s return) and what `av_buffersink_get_frame()` returns.
The macOS values are `EAGAIN` = 35, `EINVAL` = 22 and `EPERM` = 1.
| injected `pthread_create()` error | before: `activate()` at EOF | before:
sink | after: `activate()` at EOF | after: sink |
|---|---|---|---|---|
| `EAGAIN` (35) | **35** | `AVERROR_EOF` | `AVERROR_EXTERNAL` |
`AVERROR_EXTERNAL`, then `AVERROR_EOF` |
| `EINVAL` (22) | **22** | `AVERROR_EOF` | `AVERROR(EINVAL)` (-22) | -22, then
`AVERROR_EOF` |
| `EPERM` (1) | **1** | `AVERROR_EOF` | `AVERROR(EPERM)` (-1) | -1, then
`AVERROR_EOF` |
| none; the flush returns `AVERROR(EAGAIN)` | -35 | `AVERROR_EOF` | -35 |
`AVERROR_EOF` |
All three backends gave the same results, 24 runs in total. In each run with
injection, exactly one `pthread_create()` call failed. Before the fix, the log
says only "Unable to start async inference.". After it, the log names the
cause, e.g. "Unable to start async inference: Resource temporarily
unavailable". The last row is why `EAGAIN` is not passed on as
`AVERROR(EAGAIN)`: at EOF that value is lost too.
Linux. An independent rerun used the same harness unchanged on Linux: Debian
bookworm arm64 in Docker on the same Mac Studio (kernel 7.0.12-linuxkit), gcc
12.2.0, glibc 2.36, ONNX Runtime 1.22.0 (the official linux-aarch64 release),
with separate before and after trees built with `--disable-everything
--enable-filter=buffer,buffersink,format,scale,dnn_processing
--enable-libonnxruntime`. The harness was linked with
`-Wl,--wrap=pthread_create` and a `__wrap_pthread_create` injector in place of
the macOS interposer. There `EAGAIN` = 11, `EINVAL` = 22 and `EPERM` = 1. The
results match the macOS table: before the fix `activate()` returned 11, 22 and
1 and the sink returned only `AVERROR_EOF`; after it, `activate()` and then the
sink returned `AVERROR_EXTERNAL` (-542398533), -22 and -1, followed by
`AVERROR_EOF`; the raw `AVERROR(EAGAIN)` flush returned -11 and was lost behind
EOF in both builds. That is 8 runs (ONNX Runtime only), exactly one injected
failure in each of the 6 injected runs, and the after-build log names the
cause. Both Linux builds had the same 10 warning lines (ONNX Runtime header
`-Wstrict-prototypes`), none in `dnn_backend_common.c`.
dnn_detect and dnn_classify. The same test was run through the other two
filters whose `activate()` returns the flush result, with a copy of the harness
that takes the filter name (`flush_harness_df.c`; the only other change is that
it prints the detection side data of the two frames drained before EOF). It
links against separate before and after build trees. Three graphs, on macOS:
- `dnn_detect` on TensorFlow with `tf_det.pb`. This is a minimal TF graph, not
a trained model: input `image_tensor` float [1,H,W,3], and four rank-4 constant
outputs `num_detections`, `detection_scores`, `detection_classes` and
`detection_boxes` giving one box (0.25, 0.25, 0.75, 0.75) with confidence 0.9.
On TensorFlow every frame's inference is started by
`ff_dnn_start_inference_async()`, so the success path of the changed function
runs on both frames. Each frame came out with one 32x32+16+16 box, confidence
9000/10000.
- `dnn_classify` on OpenVINO with ConvNeXt-Base (`convnext-base-224-22k-1k`,
the ONNX model from #24702's tests), after the TF `dnn_detect` above for the
boxes. `dnn_classify` accepts only OpenVINO. With `confidence=0`, each frame's
box got one classification label from the model.
- `dnn_detect` on ONNX Runtime with D-FINE S (`dfine_s_obj2coco`, from #24702's
tests). The frames are flat grey, so it detected nothing and attached no side
data; both frames passed through the filter before EOF.
| filter / backend | injected | before: `activate()` at EOF | before: sink |
after: `activate()` at EOF | after: sink |
|---|---|---|---|---|---|
| all three | `EAGAIN` (35) | **35** | `AVERROR_EOF` | `AVERROR_EXTERNAL` |
`AVERROR_EXTERNAL`, then `AVERROR_EOF` |
| all three | `EINVAL` (22) | **22** | `AVERROR_EOF` | -22 | -22, then
`AVERROR_EOF` |
| all three | `EPERM` (1) | **1** | `AVERROR_EOF` | -1 | -1, then `AVERROR_EOF`
|
| all three | none; the flush returns `AVERROR(EAGAIN)` | -35 | `AVERROR_EOF` |
-35 | `AVERROR_EOF` |
The three graphs gave identical rows, 24 runs in total, with exactly one
injected failure in each of the 18 injected runs, and the after-build log names
the cause in all 9 injected after runs.
Unchanged output. Without injection, `ffmpeg -f lavfi -i testsrc=s=64x64:d=0.4
-vf format=rgb24,dnn_processing=... -f md5 -` gives the same MD5
(`eadd543ed2bc6e4e16e9295f92a11f5c`), exit 0, for both builds on each of the
three backends. Those processing models are identity (the output equals the
input), so that check shows the frames and the frame count are unchanged, not
model numerics. For that, both builds were run over a whole real clip with
real, non-identity models, using the same method as #24702's tests:
`puppies.ogv` (294.6 s, Theora 1280x720), `fps=2`, decoder `-threads 1`, and
the `showinfo` output (the frame checksums plus every detection box, label and
confidence) hashed with MD5. Binaries are the verifier's separate before/after
builds (`0d0ba85092` and `52e469dcbd`, ONNX Runtime, OpenVINO and TensorFlow
all enabled). One `ffmpeg` process at a time:
| filter chain (`fps=2,` + ...) | backend / model(s) | before MD5 | after MD5 |
boxes | classify |
|---|---|---|---|---|---|
| (no DNN filter) | — | `28ff75129334b5c317a13eb165aa5d98` |
`28ff75129334b5c317a13eb165aa5d98` | 0 | 0 |
| `dnn_detect` | ONNX Runtime, D-FINE S (`dfine_s_obj2coco`) |
`9c444328a4c6bdc579ecf8d43828c549` | `9c444328a4c6bdc579ecf8d43828c549` | 4209
| 0 |
| `dnn_detect` | ONNX Runtime, RF-DETR S (`rfdetr_s`) |
`2c38e871b57295275a0c96b720d04693` | `2c38e871b57295275a0c96b720d04693` | 2500
| 0 |
| `dnn_detect,dnn_classify` | ONNX Runtime D-FINE S + OpenVINO ConvNeXt-Base |
`b5b4189fef25bfa8195627996e46ac8f` | `b5b4189fef25bfa8195627996e46ac8f` | 4209
| 229 |
| `dnn_detect` | TensorFlow, `tf_det.pb` (minimal constant-output graph, not
trained) | `3d6d4a722c56211b5f3b5942c79a0e3f` |
`3d6d4a722c56211b5f3b5942c79a0e3f` | 589 | 0 |
All 589 decoded frames in every row, exit 0 in all ten runs. Each before/after
pair is identical, and every filtered MD5 differs from both the no-filter MD5
and from each other. D-FINE S and RF-DETR are real, trained, non-identity
detectors (different architectures, different box counts on the same clip);
ConvNeXt-Base is a real, trained, non-identity classifier. `tf_det.pb` is the
same minimal constant-output graph used in the dnn_detect/dnn_classify
injection test above, included because it is the only graph that exercises
`ff_dnn_start_inference_async()` on every frame; it is not a trained model and
is not one of the two non-identity models this check relies on.
Reproducer. The scripts and raw outputs are: `run.sh`, `flush_harness.c` and
`inject_pthread_create.c` (build and the `dnn_processing` injection on macOS,
output `run-final.out`); `verify-mac.sh` (separate before and after build
trees) and `verify-linux-inner.sh` with `wrap_pthread_create.c` (the Linux run
inside the container, output `verify-linux.out`); `run-df.sh` with
`flush_harness_df.c` (the `dnn_detect` and `dnn_classify` injection, output
`run-df.out`); and `run-unchanged.sh` (the whole-clip runs, output
`run-unchanged.out`). The models are the ones from #24702's tests:
`pr24702_tf_models.py` writes `tf_det.pb`, and the D-FINE, RF-DETR and ConvNeXt
files are the pinned Hugging Face revisions fetched by #24702's evidence script.
**Limits.**
- The flush is a stand-in for Torch's code path, not LibTorch itself, and only
the thread-start result is simulated. Everything above and below it is real:
the common helper, `ff_dnn_flush()`, `dnn_processing`'s EOF handling, the graph
and the buffersink.
- The Linux run covers `dnn_processing` on ONNX Runtime in a minimal build.
TensorFlow, OpenVINO, `dnn_detect` and `dnn_classify` were run with injection
on macOS only.
- In every injection run the failing thread start comes from the stand-in
flush. `dnn_detect` does not accept Torch and `dnn_classify` accepts only
OpenVINO, which does not call `ff_dnn_start_inference_async()`, so for these
two filters the runs show that their EOF handling now returns the error. The
Torch route itself exists only in `dnn_processing`.
- `tf_det.pb` is a minimal constant-output graph, not a trained detector. The
real detectors and the classifier run on ONNX Runtime and OpenVINO, which do
not use the changed function; the whole-clip TensorFlow row is where its
success path runs on every frame.
**Not changed.** `execute_model_tf()` ignores the
`ff_dnn_start_inference_async()` result in async mode and returns 0, so a
TensorFlow frame whose thread cannot start is not reported by `execute_model`
(pre-existing). That return value is a separate issue with its own
task-completion consequences, so it is left for a separate patch.
From 26a4f676f8d936968bdeec6eb5f2d63a4a3baa06 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 1/9] avfilter/dnn: fix single-pointer arrays passed as 4-plane
sws_scale buffers
ff_proc_from_dnn_to_frame() and ff_proc_from_frame_to_dnn() take the
address of a single `void *middle_data` local (`&middle_data`) for the
DL_NCHW intermediate buffer and pass it to sws_scale() cast to a
4-element plane pointer array (`uint8_t * const*` / `const uint8_t
**`). &middle_data is the address of one pointer-sized stack slot, not
an array of four; sws_scale() indexes dst[1..3]/src[1..3] even for a
single-plane GRAY8 conversion (to determine plane strides/validity),
so this reads three pointer-sized values past the stack slot holding
middle_data.
On common ABIs the extra slots are only read, never written or
dereferenced, for these single-plane conversions, so the output is
unaffected. It is still undefined behavior: an ASan build of master
reports a stack-buffer-overflow in sws_scale() on these paths. It also
makes the surrounding buffer-safety work in these two functions harder
to reason about.
Replace the single pointer with a real, zero-initialized 4-entry
`void *middle_data_planes[4]` array, storing the allocated buffer in
slot 0 and leaving the unused plane slots explicitly NULL, matching
how frame->data (a genuine 4-element array) is used everywhere else
in this file.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_io_proc.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 0a18c56b84..08c663be04 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -46,6 +46,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
int linesize[4] = { 0 };
void **dst_data = NULL;
void *middle_data = NULL;
+ void *middle_data_planes[4] = { NULL, NULL, NULL, NULL };
uint8_t *planar_data[4] = { 0 };
int plane_size = frame->width * frame->height * sizeof(uint8_t);
enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE;
@@ -76,7 +77,8 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
ret = AVERROR(ENOMEM);
goto err;
}
- dst_data = &middle_data;
+ middle_data_planes[0] = middle_data;
+ dst_data = middle_data_planes;
linesize[0] = frame->width * 3;
}
@@ -186,6 +188,7 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData
*input, void *log_ctx)
int linesize[4] = { 0 };
void **src_data = NULL;
void *middle_data = NULL;
+ void *middle_data_planes[4] = { NULL, NULL, NULL, NULL };
uint8_t *planar_data[4] = { 0 };
int plane_size = frame->width * frame->height * sizeof(uint8_t);
enum AVPixelFormat dst_fmt = AV_PIX_FMT_NONE;
@@ -215,7 +218,8 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData
*input, void *log_ctx)
ret = AVERROR(ENOMEM);
goto err;
}
- src_data = &middle_data;
+ middle_data_planes[0] = middle_data;
+ src_data = middle_data_planes;
linesize[0] = frame->width * 3;
}
--
2.52.0
From 072f973d7e2ebea8170a56def58df4d45457b489 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 2/9] avfilter/dnn: synchronize task completion with the result
reader
A task is complete when its inference_done count reaches
inference_todo. The completion callbacks increment the count on the
inference thread: an OpenVINO async request's callback thread, or the
thread started by the common async module for TensorFlow and Torch.
ff_dnn_get_result_common() polls it from the filter thread without any
synchronization, and then hands the task's output frame, written by the
callback, to the filter. These are conflicting accesses with no
happens-before relation, i.e. a data race under the C memory model.
OpenVINO classification can also run several inferences of one task
concurrently on different requests, whose callbacks then increment the
same count concurrently.
Add ff_dnn_complete_lltask(), which counts one inference as done under
a lock, and take the same lock in ff_dnn_get_result_common() to read
the count. Everything a callback writes before completing its
inference is thereby visible to the filter thread that takes the
result. The lock is only held around these two accesses. All four
backends' completion callbacks use the helper; the synchronous checks
in their execute functions run on the thread that ran the callback and
are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 20 +++++++++++++++++++-
libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++
libavfilter/dnn/dnn_backend_onnx.c | 2 +-
libavfilter/dnn/dnn_backend_openvino.c | 2 +-
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn/dnn_backend_torch.cpp | 2 +-
6 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 82627c7a96..5a73b1d15d 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -67,6 +67,20 @@ int ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams
*exec_params, void *backe
return 0;
}
+/**
+ * Protects TaskItem.inference_done, which completion callbacks update
+ * from inference threads and ff_dnn_get_result_common() reads from the
+ * filter thread. It is only held for those accesses.
+ */
+static AVMutex task_completion_lock = AV_MUTEX_INITIALIZER;
+
+void ff_dnn_complete_lltask(TaskItem *task)
+{
+ ff_mutex_lock(&task_completion_lock);
+ task->inference_done++;
+ ff_mutex_unlock(&task_completion_lock);
+}
+
/**
* Thread routine for async execution.
* @param args pointer to DNNAsyncExecModule module
@@ -144,12 +158,16 @@ int ff_dnn_start_inference_async(void *ctx,
DNNAsyncExecModule *async_module)
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out)
{
TaskItem *task = ff_queue_peek_front(task_queue);
+ int done;
if (!task) {
return DAST_EMPTY_QUEUE;
}
- if (task->inference_done != task->inference_todo) {
+ ff_mutex_lock(&task_completion_lock);
+ done = task->inference_done == task->inference_todo;
+ ff_mutex_unlock(&task_completion_lock);
+ if (!done) {
return DAST_NOT_READY;
}
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index f0b8b47397..f364f1739a 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -139,6 +139,21 @@ void ff_dnn_wait_requests(SafeQueue *request_queue, int
nireq);
*/
int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module);
+/**
+ * Count one inference (LastLevelTaskItem) of a task as done.
+ *
+ * Completion callbacks may run on other threads than the filter (async
+ * OpenVINO requests, and the TensorFlow and Torch async modules), while
+ * ff_dnn_get_result_common() polls the count from the filter thread.
+ * Backends therefore update task->inference_done only through this
+ * function, which does so under the lock ff_dnn_get_result_common()
+ * takes to read it: everything the callback wrote before this call,
+ * such as the output frame or its side data, is visible to the thread
+ * that takes the result. The call that completes the task hands it
+ * over to that thread, which may free it at any time after.
+ */
+void ff_dnn_complete_lltask(TaskItem *task);
+
/**
* Extract input and output frame from the Task Queue after
* asynchronous inference.
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index 009387b36d..08ff70099a 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -720,7 +720,7 @@ static void infer_completion_callback(void *args)
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
err:
av_freep(&outputs);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 0df4d585e8..b5ccb938c0 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -479,7 +479,7 @@ static void infer_completion_callback(void *args)
break;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
av_freep(&request->lltasks[i]);
for (int i = 0; i < ov_model->nb_outputs; i++)
outputs[i].data = (uint8_t *)outputs[i].data +
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index f8a4d9bc62..876f95619e 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -744,7 +744,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
err:
tf_free_request(infer_request);
av_freep(&outputs);
diff --git a/libavfilter/dnn/dnn_backend_torch.cpp
b/libavfilter/dnn/dnn_backend_torch.cpp
index 1ecf95e938..afcac800ec 100644
--- a/libavfilter/dnn/dnn_backend_torch.cpp
+++ b/libavfilter/dnn/dnn_backend_torch.cpp
@@ -471,7 +471,7 @@ static void infer_completion_callback(void *args) {
avpriv_report_missing_feature(th_model->ctx, "model function type
%d", th_model->model.func_type);
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
}
err:
--
2.52.0
From 5fba7edbc44179db32911c530b6c86c58791a02f Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 3/9] avfilter/dnn: propagate post-processing failures through
async results
Every backend's default post-processing call site (ONNX, OpenVINO,
TensorFlow, Torch, in their inference-completion callbacks) calls
ff_proc_from_dnn_to_frame() -- or a filter-supplied FramePrePostProc
callback -- to convert the raw inference output into out_frame, but
discards the returned AVERROR. The task is then unconditionally marked
complete (ff_dnn_complete_lltask()), so ff_dnn_get_result_common() always
returns DAST_SUCCESS with out_frame handed back as though it had been
validly written, even when post-processing failed and out_frame is
untouched or partially written.
vf_dnn_processing.c allocates out_frame separately from in_frame
(av_frame_copy_props() only copies metadata, not pixel data), so on a
post-processing failure it forwards this unwritten buffer downstream
as a normal successful frame. This is exactly the class of bug that
motivates rejecting channel-mismatched model output in
ff_proc_from_dnn_to_frame() with AVERROR(ENOSYS), as proposed in
PR #24048: rejecting bad output at the post-proc call site only helps
if callers actually see the failure instead of a synthetic success.
Fix this generically, independent of any one post-proc failure mode:
- Add TaskItem.status (dnn_backend_common.h), initialized to 0 by
ff_dnn_fill_task(). All four backends now pass the return value of
ff_proc_from_dnn_to_frame()/frame_post_proc() to
ff_dnn_complete_lltask() instead of discarding it. That helper keeps
the first error in task->status, under the same lock as the
inference_done update, so the status is published to the filter
thread together with the completion.
- ff_dnn_get_result_common() reads task->status under that lock,
together with inference_done, and checks it before handing back
the frames: on failure it releases in_frame and out_frame (only
once when they are the same frame, as on vf_sr's pre-scale path),
sets *in/*out to NULL, and returns DAST_FAIL (already defined in
DNNAsyncStatusType, previously unused by this path) instead of
DAST_SUCCESS. The task is popped and freed either way, so a failed
post-proc can't stall async draining.
- ff_dnn_get_result_common(), the get_result vtable entry in
DNNModule, and the ff_dnn_get_result() wrapper all gain an optional
`int *status` out-parameter carrying the stored AVERROR back to the
caller when DAST_FAIL is returned; NULL is safe to pass when the
specific code isn't needed.
- vf_dnn_processing.c's activate() and flush_frame() check for
DAST_FAIL and return the stored AVERROR (or AVERROR_EXTERNAL as a
fallback) instead of silently continuing.
- vf_dnn_classify.c, vf_dnn_detect.c (DFT_ANALYTICS_* function types,
which never touch this post-proc path, so task->status stays 0) and
vf_derain.c/vf_sr.c (which already treat any async_state !=
DAST_SUCCESS as AVERROR(EINVAL)) are updated only for the new
ff_dnn_get_result() parameter and pass NULL; their existing
behavior is unchanged, and derain/sr now also correctly fail
instead of forwarding a bad frame if their post-proc callback ever
errors.
The OpenVINO and TensorFlow backends were built and run with
OpenVINO 2026.4.0 and the TensorFlow 2.18.0 C library. The Torch
backend was not built, as libtorch is not installed here; its change
is the same as in the ONNX and TensorFlow backends and was checked by
reading. The CUDA zero-copy output path in dnn_backend_torch.cpp
(fill_model_output_th_cuda) is a separate conversion path that never
calls ff_proc_from_dnn_to_frame/frame_post_proc and is left as-is.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 39 +++++++++++++++++++++-----
libavfilter/dnn/dnn_backend_common.h | 33 ++++++++++++++++++----
libavfilter/dnn/dnn_backend_onnx.c | 12 ++++----
libavfilter/dnn/dnn_backend_openvino.c | 12 ++++----
libavfilter/dnn/dnn_backend_tf.c | 11 ++++----
libavfilter/dnn/dnn_backend_torch.cpp | 15 +++++-----
libavfilter/dnn_filter_common.c | 4 +--
libavfilter/dnn_filter_common.h | 2 +-
libavfilter/dnn_interface.h | 7 +++--
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_classify.c | 4 +--
libavfilter/vf_dnn_detect.c | 4 +--
libavfilter/vf_dnn_processing.c | 18 ++++++++++--
libavfilter/vf_sr.c | 2 +-
14 files changed, 116 insertions(+), 49 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 5a73b1d15d..1c6ccb25bb 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -63,20 +63,23 @@ int ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams
*exec_params, void *backe
task->model = backend_model;
task->nb_output = exec_params->nb_output;
task->output_names = exec_params->output_names;
+ task->status = 0;
return 0;
}
/**
- * Protects TaskItem.inference_done, which completion callbacks update
- * from inference threads and ff_dnn_get_result_common() reads from the
- * filter thread. It is only held for those accesses.
+ * Protects TaskItem.inference_done and TaskItem.status, which completion
+ * callbacks update from inference threads and ff_dnn_get_result_common()
+ * reads from the filter thread. It is only held for those accesses.
*/
static AVMutex task_completion_lock = AV_MUTEX_INITIALIZER;
-void ff_dnn_complete_lltask(TaskItem *task)
+void ff_dnn_complete_lltask(TaskItem *task, int status)
{
ff_mutex_lock(&task_completion_lock);
+ if (status < 0 && task->status >= 0)
+ task->status = status;
task->inference_done++;
ff_mutex_unlock(&task_completion_lock);
}
@@ -155,10 +158,10 @@ int ff_dnn_start_inference_async(void *ctx,
DNNAsyncExecModule *async_module)
return 0;
}
-DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out)
+DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status)
{
TaskItem *task = ff_queue_peek_front(task_queue);
- int done;
+ int done, task_status;
if (!task) {
return DAST_EMPTY_QUEUE;
@@ -166,14 +169,36 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_mutex_lock(&task_completion_lock);
done = task->inference_done == task->inference_todo;
+ task_status = task->status;
ff_mutex_unlock(&task_completion_lock);
if (!done) {
return DAST_NOT_READY;
}
+ ff_queue_pop_front(task_queue);
+
+ if (task_status < 0) {
+ // Post-processing of the inference output into out_frame failed
+ // (either the default ff_proc_from_dnn_to_frame() or a custom
+ // FramePrePostProc callback). out_frame was never validly
+ // written, so it must not be handed back as a result; release
+ // both frames here since the caller has no valid pointer to
+ // free them through once DAST_FAIL is returned. vf_sr passes the
+ // same frame as input and output, so free it only once then.
+ if (status)
+ *status = task_status;
+ if (task->out_frame == task->in_frame)
+ task->out_frame = NULL;
+ av_frame_free(&task->in_frame);
+ av_frame_free(&task->out_frame);
+ *in = NULL;
+ *out = NULL;
+ av_freep(&task);
+ return DAST_FAIL;
+ }
+
*in = task->in_frame;
*out = task->out_frame;
- ff_queue_pop_front(task_queue);
av_freep(&task);
return DAST_SUCCESS;
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index f364f1739a..44e39f8bc9 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -52,6 +52,15 @@ typedef struct TaskItem {
uint32_t nb_output;
uint32_t inference_todo;
uint32_t inference_done;
+ /**
+ * Zero on success, or a negative AVERROR from frame post-processing
+ * (ff_proc_from_dnn_to_frame() or a FramePrePostProc callback).
+ * Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
+ * which keeps the first error, under the same lock as inference_done.
+ * Consumed by ff_dnn_get_result_common(), which releases failed tasks
+ * and their frames.
+ */
+ int status;
} TaskItem;
// one task might have multiple inferences
@@ -140,19 +149,24 @@ void ff_dnn_wait_requests(SafeQueue *request_queue, int
nireq);
int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module);
/**
- * Count one inference (LastLevelTaskItem) of a task as done.
+ * Count one inference (LastLevelTaskItem) of a task as done, and record
+ * its result in task->status if it is the task's first error.
*
* Completion callbacks may run on other threads than the filter (async
* OpenVINO requests, and the TensorFlow and Torch async modules), while
* ff_dnn_get_result_common() polls the count from the filter thread.
- * Backends therefore update task->inference_done only through this
- * function, which does so under the lock ff_dnn_get_result_common()
- * takes to read it: everything the callback wrote before this call,
+ * Backends therefore update task->inference_done and task->status only
+ * through this function, which does so under the lock
+ * ff_dnn_get_result_common() takes to read them: everything the callback
wrote before this call,
* such as the output frame or its side data, is visible to the thread
* that takes the result. The call that completes the task hands it
* over to that thread, which may free it at any time after.
+ *
+ * @param task the task the inference belongs to
+ * @param status 0 or another non-negative value on success, a negative
+ * AVERROR if the inference output could not be processed
*/
-void ff_dnn_complete_lltask(TaskItem *task);
+void ff_dnn_complete_lltask(TaskItem *task, int status);
/**
* Extract input and output frame from the Task Queue after
@@ -161,12 +175,19 @@ void ff_dnn_complete_lltask(TaskItem *task);
* @param task_queue pointer to the task queue of the backend
* @param in double pointer to the input frame
* @param out double pointer to the output frame
+ * @param status optional pointer to receive the negative AVERROR
+ * stored on the task when post-processing failed.
+ * Only written when DAST_FAIL is returned; may be NULL
+ * if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
+ * @retval DAST_FAIL if inference completed but post-processing of the
+ * output into out_frame failed; *in and *out are
+ * set to NULL and both frames are released.
* @retval DAST_SUCCESS if result successfully extracted
*/
-DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out);
+DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
/**
* Allocate input and output frames and fill the Task
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index 08ff70099a..fcbba7cf7b 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -588,7 +588,7 @@ static void infer_completion_callback(void *args)
size_t num_dims;
int64_t *dims;
OrtStatus *status;
- int ret;
+ int ret = 0;
outputs = av_calloc(infer_request->nb_outputs, sizeof(*outputs));
if (!outputs) {
@@ -699,9 +699,9 @@ static void infer_completion_callback(void *args)
if (task->do_ioproc) {
outputs[0].scale = 255;
if (onnx_model->model.frame_post_proc != NULL) {
- onnx_model->model.frame_post_proc(task->out_frame, outputs,
onnx_model->model.filter_ctx);
+ ret = onnx_model->model.frame_post_proc(task->out_frame,
outputs, onnx_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
}
} else {
task->out_frame->width =
outputs[0].dims[dnn_get_width_idx_by_layout(outputs[0].layout)];
@@ -720,7 +720,7 @@ static void infer_completion_callback(void *args)
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
err:
av_freep(&outputs);
@@ -1189,10 +1189,10 @@ static int dnn_execute_model_onnx(const DNNModel
*model, DNNExecBaseParams *exec
return execute_model_onnx(request, onnx_model->lltask_queue);
}
-static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
ONNXModel *onnx_model = (ONNXModel *)model;
- return ff_dnn_get_result_common(onnx_model->task_queue, in, out);
+ return ff_dnn_get_result_common(onnx_model->task_queue, in, out, status);
}
static int dnn_flush_onnx(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index b5ccb938c0..57ef4e18dd 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -338,6 +338,7 @@ static void infer_completion_callback(void *args)
OVModel *ov_model = task->model;
SafeQueue *requestq = ov_model->request_queue;
DNNData *outputs;
+ int ret;
DnnContext *ctx = ov_model->ctx;
#if HAVE_OPENVINO2
size_t* dims;
@@ -439,14 +440,15 @@ static void infer_completion_callback(void *args)
av_assert0(request->lltask_count >= 1);
for (int i = 0; i < request->lltask_count; ++i) {
task = request->lltasks[i]->task;
+ ret = 0;
switch (ov_model->model.func_type) {
case DFT_PROCESS_FRAME:
if (task->do_ioproc) {
if (ov_model->model.frame_post_proc != NULL) {
- ov_model->model.frame_post_proc(task->out_frame, outputs,
ov_model->model.filter_ctx);
+ ret = ov_model->model.frame_post_proc(task->out_frame,
outputs, ov_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs,
ctx);
}
} else {
task->out_frame->width =
@@ -479,7 +481,7 @@ static void infer_completion_callback(void *args)
break;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
av_freep(&request->lltasks[i]);
for (int i = 0; i < ov_model->nb_outputs; i++)
outputs[i].data = (uint8_t *)outputs[i].data +
@@ -1558,10 +1560,10 @@ static int dnn_execute_model_ov(const DNNModel *model,
DNNExecBaseParams *exec_p
}
}
-static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
OVModel *ov_model = (OVModel *)model;
- return ff_dnn_get_result_common(ov_model->task_queue, in, out);
+ return ff_dnn_get_result_common(ov_model->task_queue, in, out, status);
}
static int dnn_flush_ov(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 876f95619e..dc895f554a 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -696,6 +696,7 @@ static void infer_completion_callback(void *args) {
LastLevelTaskItem *lltask = request->lltask;
TaskItem *task = lltask->task;
DNNData *outputs;
+ int ret = 0;
TFInferRequest *infer_request = request->infer_request;
TFModel *tf_model = task->model;
DnnContext *ctx = tf_model->ctx;
@@ -722,9 +723,9 @@ static void infer_completion_callback(void *args) {
//it only support 1 output if it's frame in & frame out
if (task->do_ioproc) {
if (tf_model->model.frame_post_proc != NULL) {
- tf_model->model.frame_post_proc(task->out_frame, outputs,
tf_model->model.filter_ctx);
+ ret = tf_model->model.frame_post_proc(task->out_frame,
outputs, tf_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
}
} else {
task->out_frame->width =
@@ -744,7 +745,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
err:
tf_free_request(infer_request);
av_freep(&outputs);
@@ -847,10 +848,10 @@ static int dnn_execute_model_tf(const DNNModel *model,
DNNExecBaseParams *exec_p
return execute_model_tf(request, tf_model->lltask_queue);
}
-static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
TFModel *tf_model = (TFModel *)model;
- return ff_dnn_get_result_common(tf_model->task_queue, in, out);
+ return ff_dnn_get_result_common(tf_model->task_queue, in, out, status);
}
static int dnn_flush_tf(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_torch.cpp
b/libavfilter/dnn/dnn_backend_torch.cpp
index afcac800ec..6f00673aeb 100644
--- a/libavfilter/dnn/dnn_backend_torch.cpp
+++ b/libavfilter/dnn/dnn_backend_torch.cpp
@@ -421,6 +421,7 @@ static void infer_completion_callback(void *args) {
for (uint32_t i = 0; i < request->lltask_count; i++) {
lltask = request->lltasks[i];
TaskItem *task = lltask->task;
+ int ret = 0;
torch::Tensor out_slice = slices[i];
c10::IntArrayRef sizes = out_slice.sizes();
@@ -453,11 +454,11 @@ static void infer_completion_callback(void *args) {
outputs.scale = 255;
outputs.data = out_slice.data_ptr();
if (th_model->model.frame_post_proc != NULL) {
- th_model->model.frame_post_proc(task->out_frame,
&outputs,
-
th_model->model.filter_ctx);
+ ret = th_model->model.frame_post_proc(task->out_frame,
&outputs,
+
th_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, &outputs,
- th_model->ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame,
&outputs,
+ th_model->ctx);
}
#if CONFIG_CUDA
}
@@ -471,7 +472,7 @@ static void infer_completion_callback(void *args) {
avpriv_report_missing_feature(th_model->ctx, "model function type
%d", th_model->model.func_type);
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
}
err:
@@ -752,10 +753,10 @@ static int dnn_execute_model_th(const DNNModel *model,
DNNExecBaseParams *exec_p
return 0;
}
-static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
THModel *th_model = (THModel *)model;
- return ff_dnn_get_result_common(th_model->task_queue, in, out);
+ return ff_dnn_get_result_common(th_model->task_queue, in, out, status);
}
static int dnn_flush_th(const DNNModel *model)
diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c
index b644a6280f..b123a4c608 100644
--- a/libavfilter/dnn_filter_common.c
+++ b/libavfilter/dnn_filter_common.c
@@ -216,9 +216,9 @@ int ff_dnn_execute_model_classification(DnnContext *ctx,
AVFrame *in_frame, AVFr
return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base);
}
-DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame)
+DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame, int *status)
{
- return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame);
+ return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame,
status);
}
int ff_dnn_flush(DnnContext *ctx)
diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h
index 1e89ab42ae..596cab55a4 100644
--- a/libavfilter/dnn_filter_common.h
+++ b/libavfilter/dnn_filter_common.h
@@ -59,7 +59,7 @@ int ff_dnn_get_input(DnnContext *ctx, DNNData *input);
int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int
*output_width, int *output_height);
int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame
*out_frame);
int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame,
AVFrame *out_frame, const char *target);
-DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame);
+DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame, int *status);
int ff_dnn_flush(DnnContext *ctx);
void ff_dnn_uninit(DnnContext *ctx);
#if CONFIG_CUDA
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 632ae6be84..810d3c4bdf 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -191,8 +191,11 @@ struct DNNModule {
DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type,
AVFilterContext *filter_ctx);
// Executes model with specified input and output. Returns the error code
otherwise.
int (*execute_model)(const DNNModel *model, DNNExecBaseParams
*exec_params);
- // Retrieve inference result.
- DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out);
+ // Retrieve inference result. status, if non-NULL, receives the
+ // negative AVERROR stored on the task when DAST_FAIL is returned
+ // (i.e. inference completed but post-processing the output into
+ // *out failed); *in and *out are set to NULL in that case.
+ DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
// Frees memory allocated for model.
diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c
index 690a797bdb..c4910d9a1e 100644
--- a/libavfilter/vf_derain.c
+++ b/libavfilter/vf_derain.c
@@ -75,7 +75,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
return dnn_result;
}
do {
- async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out);
+ async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL);
} while (async_state == DAST_NOT_READY);
if (async_state != DAST_SUCCESS)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index f92c41ab76..5751e7c4bc 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -216,7 +216,7 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -259,7 +259,7 @@ static int dnn_classify_activate(AVFilterContext
*filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 99dca8546c..846e4c11fc 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -882,7 +882,7 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -925,7 +925,7 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 16904a2c20..5f88734e44 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -276,7 +276,14 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts,
int64_t *out_pts)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (out_frame) {
if (isPlanarYUV(in_frame->format))
copy_uv_planes(ctx, out_frame, in_frame);
@@ -328,7 +335,14 @@ static int activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (out_frame) {
if (isPlanarYUV(in_frame->format))
copy_uv_planes(ctx, out_frame, in_frame);
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index 8f1ccee56a..7eac5ca18d 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -142,7 +142,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
}
do {
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL);
} while (async_state == DAST_NOT_READY);
if (async_state != DAST_SUCCESS)
--
2.52.0
From a229e63a830d2cfaf103b4e3575bfa690400c7b8 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 4/9] avfilter/vf_derain, vf_sr: return the post-processing
error
ff_dnn_get_result() now reports DAST_FAIL together with the error
returned by the frame post-processing step, but vf_derain and vf_sr
still passed NULL for the status and collapsed every non-success
result into AVERROR(EINVAL). A failed conversion of the model output
(for example, with PR #24048 applied, an output shape that does not
match the output frame) was therefore reported as a generic
invalid-argument error with no hint of what went wrong.
Pass a status pointer, log the failure and return the real error, as
vf_dnn_processing already does. Fall back to AVERROR_EXTERNAL if the
backend did not store a negative status. On DAST_FAIL the frames
passed to ff_dnn_execute_model() have already been released by
ff_dnn_get_result(). vf_sr's pre-scale path passes its scaled output
frame as both input and output, so its original input frame is not
released there; that leak happens on success too and predates this
change.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/vf_derain.c | 9 ++++++++-
libavfilter/vf_sr.c | 9 ++++++++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c
index c4910d9a1e..4242e2f518 100644
--- a/libavfilter/vf_derain.c
+++ b/libavfilter/vf_derain.c
@@ -54,6 +54,7 @@ AVFILTER_DNN_DEFINE_CLASS(derain, DNN_TF);
static int filter_frame(AVFilterLink *inlink, AVFrame *in)
{
DNNAsyncStatusType async_state = 0;
+ int proc_status = 0;
AVFilterContext *ctx = inlink->dst;
AVFilterLink *outlink = ctx->outputs[0];
DRContext *dr_context = ctx->priv;
@@ -75,9 +76,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
return dnn_result;
}
do {
- async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL);
+ async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out,
&proc_status);
} while (async_state == DAST_NOT_READY);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state != DAST_SUCCESS)
return AVERROR(EINVAL);
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index 7eac5ca18d..13d3f22919 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -112,6 +112,7 @@ static int config_output(AVFilterLink *outlink)
static int filter_frame(AVFilterLink *inlink, AVFrame *in)
{
DNNAsyncStatusType async_state = 0;
+ int proc_status = 0;
AVFilterContext *context = inlink->dst;
SRContext *ctx = context->priv;
AVFilterLink *outlink = context->outputs[0];
@@ -142,9 +143,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
}
do {
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, &proc_status);
} while (async_state == DAST_NOT_READY);
+ if (async_state == DAST_FAIL) {
+ av_log(context, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state != DAST_SUCCESS)
return AVERROR(EINVAL);
--
2.52.0
From c55405dd38da9513aafdb6d32d837bbb70639714 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 5/9] avfilter/dnn: return AVERROR codes from analytics
post-processing
Several of the detect_post_proc() and classify_post_proc() callbacks of
dnn_detect and dnn_classify return a bare -1 on failure, which is
AVERROR(EPERM). The backends do not report these values to the filter
yet: OpenVINO and TensorFlow ignore them, and ONNX turns a failed
detect_post_proc() into DNN_GENERIC_ERROR. The next commit makes them
the error of the filter, and of ffmpeg, where -1 would read as
"Operation not permitted".
Return AVERROR_INVALIDDATA, and log it, for a classifier output without
any scores, which was not logged at all. Return AVERROR(EINVAL) for
missing or already present detection side data, and AVERROR(ENOMEM)
when the side data cannot be allocated.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/vf_dnn_classify.c | 5 +++--
libavfilter/vf_dnn_detect.c | 10 +++++-----
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index 5751e7c4bc..ab79030165 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -69,13 +69,14 @@ static int dnn_classify_post_proc(AVFrame *frame, DNNData
*output, uint32_t bbox
AVFrameSideData *sd;
int output_size = output->dims[3] * output->dims[2] * output->dims[1];
if (output_size <= 0) {
- return -1;
+ av_log(filter_ctx, AV_LOG_ERROR, "Model output has no class
scores.\n");
+ return AVERROR_INVALIDDATA;
}
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (!sd) {
av_log(filter_ctx, AV_LOG_ERROR, "Cannot get side data in
dnn_classify_post_proc\n");
- return -1;
+ return AVERROR(EINVAL);
}
header = (AVDetectionBBoxHeader *)sd->data;
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 846e4c11fc..2382562daf 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -392,7 +392,7 @@ static int dnn_detect_fill_side_data(AVFrame *frame,
AVFilterContext *filter_ctx
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
@@ -551,7 +551,7 @@ static int dnn_detect_post_proc_ssd(AVFrame *frame, DNNData
*output, int nb_outp
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
@@ -618,7 +618,7 @@ static int dnn_detect_post_proc_anchored(AVFrame *frame,
DNNData *output, int nb
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (sd) {
av_log(filter_ctx, AV_LOG_ERROR, "already have bounding boxes in side
data.\n");
- return -1;
+ return AVERROR(EINVAL);
}
switch (ctx->model_type) {
@@ -661,7 +661,7 @@ static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData
*output, AVFilterCont
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (sd) {
av_log(filter_ctx, AV_LOG_ERROR, "already have dnn bounding boxes in
side data.\n");
- return -1;
+ return AVERROR(EINVAL);
}
for (int i = 0; i < proposal_count; ++i) {
@@ -678,7 +678,7 @@ static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData
*output, AVFilterCont
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
--
2.52.0
From 9e31d945af4333ea729afb563366110b973c20a6 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 6/9] avfilter/dnn: complete analytics tasks whose
post-processing failed
When detect_post_proc() failed, the ONNX backend jumped past the
completion of the task, so it could never satisfy
inference_done == inference_todo. The ONNX backend only runs
requests synchronously, so execute_model_onnx() then returned
DNN_GENERIC_ERROR and dnn_detect failed with a bare AVERROR(EIO)
("Input/output error") instead of the post-processing error. The
OpenVINO and TensorFlow backends ignored the return values of
detect_post_proc() and classify_post_proc() altogether, so failures
there went unnoticed.
Pass the post-processing result to ff_dnn_complete_lltask(), which
keeps the first failure in task->status and still counts the lltask
as done, so the task completes and the result path reports DAST_FAIL
with the error. An OpenVINO classification task can be split over
several requests whose callbacks run concurrently; the helper updates
the status under its lock, so they do not race on it. dnn_detect and
dnn_classify now pass a status to ff_dnn_get_result() and return that
error, like dnn_processing does, instead of silently forwarding the
frame.
The Torch backend has no analytics post-processing and is unchanged.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 7 +++----
libavfilter/dnn/dnn_backend_common.h | 13 +++++++------
libavfilter/dnn/dnn_backend_onnx.c | 2 --
libavfilter/dnn/dnn_backend_openvino.c | 17 ++++++++++-------
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn_interface.h | 5 +++--
libavfilter/vf_dnn_classify.c | 18 ++++++++++++++++--
libavfilter/vf_dnn_detect.c | 18 ++++++++++++++++--
8 files changed, 56 insertions(+), 26 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 1c6ccb25bb..077e0211b8 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -178,10 +178,9 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_queue_pop_front(task_queue);
if (task_status < 0) {
- // Post-processing of the inference output into out_frame failed
- // (either the default ff_proc_from_dnn_to_frame() or a custom
- // FramePrePostProc callback). out_frame was never validly
- // written, so it must not be handed back as a result; release
+ // Frame/detect/classify post-processing failed, so out_frame was
+ // not validly written, or in_frame lacks the analytics side data
+ // it should carry. Neither is handed back as a result; release
// both frames here since the caller has no valid pointer to
// free them through once DAST_FAIL is returned. vf_sr passes the
// same frame as input and output, so free it only once then.
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index 44e39f8bc9..6fff571009 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -53,8 +53,8 @@ typedef struct TaskItem {
uint32_t inference_todo;
uint32_t inference_done;
/**
- * Zero on success, or a negative AVERROR from frame post-processing
- * (ff_proc_from_dnn_to_frame() or a FramePrePostProc callback).
+ * Zero on success, or a negative AVERROR from frame/detect/classify
+ * post-processing.
* Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
* which keeps the first error, under the same lock as inference_done.
* Consumed by ff_dnn_get_result_common(), which releases failed tasks
@@ -176,15 +176,16 @@ void ff_dnn_complete_lltask(TaskItem *task, int status);
* @param in double pointer to the input frame
* @param out double pointer to the output frame
* @param status optional pointer to receive the negative AVERROR
- * stored on the task when post-processing failed.
+ * stored on the task when frame/detect/classify
+ * post-processing failed.
* Only written when DAST_FAIL is returned; may be NULL
* if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
- * @retval DAST_FAIL if inference completed but post-processing of the
- * output into out_frame failed; *in and *out are
- * set to NULL and both frames are released.
+ * @retval DAST_FAIL if inference completed but frame/detect/classify
+ * post-processing failed; the task and both frames
+ * are released, and *in and *out are set to NULL.
* @retval DAST_SUCCESS if result successfully extracted
*/
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index fcbba7cf7b..ffa150034f 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -712,8 +712,6 @@ static void infer_completion_callback(void *args)
ret = onnx_model->model.detect_post_proc(task->in_frame, outputs,
infer_request->nb_outputs,
onnx_model->model.filter_ctx);
- if (ret < 0)
- goto err;
break;
default:
avpriv_report_missing_feature(ctx, "model function type %d",
onnx_model->model.func_type);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 57ef4e18dd..200096862e 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -462,19 +462,22 @@ static void infer_completion_callback(void *args)
av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post
proc\n");
goto end;
}
- ov_model->model.detect_post_proc(task->in_frame, outputs,
- ov_model->nb_outputs,
- ov_model->model.filter_ctx);
+ ret = ov_model->model.detect_post_proc(task->in_frame, outputs,
+ ov_model->nb_outputs,
+
ov_model->model.filter_ctx);
break;
case DFT_ANALYTICS_CLASSIFY:
if (!ov_model->model.classify_post_proc) {
av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide
post proc\n");
goto end;
}
- for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++)
- ov_model->model.classify_post_proc(task->in_frame, outputs,
-
request->lltasks[i]->bbox_index,
-
ov_model->model.filter_ctx);
+ for (int output_i = 0; output_i < ov_model->nb_outputs;
output_i++) {
+ int err = ov_model->model.classify_post_proc(task->in_frame,
outputs,
+
request->lltasks[i]->bbox_index,
+
ov_model->model.filter_ctx);
+ if (err < 0 && ret >= 0)
+ ret = err;
+ }
break;
default:
av_assert0(!"should not reach here");
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index dc895f554a..ac87884a46 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -739,7 +739,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post
proc\n");
return;
}
- tf_model->model.detect_post_proc(task->in_frame, outputs,
task->nb_output, tf_model->model.filter_ctx);
+ ret = tf_model->model.detect_post_proc(task->in_frame, outputs,
task->nb_output, tf_model->model.filter_ctx);
break;
default:
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 810d3c4bdf..6f1ad3b565 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -193,8 +193,9 @@ struct DNNModule {
int (*execute_model)(const DNNModel *model, DNNExecBaseParams
*exec_params);
// Retrieve inference result. status, if non-NULL, receives the
// negative AVERROR stored on the task when DAST_FAIL is returned
- // (i.e. inference completed but post-processing the output into
- // *out failed); *in and *out are set to NULL in that case.
+ // (i.e. inference completed but frame/detect/classify
+ // post-processing failed); the frames are then released and *in
+ // and *out are set to NULL.
DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index ab79030165..715426fb48 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -217,7 +217,14 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -260,7 +267,14 @@ static int dnn_classify_activate(AVFilterContext
*filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 2382562daf..68cc1e10ad 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -882,7 +882,14 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -925,7 +932,14 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
--
2.52.0
From f2a854508d07275a662e3c553ca8319a628ed65c Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 7/9] avfilter/dnn: return the ff_dnn_flush() error at EOF
dnn_processing, dnn_detect and dnn_classify return -1, i.e.
AVERROR(EPERM), when ff_dnn_flush() fails at EOF, which replaces the
backend's error with "Operation not permitted". Return the error
ff_dnn_flush() reported instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/vf_dnn_classify.c | 2 +-
libavfilter/vf_dnn_detect.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index 715426fb48..6f5103c47f 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -211,7 +211,7 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 68cc1e10ad..2d5c70b8fb 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -876,7 +876,7 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 5f88734e44..fec85ace79 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -270,7 +270,7 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts,
int64_t *out_pts)
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
--
2.52.0
From 0d0ba85092020375c5f6b7f38745dc8188372e2c Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:41 -0400
Subject: [PATCH 8/9] avfilter/dnn_backend_openvino: complete tasks when
reading the output fails
infer_completion_callback() jumps to its cleanup code when it cannot get
an output tensor, its data, shape or element type, or when the filter
has no post-processing callback. That skipped task->inference_done++
for every lltask of the request and leaked them. In async mode the task
then never completed: ff_dnn_get_result() returned DAST_NOT_READY
forever, and the flush loops of dnn_detect and dnn_classify spun at EOF.
A failed allocation of the outputs array also returned without giving
the request back to the request queue.
A model with a constant zero-element output reaches this, e.g. an SSD
detector without proposals ([1,1,0,7]) or a classifier without classes:
the OpenVINO ONNX frontend reads such an output as a rank-0 tensor, and
ov_tensor_get_shape() fails with INVALID_C_PARAM because an ov_shape_t
cannot have rank 0. Outputs that are only empty at run time are read
fine.
On failure, complete the remaining lltasks with the error through
ff_dnn_complete_lltask(), so that get_result reports DAST_FAIL and the
filter returns the error. Initialize output_tensor, which was freed while
uninitialized when getting the first output tensor failed. Replace the
assertion on the first output dimension with an error, since a rank-2
detection output with zero rows made ffmpeg abort.
get_output_ov() runs a synchronous inference on a task of its own to
query the output dimensions, and execute_model_ov() returns success
once all lltasks are counted as done, now also when they were completed
with an error. Check the task's status there as well, so that a failed
output read is returned as the error instead of 0x0 output dimensions,
and the caller's dimensions are left unchanged.
The completion of the remaining lltasks at the end label and the errors
set for a missing post-processing callback also apply to the OpenVINO
1.x API path. Its early returns when getting the output blob, its
buffer, dims or precision fails are left unchanged, since that path
cannot be built against current OpenVINO releases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 13 ++++----
libavfilter/dnn/dnn_backend_common.h | 9 +++---
libavfilter/dnn/dnn_backend_openvino.c | 42 +++++++++++++++++++++-----
libavfilter/dnn_interface.h | 4 +--
4 files changed, 49 insertions(+), 19 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 077e0211b8..193099d31f 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -178,12 +178,13 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_queue_pop_front(task_queue);
if (task_status < 0) {
- // Frame/detect/classify post-processing failed, so out_frame was
- // not validly written, or in_frame lacks the analytics side data
- // it should carry. Neither is handed back as a result; release
- // both frames here since the caller has no valid pointer to
- // free them through once DAST_FAIL is returned. vf_sr passes the
- // same frame as input and output, so free it only once then.
+ // Frame/detect/classify post-processing or reading the inference
+ // output failed, so out_frame was not validly written, or
+ // in_frame lacks the analytics side data it should carry.
+ // Neither is handed back as a result; release both frames here
+ // since the caller has no valid pointer to free them through
+ // once DAST_FAIL is returned. vf_sr passes the same frame as
+ // input and output, so free it only once then.
if (status)
*status = task_status;
if (task->out_frame == task->in_frame)
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index 6fff571009..f582ccb59d 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -54,7 +54,7 @@ typedef struct TaskItem {
uint32_t inference_done;
/**
* Zero on success, or a negative AVERROR from frame/detect/classify
- * post-processing.
+ * post-processing or reading the inference output.
* Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
* which keeps the first error, under the same lock as inference_done.
* Consumed by ff_dnn_get_result_common(), which releases failed tasks
@@ -177,15 +177,16 @@ void ff_dnn_complete_lltask(TaskItem *task, int status);
* @param out double pointer to the output frame
* @param status optional pointer to receive the negative AVERROR
* stored on the task when frame/detect/classify
- * post-processing failed.
+ * post-processing or reading the inference output failed.
* Only written when DAST_FAIL is returned; may be NULL
* if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
* @retval DAST_FAIL if inference completed but frame/detect/classify
- * post-processing failed; the task and both frames
- * are released, and *in and *out are set to NULL.
+ * post-processing or reading the inference output
+ * failed; the task and both frames are released,
+ * and *in and *out are set to NULL.
* @retval DAST_SUCCESS if result successfully extracted
*/
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 200096862e..e3d7be64b5 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -338,19 +338,20 @@ static void infer_completion_callback(void *args)
OVModel *ov_model = task->model;
SafeQueue *requestq = ov_model->request_queue;
DNNData *outputs;
- int ret;
+ int ret = 0;
DnnContext *ctx = ov_model->ctx;
#if HAVE_OPENVINO2
size_t* dims;
ov_status_e status;
- ov_tensor_t *output_tensor;
+ ov_tensor_t *output_tensor = NULL;
ov_shape_t output_shape = {0};
ov_element_type_e precision;
outputs = av_calloc(ov_model->nb_outputs, sizeof(*outputs));
if (!outputs) {
- av_log(ctx, AV_LOG_ERROR, "Failed to alloc outputs.");
- return;
+ av_log(ctx, AV_LOG_ERROR, "Failed to alloc outputs.\n");
+ ret = AVERROR(ENOMEM);
+ goto end;
}
for (int i = 0; i < ov_model->nb_outputs; i++) {
@@ -359,20 +360,23 @@ static void infer_completion_callback(void *args)
&output_tensor);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR,
- "Failed to get output tensor.");
+ "Failed to get output tensor.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
status = ov_tensor_data(output_tensor, &outputs[i].data);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR,
- "Failed to get output data.");
+ "Failed to get output data.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
status = ov_tensor_get_shape(output_tensor, &output_shape);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to get output port shape.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
dims = output_shape.dims;
@@ -380,6 +384,7 @@ static void infer_completion_callback(void *args)
status = ov_port_get_element_type(ov_model->output_ports[i],
&precision);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to get output port data
type.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
outputs[i].dt = precision_to_datatype(precision);
@@ -388,7 +393,13 @@ static void infer_completion_callback(void *args)
outputs[i].dims[1] = output_shape.rank > 2 ? dims[output_shape.rank -
3] : 1;
outputs[i].dims[2] = output_shape.rank > 1 ? dims[output_shape.rank -
2] : 1;
outputs[i].dims[3] = output_shape.rank > 0 ? dims[output_shape.rank -
1] : 1;
- av_assert0(request->lltask_count <= dims[0]);
+ if (dims[0] < request->lltask_count) {
+ av_log(ctx, AV_LOG_ERROR,
+ "Output %d has %zu entries in its first dimension, fewer
than the batch of %u.\n",
+ i, dims[0], request->lltask_count);
+ ret = AVERROR(EINVAL);
+ goto end;
+ }
outputs[i].layout = ctx->ov_option.layout;
outputs[i].scale = ctx->ov_option.scale;
outputs[i].mean = ctx->ov_option.mean;
@@ -460,6 +471,7 @@ static void infer_completion_callback(void *args)
case DFT_ANALYTICS_DETECT:
if (!ov_model->model.detect_post_proc) {
av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post
proc\n");
+ ret = AVERROR(EINVAL);
goto end;
}
ret = ov_model->model.detect_post_proc(task->in_frame, outputs,
@@ -469,6 +481,7 @@ static void infer_completion_callback(void *args)
case DFT_ANALYTICS_CLASSIFY:
if (!ov_model->model.classify_post_proc) {
av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide
post proc\n");
+ ret = AVERROR(EINVAL);
goto end;
}
for (int output_i = 0; output_i < ov_model->nb_outputs;
output_i++) {
@@ -492,6 +505,15 @@ static void infer_completion_callback(void *args)
get_datatype_size(outputs[i].dt);
}
end:
+ /* On failure, finish the lltasks that were not processed with the
+ * error, so that their tasks still complete and get_result reports
+ * DAST_FAIL instead of DAST_NOT_READY forever. */
+ for (int i = 0; i < request->lltask_count; i++) {
+ if (!request->lltasks[i])
+ continue;
+ ff_dnn_complete_lltask(request->lltasks[i]->task, ret < 0 ? ret :
AVERROR_BUG);
+ av_freep(&request->lltasks[i]);
+ }
#if HAVE_OPENVINO2
av_freep(&outputs);
ov_shape_free(&output_shape);
@@ -1364,6 +1386,12 @@ static int get_output_ov(DNNModel *model, const char
*input_name, int input_widt
}
ret = execute_model_ov(request, ov_model->lltask_queue);
+ if (ret != 0)
+ goto err;
+ if (task.status < 0) {
+ ret = task.status;
+ goto err;
+ }
*output_width = task.out_frame->width;
*output_height = task.out_frame->height;
err:
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 6f1ad3b565..da5689bc05 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -194,8 +194,8 @@ struct DNNModule {
// Retrieve inference result. status, if non-NULL, receives the
// negative AVERROR stored on the task when DAST_FAIL is returned
// (i.e. inference completed but frame/detect/classify
- // post-processing failed); the frames are then released and *in
- // and *out are set to NULL.
+ // post-processing or reading the inference output failed); the
+ // frames are then released and *in and *out are set to NULL.
DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
--
2.52.0
From 52e469dcbd68f99276e570620b69d59498a506df Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 18:04:22 -0400
Subject: [PATCH 9/9] avfilter/dnn: return AVERROR codes when an async thread
cannot start
ff_dnn_start_inference_async() returned the error number of a failed
pthread_create() as is. That is a positive value such as EAGAIN, and
since "return the ff_dnn_flush() error at EOF" dnn_processing,
dnn_detect and dnn_classify return it from activate() after setting
EOF on their output. A positive return is not treated as an error by
the graph or the buffersink, so an inference that could not be started
while flushing, e.g. the partial final batch of the Torch backend with
batch_size > 1, ended as a normal EOF.
Return AVERROR(ret) instead, and log the cause. EAGAIN is returned as
AVERROR_EXTERNAL: AVERROR(EAGAIN) means "no output yet, try again" to
the filter graph and to buffersink users, while the flush has already
given up the batch and set EOF.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 193099d31f..375b3a9eba 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -145,8 +145,9 @@ int ff_dnn_start_inference_async(void *ctx,
DNNAsyncExecModule *async_module)
}
ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine,
async_module);
if (ret != 0) {
- av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n");
- return ret;
+ av_log(ctx, AV_LOG_ERROR, "Unable to start async inference: %s\n",
+ av_err2str(AVERROR(ret)));
+ return ret == EAGAIN ? AVERROR_EXTERNAL : AVERROR(ret);
}
#else
ret = async_module->start_inference(async_module->args);
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
