PR #24761 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24761
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24761.patch
**Depends on #24702** (v3, head 0d0ba85092) and applies on top of it. It uses
`ff_dnn_complete_lltask(task, status)`, `TaskItem.status` and the `DAST_FAIL`
result from that PR, including its vf_sr fix for in == out.
v2: rebased onto #24702 v3. Both callbacks now complete the task only through
`ff_dnn_complete_lltask()`, which takes the completion lock, instead of writing
`task->status` and `task->inference_done` directly. The detect post-processing
status handling that v1 changed is dropped here, because v3 already passes that
result to the helper. `get_output_{tf,onnx}()` now leave the caller's
dimensions unchanged on failure, as `get_output_ov()` does in v3.
## What was wrong
When `infer_completion_callback()` in the TensorFlow and ONNX backends hits an
error, it jumps past the completion of the task. The task never completes, so
`ff_dnn_get_result()` keeps returning `DAST_NOT_READY` for it. The OpenVINO
backend had the same bug; #24702 fixes it there.
- **TensorFlow:** the callback fails when it cannot allocate the outputs array
or the model function type is unsupported. When a detect model has no
post-processing callback, it returns without handing the request back.
- Async mode is the default. There, dnn_processing, derain, sr and dnn_detect
hang.
- In sync mode the filter fails with `Input/output error`, or with `Error
number -558779972 occurred`, which gives no cause. derain and sr then free
frames the unfinished task still holds, and `dnn_free_model_tf()` frees them
again. ASan reports a heap-use-after-free there.
- **ONNX Runtime:** the callback fails on an output that is not float or has a
rank other than 3 or 4. A failed ORT call, a failed allocation or an
unsupported function type do the same. A detector with a uint8 output, or with
rank-2 `[N,7]` rows, triggers this with a model alone. The backend is sync
only, so it does not hang. The filter fails with `Input/output error` or
`-558779972` instead of the cause.
## The fix
Every exit of the callback now passes its error to `ff_dnn_complete_lltask()`.
The helper keeps the first error in `task->status` and counts the lltask as
done, under the lock that `ff_dnn_get_result_common()` takes. Post-processing
failures are already handled this way. `get_result` then returns `DAST_FAIL`
with the error and frees the frames once. Error codes:
- `ENOMEM` for allocation failures;
- `EINVAL` when there is no detect post-processing callback;
- `ENOSYS` for an unsupported output or function type;
- `AVERROR_EXTERNAL` for a failed ORT call.
`execute_model_*()` now succeeds for such a task, so `get_output_{tf,onnx}()`
check `task.status` and return it, as `get_output_ov()` does. A failed
output-size probe therefore stays a configuration error, and the caller's
dimensions are left unchanged.
## Testing
`cb-inference-done-evidence-v2.sh` regenerates every number below. It runs on
macOS 27 arm64 (M1 Ultra) with TensorFlow 2.18.0 (C library), ONNX Runtime
1.30.0 and OpenVINO 2026.4.0. The release builds enable all three backends. The
ASan builds leave out OpenVINO, because its libtbb crashes at exit under ASan.
The builds compared are:
- **master**: upstream 6246f7a4cd, which #24702 v3 is based on;
- **base**: master plus #24702 v3, 0d0ba85092;
- **+tf**: base plus the TensorFlow commit, ec7e3fd87c;
- **fix**: the whole branch, 4597d09c98.
The TensorFlow callback branches cannot be reached with a model alone, so a
test-only patch (`cb_inference_done_inject.py`, not part of this PR) forces one
branch per run with `DNN_FAULT=<site>`. It only fires on filter frames, or with
`DNN_FAULT_GETOUTPUT` only on the output-size probe. The same patch is applied
to every build being compared. The ONNX `u8`, `rank2` and `rows0dyn` rows use
unmodified builds and a model alone.
The input is 5 frames of lavfi `testsrc`/`color` at 64x64. A run counts as hung
after 30 s (exit 142).
**Exit status per build**
| case | trigger | master | base | +tf | fix |
|---|---|---|---|---|---|
| tf dnn_processing, async | calloc / functype | hang / hang | hang / hang |
ENOMEM / ENOSYS | ENOMEM / ENOSYS |
| tf dnn_processing, sync | calloc / functype | EIO / EIO | EIO / EIO | ENOMEM
/ ENOSYS | ENOMEM / ENOSYS |
| tf derain, async | calloc | hang | hang | ENOMEM | ENOMEM |
| tf derain, sync | calloc | crash or 188 ¹ | crash or 188 ¹ | ENOMEM | ENOMEM |
| tf sr (pre-scale, in == out), async | calloc | hang | hang | ENOMEM | ENOMEM |
| tf sr, sync | calloc | crash (SIGTRAP) | crash (SIGTRAP) | ENOMEM | ENOMEM |
| tf dnn_detect, async | calloc / nopp / functype | hang ×3 | hang ×3 | ENOMEM
/ EINVAL / ENOSYS | same |
| tf dnn_detect, sync | calloc / nopp / functype | EIO / **hang** / EIO | EIO /
**hang** / EIO | ENOMEM / EINVAL / ENOSYS | same |
| tf output-size probe (dnn_processing, sr) | calloc, GETOUTPUT | 188 | 188 |
ENOMEM | ENOMEM |
| onnx dnn_processing / dnn_detect | calloc, ortdata, functype | EIO | EIO |
EIO | ENOMEM, EXTERNAL, ENOSYS |
| onnx dnn_processing probe | calloc / ortdata, GETOUTPUT | 188 | 188 | 188 |
ENOMEM / EXTERNAL |
| onnx dnn_detect, uint8 output (`ssd_u8`) | model | EIO | EIO | EIO | ENOSYS |
| onnx dnn_detect, rank-2 output (`ssd_rank2`, `ssd_rows0dyn`) | model | EIO |
EIO | EIO | ENOSYS |
| onnx dnn_processing, uint8 output (`proc_u8_64`) | model | 188 | 188 | 188 |
ENOSYS |
| valid models (tf async + sync, onnx proc + detect) | none | 0, 5 frames | 0,
5 frames | 0, 5 frames | 0, 5 frames |
188 is `DNN_GENERIC_ERROR`, printed as "Error number -558779972 occurred". With
the fix, every error run above exits in 0.7 s or less.
¹ In this run the derain sync case crashed (133) on both master and base. In
the v1 runs it crashed on only one of them and returned 188 on the other. The
same row under ASan always reports the heap-use-after-free.
**ASan** (clang-asan builds with the fault patch, 18 cases × 3 builds). The
builds run with `ASAN_OPTIONS=detect_invalid_join=0`. Without it, every DNN run
of all three builds, master included, aborts at start with "Joining already
joined thread". #24702 v3 is based on a master from before 6a030f3dd2
("libavfilter/dnn: guard pthread_join on never-started async threads"), which
fixes that join.
- fix: no report in any case.
- master and base: tf sr sync and tf derain sync report `heap-use-after-free`
in `av_frame_free()` ← `dnn_free_model_tf()` (first frame
`av_frame_side_data_free()`). The memory was freed by `filter_frame()` in
vf_sr.c / vf_derain.c.
- master only: onnx dnn_processing (calloc and ortdata) reports a
`stack-buffer-overflow` in swscale (`scale_internal`). This is the bug that the
first commit of #24702 fixes.
**Unchanged paths:** the whole 294.6 s clip, decoded with `-threads 1` at
`fps=2`, gives 589 frames. The output is byte-identical across master, base and
fix for every model:
| model | output | frames | boxes |
|---|---|---|---|
| D-FINE S (onnx, dnn_detect) | showinfo + boxes | 589 | 4209 |
| RF-DETR S (onnx, dnn_detect) | showinfo + boxes | 589 | 2500 |
| identity (onnx, dnn_processing) | framemd5 | 589 | - |
| identity (tf, dnn_processing async and sync, derain) | framemd5 | 589 | - |
| identity Y (tf, sr ×2) | framemd5 | 589 | - |
| constant one-box detector (tf, dnn_detect) | showinfo + boxes | 589 | 589 |
**leaks(1):** the fix adds no leak. `leaks --atExit` also shows two existing
leaks on master, in both success and error runs. Neither is touched here:
- `tf_free_request()` deletes only `output_tensors[0]`, because `nb_output` is
`sizeof(pointer)/sizeof(pointer)`. The 4-output detector leaks 15 to 16 root
tensors over 5 frames.
- sr with pre-scaling (`ff_dnn_execute_model(out, out)`) loses its input frame,
because `ff_dnn_get_result()` overwrites `in`. That is 6 root leaks for 5
frames.
**Not fixed here:** in async mode, a `TF_SessionRun()` failure never reaches
the callback, because `async_thread_routine()` returns before calling it. That
still hangs on every build (`tf_rgb_run2fail`, a graph whose
`tf.debugging.Assert` fails from the second run on). It is a separate change in
the shared async code.
**Warnings:** the builds have the same 4 existing `-Wunused-result` warnings in
dnn_backend_onnx.c, outside the callback.
## Reproducer
Put the six attached files in one directory:
`cb-inference-done-evidence-v2.sh`, `cb_inference_done_models.py`,
`cb_inference_done_inject.py`, `wrap_models.py`, `pr24702_models.py` and
`pr24702_tf_models.py`. All downloads (clip, models, libtensorflow) are pinned
by sha256. The script fetches `dnn-cb-inference-done` from `BRANCH_SRC`, which
defaults to the fork.
```sh
# needs: Homebrew onnxruntime and openvino, python3 (onnx), python3.12
(tensorflow 2.18.0); ~1.3 GB downloads
WORK=$PWD/cb-work bash cb-inference-done-evidence-v2.sh # setup build errors
unchanged asan leaks report
cat cb-work/RESULTS.md
```
A single hang, by hand: build base with the fault patch, then run
`DNN_FAULT=calloc ./ffmpeg -f lavfi -i testsrc=s=64x64:d=0.2,format=yuv420p -vf
sr=dnn_backend=tensorflow:model=tf_y_valid.pb:scale_factor=2:input=x:output=y
-f null -`.
It never finishes; the script kills it after 30 s. With this PR it exits with
`Cannot allocate memory`.
From 26a4f676f8d936968bdeec6eb5f2d63a4a3baa06 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 01/10] avfilter/dnn: fix single-pointer arrays passed as
4-plane sws_scale buffers
ff_proc_from_dnn_to_frame() and ff_proc_from_frame_to_dnn() take the
address of a single `void *middle_data` local (`&middle_data`) for the
DL_NCHW intermediate buffer and pass it to sws_scale() cast to a
4-element plane pointer array (`uint8_t * const*` / `const uint8_t
**`). &middle_data is the address of one pointer-sized stack slot, not
an array of four; sws_scale() indexes dst[1..3]/src[1..3] even for a
single-plane GRAY8 conversion (to determine plane strides/validity),
so this reads three pointer-sized values past the stack slot holding
middle_data.
On common ABIs the extra slots are only read, never written or
dereferenced, for these single-plane conversions, so the output is
unaffected. It is still undefined behavior: an ASan build of master
reports a stack-buffer-overflow in sws_scale() on these paths. It also
makes the surrounding buffer-safety work in these two functions harder
to reason about.
Replace the single pointer with a real, zero-initialized 4-entry
`void *middle_data_planes[4]` array, storing the allocated buffer in
slot 0 and leaving the unused plane slots explicitly NULL, matching
how frame->data (a genuine 4-element array) is used everywhere else
in this file.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_io_proc.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 0a18c56b84..08c663be04 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -46,6 +46,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
int linesize[4] = { 0 };
void **dst_data = NULL;
void *middle_data = NULL;
+ void *middle_data_planes[4] = { NULL, NULL, NULL, NULL };
uint8_t *planar_data[4] = { 0 };
int plane_size = frame->width * frame->height * sizeof(uint8_t);
enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE;
@@ -76,7 +77,8 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
ret = AVERROR(ENOMEM);
goto err;
}
- dst_data = &middle_data;
+ middle_data_planes[0] = middle_data;
+ dst_data = middle_data_planes;
linesize[0] = frame->width * 3;
}
@@ -186,6 +188,7 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData
*input, void *log_ctx)
int linesize[4] = { 0 };
void **src_data = NULL;
void *middle_data = NULL;
+ void *middle_data_planes[4] = { NULL, NULL, NULL, NULL };
uint8_t *planar_data[4] = { 0 };
int plane_size = frame->width * frame->height * sizeof(uint8_t);
enum AVPixelFormat dst_fmt = AV_PIX_FMT_NONE;
@@ -215,7 +218,8 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData
*input, void *log_ctx)
ret = AVERROR(ENOMEM);
goto err;
}
- src_data = &middle_data;
+ middle_data_planes[0] = middle_data;
+ src_data = middle_data_planes;
linesize[0] = frame->width * 3;
}
--
2.52.0
From 072f973d7e2ebea8170a56def58df4d45457b489 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 02/10] avfilter/dnn: synchronize task completion with the
result reader
A task is complete when its inference_done count reaches
inference_todo. The completion callbacks increment the count on the
inference thread: an OpenVINO async request's callback thread, or the
thread started by the common async module for TensorFlow and Torch.
ff_dnn_get_result_common() polls it from the filter thread without any
synchronization, and then hands the task's output frame, written by the
callback, to the filter. These are conflicting accesses with no
happens-before relation, i.e. a data race under the C memory model.
OpenVINO classification can also run several inferences of one task
concurrently on different requests, whose callbacks then increment the
same count concurrently.
Add ff_dnn_complete_lltask(), which counts one inference as done under
a lock, and take the same lock in ff_dnn_get_result_common() to read
the count. Everything a callback writes before completing its
inference is thereby visible to the filter thread that takes the
result. The lock is only held around these two accesses. All four
backends' completion callbacks use the helper; the synchronous checks
in their execute functions run on the thread that ran the callback and
are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 20 +++++++++++++++++++-
libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++
libavfilter/dnn/dnn_backend_onnx.c | 2 +-
libavfilter/dnn/dnn_backend_openvino.c | 2 +-
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn/dnn_backend_torch.cpp | 2 +-
6 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 82627c7a96..5a73b1d15d 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -67,6 +67,20 @@ int ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams
*exec_params, void *backe
return 0;
}
+/**
+ * Protects TaskItem.inference_done, which completion callbacks update
+ * from inference threads and ff_dnn_get_result_common() reads from the
+ * filter thread. It is only held for those accesses.
+ */
+static AVMutex task_completion_lock = AV_MUTEX_INITIALIZER;
+
+void ff_dnn_complete_lltask(TaskItem *task)
+{
+ ff_mutex_lock(&task_completion_lock);
+ task->inference_done++;
+ ff_mutex_unlock(&task_completion_lock);
+}
+
/**
* Thread routine for async execution.
* @param args pointer to DNNAsyncExecModule module
@@ -144,12 +158,16 @@ int ff_dnn_start_inference_async(void *ctx,
DNNAsyncExecModule *async_module)
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out)
{
TaskItem *task = ff_queue_peek_front(task_queue);
+ int done;
if (!task) {
return DAST_EMPTY_QUEUE;
}
- if (task->inference_done != task->inference_todo) {
+ ff_mutex_lock(&task_completion_lock);
+ done = task->inference_done == task->inference_todo;
+ ff_mutex_unlock(&task_completion_lock);
+ if (!done) {
return DAST_NOT_READY;
}
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index f0b8b47397..f364f1739a 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -139,6 +139,21 @@ void ff_dnn_wait_requests(SafeQueue *request_queue, int
nireq);
*/
int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module);
+/**
+ * Count one inference (LastLevelTaskItem) of a task as done.
+ *
+ * Completion callbacks may run on other threads than the filter (async
+ * OpenVINO requests, and the TensorFlow and Torch async modules), while
+ * ff_dnn_get_result_common() polls the count from the filter thread.
+ * Backends therefore update task->inference_done only through this
+ * function, which does so under the lock ff_dnn_get_result_common()
+ * takes to read it: everything the callback wrote before this call,
+ * such as the output frame or its side data, is visible to the thread
+ * that takes the result. The call that completes the task hands it
+ * over to that thread, which may free it at any time after.
+ */
+void ff_dnn_complete_lltask(TaskItem *task);
+
/**
* Extract input and output frame from the Task Queue after
* asynchronous inference.
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index 009387b36d..08ff70099a 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -720,7 +720,7 @@ static void infer_completion_callback(void *args)
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
err:
av_freep(&outputs);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 0df4d585e8..b5ccb938c0 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -479,7 +479,7 @@ static void infer_completion_callback(void *args)
break;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
av_freep(&request->lltasks[i]);
for (int i = 0; i < ov_model->nb_outputs; i++)
outputs[i].data = (uint8_t *)outputs[i].data +
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index f8a4d9bc62..876f95619e 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -744,7 +744,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
err:
tf_free_request(infer_request);
av_freep(&outputs);
diff --git a/libavfilter/dnn/dnn_backend_torch.cpp
b/libavfilter/dnn/dnn_backend_torch.cpp
index 1ecf95e938..afcac800ec 100644
--- a/libavfilter/dnn/dnn_backend_torch.cpp
+++ b/libavfilter/dnn/dnn_backend_torch.cpp
@@ -471,7 +471,7 @@ static void infer_completion_callback(void *args) {
avpriv_report_missing_feature(th_model->ctx, "model function type
%d", th_model->model.func_type);
goto err;
}
- task->inference_done++;
+ ff_dnn_complete_lltask(task);
}
err:
--
2.52.0
From 5fba7edbc44179db32911c530b6c86c58791a02f Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 03/10] avfilter/dnn: propagate post-processing failures
through async results
Every backend's default post-processing call site (ONNX, OpenVINO,
TensorFlow, Torch, in their inference-completion callbacks) calls
ff_proc_from_dnn_to_frame() -- or a filter-supplied FramePrePostProc
callback -- to convert the raw inference output into out_frame, but
discards the returned AVERROR. The task is then unconditionally marked
complete (ff_dnn_complete_lltask()), so ff_dnn_get_result_common() always
returns DAST_SUCCESS with out_frame handed back as though it had been
validly written, even when post-processing failed and out_frame is
untouched or partially written.
vf_dnn_processing.c allocates out_frame separately from in_frame
(av_frame_copy_props() only copies metadata, not pixel data), so on a
post-processing failure it forwards this unwritten buffer downstream
as a normal successful frame. This is exactly the class of bug that
motivates rejecting channel-mismatched model output in
ff_proc_from_dnn_to_frame() with AVERROR(ENOSYS), as proposed in
PR #24048: rejecting bad output at the post-proc call site only helps
if callers actually see the failure instead of a synthetic success.
Fix this generically, independent of any one post-proc failure mode:
- Add TaskItem.status (dnn_backend_common.h), initialized to 0 by
ff_dnn_fill_task(). All four backends now pass the return value of
ff_proc_from_dnn_to_frame()/frame_post_proc() to
ff_dnn_complete_lltask() instead of discarding it. That helper keeps
the first error in task->status, under the same lock as the
inference_done update, so the status is published to the filter
thread together with the completion.
- ff_dnn_get_result_common() reads task->status under that lock,
together with inference_done, and checks it before handing back
the frames: on failure it releases in_frame and out_frame (only
once when they are the same frame, as on vf_sr's pre-scale path),
sets *in/*out to NULL, and returns DAST_FAIL (already defined in
DNNAsyncStatusType, previously unused by this path) instead of
DAST_SUCCESS. The task is popped and freed either way, so a failed
post-proc can't stall async draining.
- ff_dnn_get_result_common(), the get_result vtable entry in
DNNModule, and the ff_dnn_get_result() wrapper all gain an optional
`int *status` out-parameter carrying the stored AVERROR back to the
caller when DAST_FAIL is returned; NULL is safe to pass when the
specific code isn't needed.
- vf_dnn_processing.c's activate() and flush_frame() check for
DAST_FAIL and return the stored AVERROR (or AVERROR_EXTERNAL as a
fallback) instead of silently continuing.
- vf_dnn_classify.c, vf_dnn_detect.c (DFT_ANALYTICS_* function types,
which never touch this post-proc path, so task->status stays 0) and
vf_derain.c/vf_sr.c (which already treat any async_state !=
DAST_SUCCESS as AVERROR(EINVAL)) are updated only for the new
ff_dnn_get_result() parameter and pass NULL; their existing
behavior is unchanged, and derain/sr now also correctly fail
instead of forwarding a bad frame if their post-proc callback ever
errors.
The OpenVINO and TensorFlow backends were built and run with
OpenVINO 2026.4.0 and the TensorFlow 2.18.0 C library. The Torch
backend was not built, as libtorch is not installed here; its change
is the same as in the ONNX and TensorFlow backends and was checked by
reading. The CUDA zero-copy output path in dnn_backend_torch.cpp
(fill_model_output_th_cuda) is a separate conversion path that never
calls ff_proc_from_dnn_to_frame/frame_post_proc and is left as-is.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 39 +++++++++++++++++++++-----
libavfilter/dnn/dnn_backend_common.h | 33 ++++++++++++++++++----
libavfilter/dnn/dnn_backend_onnx.c | 12 ++++----
libavfilter/dnn/dnn_backend_openvino.c | 12 ++++----
libavfilter/dnn/dnn_backend_tf.c | 11 ++++----
libavfilter/dnn/dnn_backend_torch.cpp | 15 +++++-----
libavfilter/dnn_filter_common.c | 4 +--
libavfilter/dnn_filter_common.h | 2 +-
libavfilter/dnn_interface.h | 7 +++--
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_classify.c | 4 +--
libavfilter/vf_dnn_detect.c | 4 +--
libavfilter/vf_dnn_processing.c | 18 ++++++++++--
libavfilter/vf_sr.c | 2 +-
14 files changed, 116 insertions(+), 49 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 5a73b1d15d..1c6ccb25bb 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -63,20 +63,23 @@ int ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams
*exec_params, void *backe
task->model = backend_model;
task->nb_output = exec_params->nb_output;
task->output_names = exec_params->output_names;
+ task->status = 0;
return 0;
}
/**
- * Protects TaskItem.inference_done, which completion callbacks update
- * from inference threads and ff_dnn_get_result_common() reads from the
- * filter thread. It is only held for those accesses.
+ * Protects TaskItem.inference_done and TaskItem.status, which completion
+ * callbacks update from inference threads and ff_dnn_get_result_common()
+ * reads from the filter thread. It is only held for those accesses.
*/
static AVMutex task_completion_lock = AV_MUTEX_INITIALIZER;
-void ff_dnn_complete_lltask(TaskItem *task)
+void ff_dnn_complete_lltask(TaskItem *task, int status)
{
ff_mutex_lock(&task_completion_lock);
+ if (status < 0 && task->status >= 0)
+ task->status = status;
task->inference_done++;
ff_mutex_unlock(&task_completion_lock);
}
@@ -155,10 +158,10 @@ int ff_dnn_start_inference_async(void *ctx,
DNNAsyncExecModule *async_module)
return 0;
}
-DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out)
+DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status)
{
TaskItem *task = ff_queue_peek_front(task_queue);
- int done;
+ int done, task_status;
if (!task) {
return DAST_EMPTY_QUEUE;
@@ -166,14 +169,36 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_mutex_lock(&task_completion_lock);
done = task->inference_done == task->inference_todo;
+ task_status = task->status;
ff_mutex_unlock(&task_completion_lock);
if (!done) {
return DAST_NOT_READY;
}
+ ff_queue_pop_front(task_queue);
+
+ if (task_status < 0) {
+ // Post-processing of the inference output into out_frame failed
+ // (either the default ff_proc_from_dnn_to_frame() or a custom
+ // FramePrePostProc callback). out_frame was never validly
+ // written, so it must not be handed back as a result; release
+ // both frames here since the caller has no valid pointer to
+ // free them through once DAST_FAIL is returned. vf_sr passes the
+ // same frame as input and output, so free it only once then.
+ if (status)
+ *status = task_status;
+ if (task->out_frame == task->in_frame)
+ task->out_frame = NULL;
+ av_frame_free(&task->in_frame);
+ av_frame_free(&task->out_frame);
+ *in = NULL;
+ *out = NULL;
+ av_freep(&task);
+ return DAST_FAIL;
+ }
+
*in = task->in_frame;
*out = task->out_frame;
- ff_queue_pop_front(task_queue);
av_freep(&task);
return DAST_SUCCESS;
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index f364f1739a..44e39f8bc9 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -52,6 +52,15 @@ typedef struct TaskItem {
uint32_t nb_output;
uint32_t inference_todo;
uint32_t inference_done;
+ /**
+ * Zero on success, or a negative AVERROR from frame post-processing
+ * (ff_proc_from_dnn_to_frame() or a FramePrePostProc callback).
+ * Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
+ * which keeps the first error, under the same lock as inference_done.
+ * Consumed by ff_dnn_get_result_common(), which releases failed tasks
+ * and their frames.
+ */
+ int status;
} TaskItem;
// one task might have multiple inferences
@@ -140,19 +149,24 @@ void ff_dnn_wait_requests(SafeQueue *request_queue, int
nireq);
int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module);
/**
- * Count one inference (LastLevelTaskItem) of a task as done.
+ * Count one inference (LastLevelTaskItem) of a task as done, and record
+ * its result in task->status if it is the task's first error.
*
* Completion callbacks may run on other threads than the filter (async
* OpenVINO requests, and the TensorFlow and Torch async modules), while
* ff_dnn_get_result_common() polls the count from the filter thread.
- * Backends therefore update task->inference_done only through this
- * function, which does so under the lock ff_dnn_get_result_common()
- * takes to read it: everything the callback wrote before this call,
+ * Backends therefore update task->inference_done and task->status only
+ * through this function, which does so under the lock
+ * ff_dnn_get_result_common() takes to read them: everything the callback
wrote before this call,
* such as the output frame or its side data, is visible to the thread
* that takes the result. The call that completes the task hands it
* over to that thread, which may free it at any time after.
+ *
+ * @param task the task the inference belongs to
+ * @param status 0 or another non-negative value on success, a negative
+ * AVERROR if the inference output could not be processed
*/
-void ff_dnn_complete_lltask(TaskItem *task);
+void ff_dnn_complete_lltask(TaskItem *task, int status);
/**
* Extract input and output frame from the Task Queue after
@@ -161,12 +175,19 @@ void ff_dnn_complete_lltask(TaskItem *task);
* @param task_queue pointer to the task queue of the backend
* @param in double pointer to the input frame
* @param out double pointer to the output frame
+ * @param status optional pointer to receive the negative AVERROR
+ * stored on the task when post-processing failed.
+ * Only written when DAST_FAIL is returned; may be NULL
+ * if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
+ * @retval DAST_FAIL if inference completed but post-processing of the
+ * output into out_frame failed; *in and *out are
+ * set to NULL and both frames are released.
* @retval DAST_SUCCESS if result successfully extracted
*/
-DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out);
+DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
/**
* Allocate input and output frames and fill the Task
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index 08ff70099a..fcbba7cf7b 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -588,7 +588,7 @@ static void infer_completion_callback(void *args)
size_t num_dims;
int64_t *dims;
OrtStatus *status;
- int ret;
+ int ret = 0;
outputs = av_calloc(infer_request->nb_outputs, sizeof(*outputs));
if (!outputs) {
@@ -699,9 +699,9 @@ static void infer_completion_callback(void *args)
if (task->do_ioproc) {
outputs[0].scale = 255;
if (onnx_model->model.frame_post_proc != NULL) {
- onnx_model->model.frame_post_proc(task->out_frame, outputs,
onnx_model->model.filter_ctx);
+ ret = onnx_model->model.frame_post_proc(task->out_frame,
outputs, onnx_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
}
} else {
task->out_frame->width =
outputs[0].dims[dnn_get_width_idx_by_layout(outputs[0].layout)];
@@ -720,7 +720,7 @@ static void infer_completion_callback(void *args)
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
err:
av_freep(&outputs);
@@ -1189,10 +1189,10 @@ static int dnn_execute_model_onnx(const DNNModel
*model, DNNExecBaseParams *exec
return execute_model_onnx(request, onnx_model->lltask_queue);
}
-static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
ONNXModel *onnx_model = (ONNXModel *)model;
- return ff_dnn_get_result_common(onnx_model->task_queue, in, out);
+ return ff_dnn_get_result_common(onnx_model->task_queue, in, out, status);
}
static int dnn_flush_onnx(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index b5ccb938c0..57ef4e18dd 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -338,6 +338,7 @@ static void infer_completion_callback(void *args)
OVModel *ov_model = task->model;
SafeQueue *requestq = ov_model->request_queue;
DNNData *outputs;
+ int ret;
DnnContext *ctx = ov_model->ctx;
#if HAVE_OPENVINO2
size_t* dims;
@@ -439,14 +440,15 @@ static void infer_completion_callback(void *args)
av_assert0(request->lltask_count >= 1);
for (int i = 0; i < request->lltask_count; ++i) {
task = request->lltasks[i]->task;
+ ret = 0;
switch (ov_model->model.func_type) {
case DFT_PROCESS_FRAME:
if (task->do_ioproc) {
if (ov_model->model.frame_post_proc != NULL) {
- ov_model->model.frame_post_proc(task->out_frame, outputs,
ov_model->model.filter_ctx);
+ ret = ov_model->model.frame_post_proc(task->out_frame,
outputs, ov_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs,
ctx);
}
} else {
task->out_frame->width =
@@ -479,7 +481,7 @@ static void infer_completion_callback(void *args)
break;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
av_freep(&request->lltasks[i]);
for (int i = 0; i < ov_model->nb_outputs; i++)
outputs[i].data = (uint8_t *)outputs[i].data +
@@ -1558,10 +1560,10 @@ static int dnn_execute_model_ov(const DNNModel *model,
DNNExecBaseParams *exec_p
}
}
-static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
OVModel *ov_model = (OVModel *)model;
- return ff_dnn_get_result_common(ov_model->task_queue, in, out);
+ return ff_dnn_get_result_common(ov_model->task_queue, in, out, status);
}
static int dnn_flush_ov(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 876f95619e..dc895f554a 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -696,6 +696,7 @@ static void infer_completion_callback(void *args) {
LastLevelTaskItem *lltask = request->lltask;
TaskItem *task = lltask->task;
DNNData *outputs;
+ int ret = 0;
TFInferRequest *infer_request = request->infer_request;
TFModel *tf_model = task->model;
DnnContext *ctx = tf_model->ctx;
@@ -722,9 +723,9 @@ static void infer_completion_callback(void *args) {
//it only support 1 output if it's frame in & frame out
if (task->do_ioproc) {
if (tf_model->model.frame_post_proc != NULL) {
- tf_model->model.frame_post_proc(task->out_frame, outputs,
tf_model->model.filter_ctx);
+ ret = tf_model->model.frame_post_proc(task->out_frame,
outputs, tf_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx);
}
} else {
task->out_frame->width =
@@ -744,7 +745,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
err:
tf_free_request(infer_request);
av_freep(&outputs);
@@ -847,10 +848,10 @@ static int dnn_execute_model_tf(const DNNModel *model,
DNNExecBaseParams *exec_p
return execute_model_tf(request, tf_model->lltask_queue);
}
-static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
TFModel *tf_model = (TFModel *)model;
- return ff_dnn_get_result_common(tf_model->task_queue, in, out);
+ return ff_dnn_get_result_common(tf_model->task_queue, in, out, status);
}
static int dnn_flush_tf(const DNNModel *model)
diff --git a/libavfilter/dnn/dnn_backend_torch.cpp
b/libavfilter/dnn/dnn_backend_torch.cpp
index afcac800ec..6f00673aeb 100644
--- a/libavfilter/dnn/dnn_backend_torch.cpp
+++ b/libavfilter/dnn/dnn_backend_torch.cpp
@@ -421,6 +421,7 @@ static void infer_completion_callback(void *args) {
for (uint32_t i = 0; i < request->lltask_count; i++) {
lltask = request->lltasks[i];
TaskItem *task = lltask->task;
+ int ret = 0;
torch::Tensor out_slice = slices[i];
c10::IntArrayRef sizes = out_slice.sizes();
@@ -453,11 +454,11 @@ static void infer_completion_callback(void *args) {
outputs.scale = 255;
outputs.data = out_slice.data_ptr();
if (th_model->model.frame_post_proc != NULL) {
- th_model->model.frame_post_proc(task->out_frame,
&outputs,
-
th_model->model.filter_ctx);
+ ret = th_model->model.frame_post_proc(task->out_frame,
&outputs,
+
th_model->model.filter_ctx);
} else {
- ff_proc_from_dnn_to_frame(task->out_frame, &outputs,
- th_model->ctx);
+ ret = ff_proc_from_dnn_to_frame(task->out_frame,
&outputs,
+ th_model->ctx);
}
#if CONFIG_CUDA
}
@@ -471,7 +472,7 @@ static void infer_completion_callback(void *args) {
avpriv_report_missing_feature(th_model->ctx, "model function type
%d", th_model->model.func_type);
goto err;
}
- ff_dnn_complete_lltask(task);
+ ff_dnn_complete_lltask(task, ret);
}
err:
@@ -752,10 +753,10 @@ static int dnn_execute_model_th(const DNNModel *model,
DNNExecBaseParams *exec_p
return 0;
}
-static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame
**in, AVFrame **out)
+static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame
**in, AVFrame **out, int *status)
{
THModel *th_model = (THModel *)model;
- return ff_dnn_get_result_common(th_model->task_queue, in, out);
+ return ff_dnn_get_result_common(th_model->task_queue, in, out, status);
}
static int dnn_flush_th(const DNNModel *model)
diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c
index b644a6280f..b123a4c608 100644
--- a/libavfilter/dnn_filter_common.c
+++ b/libavfilter/dnn_filter_common.c
@@ -216,9 +216,9 @@ int ff_dnn_execute_model_classification(DnnContext *ctx,
AVFrame *in_frame, AVFr
return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base);
}
-DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame)
+DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame, int *status)
{
- return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame);
+ return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame,
status);
}
int ff_dnn_flush(DnnContext *ctx)
diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h
index 1e89ab42ae..596cab55a4 100644
--- a/libavfilter/dnn_filter_common.h
+++ b/libavfilter/dnn_filter_common.h
@@ -59,7 +59,7 @@ int ff_dnn_get_input(DnnContext *ctx, DNNData *input);
int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int
*output_width, int *output_height);
int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame
*out_frame);
int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame,
AVFrame *out_frame, const char *target);
-DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame);
+DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame,
AVFrame **out_frame, int *status);
int ff_dnn_flush(DnnContext *ctx);
void ff_dnn_uninit(DnnContext *ctx);
#if CONFIG_CUDA
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 632ae6be84..810d3c4bdf 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -191,8 +191,11 @@ struct DNNModule {
DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type,
AVFilterContext *filter_ctx);
// Executes model with specified input and output. Returns the error code
otherwise.
int (*execute_model)(const DNNModel *model, DNNExecBaseParams
*exec_params);
- // Retrieve inference result.
- DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out);
+ // Retrieve inference result. status, if non-NULL, receives the
+ // negative AVERROR stored on the task when DAST_FAIL is returned
+ // (i.e. inference completed but post-processing the output into
+ // *out failed); *in and *out are set to NULL in that case.
+ DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
// Frees memory allocated for model.
diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c
index 690a797bdb..c4910d9a1e 100644
--- a/libavfilter/vf_derain.c
+++ b/libavfilter/vf_derain.c
@@ -75,7 +75,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
return dnn_result;
}
do {
- async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out);
+ async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL);
} while (async_state == DAST_NOT_READY);
if (async_state != DAST_SUCCESS)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index f92c41ab76..5751e7c4bc 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -216,7 +216,7 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -259,7 +259,7 @@ static int dnn_classify_activate(AVFilterContext
*filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 99dca8546c..846e4c11fc 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -882,7 +882,7 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -925,7 +925,7 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 16904a2c20..5f88734e44 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -276,7 +276,14 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts,
int64_t *out_pts)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (out_frame) {
if (isPlanarYUV(in_frame->format))
copy_uv_planes(ctx, out_frame, in_frame);
@@ -328,7 +335,14 @@ static int activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (out_frame) {
if (isPlanarYUV(in_frame->format))
copy_uv_planes(ctx, out_frame, in_frame);
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index 8f1ccee56a..7eac5ca18d 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -142,7 +142,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
}
do {
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL);
} while (async_state == DAST_NOT_READY);
if (async_state != DAST_SUCCESS)
--
2.52.0
From a229e63a830d2cfaf103b4e3575bfa690400c7b8 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:39 -0400
Subject: [PATCH 04/10] avfilter/vf_derain, vf_sr: return the post-processing
error
ff_dnn_get_result() now reports DAST_FAIL together with the error
returned by the frame post-processing step, but vf_derain and vf_sr
still passed NULL for the status and collapsed every non-success
result into AVERROR(EINVAL). A failed conversion of the model output
(for example, with PR #24048 applied, an output shape that does not
match the output frame) was therefore reported as a generic
invalid-argument error with no hint of what went wrong.
Pass a status pointer, log the failure and return the real error, as
vf_dnn_processing already does. Fall back to AVERROR_EXTERNAL if the
backend did not store a negative status. On DAST_FAIL the frames
passed to ff_dnn_execute_model() have already been released by
ff_dnn_get_result(). vf_sr's pre-scale path passes its scaled output
frame as both input and output, so its original input frame is not
released there; that leak happens on success too and predates this
change.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/vf_derain.c | 9 ++++++++-
libavfilter/vf_sr.c | 9 ++++++++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c
index c4910d9a1e..4242e2f518 100644
--- a/libavfilter/vf_derain.c
+++ b/libavfilter/vf_derain.c
@@ -54,6 +54,7 @@ AVFILTER_DNN_DEFINE_CLASS(derain, DNN_TF);
static int filter_frame(AVFilterLink *inlink, AVFrame *in)
{
DNNAsyncStatusType async_state = 0;
+ int proc_status = 0;
AVFilterContext *ctx = inlink->dst;
AVFilterLink *outlink = ctx->outputs[0];
DRContext *dr_context = ctx->priv;
@@ -75,9 +76,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
return dnn_result;
}
do {
- async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL);
+ async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out,
&proc_status);
} while (async_state == DAST_NOT_READY);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state != DAST_SUCCESS)
return AVERROR(EINVAL);
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index 7eac5ca18d..13d3f22919 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -112,6 +112,7 @@ static int config_output(AVFilterLink *outlink)
static int filter_frame(AVFilterLink *inlink, AVFrame *in)
{
DNNAsyncStatusType async_state = 0;
+ int proc_status = 0;
AVFilterContext *context = inlink->dst;
SRContext *ctx = context->priv;
AVFilterLink *outlink = context->outputs[0];
@@ -142,9 +143,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
}
do {
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL);
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, &proc_status);
} while (async_state == DAST_NOT_READY);
+ if (async_state == DAST_FAIL) {
+ av_log(context, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state != DAST_SUCCESS)
return AVERROR(EINVAL);
--
2.52.0
From c55405dd38da9513aafdb6d32d837bbb70639714 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 05/10] avfilter/dnn: return AVERROR codes from analytics
post-processing
Several of the detect_post_proc() and classify_post_proc() callbacks of
dnn_detect and dnn_classify return a bare -1 on failure, which is
AVERROR(EPERM). The backends do not report these values to the filter
yet: OpenVINO and TensorFlow ignore them, and ONNX turns a failed
detect_post_proc() into DNN_GENERIC_ERROR. The next commit makes them
the error of the filter, and of ffmpeg, where -1 would read as
"Operation not permitted".
Return AVERROR_INVALIDDATA, and log it, for a classifier output without
any scores, which was not logged at all. Return AVERROR(EINVAL) for
missing or already present detection side data, and AVERROR(ENOMEM)
when the side data cannot be allocated.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/vf_dnn_classify.c | 5 +++--
libavfilter/vf_dnn_detect.c | 10 +++++-----
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index 5751e7c4bc..ab79030165 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -69,13 +69,14 @@ static int dnn_classify_post_proc(AVFrame *frame, DNNData
*output, uint32_t bbox
AVFrameSideData *sd;
int output_size = output->dims[3] * output->dims[2] * output->dims[1];
if (output_size <= 0) {
- return -1;
+ av_log(filter_ctx, AV_LOG_ERROR, "Model output has no class
scores.\n");
+ return AVERROR_INVALIDDATA;
}
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (!sd) {
av_log(filter_ctx, AV_LOG_ERROR, "Cannot get side data in
dnn_classify_post_proc\n");
- return -1;
+ return AVERROR(EINVAL);
}
header = (AVDetectionBBoxHeader *)sd->data;
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 846e4c11fc..2382562daf 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -392,7 +392,7 @@ static int dnn_detect_fill_side_data(AVFrame *frame,
AVFilterContext *filter_ctx
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
@@ -551,7 +551,7 @@ static int dnn_detect_post_proc_ssd(AVFrame *frame, DNNData
*output, int nb_outp
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
@@ -618,7 +618,7 @@ static int dnn_detect_post_proc_anchored(AVFrame *frame,
DNNData *output, int nb
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (sd) {
av_log(filter_ctx, AV_LOG_ERROR, "already have bounding boxes in side
data.\n");
- return -1;
+ return AVERROR(EINVAL);
}
switch (ctx->model_type) {
@@ -661,7 +661,7 @@ static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData
*output, AVFilterCont
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (sd) {
av_log(filter_ctx, AV_LOG_ERROR, "already have dnn bounding boxes in
side data.\n");
- return -1;
+ return AVERROR(EINVAL);
}
for (int i = 0; i < proposal_count; ++i) {
@@ -678,7 +678,7 @@ static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData
*output, AVFilterCont
header = av_detection_bbox_create_side_data(frame, nb_bboxes);
if (!header) {
av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
- return -1;
+ return AVERROR(ENOMEM);
}
av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
--
2.52.0
From 9e31d945af4333ea729afb563366110b973c20a6 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 06/10] avfilter/dnn: complete analytics tasks whose
post-processing failed
When detect_post_proc() failed, the ONNX backend jumped past the
completion of the task, so it could never satisfy
inference_done == inference_todo. The ONNX backend only runs
requests synchronously, so execute_model_onnx() then returned
DNN_GENERIC_ERROR and dnn_detect failed with a bare AVERROR(EIO)
("Input/output error") instead of the post-processing error. The
OpenVINO and TensorFlow backends ignored the return values of
detect_post_proc() and classify_post_proc() altogether, so failures
there went unnoticed.
Pass the post-processing result to ff_dnn_complete_lltask(), which
keeps the first failure in task->status and still counts the lltask
as done, so the task completes and the result path reports DAST_FAIL
with the error. An OpenVINO classification task can be split over
several requests whose callbacks run concurrently; the helper updates
the status under its lock, so they do not race on it. dnn_detect and
dnn_classify now pass a status to ff_dnn_get_result() and return that
error, like dnn_processing does, instead of silently forwarding the
frame.
The Torch backend has no analytics post-processing and is unchanged.
Signed-off-by: Jake Fineman <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 7 +++----
libavfilter/dnn/dnn_backend_common.h | 13 +++++++------
libavfilter/dnn/dnn_backend_onnx.c | 2 --
libavfilter/dnn/dnn_backend_openvino.c | 17 ++++++++++-------
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn_interface.h | 5 +++--
libavfilter/vf_dnn_classify.c | 18 ++++++++++++++++--
libavfilter/vf_dnn_detect.c | 18 ++++++++++++++++--
8 files changed, 56 insertions(+), 26 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 1c6ccb25bb..077e0211b8 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -178,10 +178,9 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_queue_pop_front(task_queue);
if (task_status < 0) {
- // Post-processing of the inference output into out_frame failed
- // (either the default ff_proc_from_dnn_to_frame() or a custom
- // FramePrePostProc callback). out_frame was never validly
- // written, so it must not be handed back as a result; release
+ // Frame/detect/classify post-processing failed, so out_frame was
+ // not validly written, or in_frame lacks the analytics side data
+ // it should carry. Neither is handed back as a result; release
// both frames here since the caller has no valid pointer to
// free them through once DAST_FAIL is returned. vf_sr passes the
// same frame as input and output, so free it only once then.
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index 44e39f8bc9..6fff571009 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -53,8 +53,8 @@ typedef struct TaskItem {
uint32_t inference_todo;
uint32_t inference_done;
/**
- * Zero on success, or a negative AVERROR from frame post-processing
- * (ff_proc_from_dnn_to_frame() or a FramePrePostProc callback).
+ * Zero on success, or a negative AVERROR from frame/detect/classify
+ * post-processing.
* Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
* which keeps the first error, under the same lock as inference_done.
* Consumed by ff_dnn_get_result_common(), which releases failed tasks
@@ -176,15 +176,16 @@ void ff_dnn_complete_lltask(TaskItem *task, int status);
* @param in double pointer to the input frame
* @param out double pointer to the output frame
* @param status optional pointer to receive the negative AVERROR
- * stored on the task when post-processing failed.
+ * stored on the task when frame/detect/classify
+ * post-processing failed.
* Only written when DAST_FAIL is returned; may be NULL
* if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
- * @retval DAST_FAIL if inference completed but post-processing of the
- * output into out_frame failed; *in and *out are
- * set to NULL and both frames are released.
+ * @retval DAST_FAIL if inference completed but frame/detect/classify
+ * post-processing failed; the task and both frames
+ * are released, and *in and *out are set to NULL.
* @retval DAST_SUCCESS if result successfully extracted
*/
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index fcbba7cf7b..ffa150034f 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -712,8 +712,6 @@ static void infer_completion_callback(void *args)
ret = onnx_model->model.detect_post_proc(task->in_frame, outputs,
infer_request->nb_outputs,
onnx_model->model.filter_ctx);
- if (ret < 0)
- goto err;
break;
default:
avpriv_report_missing_feature(ctx, "model function type %d",
onnx_model->model.func_type);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 57ef4e18dd..200096862e 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -462,19 +462,22 @@ static void infer_completion_callback(void *args)
av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post
proc\n");
goto end;
}
- ov_model->model.detect_post_proc(task->in_frame, outputs,
- ov_model->nb_outputs,
- ov_model->model.filter_ctx);
+ ret = ov_model->model.detect_post_proc(task->in_frame, outputs,
+ ov_model->nb_outputs,
+
ov_model->model.filter_ctx);
break;
case DFT_ANALYTICS_CLASSIFY:
if (!ov_model->model.classify_post_proc) {
av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide
post proc\n");
goto end;
}
- for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++)
- ov_model->model.classify_post_proc(task->in_frame, outputs,
-
request->lltasks[i]->bbox_index,
-
ov_model->model.filter_ctx);
+ for (int output_i = 0; output_i < ov_model->nb_outputs;
output_i++) {
+ int err = ov_model->model.classify_post_proc(task->in_frame,
outputs,
+
request->lltasks[i]->bbox_index,
+
ov_model->model.filter_ctx);
+ if (err < 0 && ret >= 0)
+ ret = err;
+ }
break;
default:
av_assert0(!"should not reach here");
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index dc895f554a..ac87884a46 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -739,7 +739,7 @@ static void infer_completion_callback(void *args) {
av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post
proc\n");
return;
}
- tf_model->model.detect_post_proc(task->in_frame, outputs,
task->nb_output, tf_model->model.filter_ctx);
+ ret = tf_model->model.detect_post_proc(task->in_frame, outputs,
task->nb_output, tf_model->model.filter_ctx);
break;
default:
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 810d3c4bdf..6f1ad3b565 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -193,8 +193,9 @@ struct DNNModule {
int (*execute_model)(const DNNModel *model, DNNExecBaseParams
*exec_params);
// Retrieve inference result. status, if non-NULL, receives the
// negative AVERROR stored on the task when DAST_FAIL is returned
- // (i.e. inference completed but post-processing the output into
- // *out failed); *in and *out are set to NULL in that case.
+ // (i.e. inference completed but frame/detect/classify
+ // post-processing failed); the frames are then released and *in
+ // and *out are set to NULL.
DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index ab79030165..715426fb48 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -217,7 +217,14 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -260,7 +267,14 @@ static int dnn_classify_activate(AVFilterContext
*filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 2382562daf..68cc1e10ad 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -882,7 +882,14 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed while
flushing: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
@@ -925,7 +932,14 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx)
do {
AVFrame *in_frame = NULL;
AVFrame *out_frame = NULL;
- async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
NULL);
+ int proc_status = 0;
+ async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame,
&proc_status);
+ if (async_state == DAST_FAIL) {
+ av_log(ctx, AV_LOG_ERROR,
+ "DNN post-processing of inference output failed: %s\n",
+ av_err2str(proc_status));
+ return proc_status < 0 ? proc_status : AVERROR_EXTERNAL;
+ }
if (async_state == DAST_SUCCESS) {
ret = ff_filter_frame(outlink, in_frame);
if (ret < 0)
--
2.52.0
From f2a854508d07275a662e3c553ca8319a628ed65c Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:40 -0400
Subject: [PATCH 07/10] avfilter/dnn: return the ff_dnn_flush() error at EOF
dnn_processing, dnn_detect and dnn_classify return -1, i.e.
AVERROR(EPERM), when ff_dnn_flush() fails at EOF, which replaces the
backend's error with "Operation not permitted". Return the error
ff_dnn_flush() reported instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/vf_dnn_classify.c | 2 +-
libavfilter/vf_dnn_detect.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index 715426fb48..6f5103c47f 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -211,7 +211,7 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 68cc1e10ad..2d5c70b8fb 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -876,7 +876,7 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink,
int64_t pts, int64_t *o
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 5f88734e44..fec85ace79 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -270,7 +270,7 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts,
int64_t *out_pts)
ret = ff_dnn_flush(&ctx->dnnctx);
if (ret != 0) {
- return -1;
+ return ret;
}
do {
--
2.52.0
From 0d0ba85092020375c5f6b7f38745dc8188372e2c Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 08:55:41 -0400
Subject: [PATCH 08/10] avfilter/dnn_backend_openvino: complete tasks when
reading the output fails
infer_completion_callback() jumps to its cleanup code when it cannot get
an output tensor, its data, shape or element type, or when the filter
has no post-processing callback. That skipped task->inference_done++
for every lltask of the request and leaked them. In async mode the task
then never completed: ff_dnn_get_result() returned DAST_NOT_READY
forever, and the flush loops of dnn_detect and dnn_classify spun at EOF.
A failed allocation of the outputs array also returned without giving
the request back to the request queue.
A model with a constant zero-element output reaches this, e.g. an SSD
detector without proposals ([1,1,0,7]) or a classifier without classes:
the OpenVINO ONNX frontend reads such an output as a rank-0 tensor, and
ov_tensor_get_shape() fails with INVALID_C_PARAM because an ov_shape_t
cannot have rank 0. Outputs that are only empty at run time are read
fine.
On failure, complete the remaining lltasks with the error through
ff_dnn_complete_lltask(), so that get_result reports DAST_FAIL and the
filter returns the error. Initialize output_tensor, which was freed while
uninitialized when getting the first output tensor failed. Replace the
assertion on the first output dimension with an error, since a rank-2
detection output with zero rows made ffmpeg abort.
get_output_ov() runs a synchronous inference on a task of its own to
query the output dimensions, and execute_model_ov() returns success
once all lltasks are counted as done, now also when they were completed
with an error. Check the task's status there as well, so that a failed
output read is returned as the error instead of 0x0 output dimensions,
and the caller's dimensions are left unchanged.
The completion of the remaining lltasks at the end label and the errors
set for a missing post-processing callback also apply to the OpenVINO
1.x API path. Its early returns when getting the output blob, its
buffer, dims or precision fails are left unchanged, since that path
cannot be built against current OpenVINO releases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_common.c | 13 ++++----
libavfilter/dnn/dnn_backend_common.h | 9 +++---
libavfilter/dnn/dnn_backend_openvino.c | 42 +++++++++++++++++++++-----
libavfilter/dnn_interface.h | 4 +--
4 files changed, 49 insertions(+), 19 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_common.c
b/libavfilter/dnn/dnn_backend_common.c
index 077e0211b8..193099d31f 100644
--- a/libavfilter/dnn/dnn_backend_common.c
+++ b/libavfilter/dnn/dnn_backend_common.c
@@ -178,12 +178,13 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue
*task_queue, AVFrame **in, AVF
ff_queue_pop_front(task_queue);
if (task_status < 0) {
- // Frame/detect/classify post-processing failed, so out_frame was
- // not validly written, or in_frame lacks the analytics side data
- // it should carry. Neither is handed back as a result; release
- // both frames here since the caller has no valid pointer to
- // free them through once DAST_FAIL is returned. vf_sr passes the
- // same frame as input and output, so free it only once then.
+ // Frame/detect/classify post-processing or reading the inference
+ // output failed, so out_frame was not validly written, or
+ // in_frame lacks the analytics side data it should carry.
+ // Neither is handed back as a result; release both frames here
+ // since the caller has no valid pointer to free them through
+ // once DAST_FAIL is returned. vf_sr passes the same frame as
+ // input and output, so free it only once then.
if (status)
*status = task_status;
if (task->out_frame == task->in_frame)
diff --git a/libavfilter/dnn/dnn_backend_common.h
b/libavfilter/dnn/dnn_backend_common.h
index 6fff571009..f582ccb59d 100644
--- a/libavfilter/dnn/dnn_backend_common.h
+++ b/libavfilter/dnn/dnn_backend_common.h
@@ -54,7 +54,7 @@ typedef struct TaskItem {
uint32_t inference_done;
/**
* Zero on success, or a negative AVERROR from frame/detect/classify
- * post-processing.
+ * post-processing or reading the inference output.
* Set to 0 by ff_dnn_fill_task(); written by ff_dnn_complete_lltask(),
* which keeps the first error, under the same lock as inference_done.
* Consumed by ff_dnn_get_result_common(), which releases failed tasks
@@ -177,15 +177,16 @@ void ff_dnn_complete_lltask(TaskItem *task, int status);
* @param out double pointer to the output frame
* @param status optional pointer to receive the negative AVERROR
* stored on the task when frame/detect/classify
- * post-processing failed.
+ * post-processing or reading the inference output failed.
* Only written when DAST_FAIL is returned; may be NULL
* if the caller does not need the specific error code.
*
* @retval DAST_EMPTY_QUEUE if task queue is empty
* @retval DAST_NOT_READY if inference not completed yet.
* @retval DAST_FAIL if inference completed but frame/detect/classify
- * post-processing failed; the task and both frames
- * are released, and *in and *out are set to NULL.
+ * post-processing or reading the inference output
+ * failed; the task and both frames are released,
+ * and *in and *out are set to NULL.
* @retval DAST_SUCCESS if result successfully extracted
*/
DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in,
AVFrame **out, int *status);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 200096862e..e3d7be64b5 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -338,19 +338,20 @@ static void infer_completion_callback(void *args)
OVModel *ov_model = task->model;
SafeQueue *requestq = ov_model->request_queue;
DNNData *outputs;
- int ret;
+ int ret = 0;
DnnContext *ctx = ov_model->ctx;
#if HAVE_OPENVINO2
size_t* dims;
ov_status_e status;
- ov_tensor_t *output_tensor;
+ ov_tensor_t *output_tensor = NULL;
ov_shape_t output_shape = {0};
ov_element_type_e precision;
outputs = av_calloc(ov_model->nb_outputs, sizeof(*outputs));
if (!outputs) {
- av_log(ctx, AV_LOG_ERROR, "Failed to alloc outputs.");
- return;
+ av_log(ctx, AV_LOG_ERROR, "Failed to alloc outputs.\n");
+ ret = AVERROR(ENOMEM);
+ goto end;
}
for (int i = 0; i < ov_model->nb_outputs; i++) {
@@ -359,20 +360,23 @@ static void infer_completion_callback(void *args)
&output_tensor);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR,
- "Failed to get output tensor.");
+ "Failed to get output tensor.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
status = ov_tensor_data(output_tensor, &outputs[i].data);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR,
- "Failed to get output data.");
+ "Failed to get output data.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
status = ov_tensor_get_shape(output_tensor, &output_shape);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to get output port shape.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
dims = output_shape.dims;
@@ -380,6 +384,7 @@ static void infer_completion_callback(void *args)
status = ov_port_get_element_type(ov_model->output_ports[i],
&precision);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to get output port data
type.\n");
+ ret = ov2_map_error(status, NULL);
goto end;
}
outputs[i].dt = precision_to_datatype(precision);
@@ -388,7 +393,13 @@ static void infer_completion_callback(void *args)
outputs[i].dims[1] = output_shape.rank > 2 ? dims[output_shape.rank -
3] : 1;
outputs[i].dims[2] = output_shape.rank > 1 ? dims[output_shape.rank -
2] : 1;
outputs[i].dims[3] = output_shape.rank > 0 ? dims[output_shape.rank -
1] : 1;
- av_assert0(request->lltask_count <= dims[0]);
+ if (dims[0] < request->lltask_count) {
+ av_log(ctx, AV_LOG_ERROR,
+ "Output %d has %zu entries in its first dimension, fewer
than the batch of %u.\n",
+ i, dims[0], request->lltask_count);
+ ret = AVERROR(EINVAL);
+ goto end;
+ }
outputs[i].layout = ctx->ov_option.layout;
outputs[i].scale = ctx->ov_option.scale;
outputs[i].mean = ctx->ov_option.mean;
@@ -460,6 +471,7 @@ static void infer_completion_callback(void *args)
case DFT_ANALYTICS_DETECT:
if (!ov_model->model.detect_post_proc) {
av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post
proc\n");
+ ret = AVERROR(EINVAL);
goto end;
}
ret = ov_model->model.detect_post_proc(task->in_frame, outputs,
@@ -469,6 +481,7 @@ static void infer_completion_callback(void *args)
case DFT_ANALYTICS_CLASSIFY:
if (!ov_model->model.classify_post_proc) {
av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide
post proc\n");
+ ret = AVERROR(EINVAL);
goto end;
}
for (int output_i = 0; output_i < ov_model->nb_outputs;
output_i++) {
@@ -492,6 +505,15 @@ static void infer_completion_callback(void *args)
get_datatype_size(outputs[i].dt);
}
end:
+ /* On failure, finish the lltasks that were not processed with the
+ * error, so that their tasks still complete and get_result reports
+ * DAST_FAIL instead of DAST_NOT_READY forever. */
+ for (int i = 0; i < request->lltask_count; i++) {
+ if (!request->lltasks[i])
+ continue;
+ ff_dnn_complete_lltask(request->lltasks[i]->task, ret < 0 ? ret :
AVERROR_BUG);
+ av_freep(&request->lltasks[i]);
+ }
#if HAVE_OPENVINO2
av_freep(&outputs);
ov_shape_free(&output_shape);
@@ -1364,6 +1386,12 @@ static int get_output_ov(DNNModel *model, const char
*input_name, int input_widt
}
ret = execute_model_ov(request, ov_model->lltask_queue);
+ if (ret != 0)
+ goto err;
+ if (task.status < 0) {
+ ret = task.status;
+ goto err;
+ }
*output_width = task.out_frame->width;
*output_height = task.out_frame->height;
err:
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 6f1ad3b565..da5689bc05 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -194,8 +194,8 @@ struct DNNModule {
// Retrieve inference result. status, if non-NULL, receives the
// negative AVERROR stored on the task when DAST_FAIL is returned
// (i.e. inference completed but frame/detect/classify
- // post-processing failed); the frames are then released and *in
- // and *out are set to NULL.
+ // post-processing or reading the inference output failed); the
+ // frames are then released and *in and *out are set to NULL.
DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in,
AVFrame **out, int *status);
// Flush all the pending tasks.
int (*flush)(const DNNModel *model);
--
2.52.0
From ec7e3fd87ce6d7a49aadb686c52fd201fbfaffb7 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 18:04:30 -0400
Subject: [PATCH 09/10] avfilter/dnn_backend_tf: complete the task when the
completion callback fails
infer_completion_callback() jumps past ff_dnn_complete_lltask() when it
cannot allocate the outputs array or when the model function type is not
supported, and returns early, without giving the request back to the
request queue, when a detect model has no post-processing callback. The
task then never completes:
- In async mode, the default, ff_dnn_get_result() returns
DAST_NOT_READY for it forever, and dnn_processing, derain, sr and
dnn_detect hang.
- In sync mode, execute_model_tf() returns DNN_GENERIC_ERROR, and the
filter fails with "Input/output error" or an unnamed error code
instead of the cause. derain and sr then free the frames that the
unfinished task still holds, and freeing the model frees them again,
a heap-use-after-free. With the missing post-processing callback,
sync mode hangs too.
Pass the error to ff_dnn_complete_lltask() on every path, as is already
done when post-processing fails, so that get_result reports DAST_FAIL
with the error and releases the frames once. Since execute_model_tf()
now succeeds for such a task, get_output_tf() checks the task's status,
as get_output_ov() does, so a failure while probing the output size
still fails the configuration, now with the real error, and leaves the
caller's dimensions unchanged.
The filters cannot reach these branches with a model alone. Each branch
was forced in a test build to check the new behaviour.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_tf.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index ac87884a46..dc8a10c119 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -344,6 +344,12 @@ static int get_output_tf(DNNModel *model, const char
*input_name, int input_widt
}
ret = execute_model_tf(request, tf_model->lltask_queue);
+ if (ret != 0)
+ goto err;
+ if (task.status < 0) {
+ ret = task.status;
+ goto err;
+ }
*output_width = task.out_frame->width;
*output_height = task.out_frame->height;
@@ -704,6 +710,7 @@ static void infer_completion_callback(void *args) {
outputs = av_calloc(task->nb_output, sizeof(*outputs));
if (!outputs) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n");
+ ret = AVERROR(ENOMEM);
goto err;
}
@@ -737,16 +744,20 @@ static void infer_completion_callback(void *args) {
case DFT_ANALYTICS_DETECT:
if (!tf_model->model.detect_post_proc) {
av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post
proc\n");
- return;
+ ret = AVERROR(EINVAL);
+ goto err;
}
ret = tf_model->model.detect_post_proc(task->in_frame, outputs,
task->nb_output, tf_model->model.filter_ctx);
break;
default:
av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
+ ret = AVERROR(ENOSYS);
goto err;
}
- ff_dnn_complete_lltask(task, ret);
err:
+ /* Complete the task on failure too, with the error, so that
+ * get_result reports DAST_FAIL instead of DAST_NOT_READY forever. */
+ ff_dnn_complete_lltask(task, ret);
tf_free_request(infer_request);
av_freep(&outputs);
--
2.52.0
From 4597d09c981834be05dfd9b9af7a65393f080c08 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Mon, 28 Sep 2026 18:04:31 -0400
Subject: [PATCH 10/10] avfilter/dnn_backend_onnx: complete the task when
reading the output fails
infer_completion_callback() jumps past ff_dnn_complete_lltask() on each
of its errors: an output that is not float, or whose rank is not 3 or 4,
a failed ONNX Runtime call, a failed allocation, or an unsupported model
function type. A detection model with a uint8 output, or with rank-2
[N,7] rows, reaches this. The ONNX backend runs only synchronously, so
this does not hang. execute_model_onnx() returns DNN_GENERIC_ERROR
instead, and the filter fails without the cause: dnn_detect with
"Input/output error", and dnn_processing, whose output-size probe
fails, with "Error number -558779972 occurred".
Pass the error to ff_dnn_complete_lltask() on every path, as the
TensorFlow backend now does, so that get_result reports DAST_FAIL and
the filter returns the error: ENOSYS for an output the backend cannot
read, AVERROR_EXTERNAL for a failed ONNX Runtime call, and ENOMEM.
Since execute_model_onnx() now succeeds for such a task,
get_output_onnx() checks the task's status, as get_output_ov() does,
and leaves the caller's dimensions unchanged on failure.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_onnx.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index ffa150034f..722c7b17d8 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -593,6 +593,7 @@ static void infer_completion_callback(void *args)
outputs = av_calloc(infer_request->nb_outputs, sizeof(*outputs));
if (!outputs) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate output DNNData array\n");
+ ret = AVERROR(ENOMEM);
goto err;
}
@@ -602,6 +603,7 @@ static void infer_completion_callback(void *args)
if (status != NULL) {
av_log(ctx, AV_LOG_ERROR, "Failed to get output tensor[%u]
type/shape\n", i);
g_ort->ReleaseStatus(status);
+ ret = AVERROR_EXTERNAL;
goto err;
}
@@ -610,6 +612,7 @@ static void infer_completion_callback(void *args)
av_log(ctx, AV_LOG_ERROR, "Failed to get output tensor[%u]
dimension count\n", i);
g_ort->ReleaseStatus(status);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR_EXTERNAL;
goto err;
}
@@ -617,6 +620,7 @@ static void infer_completion_callback(void *args)
if (!dims) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate dims array\n");
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR(ENOMEM);
goto err;
}
@@ -626,6 +630,7 @@ static void infer_completion_callback(void *args)
g_ort->ReleaseStatus(status);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR_EXTERNAL;
goto err;
}
@@ -636,6 +641,7 @@ static void infer_completion_callback(void *args)
i, d, dims[d]);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR(EINVAL);
goto err;
}
}
@@ -646,6 +652,7 @@ static void infer_completion_callback(void *args)
g_ort->ReleaseStatus(status);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR_EXTERNAL;
goto err;
}
if (tensor_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT) {
@@ -655,6 +662,7 @@ static void infer_completion_callback(void *args)
"Unsupported output tensor[%u] data type, only float
supported\n", i);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR(ENOSYS);
goto err;
}
@@ -678,6 +686,7 @@ static void infer_completion_callback(void *args)
"Support for %zu-dimensional output (tensor[%u])", num_dims,
i);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR(ENOSYS);
goto err;
}
@@ -687,6 +696,7 @@ static void infer_completion_callback(void *args)
g_ort->ReleaseStatus(status);
av_free(dims);
g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ ret = AVERROR_EXTERNAL;
goto err;
}
@@ -715,12 +725,15 @@ static void infer_completion_callback(void *args)
break;
default:
avpriv_report_missing_feature(ctx, "model function type %d",
onnx_model->model.func_type);
+ ret = AVERROR(ENOSYS);
goto err;
}
- ff_dnn_complete_lltask(task, ret);
-
err:
+ /* Complete the task on failure too, with the error, so that
+ * get_result reports DAST_FAIL and execute_model_onnx() does not
+ * see an unfinished task. */
+ ff_dnn_complete_lltask(task, ret);
av_freep(&outputs);
av_freep(&request->lltask);
onnx_free_request(infer_request);
@@ -813,6 +826,12 @@ static int get_output_onnx(DNNModel *model, const char
*input_name, int input_wi
}
ret = execute_model_onnx(request, onnx_model->lltask_queue);
+ if (ret != 0)
+ goto err;
+ if (task.status < 0) {
+ ret = task.status;
+ goto err;
+ }
*output_width = task.out_frame->width;
*output_height = task.out_frame->height;
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
