PR #25508 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25508 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25508.patch
Three small follow-ups to the OpenVINO backend that came up in review of #24845 and #24736. 1. **Fix the input resize for NHWC models.** With input_resizable, get_output_ov() writes the frame height and width to dims[2] and dims[3], which are the NCHW positions. For an NHWC model this resizes the wrong axes. Use dnn_get_height_idx_by_layout() and dnn_get_width_idx_by_layout() instead; for NCHW and the default layout these are still 2 and 3. 2. **Free the input tensor on fill errors.** fill_model_input_ov() returned without freeing the tensor from ov_tensor_create() when ov_tensor_data() or ov_infer_request_set_input_tensor() failed. 3. **Document the four-dimensional input requirement** for dnn_detect and dnn_processing, which share the check added in 6ef0b9b0fc. Testing, OpenVINO 2026.4 on macOS arm64, master vs this branch: - NHWC identity model with input_resizable=1: master produces output that differs from the NCHW identity model at the same size, and this branch produces identical output. NCHW with and without resizing, and NHWC without resizing, give identical output before and after. - A two-input model that makes ov_infer_request_set_input_tensor() fail: leaks(1) reports 8 leaks (164352 bytes) on master and 1 leak (32 bytes, unrelated) with this branch, for both async=1 and async=0. - Builds without new warnings; `make fate-source` and `git diff --check` pass. The classify results of dnn_detect followed by dnn_classify with OpenVINO change from run to run on master as well, so I could not use them as a before/after comparison. I will look at that separately. >From 4b36df4b29df4f395135c133e1e14b84062346ac Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Mon, 26 Oct 2026 22:39:25 -0400 Subject: [PATCH 1/3] avfilter/dnn_backend_openvino: fix the input resize for NHWC models With input_resizable=1, get_output_ov() reshapes the model input to the frame size by writing the height and width into dims[2] and dims[3] of the input port shape. Those are the H and W positions of an NCHW shape only. For an NHWC model they are W and C, so the reshape sets the width to the frame height and the channel count to the frame width, fails with "Failed to reszie model input.", and the filter cannot be configured. Index the shape with dnn_get_height_idx_by_layout() and dnn_get_width_idx_by_layout() on the input layout, which get_input_ov() has already resolved, as it does when it marks the same dimensions as resizable. For NCHW and the default layout these return 2 and 3, so the behaviour there is unchanged. Reported by Forgejo_Fairy in the review of #24845. Signed-off-by: Jake Fineman <[email protected]> --- libavfilter/dnn/dnn_backend_openvino.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 7ec3bb16328..4949a5cd089 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -1310,8 +1310,8 @@ static int get_output_ov(DNNModel *model, const char *input_name, int input_widt av_log(ctx, AV_LOG_ERROR, "Failed to create shape for model input resize.\n"); return ov2_map_error(status, NULL); } - input_shape.dims[2] = input_height; - input_shape.dims[3] = input_width; + input_shape.dims[dnn_get_height_idx_by_layout(ctx->ov_option.layout)] = input_height; + input_shape.dims[dnn_get_width_idx_by_layout(ctx->ov_option.layout)] = input_width; status = ov_shape_to_partial_shape(input_shape, &partial_shape); ov_shape_free(&input_shape); -- 2.54.0 >From f9bb871de9b7647dd0441b7dc6e88a13c9ee8e00 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Mon, 26 Oct 2026 22:39:25 -0400 Subject: [PATCH 2/3] avfilter/dnn_backend_openvino: free the input tensor on fill errors fill_model_input_ov() creates the input tensor with ov_tensor_create() and frees it after the batch has been filled. If ov_tensor_data() or ov_infer_request_set_input_tensor() fails, the function returns without freeing it, and the tensor and its buffer leak. The second failure is reachable without fault injection, e.g. with a model that has more than one input, because ov_infer_request_set_input_tensor() requires a model with exactly one input. Free the tensor on both error paths. Reported by Forgejo_Fairy in the review of #24845. Signed-off-by: Jake Fineman <[email protected]> --- libavfilter/dnn/dnn_backend_openvino.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 4949a5cd089..257e79f524a 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -291,11 +291,13 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) status = ov_tensor_data(tensor, &input.data); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to get input data.\n"); + ov_tensor_free(tensor); return ov2_map_error(status, NULL); } status = ov_infer_request_set_input_tensor(request->infer_request, tensor); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to Set an input tensor for the model.\n"); + ov_tensor_free(tensor); return ov2_map_error(status, NULL); } #endif -- 2.54.0 >From 42934fd587f63de91c499d9a0fa8652273bbf223 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Mon, 26 Oct 2026 22:39:25 -0400 Subject: [PATCH 3/3] doc/filters: note that OpenVINO models need a 4-D input Since 6ef0b9b0fc the OpenVINO backend rejects a model whose input is not 4-dimensional when the filter queries the model input, which dnn_detect and dnn_processing both do while configuring. Document the requirement in the model option of both filters. Suggested by Forgejo_Fairy in the review of #24736. Signed-off-by: Jake Fineman <[email protected]> --- doc/filters.texi | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/filters.texi b/doc/filters.texi index 37f9123df49..c92ac6c48be 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -12214,6 +12214,7 @@ ONNX Runtime backend. @item model Set path to model file specifying network architecture and its parameters. Note that different backends use different file formats. +With the OpenVINO backend, the model input must have four dimensions. @item input Set the input name of the dnn network. @@ -12373,6 +12374,7 @@ must either be consumed by a D3D12 encoder or downloaded explicitly with Set path to model file specifying network architecture and its parameters. Note that different backends use different file formats. TensorFlow, OpenVINO, Libtorch, and ONNX Runtime backends can load files for only their respective formats. +With the OpenVINO backend, the model input must have four dimensions. @item input Set the input name of the dnn network. Required for the TensorFlow backend; -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
