[FFmpeg-devel] [PATCH V7 2/3] lavfi/dnn: Modified DNN native backend related tools and docs.
Will remove native backend, so change the default backend in filters,
and also remove the python scripts which generate native model file.
Signed-off-by: Ting Fu
---
doc/filters.texi| 39 +-
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
libavfilter/vf_sr.c | 2 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607
tools/python/convert_header.py | 26 -
7 files changed, 7 insertions(+), 727 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 5dde79919a..f1f87a24fd 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11338,9 +11338,6 @@ See
@url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11361,21 +11358,16 @@ Specify which DNN backend to use for model loading
and execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load
files for only its format.
@end table
To get full functionality (such as async execution), please use the
@ref{dnn_processing} filter.
@@ -11699,9 +11691,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11717,14 +11706,9 @@ be needed if the header files and libraries are not
installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO
and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb)
by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11750,12 +11734,6 @@ Remove rain in rgb24 frame with can.pb (see
@ref{derain} filter):
./ffmpeg -i rain.jpg -vf
format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y
derain.jpg
@end example
-@item
-Halve the pixel value of the frame with format gray32f:
-@example
-ffmpeg -i input.jpg -vf
format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-y out.native.png
-@end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with
yuv420p (planar YUV formats supported):
@example
@@ -21813,9 +21791,6 @@ Training scripts as well as scripts for model file
(.pb) saving can be found at
@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21824,9 +21799,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21834,13 +21806,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow backend
-can load files for both formats, while native backend can load files for only
-its format
[FFmpeg-devel] [PATCH V7 1/3] lavfi/dnn: modify dnn interface for removing native backend
Native backend will be removed in following commits, so change the
dnn interface and modify the error message in it first.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_interface.c | 10 +-
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c
index 554a36b0dc..5b1695a1dd 100644
--- a/libavfilter/dnn/dnn_interface.c
+++ b/libavfilter/dnn/dnn_interface.c
@@ -24,7 +24,6 @@
*/
#include "../dnn_interface.h"
-#include "dnn_backend_native.h"
#include "dnn_backend_tf.h"
#include "dnn_backend_openvino.h"
#include "libavutil/mem.h"
@@ -39,13 +38,6 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
}
switch(backend_type){
-case DNN_NATIVE:
-dnn_module->load_model = _dnn_load_model_native;
-dnn_module->execute_model = _dnn_execute_model_native;
-dnn_module->get_result = _dnn_get_result_native;
-dnn_module->flush = _dnn_flush_native;
-dnn_module->free_model = _dnn_free_model_native;
-break;
case DNN_TF:
#if (CONFIG_LIBTENSORFLOW == 1)
dnn_module->load_model = _dnn_load_model_tf;
@@ -71,7 +63,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
#endif
break;
default:
-av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or
tensorflow\n");
+av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or
enabled.\n");
av_freep(_module);
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 3/3] lavfi/dnn: add error info for TF backend filling task failure
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 5d809a8694..bafd802abf 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -1168,6 +1168,7 @@ int ff_dnn_execute_model_tf(const DNNModel *model,
DNNExecBaseParams *exec_param
ret = ff_dnn_fill_task(task, exec_params, tf_model, ctx->options.async, 1);
if (ret != 0) {
+av_log(ctx, AV_LOG_ERROR, "Fill task with invalid parameter(s).\n");
av_freep();
return ret;
}
--
2.25.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/3] lavfi/dnn: fix mem leak in TF backend error handle
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index fb1a5f1350..5d809a8694 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -176,6 +176,7 @@ static int tf_start_inference(void *args)
if (TF_GetCode(request->status) != TF_OK) {
av_log(_model->ctx, AV_LOG_ERROR, "%s",
TF_Message(request->status));
tf_free_request(infer_request);
+av_freep();
return DNN_GENERIC_ERROR;
}
return 0;
@@ -466,6 +467,7 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
{
TF_DeleteGraph(tf_model->graph);
tf_model->graph = NULL;
+av_freep(_config);
av_log(ctx, AV_LOG_ERROR, "Failed to create new session with model
graph\n");
return DNN_GENERIC_ERROR;
}
@@ -484,6 +486,7 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
tf_model->graph = NULL;
TF_DeleteStatus(tf_model->status);
tf_model->status = NULL;
+av_freep(_config);
av_log(ctx, AV_LOG_ERROR, "Failed to run session when
initializing\n");
return DNN_GENERIC_ERROR;
}
@@ -1177,12 +1180,14 @@ int ff_dnn_execute_model_tf(const DNNModel *model,
DNNExecBaseParams *exec_param
ret = extract_lltask_from_task(task, tf_model->lltask_queue);
if (ret != 0) {
+av_freep();
av_log(ctx, AV_LOG_ERROR, "unable to extract last level task from
task.\n");
return ret;
}
request = ff_safe_queue_pop_front(tf_model->request_queue);
if (!request) {
+av_freep();
av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n");
return AVERROR(EINVAL);
}
--
2.25.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 1/3] lavfi/dnn: fix corruption when TF backend infer failed
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 13 -
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 3b5084b67b..fb1a5f1350 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -176,9 +176,6 @@ static int tf_start_inference(void *args)
if (TF_GetCode(request->status) != TF_OK) {
av_log(_model->ctx, AV_LOG_ERROR, "%s",
TF_Message(request->status));
tf_free_request(infer_request);
-if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) {
-destroy_request_item();
-}
return DNN_GENERIC_ERROR;
}
return 0;
@@ -439,7 +436,9 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
TF_DeleteBuffer(graph_def);
if (TF_GetCode(tf_model->status) != TF_OK){
TF_DeleteGraph(tf_model->graph);
+tf_model->graph = NULL;
TF_DeleteStatus(tf_model->status);
+tf_model->status = NULL;
av_log(ctx, AV_LOG_ERROR, "Failed to import serialized graph to model
graph\n");
av_freep(_config);
return DNN_GENERIC_ERROR;
@@ -453,7 +452,7 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
av_freep(_config);
if (TF_GetCode(tf_model->status) != TF_OK) {
TF_DeleteGraph(tf_model->graph);
-TF_DeleteStatus(tf_model->status);
+tf_model->graph = NULL;
TF_DeleteSessionOptions(sess_opts);
av_log(ctx, AV_LOG_ERROR, "Failed to set config for sess options
with %s\n",
tf_model->ctx.options.sess_config);
@@ -466,7 +465,7 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
if (TF_GetCode(tf_model->status) != TF_OK)
{
TF_DeleteGraph(tf_model->graph);
-TF_DeleteStatus(tf_model->status);
+tf_model->graph = NULL;
av_log(ctx, AV_LOG_ERROR, "Failed to create new session with model
graph\n");
return DNN_GENERIC_ERROR;
}
@@ -480,8 +479,11 @@ static int load_tf_model(TFModel *tf_model, const char
*model_filename)
if (TF_GetCode(tf_model->status) != TF_OK)
{
TF_DeleteSession(tf_model->session, tf_model->status);
+tf_model->session = NULL;
TF_DeleteGraph(tf_model->graph);
+tf_model->graph = NULL;
TF_DeleteStatus(tf_model->status);
+tf_model->status = NULL;
av_log(ctx, AV_LOG_ERROR, "Failed to run session when
initializing\n");
return DNN_GENERIC_ERROR;
}
@@ -940,6 +942,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename,
DNNFunctionType func_
return model;
err:
+model->model = tf_model;
ff_dnn_free_model_tf();
return NULL;
}
--
2.25.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V6 2/3] lavfi/dnn: Modified DNN native backend related tools and docs.
Deleted the native backend related files in 'tools' dir. Modify its'
docs and codes mentioned in such docs.
Signed-off-by: Ting Fu
---
doc/filters.texi| 39 +-
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
libavfilter/vf_sr.c | 2 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607
tools/python/convert_header.py | 26 -
7 files changed, 7 insertions(+), 727 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 7a7b2ba4e7..726d2fd7e2 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11305,9 +11305,6 @@ See
@url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11328,21 +11325,16 @@ Specify which DNN backend to use for model loading
and execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load
files for only its format.
@end table
To get full functionality (such as async execution), please use the
@ref{dnn_processing} filter.
@@ -11666,9 +11658,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11684,14 +11673,9 @@ be needed if the header files and libraries are not
installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO
and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb)
by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11717,12 +11701,6 @@ Remove rain in rgb24 frame with can.pb (see
@ref{derain} filter):
./ffmpeg -i rain.jpg -vf
format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y
derain.jpg
@end example
-@item
-Halve the pixel value of the frame with format gray32f:
-@example
-ffmpeg -i input.jpg -vf
format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-y out.native.png
-@end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with
yuv420p (planar YUV formats supported):
@example
@@ -21750,9 +21728,6 @@ Training scripts as well as scripts for model file
(.pb) saving can be found at
@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21761,9 +21736,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21771,13 +21743,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow backend
-can load files for both formats, while native backend can load files for only
-its format.
+Note that different backends use
[FFmpeg-devel] [PATCH V6 1/3] lavfi/dnn: Mark native backend as unsupported
Native is deprecated value for backed_type option. Modify related error
message.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_interface.c | 10 +-
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c
index 554a36b0dc..5b1695a1dd 100644
--- a/libavfilter/dnn/dnn_interface.c
+++ b/libavfilter/dnn/dnn_interface.c
@@ -24,7 +24,6 @@
*/
#include "../dnn_interface.h"
-#include "dnn_backend_native.h"
#include "dnn_backend_tf.h"
#include "dnn_backend_openvino.h"
#include "libavutil/mem.h"
@@ -39,13 +38,6 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
}
switch(backend_type){
-case DNN_NATIVE:
-dnn_module->load_model = _dnn_load_model_native;
-dnn_module->execute_model = _dnn_execute_model_native;
-dnn_module->get_result = _dnn_get_result_native;
-dnn_module->flush = _dnn_flush_native;
-dnn_module->free_model = _dnn_free_model_native;
-break;
case DNN_TF:
#if (CONFIG_LIBTENSORFLOW == 1)
dnn_module->load_model = _dnn_load_model_tf;
@@ -71,7 +63,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
#endif
break;
default:
-av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or
tensorflow\n");
+av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or
enabled.\n");
av_freep(_module);
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V5 2/3] lavfi/dnn: Modified DNN native backend related tools and docs.
Deleted the native backend related files in 'tools' dir. Modify its'
docs and codes mentioned in such docs.
Signed-off-by: Ting Fu
---
doc/filters.texi| 43 +-
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
libavfilter/vf_sr.c | 2 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607
tools/python/convert_header.py | 26 -
7 files changed, 7 insertions(+), 731 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 3a54c68f3e..6fcb506b38 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11275,9 +11275,6 @@ See
@url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11298,21 +11295,16 @@ Specify which DNN backend to use for model loading
and execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load
files for only its format.
@end table
To get full functionality (such as async execution), please use the
@ref{dnn_processing} filter.
@@ -11636,9 +11628,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11654,14 +11643,9 @@ be needed if the header files and libraries are not
installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO
and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb)
by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11687,12 +11671,6 @@ Remove rain in rgb24 frame with can.pb (see
@ref{derain} filter):
./ffmpeg -i rain.jpg -vf
format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y
derain.jpg
@end example
-@item
-Halve the pixel value of the frame with format gray32f:
-@example
-ffmpeg -i input.jpg -vf
format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-y out.native.png
-@end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with
yuv420p (planar YUV formats supported):
@example
@@ -21702,13 +21680,6 @@ Efficient Sub-Pixel Convolutional Neural Network model
(ESPCN).
See @url{https://arxiv.org/abs/1609.05158}.
@end itemize
-Training scripts as well as scripts for model file (.pb) saving can be found at
-@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
-is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21717,9 +21688,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21727,13 +21695,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow
[FFmpeg-devel] [PATCH V5 1/3] lavfi/dnn: Mark native backend as unsupported
Native is deprecated value for backed_type option. Modify related error
message.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_interface.c | 10 +-
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c
index 554a36b0dc..5b1695a1dd 100644
--- a/libavfilter/dnn/dnn_interface.c
+++ b/libavfilter/dnn/dnn_interface.c
@@ -24,7 +24,6 @@
*/
#include "../dnn_interface.h"
-#include "dnn_backend_native.h"
#include "dnn_backend_tf.h"
#include "dnn_backend_openvino.h"
#include "libavutil/mem.h"
@@ -39,13 +38,6 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
}
switch(backend_type){
-case DNN_NATIVE:
-dnn_module->load_model = _dnn_load_model_native;
-dnn_module->execute_model = _dnn_execute_model_native;
-dnn_module->get_result = _dnn_get_result_native;
-dnn_module->flush = _dnn_flush_native;
-dnn_module->free_model = _dnn_free_model_native;
-break;
case DNN_TF:
#if (CONFIG_LIBTENSORFLOW == 1)
dnn_module->load_model = _dnn_load_model_tf;
@@ -71,7 +63,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
#endif
break;
default:
-av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or
tensorflow\n");
+av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or
enabled.\n");
av_freep(_module);
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V4 2/3] lavfi/dnn: Delete DNN native backend releated tools and docs.
Signed-off-by: Ting Fu
---
doc/filters.texi| 43 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607
tools/python/convert_header.py | 26 -
4 files changed, 4 insertions(+), 728 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 9c32339141..797d1c9fe2 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11222,9 +11222,6 @@ See
@url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11245,21 +11242,16 @@ Specify which DNN backend to use for model loading
and execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load
files for only its format.
@end table
To get full functionality (such as async execution), please use the
@ref{dnn_processing} filter.
@@ -11583,9 +11575,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11601,14 +11590,9 @@ be needed if the header files and libraries are not
installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO
and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb)
by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11634,12 +11618,6 @@ Remove rain in rgb24 frame with can.pb (see
@ref{derain} filter):
./ffmpeg -i rain.jpg -vf
format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y
derain.jpg
@end example
-@item
-Halve the pixel value of the frame with format gray32f:
-@example
-ffmpeg -i input.jpg -vf
format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-y out.native.png
-@end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with
yuv420p (planar YUV formats supported):
@example
@@ -21648,13 +21626,6 @@ Efficient Sub-Pixel Convolutional Neural Network model
(ESPCN).
See @url{https://arxiv.org/abs/1609.05158}.
@end itemize
-Training scripts as well as scripts for model file (.pb) saving can be found at
-@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
-is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21663,9 +21634,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21673,13 +21641,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow backend
-can load files for both formats, while native backend can load files for only
-its format.
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend
+can load files for only its format.
@item scale_factor
Set scale factor
[FFmpeg-devel] [PATCH V4 1/3] lavfi/dnn: Mark native backend as unsupported
Native is deprecated value for backed_type option. Modify realted error
message.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_interface.c | 10 +-
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c
index 554a36b0dc..5b1695a1dd 100644
--- a/libavfilter/dnn/dnn_interface.c
+++ b/libavfilter/dnn/dnn_interface.c
@@ -24,7 +24,6 @@
*/
#include "../dnn_interface.h"
-#include "dnn_backend_native.h"
#include "dnn_backend_tf.h"
#include "dnn_backend_openvino.h"
#include "libavutil/mem.h"
@@ -39,13 +38,6 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
}
switch(backend_type){
-case DNN_NATIVE:
-dnn_module->load_model = _dnn_load_model_native;
-dnn_module->execute_model = _dnn_execute_model_native;
-dnn_module->get_result = _dnn_get_result_native;
-dnn_module->flush = _dnn_flush_native;
-dnn_module->free_model = _dnn_free_model_native;
-break;
case DNN_TF:
#if (CONFIG_LIBTENSORFLOW == 1)
dnn_module->load_model = _dnn_load_model_tf;
@@ -71,7 +63,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
#endif
break;
default:
-av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or
tensorflow\n");
+av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or
enabled.\n");
av_freep(_module);
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V3 2/3] lavfi/dnn: Delete DNN native backend releated tools and docs.
Signed-off-by: Ting Fu
---
doc/filters.texi| 43 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607
tools/python/convert_header.py | 26 -
4 files changed, 4 insertions(+), 728 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 9c32339141..797d1c9fe2 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11222,9 +11222,6 @@ See
@url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11245,21 +11242,16 @@ Specify which DNN backend to use for model loading
and execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load
files for only its format.
@end table
To get full functionality (such as async execution), please use the
@ref{dnn_processing} filter.
@@ -11583,9 +11575,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11601,14 +11590,9 @@ be needed if the header files and libraries are not
installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO
and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb)
by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11634,12 +11618,6 @@ Remove rain in rgb24 frame with can.pb (see
@ref{derain} filter):
./ffmpeg -i rain.jpg -vf
format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y
derain.jpg
@end example
-@item
-Halve the pixel value of the frame with format gray32f:
-@example
-ffmpeg -i input.jpg -vf
format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-y out.native.png
-@end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with
yuv420p (planar YUV formats supported):
@example
@@ -21648,13 +21626,6 @@ Efficient Sub-Pixel Convolutional Neural Network model
(ESPCN).
See @url{https://arxiv.org/abs/1609.05158}.
@end itemize
-Training scripts as well as scripts for model file (.pb) saving can be found at
-@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
-is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21663,9 +21634,6 @@ Specify which DNN backend to use for model loading and
execution. This option ac
the following values:
@table @samp
-@item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21673,13 +21641,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow backend
-can load files for both formats, while native backend can load files for only
-its format.
+Note that different backends use different file formats. TensorFlow, OpenVINO
backend
+can load files for only its format.
@item scale_factor
Set scale factor
[FFmpeg-devel] [PATCH V3 1/3] lavfi/dnn: Mark native backend as deprecated
Mark native as deprecated for backed_type option. Modify realted error
message.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_interface.c | 12
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c
index 554a36b0dc..12d36f7fed 100644
--- a/libavfilter/dnn/dnn_interface.c
+++ b/libavfilter/dnn/dnn_interface.c
@@ -24,7 +24,6 @@
*/
#include "../dnn_interface.h"
-#include "dnn_backend_native.h"
#include "dnn_backend_tf.h"
#include "dnn_backend_openvino.h"
#include "libavutil/mem.h"
@@ -40,12 +39,9 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
switch(backend_type){
case DNN_NATIVE:
-dnn_module->load_model = _dnn_load_model_native;
-dnn_module->execute_model = _dnn_execute_model_native;
-dnn_module->get_result = _dnn_get_result_native;
-dnn_module->flush = _dnn_flush_native;
-dnn_module->free_model = _dnn_free_model_native;
-break;
+av_log(NULL, AV_LOG_ERROR, "Native backend is deprecated, please use
other supported DNN backends.\n");
+av_freep(_module);
+return NULL;
case DNN_TF:
#if (CONFIG_LIBTENSORFLOW == 1)
dnn_module->load_model = _dnn_load_model_tf;
@@ -71,7 +67,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type)
#endif
break;
default:
-av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or
tensorflow\n");
+av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or
enabled.\n");
av_freep(_module);
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 1/2] lavfi/dnn: Modify error message for incorrect backend_type
Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_interface.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 554a36b0dc..fa484c0905 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -71,7 +71,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #endif break; default: -av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or tensorflow\n"); +av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or enabled.\n"); av_freep(_module); return NULL; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 1/2] lavfi/dnn: Modify error message for incorrect backend_type
Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_interface.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 554a36b0dc..fa484c0905 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -71,7 +71,7 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #endif break; default: -av_log(NULL, AV_LOG_ERROR, "Module backend_type is not native or tensorflow\n"); +av_log(NULL, AV_LOG_ERROR, "Module backend_type is not supported or enabled.\n"); av_freep(_module); return NULL; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] lavfi/dnn: Fix OpenVINO missing model file corrupt issue.
DNN OpenVINO backend would not report missing model file if it does not
exist. It would corrupt directly with out any error infomation. This commit
would check both .xml and .bin file existance before loading model.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index b494f26f55..47e3fc8280 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -34,6 +34,7 @@
#include "../internal.h"
#include "safe_queue.h"
#include
+#include
#include "dnn_backend_common.h"
typedef struct OVOptions{
@@ -728,7 +729,7 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
OVContext *ctx = NULL;
IEStatusCode status;
size_t node_count = 0;
-char *node_name = NULL;
+char *node_name, *model_binary_file, *extension_chr = NULL;
model = av_mallocz(sizeof(DNNModel));
if (!model){
@@ -758,6 +759,22 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
if (status != OK)
goto err;
+//file existance check
+model_binary_file = (char *)malloc(sizeof(model_filename));
+strcpy(model_binary_file, model_filename);
+extension_chr = strstr(model_binary_file, ".xml");
+if (!extension_chr) {
+av_log(ctx, AV_LOG_ERROR, "Model file name \"%s\" with incorrect
extension.",
+ model_filename);
+goto err;
+}
+strcpy(extension_chr, ".bin");
+if (access(model_filename, F_OK) || access(model_binary_file, F_OK)) {
+av_log(ctx, AV_LOG_ERROR, "Model file \"%s\" or \"%s\" does not
exist.",
+ model_filename, model_binary_file);
+goto err;
+}
+
status = ie_core_read_network(ov_model->core, model_filename, NULL,
_model->network);
if (status != OK) {
ie_version_t ver;
@@ -806,6 +823,7 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
return model;
err:
+free(model_binary_file);
ff_dnn_free_model_ov();
return NULL;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2] lavf/dnn: dump OpenVINO model input/output names to OVMdel struct.
Dump all input/output names to OVModel struct. In case other funcs use
them for reporting errors or locating issues.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 66 +++---
1 file changed, 48 insertions(+), 18 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index cf012aca4c..b494f26f55 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -58,6 +58,8 @@ typedef struct OVModel{
SafeQueue *request_queue; // holds OVRequestItem
Queue *task_queue; // holds TaskItem
Queue *lltask_queue; // holds LastLevelTaskItem
+const char *all_input_names;
+const char *all_output_names;
} OVModel;
// one request for one call to openvino
@@ -211,19 +213,9 @@ static void infer_completion_callback(void *args)
status = ie_infer_request_get_blob(request->infer_request,
task->output_names[0], _blob);
if (status != OK) {
-//incorrect output name
-char *model_output_name = NULL;
-char *all_output_names = NULL;
-size_t model_output_count = 0;
-av_log(ctx, AV_LOG_ERROR, "Failed to get model output data\n");
-status = ie_network_get_outputs_number(ov_model->network,
_output_count);
-for (size_t i = 0; i < model_output_count; i++) {
-status = ie_network_get_output_name(ov_model->network, i,
_output_name);
-APPEND_STRING(all_output_names, model_output_name)
-}
av_log(ctx, AV_LOG_ERROR,
"output \"%s\" may not correct, all output(s) are: \"%s\"\n",
- task->output_names[0], all_output_names);
+ task->output_names[0], ov_model->all_output_names);
return;
}
@@ -336,13 +328,23 @@ static int init_model_ov(OVModel *ov_model, const char
*input_name, const char *
// while we pass NHWC data from FFmpeg to openvino
status = ie_network_set_input_layout(ov_model->network, input_name, NHWC);
if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input
%s\n", input_name);
+if (status == NOT_FOUND) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, failed
to set input layout as NHWC, "\
+ "all input(s) are: \"%s\"\n",
input_name, ov_model->all_input_names);
+} else{
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input
%s\n", input_name);
+}
ret = DNN_GENERIC_ERROR;
goto err;
}
status = ie_network_set_output_layout(ov_model->network, output_name,
NHWC);
if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output
%s\n", output_name);
+if (status == NOT_FOUND) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, failed
to set output layout as NHWC, "\
+ "all output(s) are: \"%s\"\n",
input_name, ov_model->all_output_names);
+} else{
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output
%s\n", output_name);
+}
ret = DNN_GENERIC_ERROR;
goto err;
}
@@ -505,7 +507,6 @@ static int get_input_ov(void *model, DNNData *input, const
char *input_name)
OVModel *ov_model = model;
OVContext *ctx = _model->ctx;
char *model_input_name = NULL;
-char *all_input_names = NULL;
IEStatusCode status;
size_t model_input_count = 0;
dimensions_t dims;
@@ -538,15 +539,12 @@ static int get_input_ov(void *model, DNNData *input,
const char *input_name)
input->width= input_resizable ? -1 : dims.dims[3];
input->dt = precision_to_datatype(precision);
return 0;
-} else {
-//incorrect input name
-APPEND_STRING(all_input_names, model_input_name)
}
ie_network_name_free(_input_name);
}
-av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, all input(s)
are: \"%s\"\n", input_name, all_input_names);
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, all input(s)
are: \"%s\"\n", input_name, ov_model->all_input_names);
return AVERROR(EINVAL);
}
@@ -729,6 +727,8 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
OVModel *ov_model = NULL;
OVContext *ctx = NULL;
IEStatusCode status;
+size_t node_count = 0;
+char *node_name = NULL;
model = av_mallocz(sizeof(DNNModel));
if (!model){
@@ -744,6 +744,8 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionTyp
[FFmpeg-devel] [PATCH] lavf/dnn: refine the error log of OpenVINO incorrect input/output name.
Refine the error report of OpenVINO backend incorrect input or output node
name, help the users to locate issue.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 54 ++
1 file changed, 47 insertions(+), 7 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index cf012aca4c..02d40327a5 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -58,6 +58,8 @@ typedef struct OVModel{
SafeQueue *request_queue; // holds OVRequestItem
Queue *task_queue; // holds TaskItem
Queue *lltask_queue; // holds LastLevelTaskItem
+char *all_input_names;
+char *all_output_names;
} OVModel;
// one request for one call to openvino
@@ -336,13 +338,23 @@ static int init_model_ov(OVModel *ov_model, const char
*input_name, const char *
// while we pass NHWC data from FFmpeg to openvino
status = ie_network_set_input_layout(ov_model->network, input_name, NHWC);
if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input
%s\n", input_name);
+if (status == NOT_FOUND) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, failed
to set input layout as NHWC, "\
+ "all input(s) are: \"%s\"\n",
input_name, ov_model->all_input_names);
+} else{
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input
%s\n", input_name);
+}
ret = DNN_GENERIC_ERROR;
goto err;
}
status = ie_network_set_output_layout(ov_model->network, output_name,
NHWC);
if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output
%s\n", output_name);
+if (status == NOT_FOUND) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, failed
to set output layout as NHWC, "\
+ "all output(s) are: \"%s\"\n",
input_name, ov_model->all_output_names);
+} else{
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output
%s\n", output_name);
+}
ret = DNN_GENERIC_ERROR;
goto err;
}
@@ -505,7 +517,6 @@ static int get_input_ov(void *model, DNNData *input, const
char *input_name)
OVModel *ov_model = model;
OVContext *ctx = _model->ctx;
char *model_input_name = NULL;
-char *all_input_names = NULL;
IEStatusCode status;
size_t model_input_count = 0;
dimensions_t dims;
@@ -538,15 +549,12 @@ static int get_input_ov(void *model, DNNData *input,
const char *input_name)
input->width= input_resizable ? -1 : dims.dims[3];
input->dt = precision_to_datatype(precision);
return 0;
-} else {
-//incorrect input name
-APPEND_STRING(all_input_names, model_input_name)
}
ie_network_name_free(_input_name);
}
-av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, all input(s)
are: \"%s\"\n", input_name, all_input_names);
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, all input(s)
are: \"%s\"\n", input_name, ov_model->all_input_names);
return AVERROR(EINVAL);
}
@@ -729,6 +737,8 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
OVModel *ov_model = NULL;
OVContext *ctx = NULL;
IEStatusCode status;
+size_t node_count = 0;
+char *node_name = NULL;
model = av_mallocz(sizeof(DNNModel));
if (!model){
@@ -744,6 +754,8 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
ov_model->model = model;
ov_model->ctx.class = _openvino_class;
ctx = _model->ctx;
+ov_model->all_input_names = NULL;
+ov_model->all_output_names = NULL;
//parse options
av_opt_set_defaults(ctx);
@@ -767,6 +779,34 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
DNNFunctionType func_
goto err;
}
+//get all the input and output names
+status = ie_network_get_inputs_number(ov_model->network, _count);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get input count\n");
+goto err;
+}
+for (size_t i = 0; i < node_count; i++) {
+status = ie_network_get_input_name(ov_model->network, i, _name);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get No.%d input's name\n",
(int)i);
+goto err;
+}
+APPEND_STRING(ov_model->all_input_names, node_name)
+}
+status = ie_network_get_outputs_number(ov_model->network, _count);
+if (sta
[FFmpeg-devel] [PATCH] lavf/sr: fix the segmentation fault caused by incorrect input frame free.
This issue would cause segmetaion fault when running srcnn model with
sr filter by TensorFlow backend. This filter would free the frame incorectly.
Signed-off-by: Ting Fu
---
libavfilter/vf_sr.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index 0890c8ba18..cb24c096ce 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -159,8 +159,9 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
sws_scale(ctx->sws_uv_scale, (const uint8_t **)(in->data + 2),
in->linesize + 2,
0, ctx->sws_uv_height, out->data + 2, out->linesize + 2);
}
-
-av_frame_free();
+if (in != out) {
+av_frame_free();
+}
return ff_filter_frame(outlink, out);
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/2] libavfi/dnn: add LibTorch as one of DNN backend
PyTorch is an open source machine learning framework that accelerates
the path from research prototyping to production deployment. Official
websit: https://pytorch.org/. We call the C++ library of PyTorch as
LibTorch, the same below.
To build FFmpeg with LibTorch, please take following steps as reference:
1. download LibTorch C++ library in https://pytorch.org/get-started/locally/,
please select C++/Java for language, and other options as your need.
2. unzip the file to your own dir, with command
unzip libtorch-shared-with-deps-latest.zip -d your_dir
3. export libtorch_root/libtorch/include and
libtorch_root/libtorch/include/torch/csrc/api/include to $PATH
export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH
4. config FFmpeg with ../configure --enable-libtorch
--extra-cflag=-I/libtorch_root/libtorch/include
--extra-cflag=-I/libtorch_root/libtorch/include/torch/csrc/api/include
--extra-ldflags=-L/libtorch_root/libtorch/lib/
5. make
To run FFmpeg DNN inference with LibTorch backend:
./ffmpeg -i input.jpg -vf
dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg
The LibTorch_model.pt can be generated by Python with torch.jit.script() api.
Please note, torch.jit.trace() is not recommanded, since it does not support
ambiguous input size.
Signed-off-by: Ting Fu
---
configure | 7 +-
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_torch.cpp | 567 ++
libavfilter/dnn/dnn_backend_torch.h | 47 +++
libavfilter/dnn/dnn_interface.c | 12 +
libavfilter/dnn/dnn_io_proc.c | 117 +-
libavfilter/dnn_filter_common.c | 31 +-
libavfilter/dnn_interface.h | 3 +-
libavfilter/vf_dnn_processing.c | 3 +
9 files changed, 774 insertions(+), 14 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp
create mode 100644 libavfilter/dnn/dnn_backend_torch.h
diff --git a/configure b/configure
index f115b21064..85ce3e67a3 100755
--- a/configure
+++ b/configure
@@ -279,6 +279,7 @@ External library support:
--enable-libtheora enable Theora encoding via libtheora [no]
--enable-libtls enable LibreSSL (via libtls), needed for https
support
if openssl, gnutls or mbedtls is not used [no]
+ --enable-libtorchenable Torch as one DNN backend
--enable-libtwolame enable MP2 encoding via libtwolame [no]
--enable-libuavs3d enable AVS3 decoding via libuavs3d [no]
--enable-libv4l2 enable libv4l2/v4l-utils [no]
@@ -1850,6 +1851,7 @@ EXTERNAL_LIBRARY_LIST="
libopus
libplacebo
libpulse
+libtorch
librabbitmq
librav1e
librist
@@ -2719,7 +2721,7 @@ dct_select="rdft"
deflate_wrapper_deps="zlib"
dirac_parse_select="golomb"
dovi_rpu_select="golomb"
-dnn_suggest="libtensorflow libopenvino"
+dnn_suggest="libtensorflow libopenvino libtorch"
dnn_deps="avformat swscale"
error_resilience_select="me_cmp"
faandct_deps="faan"
@@ -6600,6 +6602,7 @@ enabled libopus && {
}
enabled libplacebo&& require_pkg_config libplacebo "libplacebo >=
4.192.0" libplacebo/vulkan.h pl_vulkan_create
enabled libpulse && require_pkg_config libpulse libpulse
pulse/pulseaudio.h pa_context_new
+enabled libtorch && add_cppflags -D_GLIBCXX_USE_CXX11_ABI=0 &&
check_cxxflags -std=c++14 && require_cpp libtorch torch/torch.h "torch::Tensor"
-ltorch -lc10 -ltorch_cpu -lstdc++ -lpthread
enabled librabbitmq && require_pkg_config librabbitmq "librabbitmq >=
0.7.1" amqp.h amqp_new_connection
enabled librav1e && require_pkg_config librav1e "rav1e >= 0.4.0"
rav1e.h rav1e_context_new
enabled librist && require_pkg_config librist "librist >= 0.2"
librist/librist.h rist_receiver_create
@@ -7025,6 +7028,8 @@ check_disable_warning -Wno-pointer-sign
check_disable_warning -Wno-unused-const-variable
check_disable_warning -Wno-bool-operation
check_disable_warning -Wno-char-subscripts
+#this option is for supress redundant-decls warning in compile libtorch
+check_disable_warning -Wno-redundant-decls
check_disable_warning_headers(){
warning_flag=-W${1#-Wno-}
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index 4cfbce0efc..d44dcb847e 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -16,5 +16,6 @@ OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_mat
DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn/dnn_backend_tf.o
DNN-OBJS-$(CONFIG_LIBOPENVINO) += dnn/dnn_backend_openvino.o
+DNN-OBJS-$(CONFIG_LIBTORCH) += dnn/dnn_backend_torch.o
OBJS-$(CONFIG_DNN) += $(DNN-OBJS-yes)
diff --gi
[FFmpeg-devel] [PATCH 1/2] libavfi/dnn: refine enum DNNColorOrder
Change the DCO_RGB and DCO_BGR color order in DNNColorOrder to
DCO_RGB_PACKED and DCO_GBR_PACKED for following RGB planar support.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 2 +-
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn/dnn_io_proc.c | 4 ++--
libavfilter/dnn_interface.h| 4 ++--
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index cf012aca4c..92e180a0eb 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -156,7 +156,7 @@ static int fill_model_input_ov(OVModel *ov_model,
OVRequestItem *request)
input.dt = precision_to_datatype(precision);
// all models in openvino open model zoo use BGR as input,
// change to be an option when necessary.
-input.order = DCO_BGR;
+input.order = DCO_BGR_PACKED;
for (int i = 0; i < ctx->options.batch_size; ++i) {
lltask = ff_queue_pop_front(ov_model->lltask_queue);
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 3b5084b67b..e639b3cecd 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -294,7 +294,7 @@ static int get_input_tf(void *model, DNNData *input, const
char *input_name)
tf_output.index = 0;
input->dt = TF_OperationOutputType(tf_output);
-input->order = DCO_RGB;
+input->order = DCO_RGB_PACKED;
status = TF_NewStatus();
TF_GraphGetTensorShape(tf_model->graph, tf_output, dims, 4, status);
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 7961bf6b95..532b089002 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -176,9 +176,9 @@ static enum AVPixelFormat get_pixel_format(DNNData *data)
{
if (data->dt == DNN_UINT8) {
switch (data->order) {
-case DCO_BGR:
+case DCO_BGR_PACKED:
return AV_PIX_FMT_BGR24;
-case DCO_RGB:
+case DCO_RGB_PACKED:
return AV_PIX_FMT_RGB24;
default:
av_assert0(!"unsupported data pixel format.\n");
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index ef8d7ae66f..d94baa90c4 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -38,8 +38,8 @@ typedef enum {DNN_FLOAT = 1, DNN_UINT8 = 4} DNNDataType;
typedef enum {
DCO_NONE,
-DCO_BGR,
-DCO_RGB,
+DCO_BGR_PACKED,
+DCO_RGB_PACKED,
} DNNColorOrder;
typedef enum {
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/2] lavfi/vf_drawtext.c: fix CID 1485003
CID 1485003: Memory - illegal accesses (UNINIT)
Using uninitialized value "sd".
Signed-off-by: Ting Fu
---
libavfilter/vf_drawtext.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libavfilter/vf_drawtext.c b/libavfilter/vf_drawtext.c
index 382d589e26..c4c09894e4 100644
--- a/libavfilter/vf_drawtext.c
+++ b/libavfilter/vf_drawtext.c
@@ -1554,7 +1554,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame
*frame)
AVFrameSideData *sd;
int loop = 1;
-if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES && sd) {
+if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES) {
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
if (sd) {
header = (AVDetectionBBoxHeader *)sd->data;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 1/2] lavfi/vf_drawbox.c: fix CID 1485004
CID 1485004: Uninitialized variables (UNINIT)
Using uninitialized value "x" when calling "*pixel_belongs_to_region".
Signed-off-by: Ting Fu
---
libavfilter/vf_drawbox.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c
index fff78862e9..1e9e028650 100644
--- a/libavfilter/vf_drawbox.c
+++ b/libavfilter/vf_drawbox.c
@@ -126,8 +126,9 @@ static void draw_region(AVFrame *frame, DrawBoxContext
*ctx, int left, int top,
for (y = top; y < down; y++) {
ASSIGN_THREE_CHANNELS
if (ctx->invert_color) {
-if (pixel_belongs_to_region(ctx, x, y))
-row[0][x] = 0xff - row[0][x];
+for (x = left; x < right; x++)
+if (pixel_belongs_to_region(ctx, x, y))
+row[0][x] = 0xff - row[0][x];
} else {
for (x = left; x < right; x++) {
double alpha = (double)ctx->yuv_color[A] / 255;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 3/3] libavfilter: vf_drawtext filter support draw text with detection bounding boxes in side_data
This feature can be used with dnn detection by setting vf_drawtext's option
text_source=side_data_detection_bboxes, for example:
./ffmpeg -i face.jpeg -vf
dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:\
input=data:output=detection_out:labels=face-detection-adas-0001.label,drawbox=box_source=
side_data_detection_bboxes,drawtext=text_source=side_data_detection_bboxes:fontcolor=green:\
fontsize=40, -y face_detect.jpeg
Please note, the default fontsize of vf_drawtext is 12, which may be too
small to be seen clearly.
Signed-off-by: Ting Fu
---
doc/filters.texi | 8
libavfilter/vf_drawtext.c | 77 ---
2 files changed, 79 insertions(+), 6 deletions(-)
diff --git a/doc/filters.texi b/doc/filters.texi
index f2ac8c4cc8..d10e6de03d 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -10788,6 +10788,14 @@ parameter @var{text}.
If both @var{text} and @var{textfile} are specified, an error is thrown.
+@item text_source
+Text source should be set as side_data_detection_bboxes if you want to use
text data in
+detection bboxes of side data.
+
+If text source is set, @var{text} and @var{textfile} will be ignored and still
use
+text data in detection bboxes of side data. So please do not use this parameter
+if you are not sure about the text source.
+
@item reload
If set to 1, the @var{textfile} will be reloaded before each frame.
Be sure to update it atomically, or it may be read partially, or even fail.
diff --git a/libavfilter/vf_drawtext.c b/libavfilter/vf_drawtext.c
index 7ea057b812..382d589e26 100644
--- a/libavfilter/vf_drawtext.c
+++ b/libavfilter/vf_drawtext.c
@@ -55,6 +55,7 @@
#include "libavutil/time_internal.h"
#include "libavutil/tree.h"
#include "libavutil/lfg.h"
+#include "libavutil/detection_bbox.h"
#include "avfilter.h"
#include "drawutils.h"
#include "formats.h"
@@ -199,6 +200,8 @@ typedef struct DrawTextContext {
int tc24hmax; ///< 1 if timecode is wrapped to 24 hours,
0 otherwise
int reload; ///< reload text file for each frame
int start_number; ///< starting frame number for n/frame_num
var
+char *text_source_string; ///< the string to specify text data source
+enum AVFrameSideDataType text_source;
#if CONFIG_LIBFRIBIDI
int text_shaping; ///< 1 to shape the text before drawing it
#endif
@@ -246,6 +249,7 @@ static const AVOption drawtext_options[]= {
{ "alpha", "apply alpha while rendering", OFFSET(a_expr),
AV_OPT_TYPE_STRING, { .str = "1" }, .flags = FLAGS },
{"fix_bounds", "check and fix text coords to avoid clipping",
OFFSET(fix_bounds), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS},
{"start_number", "start frame number for n/frame_num variable",
OFFSET(start_number), AV_OPT_TYPE_INT, {.i64=0}, 0, INT_MAX, FLAGS},
+{"text_source", "the source of text", OFFSET(text_source_string),
AV_OPT_TYPE_STRING, {.str=NULL}, 0, 1, FLAGS },
#if CONFIG_LIBFRIBIDI
{"text_shaping", "attempt to shape text before drawing",
OFFSET(text_shaping), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS},
@@ -690,6 +694,16 @@ out:
}
#endif
+static enum AVFrameSideDataType text_source_string_parse(const char
*text_source_string)
+{
+av_assert0(text_source_string);
+if (!strcmp(text_source_string, "side_data_detection_bboxes")) {
+return AV_FRAME_DATA_DETECTION_BBOXES;
+} else {
+return AVERROR(EINVAL);
+}
+}
+
static av_cold int init(AVFilterContext *ctx)
{
int err;
@@ -731,9 +745,28 @@ static av_cold int init(AVFilterContext *ctx)
s->text = av_strdup("");
}
+if (s->text_source_string) {
+s->text_source = text_source_string_parse(s->text_source_string);
+if ((int)s->text_source < 0) {
+av_log(ctx, AV_LOG_ERROR, "Error text source: %s\n",
s->text_source_string);
+return AVERROR(EINVAL);
+}
+}
+
+if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES) {
+if (s->text) {
+av_log(ctx, AV_LOG_WARNING, "Multiple texts provided, will use
text_source only\n");
+av_free(s->text);
+}
+s->text = av_mallocz(AV_DETECTION_BBOX_LABEL_NAME_MAX_SIZE *
+ (AV_NUM_DETECTION_BBOX_CLASSIFY + 1));
+if (!s->text)
+return AVERROR(ENOMEM);
+}
+
if (!s->text) {
av_log(ctx, AV_LOG_ERROR,
- "Either text, a valid file or a timecode must be provided\n");
+ "Either text, a valid file, a timecode or text source must be
provided\n");
return AVERROR(EINVAL);
[FFmpeg-devel] [PATCH 2/3] libavfilter: vf_drawbox filter support draw box with detection bounding boxes in side_data
This feature can be used with dnn detection by setting vf_drawbox's
option box_source=side_data_detection_bboxes, for example:
./ffmpeg -i face.jpeg -vf
dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:\
input=data:output=detection_out:labels=face-detection-adas-0001.label,\
drawbox=box_source=side_data_detection_bboxes -y face_detect.jpeg
Signed-off-by: Ting Fu
---
doc/filters.texi | 8 +++
libavfilter/vf_drawbox.c | 52 ++--
2 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/doc/filters.texi b/doc/filters.texi
index a218289ddd..f2ac8c4cc8 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -10356,6 +10356,14 @@ The x and y offset coordinates where the box is drawn.
@item h
The width and height of the drawn box.
+@item box_source
+Box source can be set as side_data_detection_bboxes if you want to use box
data in
+detection bboxes of side data.
+
+If @var{box_source} is set, the @var{x}, @var{y}, @var{width} and @var{height}
will be ignored and
+still use box data in detection bboxes of side data. So please do not use this
parameter if you were
+not sure about the box source.
+
@item t
The thickness of the drawn box.
diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c
index 95e26191bd..fff78862e9 100644
--- a/libavfilter/vf_drawbox.c
+++ b/libavfilter/vf_drawbox.c
@@ -31,6 +31,7 @@
#include "libavutil/eval.h"
#include "libavutil/pixdesc.h"
#include "libavutil/parseutils.h"
+#include "libavutil/detection_bbox.h"
#include "avfilter.h"
#include "formats.h"
#include "internal.h"
@@ -79,8 +80,10 @@ typedef struct DrawBoxContext {
char *x_expr, *y_expr; ///< expression for x and y
char *w_expr, *h_expr; ///< expression for width and height
char *t_expr; ///< expression for thickness
+char *box_source_string; ///< string for box data source
int have_alpha;
int replace;
+enum AVFrameSideDataType box_source;
} DrawBoxContext;
static const int NUM_EXPR_EVALS = 5;
@@ -140,11 +143,30 @@ static void draw_region(AVFrame *frame, DrawBoxContext
*ctx, int left, int top,
}
}
+static enum AVFrameSideDataType box_source_string_parse(const char
*box_source_string)
+{
+av_assert0(box_source_string);
+if (!strcmp(box_source_string, "side_data_detection_bboxes")) {
+return AV_FRAME_DATA_DETECTION_BBOXES;
+} else {
+// will support side_data_regions_of_interest next
+return AVERROR(EINVAL);
+}
+}
+
static av_cold int init(AVFilterContext *ctx)
{
DrawBoxContext *s = ctx->priv;
uint8_t rgba_color[4];
+if (s->box_source_string) {
+s->box_source = box_source_string_parse(s->box_source_string);
+if ((int)s->box_source < 0) {
+av_log(ctx, AV_LOG_ERROR, "Error box source:
%s\n",s->box_source_string);
+return AVERROR(EINVAL);
+}
+}
+
if (!strcmp(s->color_str, "invert"))
s->invert_color = 1;
else if (av_parse_color(rgba_color, s->color_str, -1, ctx) < 0)
@@ -272,9 +294,34 @@ static av_pure av_always_inline int
pixel_belongs_to_box(DrawBoxContext *s, int
static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
{
DrawBoxContext *s = inlink->dst->priv;
+const AVDetectionBBoxHeader *header = NULL;
+const AVDetectionBBox *bbox;
+AVFrameSideData *sd;
+int loop = 1;
+
+if (s->box_source == AV_FRAME_DATA_DETECTION_BBOXES) {
+sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
+if (sd) {
+header = (AVDetectionBBoxHeader *)sd->data;
+loop = header->nb_bboxes;
+} else {
+av_log(s, AV_LOG_WARNING, "No detection bboxes.\n");
+return ff_filter_frame(inlink->dst->outputs[0], frame);
+}
+}
-draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w,
frame->width),
-FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box);
+for (int i = 0; i < loop; i++) {
+if (header) {
+bbox = av_get_detection_bbox(header, i);
+s->y = bbox->y;
+s->x = bbox->x;
+s->h = bbox->h;
+s->w = bbox->w;
+}
+
+draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x +
s->w, frame->width),
+FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box);
+}
return ff_filter_frame(inlink->dst->outputs[0], frame);
}
@@ -329,6 +376,7 @@ static const AVOption drawbox_options[] = {
{ "thickness", "set the box thickness",
OFFSET(t_expr),AV_OPT_TYPE_STRING, { .str="3" }, 0, 0, FLAG
[FFmpeg-devel] [PATCH 1/3] lavfi/drawbox: refine code
Extract common code of filter_frame() and drawgrid_filter_frame() to
draw_region().
Signed-off-by: Ting Fu
---
libavfilter/vf_drawbox.c | 160 ++-
1 file changed, 58 insertions(+), 102 deletions(-)
diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c
index 2794fc2520..95e26191bd 100644
--- a/libavfilter/vf_drawbox.c
+++ b/libavfilter/vf_drawbox.c
@@ -85,6 +85,61 @@ typedef struct DrawBoxContext {
static const int NUM_EXPR_EVALS = 5;
+typedef int (*PixelBelongsToRegion)(DrawBoxContext *s, int x, int y);
+
+#define ASSIGN_THREE_CHANNELS\
+row[0] = frame->data[0] + y * frame->linesize[0]; \
+row[1] = frame->data[1] + (y >> ctx->vsub) * frame->linesize[1]; \
+row[2] = frame->data[2] + (y >> ctx->vsub) * frame->linesize[2];
+
+#define ASSIGN_FOUR_CHANNELS \
+ASSIGN_THREE_CHANNELS \
+row[3] = frame->data[3] + y * frame->linesize[3];
+
+static void draw_region(AVFrame *frame, DrawBoxContext *ctx, int left, int
top, int right, int down,
+PixelBelongsToRegion pixel_belongs_to_region)
+{
+unsigned char *row[4];
+int x, y;
+if (ctx->have_alpha && ctx->replace) {
+for (y = top; y < down; y++) {
+ASSIGN_FOUR_CHANNELS
+if (ctx->invert_color) {
+for (x = left; x < right; x++)
+if (pixel_belongs_to_region(ctx, x, y))
+row[0][x] = 0xff - row[0][x];
+} else {
+for (x = left; x < right; x++) {
+if (pixel_belongs_to_region(ctx, x, y)) {
+row[0][x ] = ctx->yuv_color[Y];
+row[1][x >> ctx->hsub] = ctx->yuv_color[U];
+row[2][x >> ctx->hsub] = ctx->yuv_color[V];
+row[3][x ] = ctx->yuv_color[A];
+}
+}
+}
+}
+} else {
+for (y = top; y < down; y++) {
+ASSIGN_THREE_CHANNELS
+if (ctx->invert_color) {
+if (pixel_belongs_to_region(ctx, x, y))
+row[0][x] = 0xff - row[0][x];
+} else {
+for (x = left; x < right; x++) {
+double alpha = (double)ctx->yuv_color[A] / 255;
+
+if (pixel_belongs_to_region(ctx, x, y)) {
+row[0][x ] = (1 - alpha) * row[0][x
] + alpha * ctx->yuv_color[Y];
+row[1][x >> ctx->hsub] = (1 - alpha) * row[1][x >>
ctx->hsub] + alpha * ctx->yuv_color[U];
+row[2][x >> ctx->hsub] = (1 - alpha) * row[2][x >>
ctx->hsub] + alpha * ctx->yuv_color[V];
+}
+}
+}
+}
+}
+}
+
static av_cold int init(AVFilterContext *ctx)
{
DrawBoxContext *s = ctx->priv;
@@ -217,58 +272,9 @@ static av_pure av_always_inline int
pixel_belongs_to_box(DrawBoxContext *s, int
static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
{
DrawBoxContext *s = inlink->dst->priv;
-int plane, x, y, xb = s->x, yb = s->y;
-unsigned char *row[4];
-
-if (s->have_alpha && s->replace) {
-for (y = FFMAX(yb, 0); y < frame->height && y < (yb + s->h); y++) {
-row[0] = frame->data[0] + y * frame->linesize[0];
-row[3] = frame->data[3] + y * frame->linesize[3];
-
-for (plane = 1; plane < 3; plane++)
-row[plane] = frame->data[plane] +
- frame->linesize[plane] * (y >> s->vsub);
-
-if (s->invert_color) {
-for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++)
-if (pixel_belongs_to_box(s, x, y))
-row[0][x] = 0xff - row[0][x];
-} else {
-for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++)
{
-if (pixel_belongs_to_box(s, x, y)) {
-row[0][x ] = s->yuv_color[Y];
-row[1][x >> s->hsub] = s->yuv_color[U];
-row[2][x >> s->hsub] = s->yuv_color[V];
-row[3][x ] = s->yuv_color[A];
-}
-}
-}
-}
-} else {
-for (y = FFMAX(yb, 0); y < frame->height && y < (yb + s->h); y++) {
-row[0] = frame->data[0] + y * frame->linesize[0];
-for (plane = 1; plane < 3; plane++)
-row[plane
[FFmpeg-devel] [PATCH V2 4/4] dnn/vf_dnn_detect: add tensorflow output parse support
Testing model is tensorflow offical model in github repo, please refer
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md
to download the detect model as you need.
For example, local testing was carried on with
'ssd_mobilenet_v2_coco_2018_03_29.tar.gz', and
used one image of dog in
https://github.com/tensorflow/models/blob/master/research/object_detection/test_images/image1.jpg
Testing command is:
./ffmpeg -i image1.jpg -vf
dnn_detect=dnn_backend=tensorflow:input=image_tensor:output=\
"num_detections_scores_classes_boxes":model=ssd_mobilenet_v2_coco.pb,\
showinfo -f null -
We will see the result similar as below:
[Parsed_showinfo_1 @ 0x33e65f0] side data - detection bounding boxes:
[Parsed_showinfo_1 @ 0x33e65f0] source: ssd_mobilenet_v2_coco.pb
[Parsed_showinfo_1 @ 0x33e65f0] index: 0, region: (382, 60) -> (1005,
593), label: 18, confidence: 9834/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 1, region: (12, 8) -> (328, 549),
label: 18, confidence: 8555/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 2, region: (293, 7) -> (682, 458),
label: 1, confidence: 8033/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 3, region: (342, 0) -> (690, 325),
label: 1, confidence: 5878/1.
There are two boxes of dog with cores 94.05% & 93.45% and two boxes of person
with scores 80.33% & 58.78%.
Signed-off-by: Ting Fu
---
libavfilter/vf_dnn_detect.c | 95 -
1 file changed, 94 insertions(+), 1 deletion(-)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 7d39acb653..818b53a052 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -48,6 +48,9 @@ typedef struct DnnDetectContext {
#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
static const AVOption dnn_detect_options[] = {
{ "dnn_backend", "DNN backend",OFFSET(backend_type),
AV_OPT_TYPE_INT, { .i64 = 2 },INT_MIN, INT_MAX, FLAGS, "backend" },
+#if (CONFIG_LIBTENSORFLOW == 1)
+{ "tensorflow", "tensorflow backend flag",0,
AV_OPT_TYPE_CONST, { .i64 = 1 },0, 0, FLAGS, "backend" },
+#endif
#if (CONFIG_LIBOPENVINO == 1)
{ "openvino","openvino backend flag", 0,
AV_OPT_TYPE_CONST, { .i64 = 2 },0, 0, FLAGS, "backend" },
#endif
@@ -59,7 +62,7 @@ static const AVOption dnn_detect_options[] = {
AVFILTER_DEFINE_CLASS(dnn_detect);
-static int dnn_detect_post_proc(AVFrame *frame, DNNData *output, uint32_t nb,
AVFilterContext *filter_ctx)
+static int dnn_detect_post_proc_ov(AVFrame *frame, DNNData *output,
AVFilterContext *filter_ctx)
{
DnnDetectContext *ctx = filter_ctx->priv;
float conf_threshold = ctx->confidence;
@@ -136,6 +139,96 @@ static int dnn_detect_post_proc(AVFrame *frame, DNNData
*output, uint32_t nb, AV
return 0;
}
+static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData *output,
AVFilterContext *filter_ctx)
+{
+DnnDetectContext *ctx = filter_ctx->priv;
+int proposal_count;
+float conf_threshold = ctx->confidence;
+float *conf, *position, *label_id, x0, y0, x1, y1;
+int nb_bboxes = 0;
+AVFrameSideData *sd;
+AVDetectionBBox *bbox;
+AVDetectionBBoxHeader *header;
+
+proposal_count = *(float *)(output[0].data);
+conf = output[1].data;
+position = output[3].data;
+label_id = output[2].data;
+
+sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
+if (sd) {
+av_log(filter_ctx, AV_LOG_ERROR, "already have dnn bounding boxes in
side data.\n");
+return -1;
+}
+
+for (int i = 0; i < proposal_count; ++i) {
+if (conf[i] < conf_threshold)
+continue;
+nb_bboxes++;
+}
+
+if (nb_bboxes == 0) {
+av_log(filter_ctx, AV_LOG_VERBOSE, "nothing detected in this
frame.\n");
+return 0;
+}
+
+header = av_detection_bbox_create_side_data(frame, nb_bboxes);
+if (!header) {
+av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
+return -1;
+}
+
+av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
+
+for (int i = 0; i < proposal_count; ++i) {
+y0 = position[i * 4];
+x0 = position[i * 4 + 1];
+y1 = position[i * 4 + 2];
+x1 = position[i * 4 + 3];
+
+bbox = av_get_detection_bbox(header, i);
+
+if (conf[i] < conf_threshold) {
+continue;
+}
+
+bbox->x = (int)(x0 * frame->width);
+bbox->w = (int)(x1 * frame->width) - bbox->x;
+bbox->y = (int)(y0 * frame->height);
+bbox->h = (int
[FFmpeg-devel] [PATCH V2 3/4] lavfi/dnn_backend_tensorflow: support detect model
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 39 ++--
libavfilter/vf_dnn_detect.c | 32 +-
2 files changed, 63 insertions(+), 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index b6b1812cd9..622b5a8464 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -793,15 +793,40 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
outputs[i].data = TF_TensorData(output_tensors[i]);
outputs[i].dt = TF_TensorType(output_tensors[i]);
}
-if (do_ioproc) {
-if (tf_model->model->frame_post_proc != NULL) {
-tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
+switch (model->func_type) {
+case DFT_PROCESS_FRAME:
+//it only support 1 output if it's frame in & frame out
+if (do_ioproc) {
+if (tf_model->model->frame_post_proc != NULL) {
+tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
+} else {
+ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
+}
} else {
-ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
+out_frame->width = outputs[0].width;
+out_frame->height = outputs[0].height;
+}
+break;
+case DFT_ANALYTICS_DETECT:
+if (!model->detect_post_proc) {
+av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post
proc\n");
+return DNN_ERROR;
+}
+model->detect_post_proc(out_frame, outputs, nb_output,
model->filter_ctx);
+break;
+default:
+for (uint32_t i = 0; i < nb_output; ++i) {
+if (output_tensors[i]) {
+TF_DeleteTensor(output_tensors[i]);
+}
}
-} else {
-out_frame->width = outputs[0].width;
-out_frame->height = outputs[0].height;
+TF_DeleteTensor(input_tensor);
+av_freep(_tensors);
+av_freep(_outputs);
+av_freep();
+
+av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
+return DNN_ERROR;
}
for (uint32_t i = 0; i < nb_output; ++i) {
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 1dbe4f29a4..7d39acb653 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -203,10 +203,40 @@ static int read_detect_label_file(AVFilterContext
*context)
return 0;
}
+static int check_output_nb(DnnDetectContext *ctx, DNNBackendType backend_type,
int output_nb)
+{
+switch(backend_type) {
+case DNN_TF:
+if (output_nb != 4) {
+av_log(ctx, AV_LOG_ERROR, "Only support tensorflow detect model
with 4 outputs, \
+ but get %d instead\n", output_nb);
+return AVERROR(EINVAL);
+}
+return 0;
+case DNN_OV:
+if (output_nb != 1) {
+av_log(ctx, AV_LOG_ERROR, "Dnn detect filter with openvino backend
needs 1 output only, \
+ but get %d instead\n", output_nb);
+return AVERROR(EINVAL);
+}
+return 0;
+default:
+avpriv_report_missing_feature(ctx, "Dnn detect filter does not support
current backend\n");
+return AVERROR(EINVAL);
+}
+return 0;
+}
+
static av_cold int dnn_detect_init(AVFilterContext *context)
{
DnnDetectContext *ctx = context->priv;
-int ret = ff_dnn_init(>dnnctx, DFT_ANALYTICS_DETECT, context);
+DnnContext *dnn_ctx = >dnnctx;
+int ret;
+
+ret = ff_dnn_init(>dnnctx, DFT_ANALYTICS_DETECT, context);
+if (ret < 0)
+return ret;
+ret = check_output_nb(ctx, dnn_ctx->backend_type, dnn_ctx->nb_outputs);
if (ret < 0)
return ret;
ff_dnn_set_detect_post_proc(>dnnctx, dnn_detect_post_proc);
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 2/4] lavfi/dnn_backend_tensorflow: add multiple outputs support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 49 ++---
libavfilter/dnn_filter_common.c | 53 ++--
libavfilter/dnn_filter_common.h | 6 ++--
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_sr.c | 2 +-
5 files changed, 75 insertions(+), 37 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 45da29ae70..b6b1812cd9 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -155,7 +155,7 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
TF_DeleteStatus(status);
// currently only NHWC is supported
-av_assert0(dims[0] == 1);
+av_assert0(dims[0] == 1 || dims[0] == -1);
input->height = dims[1];
input->width = dims[2];
input->channels = dims[3];
@@ -707,7 +707,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
TF_Output *tf_outputs;
TFModel *tf_model = model->model;
TFContext *ctx = _model->ctx;
-DNNData input, output;
+DNNData input, *outputs;
TF_Tensor **output_tensors;
TF_Output tf_input;
TF_Tensor *input_tensor;
@@ -738,14 +738,6 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
}
}
-if (nb_output != 1) {
-// currently, the filter does not need multiple outputs,
-// so we just pending the support until we really need it.
-TF_DeleteTensor(input_tensor);
-avpriv_report_missing_feature(ctx, "multiple outputs");
-return DNN_ERROR;
-}
-
tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs));
if (tf_outputs == NULL) {
TF_DeleteTensor(input_tensor);
@@ -785,23 +777,31 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
return DNN_ERROR;
}
+outputs = av_malloc_array(nb_output, sizeof(*outputs));
+if (!outputs) {
+TF_DeleteTensor(input_tensor);
+av_freep(_outputs);
+av_freep(_tensors);
+av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n");
\
+return DNN_ERROR;
+}
+
for (uint32_t i = 0; i < nb_output; ++i) {
-output.height = TF_Dim(output_tensors[i], 1);
-output.width = TF_Dim(output_tensors[i], 2);
-output.channels = TF_Dim(output_tensors[i], 3);
-output.data = TF_TensorData(output_tensors[i]);
-output.dt = TF_TensorType(output_tensors[i]);
-
-if (do_ioproc) {
-if (tf_model->model->frame_post_proc != NULL) {
-tf_model->model->frame_post_proc(out_frame, ,
tf_model->model->filter_ctx);
-} else {
-ff_proc_from_dnn_to_frame(out_frame, , ctx);
-}
+outputs[i].height = TF_Dim(output_tensors[i], 1);
+outputs[i].width = TF_Dim(output_tensors[i], 2);
+outputs[i].channels = TF_Dim(output_tensors[i], 3);
+outputs[i].data = TF_TensorData(output_tensors[i]);
+outputs[i].dt = TF_TensorType(output_tensors[i]);
+}
+if (do_ioproc) {
+if (tf_model->model->frame_post_proc != NULL) {
+tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
} else {
-out_frame->width = output.width;
-out_frame->height = output.height;
+ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
}
+} else {
+out_frame->width = outputs[0].width;
+out_frame->height = outputs[0].height;
}
for (uint32_t i = 0; i < nb_output; ++i) {
@@ -812,6 +812,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
TF_DeleteTensor(input_tensor);
av_freep(_tensors);
av_freep(_outputs);
+av_freep();
return DNN_SUCCESS;
}
diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c
index 52c7a5392a..0ed0ac2e30 100644
--- a/libavfilter/dnn_filter_common.c
+++ b/libavfilter/dnn_filter_common.c
@@ -17,6 +17,39 @@
*/
#include "dnn_filter_common.h"
+#include "libavutil/avstring.h"
+
+#define MAX_SUPPORTED_OUTPUTS_NB 4
+
+static char **separate_output_names(const char *expr, const char *val_sep, int
*separated_nb)
+{
+char *val, **parsed_vals = NULL;
+int val_num = 0;
+if (!expr || !val_sep || !separated_nb) {
+return NULL;
+}
+
+parsed_vals = av_mallocz_array(MAX_SUPPORTED_OUTPUTS_NB,
sizeof(*parsed_vals));
+if (!parsed_vals) {
+return NULL;
+}
+
+do {
+val = av_get_token(, val_sep);
+if(val) {
+parsed_vals[val_num] = val;
+val_num++;
+}
+if (*expr) {
+expr++;
+}
+} while(*expr);
+
+parsed_vals[val_num] = NULL;
+*separa
[FFmpeg-devel] [PATCH V2 1/4] dnn: add DCO_RGB color order to enum DNNColorOrder
Adding DCO_RGB color order to DNNColorOrder, since tensorflow model
needs this kind of color oder as input.
Signed-off-by: Ting Fu
---
V2:
Rebase patch to latest code
libavfilter/dnn/dnn_backend_tf.c | 1 +
libavfilter/dnn/dnn_io_proc.c| 14 +++---
libavfilter/dnn_interface.h | 1 +
3 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 03fe310b03..45da29ae70 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -143,6 +143,7 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
tf_output.index = 0;
input->dt = TF_OperationOutputType(tf_output);
+input->order = DCO_RGB;
status = TF_NewStatus();
TF_GraphGetTensorShape(tf_model->graph, tf_output, dims, 4, status);
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 5f60d68078..1e2bef3f9a 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -168,11 +168,19 @@ static DNNReturnType
proc_from_frame_to_dnn_frameprocessing(AVFrame *frame, DNND
static enum AVPixelFormat get_pixel_format(DNNData *data)
{
-if (data->dt == DNN_UINT8 && data->order == DCO_BGR) {
-return AV_PIX_FMT_BGR24;
+if (data->dt == DNN_UINT8) {
+switch (data->order) {
+case DCO_BGR:
+return AV_PIX_FMT_BGR24;
+case DCO_RGB:
+return AV_PIX_FMT_RGB24;
+default:
+av_assert0(!"unsupported data pixel format.\n");
+return AV_PIX_FMT_BGR24;
+}
}
-av_assert0(!"not supported yet.\n");
+av_assert0(!"unsupported data type.\n");
return AV_PIX_FMT_BGR24;
}
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 799244ee14..5e9ffeb077 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -39,6 +39,7 @@ typedef enum {DNN_FLOAT = 1, DNN_UINT8 = 4} DNNDataType;
typedef enum {
DCO_NONE,
DCO_BGR,
+DCO_RGB,
} DNNColorOrder;
typedef enum {
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 4/4] dnn/vf_dnn_detect: add tensorflow output parse support
Testing model is tensorflow offical model in github repo, please refer
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md
to download the detect model as you need.
For example, local testing was carried on with
'ssd_mobilenet_v2_coco_2018_03_29.tar.gz', and
used one image of dog in
https://github.com/tensorflow/models/blob/master/research/object_detection/test_images/image1.jpg
Testing command is:
./ffmpeg -i image1.jpg -vf
dnn_detect=dnn_backend=tensorflow:input=image_tensor:output=\
"num_detections_scores_classes_boxes":model=ssd_mobilenet_v2_coco.pb,\
showinfo -f null -
We will see the result similar as below:
[Parsed_showinfo_1 @ 0x33e65f0] side data - detection bounding boxes:
[Parsed_showinfo_1 @ 0x33e65f0] source: ssd_mobilenet_v2_coco.pb
[Parsed_showinfo_1 @ 0x33e65f0] index: 0, region: (382, 60) -> (1005,
593), label: 18, confidence: 9834/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 1, region: (12, 8) -> (328, 549),
label: 18, confidence: 8555/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 2, region: (293, 7) -> (682, 458),
label: 1, confidence: 8033/1.
[Parsed_showinfo_1 @ 0x33e65f0] index: 3, region: (342, 0) -> (690, 325),
label: 1, confidence: 5878/1.
There are two boxes of dog with cores 94.05% & 93.45% and two boxes of person
with scores 80.33% & 58.78%.
Signed-off-by: Ting Fu
---
libavfilter/vf_dnn_detect.c | 95 -
1 file changed, 94 insertions(+), 1 deletion(-)
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 7d39acb653..818b53a052 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -48,6 +48,9 @@ typedef struct DnnDetectContext {
#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
static const AVOption dnn_detect_options[] = {
{ "dnn_backend", "DNN backend",OFFSET(backend_type),
AV_OPT_TYPE_INT, { .i64 = 2 },INT_MIN, INT_MAX, FLAGS, "backend" },
+#if (CONFIG_LIBTENSORFLOW == 1)
+{ "tensorflow", "tensorflow backend flag",0,
AV_OPT_TYPE_CONST, { .i64 = 1 },0, 0, FLAGS, "backend" },
+#endif
#if (CONFIG_LIBOPENVINO == 1)
{ "openvino","openvino backend flag", 0,
AV_OPT_TYPE_CONST, { .i64 = 2 },0, 0, FLAGS, "backend" },
#endif
@@ -59,7 +62,7 @@ static const AVOption dnn_detect_options[] = {
AVFILTER_DEFINE_CLASS(dnn_detect);
-static int dnn_detect_post_proc(AVFrame *frame, DNNData *output, uint32_t nb,
AVFilterContext *filter_ctx)
+static int dnn_detect_post_proc_ov(AVFrame *frame, DNNData *output,
AVFilterContext *filter_ctx)
{
DnnDetectContext *ctx = filter_ctx->priv;
float conf_threshold = ctx->confidence;
@@ -136,6 +139,96 @@ static int dnn_detect_post_proc(AVFrame *frame, DNNData
*output, uint32_t nb, AV
return 0;
}
+static int dnn_detect_post_proc_tf(AVFrame *frame, DNNData *output,
AVFilterContext *filter_ctx)
+{
+DnnDetectContext *ctx = filter_ctx->priv;
+int proposal_count;
+float conf_threshold = ctx->confidence;
+float *conf, *position, *label_id, x0, y0, x1, y1;
+int nb_bboxes = 0;
+AVFrameSideData *sd;
+AVDetectionBBox *bbox;
+AVDetectionBBoxHeader *header;
+
+proposal_count = *(float *)(output[0].data);
+conf = output[1].data;
+position = output[3].data;
+label_id = output[2].data;
+
+sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES);
+if (sd) {
+av_log(filter_ctx, AV_LOG_ERROR, "already have dnn bounding boxes in
side data.\n");
+return -1;
+}
+
+for (int i = 0; i < proposal_count; ++i) {
+if (conf[i] < conf_threshold)
+continue;
+nb_bboxes++;
+}
+
+if (nb_bboxes == 0) {
+av_log(filter_ctx, AV_LOG_VERBOSE, "nothing detected in this
frame.\n");
+return 0;
+}
+
+header = av_detection_bbox_create_side_data(frame, nb_bboxes);
+if (!header) {
+av_log(filter_ctx, AV_LOG_ERROR, "failed to create side data with %d
bounding boxes\n", nb_bboxes);
+return -1;
+}
+
+av_strlcpy(header->source, ctx->dnnctx.model_filename,
sizeof(header->source));
+
+for (int i = 0; i < proposal_count; ++i) {
+y0 = position[i * 4];
+x0 = position[i * 4 + 1];
+y1 = position[i * 4 + 2];
+x1 = position[i * 4 + 3];
+
+bbox = av_get_detection_bbox(header, i);
+
+if (conf[i] < conf_threshold) {
+continue;
+}
+
+bbox->x = (int)(x0 * frame->width);
+bbox->w = (int)(x1 * frame->width) - bbox->x;
+bbox->y = (int)(y0 * frame->height);
+bbox->h = (int
[FFmpeg-devel] [PATCH 3/4] lavfi/dnn_backend_tensorflow: support detect model
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 39 ++--
libavfilter/vf_dnn_detect.c | 32 +-
2 files changed, 63 insertions(+), 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 5c85b562c4..8fb2ae8583 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -793,15 +793,40 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
outputs[i].data = TF_TensorData(output_tensors[i]);
outputs[i].dt = TF_TensorType(output_tensors[i]);
}
-if (do_ioproc) {
-if (tf_model->model->frame_post_proc != NULL) {
-tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
+switch (model->func_type) {
+case DFT_PROCESS_FRAME:
+//it only support 1 output if it's frame in & frame out
+if (do_ioproc) {
+if (tf_model->model->frame_post_proc != NULL) {
+tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
+} else {
+ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
+}
} else {
-ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
+out_frame->width = outputs[0].width;
+out_frame->height = outputs[0].height;
+}
+break;
+case DFT_ANALYTICS_DETECT:
+if (!model->detect_post_proc) {
+av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post
proc\n");
+return DNN_ERROR;
+}
+model->detect_post_proc(out_frame, outputs, nb_output,
model->filter_ctx);
+break;
+default:
+for (uint32_t i = 0; i < nb_output; ++i) {
+if (output_tensors[i]) {
+TF_DeleteTensor(output_tensors[i]);
+}
}
-} else {
-out_frame->width = outputs[0].width;
-out_frame->height = outputs[0].height;
+TF_DeleteTensor(input_tensor);
+av_freep(_tensors);
+av_freep(_outputs);
+av_freep();
+
+av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this
kind of dnn filter now\n");
+return DNN_ERROR;
}
for (uint32_t i = 0; i < nb_output; ++i) {
diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c
index 1dbe4f29a4..7d39acb653 100644
--- a/libavfilter/vf_dnn_detect.c
+++ b/libavfilter/vf_dnn_detect.c
@@ -203,10 +203,40 @@ static int read_detect_label_file(AVFilterContext
*context)
return 0;
}
+static int check_output_nb(DnnDetectContext *ctx, DNNBackendType backend_type,
int output_nb)
+{
+switch(backend_type) {
+case DNN_TF:
+if (output_nb != 4) {
+av_log(ctx, AV_LOG_ERROR, "Only support tensorflow detect model
with 4 outputs, \
+ but get %d instead\n", output_nb);
+return AVERROR(EINVAL);
+}
+return 0;
+case DNN_OV:
+if (output_nb != 1) {
+av_log(ctx, AV_LOG_ERROR, "Dnn detect filter with openvino backend
needs 1 output only, \
+ but get %d instead\n", output_nb);
+return AVERROR(EINVAL);
+}
+return 0;
+default:
+avpriv_report_missing_feature(ctx, "Dnn detect filter does not support
current backend\n");
+return AVERROR(EINVAL);
+}
+return 0;
+}
+
static av_cold int dnn_detect_init(AVFilterContext *context)
{
DnnDetectContext *ctx = context->priv;
-int ret = ff_dnn_init(>dnnctx, DFT_ANALYTICS_DETECT, context);
+DnnContext *dnn_ctx = >dnnctx;
+int ret;
+
+ret = ff_dnn_init(>dnnctx, DFT_ANALYTICS_DETECT, context);
+if (ret < 0)
+return ret;
+ret = check_output_nb(ctx, dnn_ctx->backend_type, dnn_ctx->nb_outputs);
if (ret < 0)
return ret;
ff_dnn_set_detect_post_proc(>dnnctx, dnn_detect_post_proc);
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/4] lavfi/dnn_backend_tensorflow: add multiple outputs support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 49
libavfilter/dnn_filter_common.c | 45 +
libavfilter/dnn_filter_common.h | 6 ++--
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_sr.c | 2 +-
5 files changed, 71 insertions(+), 33 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index f70e3d4659..5c85b562c4 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -155,7 +155,7 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
TF_DeleteStatus(status);
// currently only NHWC is supported
-av_assert0(dims[0] == 1);
+av_assert0(dims[0] == 1 || dims[0] == -1);
input->height = dims[1];
input->width = dims[2];
input->channels = dims[3];
@@ -707,7 +707,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
TF_Output *tf_outputs;
TFModel *tf_model = model->model;
TFContext *ctx = _model->ctx;
-DNNData input, output;
+DNNData input, *outputs;
TF_Tensor **output_tensors;
TF_Output tf_input;
TF_Tensor *input_tensor;
@@ -738,14 +738,6 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
}
}
-if (nb_output != 1) {
-// currently, the filter does not need multiple outputs,
-// so we just pending the support until we really need it.
-TF_DeleteTensor(input_tensor);
-avpriv_report_missing_feature(ctx, "multiple outputs");
-return DNN_ERROR;
-}
-
tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs));
if (tf_outputs == NULL) {
TF_DeleteTensor(input_tensor);
@@ -785,23 +777,31 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
return DNN_ERROR;
}
+outputs = av_malloc_array(nb_output, sizeof(*outputs));
+if (!outputs) {
+TF_DeleteTensor(input_tensor);
+av_freep(_outputs);
+av_freep(_tensors);
+av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n");
\
+return DNN_ERROR;
+}
+
for (uint32_t i = 0; i < nb_output; ++i) {
-output.height = TF_Dim(output_tensors[i], 1);
-output.width = TF_Dim(output_tensors[i], 2);
-output.channels = TF_Dim(output_tensors[i], 3);
-output.data = TF_TensorData(output_tensors[i]);
-output.dt = TF_TensorType(output_tensors[i]);
-
-if (do_ioproc) {
-if (tf_model->model->frame_post_proc != NULL) {
-tf_model->model->frame_post_proc(out_frame, ,
tf_model->model->filter_ctx);
-} else {
-ff_proc_from_dnn_to_frame(out_frame, , ctx);
-}
+outputs[i].height = TF_Dim(output_tensors[i], 1);
+outputs[i].width = TF_Dim(output_tensors[i], 2);
+outputs[i].channels = TF_Dim(output_tensors[i], 3);
+outputs[i].data = TF_TensorData(output_tensors[i]);
+outputs[i].dt = TF_TensorType(output_tensors[i]);
+}
+if (do_ioproc) {
+if (tf_model->model->frame_post_proc != NULL) {
+tf_model->model->frame_post_proc(out_frame, outputs,
tf_model->model->filter_ctx);
} else {
-out_frame->width = output.width;
-out_frame->height = output.height;
+ff_proc_from_dnn_to_frame(out_frame, outputs, ctx);
}
+} else {
+out_frame->width = outputs[0].width;
+out_frame->height = outputs[0].height;
}
for (uint32_t i = 0; i < nb_output; ++i) {
@@ -812,6 +812,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
TF_DeleteTensor(input_tensor);
av_freep(_tensors);
av_freep(_outputs);
+av_freep();
return DNN_SUCCESS;
}
diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c
index 1b922455a3..4cbfdbf52a 100644
--- a/libavfilter/dnn_filter_common.c
+++ b/libavfilter/dnn_filter_common.c
@@ -17,6 +17,39 @@
*/
#include "dnn_filter_common.h"
+#include "libavutil/avstring.h"
+
+#define MAX_SUPPORTED_OUTPUTS_NB 4
+
+static char **separate_output_names(const char *expr, const char *val_sep, int
*separated_nb)
+{
+char *val, **parsed_vals = NULL;
+int val_num = 0;
+if (!expr || !val_sep || !separated_nb) {
+return NULL;
+}
+
+parsed_vals = av_mallocz_array(MAX_SUPPORTED_OUTPUTS_NB,
sizeof(*parsed_vals));
+if (!parsed_vals) {
+return NULL;
+}
+
+do {
+val = av_get_token(, val_sep);
+if(val) {
+parsed_vals[val_num] = val;
+val_num++;
+}
+if (*expr) {
+expr++;
+}
+} while(*expr);
+
+parsed_vals[val_num] = NULL;
+*separa
[FFmpeg-devel] [PATCH 1/4] dnn: add DCO_RGB color order to enum DNNColorOrder
Adding DCO_RGB color order to DNNColorOrder, since tensorflow model
needs this kind of color oder as input.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 1 +
libavfilter/dnn/dnn_io_proc.c| 14 +++---
libavfilter/dnn_interface.h | 1 +
3 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 076dd3d6a9..f70e3d4659 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -143,6 +143,7 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
tf_output.index = 0;
input->dt = TF_OperationOutputType(tf_output);
+input->order = DCO_RGB;
status = TF_NewStatus();
TF_GraphGetTensorShape(tf_model->graph, tf_output, dims, 4, status);
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index e104cc5064..5f6ce36b96 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -167,11 +167,19 @@ static DNNReturnType
proc_from_frame_to_dnn_frameprocessing(AVFrame *frame, DNND
static enum AVPixelFormat get_pixel_format(DNNData *data)
{
-if (data->dt == DNN_UINT8 && data->order == DCO_BGR) {
-return AV_PIX_FMT_BGR24;
+if (data->dt == DNN_UINT8) {
+switch (data->order) {
+case DCO_BGR:
+return AV_PIX_FMT_BGR24;
+case DCO_RGB:
+return AV_PIX_FMT_RGB24;
+default:
+av_assert0(!"unsupported data pixel format.\n");
+return AV_PIX_FMT_BGR24;
+}
}
-av_assert0(!"not supported yet.\n");
+av_assert0(!"unsupported data type.\n");
return AV_PIX_FMT_BGR24;
}
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index ae5a488341..92c3b0fc6e 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -39,6 +39,7 @@ typedef enum {DNN_FLOAT = 1, DNN_UINT8 = 4} DNNDataType;
typedef enum {
DCO_NONE,
DCO_BGR,
+DCO_RGB,
} DNNColorOrder;
typedef enum {
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 3/3] lavfi/dnn_backend_tensorflow.c: fix mem leak in execute_model_tf
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index c18cb4063f..c0aa510630 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -766,18 +766,21 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
if (nb_output != 1) {
// currently, the filter does not need multiple outputs,
// so we just pending the support until we really need it.
+TF_DeleteTensor(input_tensor);
avpriv_report_missing_feature(ctx, "multiple outputs");
return DNN_ERROR;
}
tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs));
if (tf_outputs == NULL) {
+TF_DeleteTensor(input_tensor);
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for
*tf_outputs\n"); \
return DNN_ERROR;
}
output_tensors = av_mallocz_array(nb_output, sizeof(*output_tensors));
if (!output_tensors) {
+TF_DeleteTensor(input_tensor);
av_freep(_outputs);
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output
tensor\n"); \
return DNN_ERROR;
@@ -786,6 +789,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
for (int i = 0; i < nb_output; ++i) {
tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph,
output_names[i]);
if (!tf_outputs[i].oper) {
+TF_DeleteTensor(input_tensor);
av_freep(_outputs);
av_freep(_tensors);
av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in
model\n", output_names[i]); \
@@ -799,6 +803,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
tf_outputs, output_tensors, nb_output,
NULL, 0, NULL, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK) {
+TF_DeleteTensor(input_tensor);
av_freep(_outputs);
av_freep(_tensors);
av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing
model\n");
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/3] lavfi/dnn_backend_tensorflow.c: fix mem leak in load_native_model
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 55 ++--
1 file changed, 31 insertions(+), 24 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index e016571304..c18cb4063f 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -330,7 +330,7 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_OperationDescription *op_desc;
TF_Output input;
int64_t strides[] = {1, 1, 1, 1};
-TF_Tensor *tensor;
+TF_Tensor *kernel_tensor = NULL, *biases_tensor = NULL;
int64_t dims[4];
int dims_len;
char name_buffer[NAME_BUFFER_SIZE];
@@ -347,17 +347,15 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
dims[2] = params->kernel_size;
dims[3] = params->input_num;
dims_len = 4;
-tensor = TF_AllocateTensor(TF_FLOAT, dims, dims_len, size * sizeof(float));
-memcpy(TF_TensorData(tensor), params->kernel, size * sizeof(float));
-TF_SetAttrTensor(op_desc, "value", tensor, tf_model->status);
+kernel_tensor = TF_AllocateTensor(TF_FLOAT, dims, dims_len, size *
sizeof(float));
+memcpy(TF_TensorData(kernel_tensor), params->kernel, size * sizeof(float));
+TF_SetAttrTensor(op_desc, "value", kernel_tensor, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to set value for kernel of conv
layer %d\n", layer);
-return DNN_ERROR;
+goto err;
}
op = TF_FinishOperation(op_desc, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to add kernel to conv layer %d\n",
layer);
-return DNN_ERROR;
+goto err;
}
snprintf(name_buffer, NAME_BUFFER_SIZE, "transpose%d", layer);
@@ -370,8 +368,7 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_SetAttrType(op_desc, "Tperm", TF_INT32);
op = TF_FinishOperation(op_desc, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to add transpose to conv layer
%d\n", layer);
-return DNN_ERROR;
+goto err;
}
snprintf(name_buffer, NAME_BUFFER_SIZE, "conv2d%d", layer);
@@ -385,8 +382,7 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_SetAttrString(op_desc, "padding", "VALID", 5);
*cur_op = TF_FinishOperation(op_desc, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to add conv2d to conv layer %d\n",
layer);
-return DNN_ERROR;
+goto err;
}
snprintf(name_buffer, NAME_BUFFER_SIZE, "conv_biases%d", layer);
@@ -394,17 +390,15 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_SetAttrType(op_desc, "dtype", TF_FLOAT);
dims[0] = params->output_num;
dims_len = 1;
-tensor = TF_AllocateTensor(TF_FLOAT, dims, dims_len, params->output_num *
sizeof(float));
-memcpy(TF_TensorData(tensor), params->biases, params->output_num *
sizeof(float));
-TF_SetAttrTensor(op_desc, "value", tensor, tf_model->status);
+biases_tensor = TF_AllocateTensor(TF_FLOAT, dims, dims_len,
params->output_num * sizeof(float));
+memcpy(TF_TensorData(biases_tensor), params->biases, params->output_num *
sizeof(float));
+TF_SetAttrTensor(op_desc, "value", biases_tensor, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to set value for conv_biases of conv
layer %d\n", layer);
-return DNN_ERROR;
+goto err;
}
op = TF_FinishOperation(op_desc, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to add conv_biases to conv layer
%d\n", layer);
-return DNN_ERROR;
+goto err;
}
snprintf(name_buffer, NAME_BUFFER_SIZE, "bias_add%d", layer);
@@ -416,8 +410,7 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_SetAttrType(op_desc, "T", TF_FLOAT);
*cur_op = TF_FinishOperation(op_desc, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK){
-av_log(ctx, AV_LOG_ERROR, "Failed to add bias_add to conv layer %d\n",
layer);
-return DNN_ERROR;
+goto err;
}
snprintf(name_buffer, NAME_BUFFER_SIZE, "activation%d", layer);
@@ -440,11 +433,15 @@ static DNNReturnType add_conv_layer(TFModel *tf_model,
TF_Operation *transpose_o
TF_SetAttrType(op_desc, &q
[FFmpeg-devel] [PATCH 1/3] lavfi/dnn_backend_tensorflow.c: fix mem leak in load_tf_model
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 8
1 file changed, 8 insertions(+)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 750a476726..e016571304 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -282,6 +282,9 @@ static DNNReturnType load_tf_model(TFModel *tf_model, const
char *model_filename
TF_SetConfig(sess_opts, sess_config,
sess_config_length,tf_model->status);
av_freep(_config);
if (TF_GetCode(tf_model->status) != TF_OK) {
+TF_DeleteGraph(tf_model->graph);
+TF_DeleteStatus(tf_model->status);
+TF_DeleteSessionOptions(sess_opts);
av_log(ctx, AV_LOG_ERROR, "Failed to set config for sess options
with %s\n",
tf_model->ctx.options.sess_config);
return DNN_ERROR;
@@ -292,6 +295,8 @@ static DNNReturnType load_tf_model(TFModel *tf_model, const
char *model_filename
TF_DeleteSessionOptions(sess_opts);
if (TF_GetCode(tf_model->status) != TF_OK)
{
+TF_DeleteGraph(tf_model->graph);
+TF_DeleteStatus(tf_model->status);
av_log(ctx, AV_LOG_ERROR, "Failed to create new session with model
graph\n");
return DNN_ERROR;
}
@@ -304,6 +309,9 @@ static DNNReturnType load_tf_model(TFModel *tf_model, const
char *model_filename
_op, 1, NULL, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK)
{
+TF_DeleteSession(tf_model->session, tf_model->status);
+TF_DeleteGraph(tf_model->graph);
+TF_DeleteStatus(tf_model->status);
av_log(ctx, AV_LOG_ERROR, "Failed to run session when
initializing\n");
return DNN_ERROR;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V3 3/3] dnn/openvino: support model input resize
OpenVINO APIs require specify input size to run the model, while some
OpenVINO model does accept different input size. To enable this feature
adding input_resizable option here for easier use.
Setting bool variable input_resizable to specify if the input can be resizable
or not.
input_resizable = 1 means support input resize, aka accept different input size.
input_resizable = 0 (default) means do not support input resize.
Please make sure the inference model does accept different input size
before use this option, otherwise the inference engine may report error(s).
eg: ./ffmpeg -i video_name.mp4 -vf dnn_processing=dnn_backend=openvino:\
model=model_name.xml:input=input_name:output=output_name:\
options=device=CPU\_resizable=1 -y output_video_name.mp4
Signed-off-by: Ting Fu
---
V3:
rebase to latest code and add missing code
libavfilter/dnn/dnn_backend_openvino.c | 21 +++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index ecfd2b3f36..8a7abb33f0 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -38,6 +38,7 @@ typedef struct OVOptions{
char *device_type;
int nireq;
int batch_size;
+int input_resizable;
} OVOptions;
typedef struct OVContext {
@@ -86,6 +87,7 @@ static const AVOption dnn_openvino_options[] = {
{ "device", "device to run model", OFFSET(options.device_type),
AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS },
{ "nireq", "number of request", OFFSET(options.nireq),
AV_OPT_TYPE_INT,{ .i64 = 0 }, 0, INT_MAX, FLAGS },
{ "batch_size", "batch size per request", OFFSET(options.batch_size),
AV_OPT_TYPE_INT,{ .i64 = 1 }, 1, 1000, FLAGS},
+{ "input_resizable", "can input be resizable or not",
OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1,
FLAGS },
{ NULL }
};
@@ -400,6 +402,7 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
size_t model_input_count = 0;
dimensions_t dims;
precision_e precision;
+int input_resizable = ctx->options.input_resizable;
status = ie_network_get_inputs_number(ov_model->network,
_input_count);
if (status != OK) {
@@ -423,8 +426,8 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
}
input->channels = dims.dims[1];
-input->height = dims.dims[2];
-input->width= dims.dims[3];
+input->height = input_resizable ? -1 : dims.dims[2];
+input->width= input_resizable ? -1 : dims.dims[3];
input->dt = precision_to_datatype(precision);
return DNN_SUCCESS;
} else {
@@ -450,6 +453,8 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
AVFrame *in_frame = av_frame_alloc();
AVFrame *out_frame = NULL;
TaskItem *ptask =
+IEStatusCode status;
+input_shapes_t input_shapes;
if (!in_frame) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input
frame\n");
@@ -464,6 +469,18 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (ctx->options.input_resizable) {
+status = ie_network_get_input_shapes(ov_model->network, _shapes);
+input_shapes.shapes->shape.dims[2] = input_height;
+input_shapes.shapes->shape.dims[3] = input_width;
+status |= ie_network_reshape(ov_model->network, input_shapes);
+ie_network_input_shapes_free(_shapes);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to reshape input size for %s\n",
input_name);
+return DNN_ERROR;
+}
+}
+
if (!ov_model->exe_network) {
if (init_model_ov(ov_model) != DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V3 2/3] dnn/openvino: refine code for better model initialization
Move openvino model/inference request creation and initialization steps
from ff_dnn_load_model_ov to new function init_model_ov, for later input
resize support.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 203 ++---
1 file changed, 118 insertions(+), 85 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 8476f4fb38..ecfd2b3f36 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -248,6 +248,103 @@ static void infer_completion_callback(void *args)
}
}
+static DNNReturnType init_model_ov(OVModel *ov_model)
+{
+OVContext *ctx = _model->ctx;
+IEStatusCode status;
+ie_available_devices_t a_dev;
+ie_config_t config = {NULL, NULL, NULL};
+char *all_dev_names = NULL;
+
+// batch size
+if (ctx->options.batch_size <= 0) {
+ctx->options.batch_size = 1;
+}
+
+if (ctx->options.batch_size > 1) {
+input_shapes_t input_shapes;
+status = ie_network_get_input_shapes(ov_model->network, _shapes);
+if (status != OK)
+goto err;
+for (int i = 0; i < input_shapes.shape_num; i++)
+input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size;
+status = ie_network_reshape(ov_model->network, input_shapes);
+ie_network_input_shapes_free(_shapes);
+if (status != OK)
+goto err;
+}
+
+status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n");
+status = ie_core_get_available_devices(ov_model->core, _dev);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n");
+goto err;
+}
+for (int i = 0; i < a_dev.num_devices; i++) {
+APPEND_STRING(all_dev_names, a_dev.devices[i])
+}
+av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all
available devices are: \"%s\"\n",
+ ctx->options.device_type, all_dev_names);
+goto err;
+}
+
+// create infer_request for sync execution
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
_model->infer_request);
+if (status != OK)
+goto err;
+
+// create infer_requests for async execution
+if (ctx->options.nireq <= 0) {
+// the default value is a rough estimation
+ctx->options.nireq = av_cpu_count() / 2 + 1;
+}
+
+ov_model->request_queue = ff_safe_queue_create();
+if (!ov_model->request_queue) {
+goto err;
+}
+
+for (int i = 0; i < ctx->options.nireq; i++) {
+RequestItem *item = av_mallocz(sizeof(*item));
+if (!item) {
+goto err;
+}
+
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
>infer_request);
+if (status != OK) {
+av_freep();
+goto err;
+}
+
+item->tasks = av_malloc_array(ctx->options.batch_size,
sizeof(*item->tasks));
+if (!item->tasks) {
+av_freep();
+goto err;
+}
+item->task_count = 0;
+
+item->callback.completeCallBackFunc = infer_completion_callback;
+item->callback.args = item;
+if (ff_safe_queue_push_back(ov_model->request_queue, item) < 0) {
+av_freep();
+goto err;
+}
+}
+
+ov_model->task_queue = ff_queue_create();
+if (!ov_model->task_queue) {
+goto err;
+}
+
+return DNN_SUCCESS;
+
+err:
+ff_dnn_free_model_ov(_model->model);
+return DNN_ERROR;
+}
+
static DNNReturnType execute_model_ov(RequestItem *request)
{
IEStatusCode status;
@@ -367,6 +464,13 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (!ov_model->exe_network) {
+if (init_model_ov(ov_model) != DNN_SUCCESS) {
+av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
+return DNN_ERROR;
+};
+}
+
task.done = 0;
task.do_ioproc = 0;
task.async = 0;
@@ -391,13 +495,10 @@ static DNNReturnType get_output_ov(void *model, const
char *input_name, int inpu
DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char
*options, AVFilterContext *filter_ctx)
{
-char *all_dev_names = NULL;
DNNModel *model = NULL;
OVModel *ov_model = NULL;
OVContext *ctx = NULL;
IEStatusCode status;
-ie_config_t config = {NULL, NULL, NULL};
-ie_available_devices_t a_dev;
[FFmpeg-devel] [PATCH V3 1/3] dnn/openvino: remove unnecessary code
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 8
1 file changed, 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 5271d1caa5..8476f4fb38 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -325,14 +325,6 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
return DNN_ERROR;
}
-// The order of dims in the openvino is fixed and it is always
NCHW for 4-D data.
-// while we pass NHWC data from FFmpeg to openvino
-status = ie_network_set_input_layout(ov_model->network,
input_name, NHWC);
-if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Input \"%s\" does not match layout
NHWC\n", input_name);
-return DNN_ERROR;
-}
-
input->channels = dims.dims[1];
input->height = dims.dims[2];
input->width= dims.dims[3];
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 2/3] dnn/openvino: refine code for better model initialization
Move openvino model/inference request creation and initialization steps
from ff_dnn_load_model_ov to new function init_model_ov, for later input
resize support.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 196 ++---
1 file changed, 111 insertions(+), 85 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 8476f4fb38..0b125eef65 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -248,6 +248,96 @@ static void infer_completion_callback(void *args)
}
}
+static DNNReturnType init_model_ov(OVModel *ov_model)
+{
+OVContext *ctx = _model->ctx;
+IEStatusCode status;
+ie_available_devices_t a_dev;
+ie_config_t config = {NULL, NULL, NULL};
+char *all_dev_names = NULL;
+
+// batch size
+if (ctx->options.batch_size <= 0) {
+ctx->options.batch_size = 1;
+}
+
+if (ctx->options.batch_size > 1) {
+input_shapes_t input_shapes;
+status = ie_network_get_input_shapes(ov_model->network, _shapes);
+if (status != OK)
+goto err;
+for (int i = 0; i < input_shapes.shape_num; i++)
+input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size;
+status = ie_network_reshape(ov_model->network, input_shapes);
+ie_network_input_shapes_free(_shapes);
+if (status != OK)
+goto err;
+}
+
+status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n");
+status = ie_core_get_available_devices(ov_model->core, _dev);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n");
+goto err;
+}
+for (int i = 0; i < a_dev.num_devices; i++) {
+APPEND_STRING(all_dev_names, a_dev.devices[i])
+}
+av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all
available devices are: \"%s\"\n",
+ ctx->options.device_type, all_dev_names);
+goto err;
+}
+
+// create infer_request for sync execution
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
_model->infer_request);
+if (status != OK)
+goto err;
+
+// create infer_requests for async execution
+if (ctx->options.nireq <= 0) {
+// the default value is a rough estimation
+ctx->options.nireq = av_cpu_count() / 2 + 1;
+}
+
+ov_model->request_queue = ff_safe_queue_create();
+if (!ov_model->request_queue) {
+goto err;
+}
+
+for (int i = 0; i < ctx->options.nireq; i++) {
+ie_infer_request_t *request;
+RequestItem *item = av_mallocz(sizeof(*item));
+if (!item) {
+goto err;
+}
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
);
+if (status != OK) {
+av_freep();
+goto err;
+}
+item->infer_request = request;
+item->callback.completeCallBackFunc = infer_completion_callback;
+item->callback.args = item;
+if (ff_safe_queue_push_back(ov_model->request_queue, item) < 0) {
+av_freep();
+goto err;
+}
+}
+
+ov_model->task_queue = ff_queue_create();
+if (!ov_model->task_queue) {
+goto err;
+}
+
+return DNN_SUCCESS;
+
+err:
+ff_dnn_free_model_ov(_model->model);
+return DNN_ERROR;
+}
+
static DNNReturnType execute_model_ov(RequestItem *request)
{
IEStatusCode status;
@@ -367,6 +457,13 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (!ov_model->exe_network) {
+if (init_model_ov(ov_model) != DNN_SUCCESS) {
+av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
+return DNN_ERROR;
+};
+}
+
task.done = 0;
task.do_ioproc = 0;
task.async = 0;
@@ -391,13 +488,10 @@ static DNNReturnType get_output_ov(void *model, const
char *input_name, int inpu
DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char
*options, AVFilterContext *filter_ctx)
{
-char *all_dev_names = NULL;
DNNModel *model = NULL;
OVModel *ov_model = NULL;
OVContext *ctx = NULL;
IEStatusCode status;
-ie_config_t config = {NULL, NULL, NULL};
-ie_available_devices_t a_dev;
model = av_mallocz(sizeof(DNNModel));
if (!model){
@@ -429,88 +523,6 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options,
[FFmpeg-devel] [PATCH V2 1/3] dnn/openvino: remove unnecessary code
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 8
1 file changed, 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 5271d1caa5..8476f4fb38 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -325,14 +325,6 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
return DNN_ERROR;
}
-// The order of dims in the openvino is fixed and it is always
NCHW for 4-D data.
-// while we pass NHWC data from FFmpeg to openvino
-status = ie_network_set_input_layout(ov_model->network,
input_name, NHWC);
-if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Input \"%s\" does not match layout
NHWC\n", input_name);
-return DNN_ERROR;
-}
-
input->channels = dims.dims[1];
input->height = dims.dims[2];
input->width= dims.dims[3];
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 3/3] dnn/openvino: support model input resize
OpenVINO APIs require specify input size to run the model, while some
OpenVINO model does accept different input size. To enable this feature
adding input_resizable option here for easier use.
Setting bool variable input_resizable to specify if the input can be resizable
or not.
input_resizable = 1 means support input resize, aka accept different input size.
input_resizable = 0 (default) means do not support input resize.
Please make sure the inference model does accept different input size
before use this option, otherwise the inference engine may report error(s).
eg: ./ffmpeg -i video_name.mp4 -vf dnn_processing=dnn_backend=openvino:\
model=model_name.xml:input=input_name:output=output_name:\
options=device=CPU\_resizable=1 -y output_video_name.mp4
Signed-off-by: Ting Fu
---
V2:
rebase to latest code
libavfilter/dnn/dnn_backend_openvino.c | 21 +++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 0b125eef65..1664ff5268 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -38,6 +38,7 @@ typedef struct OVOptions{
char *device_type;
int nireq;
int batch_size;
+int input_resizable;
} OVOptions;
typedef struct OVContext {
@@ -86,6 +87,7 @@ static const AVOption dnn_openvino_options[] = {
{ "device", "device to run model", OFFSET(options.device_type),
AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS },
{ "nireq", "number of request", OFFSET(options.nireq),
AV_OPT_TYPE_INT,{ .i64 = 0 }, 0, INT_MAX, FLAGS },
{ "batch_size", "batch size per request", OFFSET(options.batch_size),
AV_OPT_TYPE_INT,{ .i64 = 1 }, 1, 1000, FLAGS},
+{ "input_resizable", "can input be resizable or not",
OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1,
FLAGS },
{ NULL }
};
@@ -393,6 +395,7 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
size_t model_input_count = 0;
dimensions_t dims;
precision_e precision;
+int input_resizable = ctx->options.input_resizable;
status = ie_network_get_inputs_number(ov_model->network,
_input_count);
if (status != OK) {
@@ -416,8 +419,8 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
}
input->channels = dims.dims[1];
-input->height = dims.dims[2];
-input->width= dims.dims[3];
+input->height = input_resizable ? -1 : dims.dims[2];
+input->width= input_resizable ? -1 : dims.dims[3];
input->dt = precision_to_datatype(precision);
return DNN_SUCCESS;
} else {
@@ -443,6 +446,8 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
AVFrame *in_frame = av_frame_alloc();
AVFrame *out_frame = NULL;
TaskItem *ptask =
+IEStatusCode status;
+input_shapes_t input_shapes;
if (!in_frame) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input
frame\n");
@@ -457,6 +462,18 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (ctx->options.input_resizable) {
+status = ie_network_get_input_shapes(ov_model->network, _shapes);
+input_shapes.shapes->shape.dims[2] = input_height;
+input_shapes.shapes->shape.dims[3] = input_width;
+status |= ie_network_reshape(ov_model->network, input_shapes);
+ie_network_input_shapes_free(_shapes);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to reshape input size for %s\n",
input_name);
+return DNN_ERROR;
+}
+}
+
if (!ov_model->exe_network) {
if (init_model_ov(ov_model) != DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 3/3] dnn/openvino: support model input resize
OpenVINO APIs require specify input size to run the model, while some
OpenVINO model does accept different input size. To enable this feature
adding input_resizable option here for easier use.
Setting bool variable input_resizable to specify if the input can be resizable
or not.
input_resizable = 1 means support input resize, aka accept different input size.
input_resizable = 0 (default) means do not support input resize.
Please make sure the inference model does accept different input size
before use this option, otherwise the inference engine may report error(s).
eg: ./ffmpeg -i video_name.mp4 -vf dnn_processing=dnn_backend=openvino:\
model=model_name.xml:input=input_name:output=output_name:\
options=device=CPU\_resizable=1 -y output_video_name.mp4
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 21 +++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index d6e0593a0b..65d74702ff 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -37,6 +37,7 @@
typedef struct OVOptions{
char *device_type;
int nireq;
+int input_resizable;
} OVOptions;
typedef struct OVContext {
@@ -83,6 +84,7 @@ typedef struct RequestItem {
static const AVOption dnn_openvino_options[] = {
{ "device", "device to run model", OFFSET(options.device_type),
AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS },
{ "nireq", "number of request", OFFSET(options.nireq),
AV_OPT_TYPE_INT,{ .i64 = 0 }, 0, INT_MAX, FLAGS },
+{ "input_resizable", "can input be resizable or not",
OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1,
FLAGS },
{ NULL }
};
@@ -334,6 +336,7 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
size_t model_input_count = 0;
dimensions_t dims;
precision_e precision;
+int input_resizable = ctx->options.input_resizable;
status = ie_network_get_inputs_number(ov_model->network,
_input_count);
if (status != OK) {
@@ -357,8 +360,8 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
}
input->channels = dims.dims[1];
-input->height = dims.dims[2];
-input->width= dims.dims[3];
+input->height = input_resizable ? -1 : dims.dims[2];
+input->width= input_resizable ? -1 : dims.dims[3];
input->dt = precision_to_datatype(precision);
return DNN_SUCCESS;
} else {
@@ -383,6 +386,8 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
RequestItem request;
AVFrame *in_frame = av_frame_alloc();
AVFrame *out_frame = NULL;
+IEStatusCode status;
+input_shapes_t input_shapes;
if (!in_frame) {
av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input
frame\n");
@@ -397,6 +402,18 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (ctx->options.input_resizable) {
+status = ie_network_get_input_shapes(ov_model->network, _shapes);
+input_shapes.shapes->shape.dims[2] = input_height;
+input_shapes.shapes->shape.dims[3] = input_width;
+status |= ie_network_reshape(ov_model->network, input_shapes);
+ie_network_input_shapes_free(_shapes);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to reshape input size for %s\n",
input_name);
+return DNN_ERROR;
+}
+}
+
if (!ov_model->exe_network) {
if (init_model_ov(ov_model) != DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/3] dnn/openvino: refine code for better model initialization
Move openvino model/inference request creation and initialization steps
from ff_dnn_load_model_ov to new function init_model_ov, for later input
resize support.
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 153 +++--
1 file changed, 93 insertions(+), 60 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 050be97209..d6e0593a0b 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -217,6 +217,78 @@ static void infer_completion_callback(void *args)
task->done = 1;
}
+static DNNReturnType init_model_ov(OVModel *ov_model)
+{
+OVContext *ctx = _model->ctx;
+IEStatusCode status;
+ie_available_devices_t a_dev;
+ie_config_t config = {NULL, NULL, NULL};
+char *all_dev_names = NULL;
+
+status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n");
+status = ie_core_get_available_devices(ov_model->core, _dev);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n");
+goto err;
+}
+for (int i = 0; i < a_dev.num_devices; i++) {
+APPEND_STRING(all_dev_names, a_dev.devices[i])
+}
+av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all
available devices are: \"%s\"\n",
+ ctx->options.device_type, all_dev_names);
+goto err;
+}
+
+// create infer_request for sync execution
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
_model->infer_request);
+if (status != OK)
+goto err;
+
+// create infer_requests for async execution
+if (ctx->options.nireq <= 0) {
+// the default value is a rough estimation
+ctx->options.nireq = av_cpu_count() / 2 + 1;
+}
+
+ov_model->request_queue = ff_safe_queue_create();
+if (!ov_model->request_queue) {
+goto err;
+}
+
+for (int i = 0; i < ctx->options.nireq; i++) {
+ie_infer_request_t *request;
+RequestItem *item = av_mallocz(sizeof(*item));
+if (!item) {
+goto err;
+}
+status = ie_exec_network_create_infer_request(ov_model->exe_network,
);
+if (status != OK) {
+av_freep();
+goto err;
+}
+item->infer_request = request;
+item->callback.completeCallBackFunc = infer_completion_callback;
+item->callback.args = item;
+if (ff_safe_queue_push_back(ov_model->request_queue, item) < 0) {
+av_freep();
+goto err;
+}
+}
+
+ov_model->task_queue = ff_queue_create();
+if (!ov_model->task_queue) {
+goto err;
+}
+
+return DNN_SUCCESS;
+
+err:
+ff_dnn_free_model_ov(_model->model);
+return DNN_ERROR;
+}
+
static DNNReturnType execute_model_ov(TaskItem *task, RequestItem *request)
{
IEStatusCode status;
@@ -325,6 +397,13 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
in_frame->width = input_width;
in_frame->height = input_height;
+if (!ov_model->exe_network) {
+if (init_model_ov(ov_model) != DNN_SUCCESS) {
+av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
+return DNN_ERROR;
+};
+}
+
task.done = 0;
task.do_ioproc = 0;
task.async = 0;
@@ -347,13 +426,10 @@ static DNNReturnType get_output_ov(void *model, const
char *input_name, int inpu
DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char
*options, AVFilterContext *filter_ctx)
{
-char *all_dev_names = NULL;
DNNModel *model = NULL;
OVModel *ov_model = NULL;
OVContext *ctx = NULL;
IEStatusCode status;
-ie_config_t config = {NULL, NULL, NULL};
-ie_available_devices_t a_dev;
model = av_mallocz(sizeof(DNNModel));
if (!model){
@@ -385,63 +461,6 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options,
if (status != OK)
goto err;
-status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
-if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to init OpenVINO model\n");
-status = ie_core_get_available_devices(ov_model->core, _dev);
-if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n");
-goto err;
-}
-for (int i = 0; i < a_dev.num_devices; i++) {
-APPEND_STRING(all_dev_names, a_dev.devices[i])
-}
[FFmpeg-devel] [PATCH 1/3] dnn/openvino: remove unnecessary code
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 8
1 file changed, 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index d27e451eea..050be97209 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -284,14 +284,6 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
return DNN_ERROR;
}
-// The order of dims in the openvino is fixed and it is always
NCHW for 4-D data.
-// while we pass NHWC data from FFmpeg to openvino
-status = ie_network_set_input_layout(ov_model->network,
input_name, NHWC);
-if (status != OK) {
-av_log(ctx, AV_LOG_ERROR, "Input \"%s\" does not match layout
NHWC\n", input_name);
-return DNN_ERROR;
-}
-
input->channels = dims.dims[1];
input->height = dims.dims[2];
input->width= dims.dims[3];
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] dnn: add NV12 pixel format support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_io_proc.c | 2 ++
libavfilter/vf_dnn_processing.c | 30 +-
2 files changed, 23 insertions(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index c9b49be3bd..2744cb6502 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -64,6 +64,7 @@ DNNReturnType proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_
case AV_PIX_FMT_YUV410P:
case AV_PIX_FMT_YUV411P:
case AV_PIX_FMT_GRAY8:
+case AV_PIX_FMT_NV12:
sws_ctx = sws_getContext(frame->width,
frame->height,
AV_PIX_FMT_GRAYF32,
@@ -135,6 +136,7 @@ DNNReturnType proc_from_frame_to_dnn(AVFrame *frame,
DNNData *input, void *log_c
case AV_PIX_FMT_YUV410P:
case AV_PIX_FMT_YUV411P:
case AV_PIX_FMT_GRAY8:
+case AV_PIX_FMT_NV12:
sws_ctx = sws_getContext(frame->width,
frame->height,
AV_PIX_FMT_GRAY8,
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 334243bd2b..76fd2e88db 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -113,6 +113,7 @@ static int query_formats(AVFilterContext *context)
AV_PIX_FMT_GRAY8, AV_PIX_FMT_GRAYF32,
AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P,
AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV410P, AV_PIX_FMT_YUV411P,
+AV_PIX_FMT_NV12,
AV_PIX_FMT_NONE
};
AVFilterFormats *fmts_list = ff_make_format_list(pix_fmts);
@@ -161,6 +162,7 @@ static int check_modelinput_inlink(const DNNData
*model_input, const AVFilterLin
case AV_PIX_FMT_YUV444P:
case AV_PIX_FMT_YUV410P:
case AV_PIX_FMT_YUV411P:
+case AV_PIX_FMT_NV12:
if (model_input->channels != 1) {
LOG_FORMAT_CHANNEL_MISMATCH();
return AVERROR(EIO);
@@ -212,15 +214,22 @@ static int prepare_uv_scale(AVFilterLink *outlink)
if (isPlanarYUV(fmt)) {
if (inlink->w != outlink->w || inlink->h != outlink->h) {
-const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(fmt);
-int sws_src_h = AV_CEIL_RSHIFT(inlink->h, desc->log2_chroma_h);
-int sws_src_w = AV_CEIL_RSHIFT(inlink->w, desc->log2_chroma_w);
-int sws_dst_h = AV_CEIL_RSHIFT(outlink->h, desc->log2_chroma_h);
-int sws_dst_w = AV_CEIL_RSHIFT(outlink->w, desc->log2_chroma_w);
-ctx->sws_uv_scale = sws_getContext(sws_src_w, sws_src_h,
AV_PIX_FMT_GRAY8,
- sws_dst_w, sws_dst_h,
AV_PIX_FMT_GRAY8,
- SWS_BICUBIC, NULL, NULL, NULL);
-ctx->sws_uv_height = sws_src_h;
+if (fmt == AV_PIX_FMT_NV12) {
+ctx->sws_uv_scale = sws_getContext(inlink->w >> 1, inlink->h
>> 1, AV_PIX_FMT_YA8,
+ outlink->w >> 1, outlink->h
>> 1, AV_PIX_FMT_YA8,
+ SWS_BICUBIC, NULL, NULL,
NULL);
+ctx->sws_uv_height = inlink->h >> 1;
+} else {
+const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(fmt);
+int sws_src_h = AV_CEIL_RSHIFT(inlink->h, desc->log2_chroma_h);
+int sws_src_w = AV_CEIL_RSHIFT(inlink->w, desc->log2_chroma_w);
+int sws_dst_h = AV_CEIL_RSHIFT(outlink->h,
desc->log2_chroma_h);
+int sws_dst_w = AV_CEIL_RSHIFT(outlink->w,
desc->log2_chroma_w);
+ctx->sws_uv_scale = sws_getContext(sws_src_w, sws_src_h,
AV_PIX_FMT_GRAY8,
+ sws_dst_w, sws_dst_h,
AV_PIX_FMT_GRAY8,
+ SWS_BICUBIC, NULL, NULL,
NULL);
+ctx->sws_uv_height = sws_src_h;
+}
}
}
@@ -262,6 +271,9 @@ static int copy_uv_planes(DnnProcessingContext *ctx,
AVFrame *out, const AVFrame
in->data[i], in->linesize[i],
bytewidth, uv_height);
}
+} else if (in->format == AV_PIX_FMT_NV12) {
+sws_scale(ctx->sws_uv_scale, (const uint8_t **)(in->data + 1),
in->linesize + 1,
+ 0, ctx->sws_uv_height, out->data + 1, out->linesize + 1);
} else {
sws_scale(ctx->sws_uv_scale, (const uint8_t **)(in->data + 1),
in->linesize + 1,
0, ctx->sws_uv_height, out->data + 1, out->linesize + 1);
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 1/2] dnn/openvino: support run inference via GPU
for enabling OpenVINO GPU please:
1. install required OpenCL drivers, see:
https://github.com/intel/compute-runtime/releases/tag/19.41.14441
2. build OpenVINO c lib with GPU enabled: use cmake config with:
-DENABLE_CLDNN=ON
3. then make, and include the OpenVINO c lib in environment variables
detailed steps please refer:
https://github.com/openvinotoolkit/openvino/blob/master/build-instruction.md
inference model with GPU please add: optioins=device=GPU
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 52 ++
1 file changed, 44 insertions(+), 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 5d6d3ed542..2f0998046a 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -26,10 +26,18 @@
#include "dnn_backend_openvino.h"
#include "libavformat/avio.h"
#include "libavutil/avassert.h"
+#include "libavutil/opt.h"
+#include "libavutil/avstring.h"
+#include "../internal.h"
#include
+typedef struct OVOptions{
+char *device_type;
+} OVOptions;
+
typedef struct OVContext {
const AVClass *class;
+OVOptions options;
} OVContext;
typedef struct OVModel{
@@ -41,14 +49,19 @@ typedef struct OVModel{
ie_blob_t *input_blob;
} OVModel;
-static const AVClass dnn_openvino_class = {
-.class_name = "dnn_openvino",
-.item_name = av_default_item_name,
-.option = NULL,
-.version= LIBAVUTIL_VERSION_INT,
-.category = AV_CLASS_CATEGORY_FILTER,
+#define APPEND_STRING(generated_string, iterate_string)
\
+generated_string = generated_string ? av_asprintf("%s %s",
generated_string, iterate_string) : \
+ av_asprintf("%s", iterate_string);
+
+#define OFFSET(x) offsetof(OVContext, x)
+#define FLAGS AV_OPT_FLAG_FILTERING_PARAM
+static const AVOption dnn_openvino_options[] = {
+{ "device", "device to run model", OFFSET(options.device_type),
AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS },
+{ NULL }
};
+AVFILTER_DEFINE_CLASS(dnn_openvino);
+
static DNNDataType precision_to_datatype(precision_e precision)
{
switch (precision)
@@ -159,10 +172,13 @@ err:
DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char *options)
{
+char *all_dev_names = NULL;
DNNModel *model = NULL;
OVModel *ov_model = NULL;
+OVContext *ctx = NULL;
IEStatusCode status;
ie_config_t config = {NULL, NULL, NULL};
+ie_available_devices_t a_dev;
model = av_malloc(sizeof(DNNModel));
if (!model){
@@ -173,6 +189,14 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options)
if (!ov_model)
goto err;
ov_model->ctx.class = _openvino_class;
+ctx = _model->ctx;
+
+//parse options
+av_opt_set_defaults(ctx);
+if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) {
+av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options);
+goto err;
+}
status = ie_core_create("", _model->core);
if (status != OK)
@@ -182,9 +206,21 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options)
if (status != OK)
goto err;
-status = ie_core_load_network(ov_model->core, ov_model->network, "CPU",
, _model->exe_network);
-if (status != OK)
+status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to init OpenVINO model\n");
+status = ie_core_get_available_devices(ov_model->core, _dev);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n");
+goto err;
+}
+for (int i = 0; i < a_dev.num_devices; i++) {
+APPEND_STRING(all_dev_names, a_dev.devices[i])
+}
+av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all
available devices are: \"%s\"\n",
+ ctx->options.device_type, all_dev_names);
goto err;
+}
model->model = (void *)ov_model;
model->set_input = _input_ov;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 2/2] dnn/openvino: add input/output name info
show all input/output names when the input or output name not correct
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 2f0998046a..e5842906d1 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -79,6 +79,7 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
OVModel *ov_model = (OVModel *)model;
OVContext *ctx = _model->ctx;
char *model_input_name = NULL;
+char *all_input_names = NULL;
IEStatusCode status;
size_t model_input_count = 0;
dimensions_t dims;
@@ -118,12 +119,15 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
input->width= dims.dims[3];
input->dt = precision_to_datatype(precision);
return DNN_SUCCESS;
+} else {
+//incorrect input name
+APPEND_STRING(all_input_names, model_input_name)
}
ie_network_name_free(_input_name);
}
-av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
model_input_name);
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model, all input(s)
are: \"%s\"\n", input_name, all_input_names);
return DNN_ERROR;
}
@@ -246,12 +250,15 @@ err:
DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNData *outputs,
const char **output_names, uint32_t nb_output)
{
+char *model_output_name = NULL;
+char *all_output_names = NULL;
dimensions_t dims;
precision_e precision;
ie_blob_buffer_t blob_buffer;
OVModel *ov_model = (OVModel *)model->model;
OVContext *ctx = _model->ctx;
IEStatusCode status = ie_infer_request_infer(ov_model->infer_request);
+size_t model_output_count = 0;
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to start synchronous model
inference\n");
return DNN_ERROR;
@@ -262,7 +269,16 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel
*model, DNNData *outputs, c
ie_blob_t *output_blob = NULL;
status = ie_infer_request_get_blob(ov_model->infer_request,
output_name, _blob);
if (status != OK) {
+//incorrect output name
av_log(ctx, AV_LOG_ERROR, "Failed to get model output data\n");
+status = ie_network_get_outputs_number(ov_model->network,
_output_count);
+for (size_t i = 0; i < model_output_count; i++) {
+status = ie_network_get_output_name(ov_model->network, i,
_output_name);
+APPEND_STRING(all_output_names, model_output_name)
+}
+av_log(ctx, AV_LOG_ERROR,
+ "output \"%s\" may not correct, all output(s) are:
\"%s\"\n",
+ output_name, all_output_names);
return DNN_ERROR;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 1/2] dnn/openvino: add log error message
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 51 ++
1 file changed, 43 insertions(+), 8 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 034dee1839..5d6d3ed542 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -28,7 +28,12 @@
#include "libavutil/avassert.h"
#include
+typedef struct OVContext {
+const AVClass *class;
+} OVContext;
+
typedef struct OVModel{
+OVContext ctx;
ie_core_t *core;
ie_network_t *network;
ie_executable_network_t *exe_network;
@@ -36,6 +41,14 @@ typedef struct OVModel{
ie_blob_t *input_blob;
} OVModel;
+static const AVClass dnn_openvino_class = {
+.class_name = "dnn_openvino",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
static DNNDataType precision_to_datatype(precision_e precision)
{
switch (precision)
@@ -51,6 +64,7 @@ static DNNDataType precision_to_datatype(precision_e
precision)
static DNNReturnType get_input_ov(void *model, DNNData *input, const char
*input_name)
{
OVModel *ov_model = (OVModel *)model;
+OVContext *ctx = _model->ctx;
char *model_input_name = NULL;
IEStatusCode status;
size_t model_input_count = 0;
@@ -58,25 +72,33 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
precision_e precision;
status = ie_network_get_inputs_number(ov_model->network,
_input_count);
-if (status != OK)
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get input count\n");
return DNN_ERROR;
+}
for (size_t i = 0; i < model_input_count; i++) {
status = ie_network_get_input_name(ov_model->network, i,
_input_name);
-if (status != OK)
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get No.%d input's name\n",
(int)i);
return DNN_ERROR;
+}
if (strcmp(model_input_name, input_name) == 0) {
ie_network_name_free(_input_name);
status |= ie_network_get_input_dims(ov_model->network, input_name,
);
status |= ie_network_get_input_precision(ov_model->network,
input_name, );
-if (status != OK)
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to get No.%d input's dims or
precision\n", (int)i);
return DNN_ERROR;
+}
// The order of dims in the openvino is fixed and it is always
NCHW for 4-D data.
// while we pass NHWC data from FFmpeg to openvino
status = ie_network_set_input_layout(ov_model->network,
input_name, NHWC);
-if (status != OK)
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Input \"%s\" does not match layout
NHWC\n", input_name);
return DNN_ERROR;
+}
input->channels = dims.dims[1];
input->height = dims.dims[2];
@@ -88,12 +110,14 @@ static DNNReturnType get_input_ov(void *model, DNNData
*input, const char *input
ie_network_name_free(_input_name);
}
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
model_input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_ov(void *model, DNNData *input, const char
*input_name)
{
OVModel *ov_model = (OVModel *)model;
+OVContext *ctx = _model->ctx;
IEStatusCode status;
dimensions_t dims;
precision_e precision;
@@ -129,6 +153,7 @@ err:
ie_blob_free(_model->input_blob);
if (ov_model->infer_request)
ie_infer_request_free(_model->infer_request);
+av_log(ctx, AV_LOG_ERROR, "Failed to create inference instance or get
input data/dims/precision/memory\n");
return DNN_ERROR;
}
@@ -147,6 +172,7 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options)
ov_model = av_mallocz(sizeof(OVModel));
if (!ov_model)
goto err;
+ov_model->ctx.class = _openvino_class;
status = ie_core_create("", _model->core);
if (status != OK)
@@ -188,25 +214,34 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel
*model, DNNData *outputs, c
precision_e precision;
ie_blob_buffer_t blob_buffer;
OVModel *ov_model = (OVModel *)model->model;
+OVContext *ctx = _model->ctx;
IEStatusCode status = ie_infer_request_infer(ov_model->infer_request);
-if (status != OK)
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to start synchronous model
inference\n");
return DNN_ERROR;
+}
for (uint32_t i = 0; i < nb_output; ++i) {
[FFmpeg-devel] [PATCH 2/2] dnn/tensorflow: add log error message
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_tf.c | 59 ++--
1 file changed, 57 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index bdc90d5063..5e7f37bb12 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -34,7 +34,12 @@
#include
+typedef struct TFContext {
+const AVClass *class;
+} TFContext;
+
typedef struct TFModel{
+TFContext ctx;
TF_Graph *graph;
TF_Session *session;
TF_Status *status;
@@ -44,6 +49,14 @@ typedef struct TFModel{
uint32_t nb_output;
} TFModel;
+static const AVClass dnn_tensorflow_class = {
+.class_name = "dnn_tensorflow",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
static void free_buffer(void *data, size_t length)
{
av_freep();
@@ -107,13 +120,16 @@ static TF_Tensor *allocate_input_tensor(const DNNData
*input)
static DNNReturnType get_input_tf(void *model, DNNData *input, const char
*input_name)
{
TFModel *tf_model = (TFModel *)model;
+TFContext *ctx = _model->ctx;
TF_Status *status;
int64_t dims[4];
TF_Output tf_output;
tf_output.oper = TF_GraphOperationByName(tf_model->graph, input_name);
-if (!tf_output.oper)
+if (!tf_output.oper) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
tf_output.index = 0;
input->dt = TF_OperationOutputType(tf_output);
@@ -122,6 +138,7 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
TF_GraphGetTensorShape(tf_model->graph, tf_output, dims, 4, status);
if (TF_GetCode(status) != TF_OK){
TF_DeleteStatus(status);
+av_log(ctx, AV_LOG_ERROR, "Failed to get input tensor shape: number of
dimension incorrect\n");
return DNN_ERROR;
}
TF_DeleteStatus(status);
@@ -138,12 +155,14 @@ static DNNReturnType get_input_tf(void *model, DNNData
*input, const char *input
static DNNReturnType set_input_tf(void *model, DNNData *input, const char
*input_name)
{
TFModel *tf_model = (TFModel *)model;
+TFContext *ctx = _model->ctx;
TF_SessionOptions *sess_opts;
const TF_Operation *init_op = TF_GraphOperationByName(tf_model->graph,
"init");
// Input operation
tf_model->input.oper = TF_GraphOperationByName(tf_model->graph,
input_name);
if (!tf_model->input.oper){
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
}
tf_model->input.index = 0;
@@ -152,6 +171,7 @@ static DNNReturnType set_input_tf(void *model, DNNData
*input, const char *input
}
tf_model->input_tensor = allocate_input_tensor(input);
if (!tf_model->input_tensor){
+av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input
tensor\n");
return DNN_ERROR;
}
input->data = (float *)TF_TensorData(tf_model->input_tensor);
@@ -167,6 +187,7 @@ static DNNReturnType set_input_tf(void *model, DNNData
*input, const char *input
TF_DeleteSessionOptions(sess_opts);
if (TF_GetCode(tf_model->status) != TF_OK)
{
+av_log(ctx, AV_LOG_ERROR, "Failed to create new session with model
graph\n");
return DNN_ERROR;
}
@@ -178,6 +199,7 @@ static DNNReturnType set_input_tf(void *model, DNNData
*input, const char *input
_op, 1, NULL, tf_model->status);
if (TF_GetCode(tf_model->status) != TF_OK)
{
+av_log(ctx, AV_LOG_ERROR, "Failed to run session when
initializing\n");
return DNN_ERROR;
}
}
@@ -187,11 +209,13 @@ static DNNReturnType set_input_tf(void *model, DNNData
*input, const char *input
static DNNReturnType load_tf_model(TFModel *tf_model, const char
*model_filename)
{
+TFContext *ctx = _model->ctx;
TF_Buffer *graph_def;
TF_ImportGraphDefOptions *graph_opts;
graph_def = read_graph(model_filename);
if (!graph_def){
+av_log(ctx, AV_LOG_ERROR, "Failed to read model \"%s\" graph\n",
model_filename);
return DNN_ERROR;
}
tf_model->graph = TF_NewGraph();
@@ -203,6 +227,7 @@ static DNNReturnType load_tf_model(TFModel *tf_model, const
char *model_filename
if (TF_GetCode(tf_model->status) != TF_OK){
TF_DeleteGraph(tf_model->graph);
TF_DeleteStatus(tf_model->status);
+av_log(ctx, AV_LOG_ERROR, "Failed to import serialized graph to model
graph\n");
return DNN_ERROR;
}
@@ -214,6 +239,7 @@ static DNNReturnType load_tf_model(TFModel *tf_model, const
char *model_fil
[FFmpeg-devel] [PATCH V5 2/2] dnn/native: add log error message
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native.c | 55 +++
libavfilter/dnn/dnn_backend_native.h | 5 ++
.../dnn/dnn_backend_native_layer_avgpool.c| 10 +++-
.../dnn/dnn_backend_native_layer_avgpool.h| 2 +-
.../dnn/dnn_backend_native_layer_conv2d.c | 10 +++-
.../dnn/dnn_backend_native_layer_conv2d.h | 2 +-
.../dnn_backend_native_layer_depth2space.c| 10 +++-
.../dnn_backend_native_layer_depth2space.h| 2 +-
.../dnn/dnn_backend_native_layer_mathbinary.c | 11 +++-
.../dnn/dnn_backend_native_layer_mathbinary.h | 2 +-
.../dnn/dnn_backend_native_layer_mathunary.c | 11 +++-
.../dnn/dnn_backend_native_layer_mathunary.h | 2 +-
.../dnn/dnn_backend_native_layer_maximum.c| 10 +++-
.../dnn/dnn_backend_native_layer_maximum.h| 2 +-
.../dnn/dnn_backend_native_layer_pad.c| 10 +++-
.../dnn/dnn_backend_native_layer_pad.h| 2 +-
libavfilter/dnn/dnn_backend_native_layers.h | 2 +-
tests/dnn/dnn-layer-avgpool-test.c| 4 +-
tests/dnn/dnn-layer-conv2d-test.c | 4 +-
tests/dnn/dnn-layer-depth2space-test.c| 2 +-
tests/dnn/dnn-layer-mathbinary-test.c | 6 +-
tests/dnn/dnn-layer-mathunary-test.c | 2 +-
tests/dnn/dnn-layer-maximum-test.c| 2 +-
tests/dnn/dnn-layer-pad-test.c| 6 +-
24 files changed, 122 insertions(+), 52 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 436ce938da..a8fe6b94eb 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -28,15 +28,26 @@
#include "dnn_backend_native_layer_conv2d.h"
#include "dnn_backend_native_layers.h"
+static const AVClass dnn_native_class = {
+.class_name = "dnn_native",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
NativeModel *native_model = (NativeModel *)model;
+NativeContext *ctx = _model->ctx;
for (int i = 0; i < native_model->operands_num; ++i) {
DnnOperand *oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
input->dt = oprd->data_type;
av_assert0(oprd->dims[0] == 1);
input->height = oprd->dims[1];
@@ -47,30 +58,37 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
}
// do not find the input operand
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_native(void *model, DNNData *input, const char
*input_name)
{
NativeModel *native_model = (NativeModel *)model;
+NativeContext *ctx = _model->ctx;
DnnOperand *oprd = NULL;
-if (native_model->layers_num <= 0 || native_model->operands_num <= 0)
+if (native_model->layers_num <= 0 || native_model->operands_num <= 0) {
+av_log(ctx, AV_LOG_ERROR, "No operands or layers in model\n");
return DNN_ERROR;
+}
/* inputs */
for (int i = 0; i < native_model->operands_num; ++i) {
oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
break;
}
oprd = NULL;
}
-
-if (!oprd)
+if (!oprd) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
oprd->dims[0] = 1;
oprd->dims[1] = input->height;
@@ -79,11 +97,15 @@ static DNNReturnType set_input_native(void *model, DNNData
*input, const char *i
av_freep(>data);
oprd->length = calculate_operand_data_length(oprd);
-if (oprd->length <= 0)
+if (oprd->length <= 0) {
+av_log(ctx, AV_LOG_ERROR, "The input data length overflow\n");
return DNN_ERROR;
+}
oprd->data = av_malloc(oprd->length);
-if (!oprd->data)
+if (!oprd->data) {
+av_log(ctx, AV_LOG_ERROR, "Failed to malloc memory for input data\n");
return DNN_ERROR;
+}
input->data = oprd->da
[FFmpeg-devel] [PATCH V5 1/2] dnn/native: unify error return to DNN_ERROR
Unify all error return as DNN_ERROR, in order to cease model executing
when return error in ff_dnn_execute_model_native layer_func.pf_exec
Signed-off-by: Ting Fu
---
V5:
rebase to latest code
libavfilter/dnn/dnn_backend_native.c | 10 ++
libavfilter/dnn/dnn_backend_native_layer_avgpool.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_conv2d.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_depth2space.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_pad.c | 4 ++--
7 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 65a56704d3..436ce938da 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -246,10 +246,12 @@ DNNReturnType ff_dnn_execute_model_native(const DNNModel
*model, DNNData *output
for (layer = 0; layer < native_model->layers_num; ++layer){
DNNLayerType layer_type = native_model->layers[layer].type;
-layer_funcs[layer_type].pf_exec(native_model->operands,
-
native_model->layers[layer].input_operand_indexes,
-
native_model->layers[layer].output_operand_index,
-native_model->layers[layer].params);
+if (layer_funcs[layer_type].pf_exec(native_model->operands,
+
native_model->layers[layer].input_operand_indexes,
+
native_model->layers[layer].output_operand_index,
+
native_model->layers[layer].params) == DNN_ERROR) {
+return DNN_ERROR;
+}
}
for (uint32_t i = 0; i < nb_output; ++i) {
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
index 8d4d8db98c..bd7bdb4c97 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -109,7 +109,7 @@ int dnn_execute_layer_avg_pool(DnnOperand *operands, const
int32_t *input_operan
output_operand->length = calculate_operand_data_length(output_operand);
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (int y = 0; y < height_end; y += kernel_strides) {
diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
index a2202e4073..25356901c2 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
@@ -114,10 +114,10 @@ int dnn_execute_layer_conv2d(DnnOperand *operands, const
int32_t *input_operand_
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
av_assert0(channel == conv_params->input_num);
diff --git a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
index 2c8bddf23d..5a61025f7a 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
@@ -76,10 +76,10 @@ int dnn_execute_layer_depth2space(DnnOperand *operands,
const int32_t *input_ope
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (y = 0; y < height; ++y){
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
index 7d81694288..6ec1f08e9f 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
@@ -186,6 +186,6 @@ int dnn_execute_layer_math_binary(DnnOperand *operands,
const int32_t *input_ope
math_binary_not_commutative(floormod, params, input, output, operands,
input_operand_indexes);
[FFmpeg-devel] [PATCH V4 2/2] dnn/native: add log error message
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native.c | 59 +++
libavfilter/dnn/dnn_backend_native.h | 5 ++
.../dnn/dnn_backend_native_layer_avgpool.c| 10 +++-
.../dnn/dnn_backend_native_layer_avgpool.h| 2 +-
.../dnn/dnn_backend_native_layer_conv2d.c | 10 +++-
.../dnn/dnn_backend_native_layer_conv2d.h | 2 +-
.../dnn_backend_native_layer_depth2space.c| 10 +++-
.../dnn_backend_native_layer_depth2space.h| 2 +-
.../dnn/dnn_backend_native_layer_mathbinary.c | 11 +++-
.../dnn/dnn_backend_native_layer_mathbinary.h | 2 +-
.../dnn/dnn_backend_native_layer_mathunary.c | 11 +++-
.../dnn/dnn_backend_native_layer_mathunary.h | 2 +-
.../dnn/dnn_backend_native_layer_maximum.c| 10 +++-
.../dnn/dnn_backend_native_layer_maximum.h| 2 +-
.../dnn/dnn_backend_native_layer_pad.c| 10 +++-
.../dnn/dnn_backend_native_layer_pad.h| 2 +-
libavfilter/dnn/dnn_backend_native_layers.h | 2 +-
tests/dnn/dnn-layer-avgpool-test.c| 4 +-
tests/dnn/dnn-layer-conv2d-test.c | 4 +-
tests/dnn/dnn-layer-depth2space-test.c| 2 +-
tests/dnn/dnn-layer-mathbinary-test.c | 6 +-
tests/dnn/dnn-layer-mathunary-test.c | 2 +-
tests/dnn/dnn-layer-maximum-test.c| 2 +-
tests/dnn/dnn-layer-pad-test.c| 6 +-
24 files changed, 125 insertions(+), 53 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 2f5095f2ee..0cb764686d 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -28,15 +28,26 @@
#include "dnn_backend_native_layer_conv2d.h"
#include "dnn_backend_native_layers.h"
+static const AVClass dnn_native_class = {
+.class_name = "dnn_native",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
NativeModel *native_model = (NativeModel *)model;
+NativeContext *ctx = _model->ctx;
for (int i = 0; i < native_model->operands_num; ++i) {
DnnOperand *oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
input->dt = oprd->data_type;
av_assert0(oprd->dims[0] == 1);
input->height = oprd->dims[1];
@@ -47,30 +58,37 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
}
// do not find the input operand
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_output_native(void *model, DNNData *input,
const char *input_name, const char **output_names, uint32_t nb_output)
{
NativeModel *native_model = (NativeModel *)model;
+NativeContext *ctx = _model->ctx;
DnnOperand *oprd = NULL;
-if (native_model->layers_num <= 0 || native_model->operands_num <= 0)
+if (native_model->layers_num <= 0 || native_model->operands_num <= 0) {
+av_log(ctx, AV_LOG_ERROR, "No operands or layers in model\n");
return DNN_ERROR;
+}
/* inputs */
for (int i = 0; i < native_model->operands_num; ++i) {
oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
break;
}
oprd = NULL;
}
-
-if (!oprd)
+if (!oprd) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
oprd->dims[0] = 1;
oprd->dims[1] = input->height;
@@ -79,11 +97,15 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
av_freep(>data);
oprd->length = calculate_operand_data_length(oprd);
-if (oprd->length <= 0)
+if (oprd->length <= 0) {
+av_log(ctx, AV_LOG_ERROR, "The input data length overflow\n");
return DNN_ERROR;
+}
oprd->data = av_malloc(oprd->length);
-if (!oprd->data)
+if (!oprd->data) {
+av_log(ctx, AV_LOG_ERROR, "Failed to malloc memory for input data\n");
return
[FFmpeg-devel] [PATCH V4 1/2] dnn/native: unify error return to DNN_ERROR
Unify all error return as DNN_ERROR, in order to cease model executing
when return error in ff_dnn_execute_model_native layer_func.pf_exec
Signed-off-by: Ting Fu
---
V4:
Rename NetworkContext to NativeContext
Move pf_exec return DNN_ERROR from PATCH 2/2 to 1/2
libavfilter/dnn/dnn_backend_native.c | 10 ++
libavfilter/dnn/dnn_backend_native_layer_avgpool.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_conv2d.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_depth2space.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_pad.c | 4 ++--
7 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 0be9c0b53c..2f5095f2ee 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -268,10 +268,12 @@ DNNReturnType ff_dnn_execute_model_native(const DNNModel
*model, DNNData *output
for (layer = 0; layer < native_model->layers_num; ++layer){
DNNLayerType layer_type = native_model->layers[layer].type;
-layer_funcs[layer_type].pf_exec(native_model->operands,
-
native_model->layers[layer].input_operand_indexes,
-
native_model->layers[layer].output_operand_index,
-native_model->layers[layer].params);
+if (layer_funcs[layer_type].pf_exec(native_model->operands,
+
native_model->layers[layer].input_operand_indexes,
+
native_model->layers[layer].output_operand_index,
+
native_model->layers[layer].params) == DNN_ERROR) {
+return DNN_ERROR;
+}
}
for (uint32_t i = 0; i < nb; ++i) {
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
index 8d4d8db98c..bd7bdb4c97 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -109,7 +109,7 @@ int dnn_execute_layer_avg_pool(DnnOperand *operands, const
int32_t *input_operan
output_operand->length = calculate_operand_data_length(output_operand);
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (int y = 0; y < height_end; y += kernel_strides) {
diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
index a2202e4073..25356901c2 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
@@ -114,10 +114,10 @@ int dnn_execute_layer_conv2d(DnnOperand *operands, const
int32_t *input_operand_
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
av_assert0(channel == conv_params->input_num);
diff --git a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
index 2c8bddf23d..5a61025f7a 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
@@ -76,10 +76,10 @@ int dnn_execute_layer_depth2space(DnnOperand *operands,
const int32_t *input_ope
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (y = 0; y < height; ++y){
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
index dd42c329a9..bffa41cdda 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
@@ -176,6 +176,6 @@ int dnn_execute_layer_math_binary(DnnOperand *operands,
const int32_t *input_ope
}
[FFmpeg-devel] [PATCH V3 2/2] dnn/native: add log error message
Signed-off-by: Ting Fu
---
V3:
1. modify log_ctx of NativeModel to non-pointer
2. modify log_ctx of NativeModel to ctx
3. delete empty line
libavfilter/dnn/dnn_backend_native.c | 67 ++-
libavfilter/dnn/dnn_backend_native.h | 5 ++
.../dnn/dnn_backend_native_layer_avgpool.c| 10 ++-
.../dnn/dnn_backend_native_layer_avgpool.h| 2 +-
.../dnn/dnn_backend_native_layer_conv2d.c | 10 ++-
.../dnn/dnn_backend_native_layer_conv2d.h | 2 +-
.../dnn_backend_native_layer_depth2space.c| 10 ++-
.../dnn_backend_native_layer_depth2space.h| 2 +-
.../dnn/dnn_backend_native_layer_mathbinary.c | 11 ++-
.../dnn/dnn_backend_native_layer_mathbinary.h | 2 +-
.../dnn/dnn_backend_native_layer_mathunary.c | 11 ++-
.../dnn/dnn_backend_native_layer_mathunary.h | 2 +-
.../dnn/dnn_backend_native_layer_maximum.c| 10 ++-
.../dnn/dnn_backend_native_layer_maximum.h| 2 +-
.../dnn/dnn_backend_native_layer_pad.c| 10 ++-
.../dnn/dnn_backend_native_layer_pad.h| 2 +-
libavfilter/dnn/dnn_backend_native_layers.h | 2 +-
tests/dnn/dnn-layer-avgpool-test.c| 4 +-
tests/dnn/dnn-layer-conv2d-test.c | 4 +-
tests/dnn/dnn-layer-depth2space-test.c| 2 +-
tests/dnn/dnn-layer-mathbinary-test.c | 6 +-
tests/dnn/dnn-layer-mathunary-test.c | 2 +-
tests/dnn/dnn-layer-maximum-test.c| 2 +-
tests/dnn/dnn-layer-pad-test.c| 6 +-
24 files changed, 130 insertions(+), 56 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 0be9c0b53c..b7de27c709 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -28,15 +28,26 @@
#include "dnn_backend_native_layer_conv2d.h"
#include "dnn_backend_native_layers.h"
+static const AVClass dnn_native_class = {
+.class_name = "dnn_native",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
NativeModel *native_model = (NativeModel *)model;
+NetworkContext *ctx = _model->ctx;
for (int i = 0; i < native_model->operands_num; ++i) {
DnnOperand *oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
input->dt = oprd->data_type;
av_assert0(oprd->dims[0] == 1);
input->height = oprd->dims[1];
@@ -47,30 +58,37 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
}
// do not find the input operand
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_output_native(void *model, DNNData *input,
const char *input_name, const char **output_names, uint32_t nb_output)
{
NativeModel *native_model = (NativeModel *)model;
+NetworkContext *ctx = _model->ctx;
DnnOperand *oprd = NULL;
-if (native_model->layers_num <= 0 || native_model->operands_num <= 0)
+if (native_model->layers_num <= 0 || native_model->operands_num <= 0) {
+av_log(ctx, AV_LOG_ERROR, "No operands or layers in model\n");
return DNN_ERROR;
+}
/* inputs */
for (int i = 0; i < native_model->operands_num; ++i) {
oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
break;
}
oprd = NULL;
}
-
-if (!oprd)
+if (!oprd) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
oprd->dims[0] = 1;
oprd->dims[1] = input->height;
@@ -79,11 +97,15 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
av_freep(>data);
oprd->length = calculate_operand_data_length(oprd);
-if (oprd->length <= 0)
+if (oprd->length <= 0) {
+av_log(ctx, AV_LOG_ERROR, "The input data length overflow\n");
return DNN_ERROR;
+}
oprd->data = av_malloc(oprd->length);
-if (!oprd->data)
+if (!op
[FFmpeg-devel] [PATCH V3 1/2] dnn/native: unify error return to DNN_ERROR
Unify all error return as DNN_ERROR, in order to cease model executing
when return error in ff_dnn_execute_model_native layer_func.pf_exec
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_avgpool.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_conv2d.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_depth2space.c | 4 ++--
libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 2 +-
libavfilter/dnn/dnn_backend_native_layer_pad.c | 4 ++--
6 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
index d745c35b4a..e21a635c82 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -109,7 +109,7 @@ int dnn_execute_layer_avg_pool(DnnOperand *operands, const
int32_t *input_operan
output_operand->length = calculate_operand_data_length(output_operand);
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (int y = 0; y < height_end; y += kernel_strides) {
diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
index a2202e4073..25356901c2 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c
@@ -114,10 +114,10 @@ int dnn_execute_layer_conv2d(DnnOperand *operands, const
int32_t *input_operand_
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
av_assert0(channel == conv_params->input_num);
diff --git a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
index 2c8bddf23d..5a61025f7a 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c
@@ -76,10 +76,10 @@ int dnn_execute_layer_depth2space(DnnOperand *operands,
const int32_t *input_ope
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
for (y = 0; y < height; ++y){
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
index dd42c329a9..bffa41cdda 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c
@@ -176,6 +176,6 @@ int dnn_execute_layer_math_binary(DnnOperand *operands,
const int32_t *input_ope
}
return 0;
default:
-return -1;
+return DNN_ERROR;
}
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 58ee0e9d3d..57bbd9d3e8 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -143,6 +143,6 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
dst[i] = round(src[i]);
return 0;
default:
-return -1;
+return DNN_ERROR;
}
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_pad.c
b/libavfilter/dnn/dnn_backend_native_layer_pad.c
index feaab001e8..5452d22878 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_pad.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_pad.c
@@ -112,10 +112,10 @@ int dnn_execute_layer_pad(DnnOperand *operands, const
int32_t *input_operand_ind
output_operand->data_type = operands[input_operand_index].data_type;
output_operand->length = calculate_operand_data_length(output_operand);
if (output_operand->length <= 0)
-return -1;
+return DNN_ERROR;
output_operand->data = av_realloc(output_operand->data,
output_operand->length);
if (!output_operand->data)
-return -1;
+return DNN_ERROR;
output = output_operand->data;
// copy the original data
--
2.17.1
_
[FFmpeg-devel] [PATCH] dnn/native: rename struct ConvolutionalNetwork to NativeModel
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native.c | 112 +--
libavfilter/dnn/dnn_backend_native.h | 4 +-
libavfilter/dnn/dnn_backend_tf.c | 24 +++---
3 files changed, 70 insertions(+), 70 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index adc652a2c4..0be9c0b53c 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -30,10 +30,10 @@
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
-ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NativeModel *native_model = (NativeModel *)model;
-for (int i = 0; i < network->operands_num; ++i) {
-DnnOperand *oprd = >operands[i];
+for (int i = 0; i < native_model->operands_num; ++i) {
+DnnOperand *oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
if (oprd->type != DOT_INPUT)
return DNN_ERROR;
@@ -52,15 +52,15 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
static DNNReturnType set_input_output_native(void *model, DNNData *input,
const char *input_name, const char **output_names, uint32_t nb_output)
{
-ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NativeModel *native_model = (NativeModel *)model;
DnnOperand *oprd = NULL;
-if (network->layers_num <= 0 || network->operands_num <= 0)
+if (native_model->layers_num <= 0 || native_model->operands_num <= 0)
return DNN_ERROR;
/* inputs */
-for (int i = 0; i < network->operands_num; ++i) {
-oprd = >operands[i];
+for (int i = 0; i < native_model->operands_num; ++i) {
+oprd = _model->operands[i];
if (strcmp(oprd->name, input_name) == 0) {
if (oprd->type != DOT_INPUT)
return DNN_ERROR;
@@ -88,24 +88,24 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
input->data = oprd->data;
/* outputs */
-network->nb_output = 0;
-av_freep(>output_indexes);
-network->output_indexes = av_mallocz_array(nb_output,
sizeof(*network->output_indexes));
-if (!network->output_indexes)
+native_model->nb_output = 0;
+av_freep(_model->output_indexes);
+native_model->output_indexes = av_mallocz_array(nb_output,
sizeof(*native_model->output_indexes));
+if (!native_model->output_indexes)
return DNN_ERROR;
for (uint32_t i = 0; i < nb_output; ++i) {
const char *output_name = output_names[i];
-for (int j = 0; j < network->operands_num; ++j) {
-oprd = >operands[j];
+for (int j = 0; j < native_model->operands_num; ++j) {
+oprd = _model->operands[j];
if (strcmp(oprd->name, output_name) == 0) {
-network->output_indexes[network->nb_output++] = j;
+native_model->output_indexes[native_model->nb_output++] = j;
break;
}
}
}
-if (network->nb_output != nb_output)
+if (native_model->nb_output != nb_output)
return DNN_ERROR;
return DNN_SUCCESS;
@@ -122,7 +122,7 @@ DNNModel *ff_dnn_load_model_native(const char
*model_filename, const char *optio
char *buf;
size_t size;
int version, header_size, major_version_expected = 1;
-ConvolutionalNetwork *network = NULL;
+NativeModel *native_model = NULL;
AVIOContext *model_file_context;
int file_size, dnn_size, parsed_size;
int32_t layer;
@@ -167,29 +167,29 @@ DNNModel *ff_dnn_load_model_native(const char
*model_filename, const char *optio
dnn_size += 4;
header_size = dnn_size;
-network = av_mallocz(sizeof(ConvolutionalNetwork));
-if (!network){
+native_model = av_mallocz(sizeof(NativeModel));
+if (!native_model){
goto fail;
}
-model->model = (void *)network;
+model->model = (void *)native_model;
avio_seek(model_file_context, file_size - 8, SEEK_SET);
-network->layers_num = (int32_t)avio_rl32(model_file_context);
-network->operands_num = (int32_t)avio_rl32(model_file_context);
+native_model->layers_num = (int32_t)avio_rl32(model_file_context);
+native_model->operands_num = (int32_t)avio_rl32(model_file_context);
dnn_size += 8;
avio_seek(model_file_context, header_size, SEEK_SET);
-network->layers = av_mallocz(network->layers_num * sizeof(Layer));
-if (!network->layers){
+native_model->layers = av_mallocz(native_model->layers_num *
sizeof(Layer));
+if (!native_model->layers){
goto fail;
}
-network->operands = av_mallocz(network->operands_num * sizeof(DnnOperand));
-if (!net
[FFmpeg-devel] [PATCH V2] dnn/native: add log error message
Signed-off-by: Ting Fu
---
V2:
Fix the issue in V1: make fate failed
libavfilter/dnn/dnn_backend_native.c | 74 +++
libavfilter/dnn/dnn_backend_native.h | 5 ++
.../dnn/dnn_backend_native_layer_avgpool.c| 2 +-
.../dnn/dnn_backend_native_layer_avgpool.h| 2 +-
.../dnn/dnn_backend_native_layer_conv2d.c | 2 +-
.../dnn/dnn_backend_native_layer_conv2d.h | 2 +-
.../dnn_backend_native_layer_depth2space.c| 2 +-
.../dnn_backend_native_layer_depth2space.h| 2 +-
.../dnn/dnn_backend_native_layer_mathbinary.c | 10 ++-
.../dnn/dnn_backend_native_layer_mathbinary.h | 2 +-
.../dnn/dnn_backend_native_layer_mathunary.c | 10 ++-
.../dnn/dnn_backend_native_layer_mathunary.h | 2 +-
.../dnn/dnn_backend_native_layer_maximum.c| 2 +-
.../dnn/dnn_backend_native_layer_maximum.h| 2 +-
.../dnn/dnn_backend_native_layer_pad.c| 2 +-
.../dnn/dnn_backend_native_layer_pad.h| 2 +-
libavfilter/dnn/dnn_backend_native_layers.h | 2 +-
tests/dnn/dnn-layer-avgpool-test.c| 4 +-
tests/dnn/dnn-layer-conv2d-test.c | 4 +-
tests/dnn/dnn-layer-depth2space-test.c| 2 +-
tests/dnn/dnn-layer-mathbinary-test.c | 6 +-
tests/dnn/dnn-layer-mathunary-test.c | 2 +-
tests/dnn/dnn-layer-maximum-test.c| 2 +-
tests/dnn/dnn-layer-pad-test.c| 6 +-
24 files changed, 103 insertions(+), 48 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index adc652a2c4..6ddffa54af 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -28,15 +28,30 @@
#include "dnn_backend_native_layer_conv2d.h"
#include "dnn_backend_native_layers.h"
+static const AVClass dnn_native_class = {
+.class_name = "dnn_native",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
+NetworkContext network_ctx = {
+.class = _native_class,
+};
+
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NetworkContext *ctx = network->log_ctx;
for (int i = 0; i < network->operands_num; ++i) {
DnnOperand *oprd = >operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
input->dt = oprd->data_type;
av_assert0(oprd->dims[0] == 1);
input->height = oprd->dims[1];
@@ -47,30 +62,37 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
}
// do not find the input operand
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_output_native(void *model, DNNData *input,
const char *input_name, const char **output_names, uint32_t nb_output)
{
ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NetworkContext *ctx = network->log_ctx;
DnnOperand *oprd = NULL;
-if (network->layers_num <= 0 || network->operands_num <= 0)
+if (network->layers_num <= 0 || network->operands_num <= 0) {
+av_log(ctx, AV_LOG_ERROR, "No operands or layers in model\n");
return DNN_ERROR;
+}
/* inputs */
for (int i = 0; i < network->operands_num; ++i) {
oprd = >operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
break;
}
oprd = NULL;
}
-
-if (!oprd)
+if (!oprd) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
oprd->dims[0] = 1;
oprd->dims[1] = input->height;
@@ -79,11 +101,15 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
av_freep(>data);
oprd->length = calculate_operand_data_length(oprd);
-if (oprd->length <= 0)
+if (oprd->length <= 0) {
+av_log(ctx, AV_LOG_ERROR, "The input data length overflow\n");
return DNN_ERROR;
+}
oprd->data = av_malloc(oprd->length);
-if (!oprd->data)
+if (!oprd->data) {
[FFmpeg-devel] [PATCH] dnn/native: add log error message
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native.c | 74 +++
libavfilter/dnn/dnn_backend_native.h | 5 ++
.../dnn/dnn_backend_native_layer_avgpool.c| 2 +-
.../dnn/dnn_backend_native_layer_avgpool.h| 2 +-
.../dnn/dnn_backend_native_layer_conv2d.c | 2 +-
.../dnn/dnn_backend_native_layer_conv2d.h | 2 +-
.../dnn_backend_native_layer_depth2space.c| 2 +-
.../dnn_backend_native_layer_depth2space.h| 2 +-
.../dnn/dnn_backend_native_layer_mathbinary.c | 10 ++-
.../dnn/dnn_backend_native_layer_mathbinary.h | 2 +-
.../dnn/dnn_backend_native_layer_mathunary.c | 10 ++-
.../dnn/dnn_backend_native_layer_mathunary.h | 2 +-
.../dnn/dnn_backend_native_layer_maximum.c| 2 +-
.../dnn/dnn_backend_native_layer_maximum.h| 2 +-
.../dnn/dnn_backend_native_layer_pad.c| 2 +-
.../dnn/dnn_backend_native_layer_pad.h| 2 +-
libavfilter/dnn/dnn_backend_native_layers.h | 2 +-
17 files changed, 90 insertions(+), 35 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index adc652a2c4..6ddffa54af 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -28,15 +28,30 @@
#include "dnn_backend_native_layer_conv2d.h"
#include "dnn_backend_native_layers.h"
+static const AVClass dnn_native_class = {
+.class_name = "dnn_native",
+.item_name = av_default_item_name,
+.option = NULL,
+.version= LIBAVUTIL_VERSION_INT,
+.category = AV_CLASS_CATEGORY_FILTER,
+};
+
+NetworkContext network_ctx = {
+.class = _native_class,
+};
+
static DNNReturnType get_input_native(void *model, DNNData *input, const char
*input_name)
{
ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NetworkContext *ctx = network->log_ctx;
for (int i = 0; i < network->operands_num; ++i) {
DnnOperand *oprd = >operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
input->dt = oprd->data_type;
av_assert0(oprd->dims[0] == 1);
input->height = oprd->dims[1];
@@ -47,30 +62,37 @@ static DNNReturnType get_input_native(void *model, DNNData
*input, const char *i
}
// do not find the input operand
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name);
return DNN_ERROR;
}
static DNNReturnType set_input_output_native(void *model, DNNData *input,
const char *input_name, const char **output_names, uint32_t nb_output)
{
ConvolutionalNetwork *network = (ConvolutionalNetwork *)model;
+NetworkContext *ctx = network->log_ctx;
DnnOperand *oprd = NULL;
-if (network->layers_num <= 0 || network->operands_num <= 0)
+if (network->layers_num <= 0 || network->operands_num <= 0) {
+av_log(ctx, AV_LOG_ERROR, "No operands or layers in model\n");
return DNN_ERROR;
+}
/* inputs */
for (int i = 0; i < network->operands_num; ++i) {
oprd = >operands[i];
if (strcmp(oprd->name, input_name) == 0) {
-if (oprd->type != DOT_INPUT)
+if (oprd->type != DOT_INPUT) {
+av_log(ctx, AV_LOG_ERROR, "Found \"%s\" in model, but it is
not input node\n", input_name);
return DNN_ERROR;
+}
break;
}
oprd = NULL;
}
-
-if (!oprd)
+if (!oprd) {
+av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n",
input_name);
return DNN_ERROR;
+}
oprd->dims[0] = 1;
oprd->dims[1] = input->height;
@@ -79,11 +101,15 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
av_freep(>data);
oprd->length = calculate_operand_data_length(oprd);
-if (oprd->length <= 0)
+if (oprd->length <= 0) {
+av_log(ctx, AV_LOG_ERROR, "The input data length overflow\n");
return DNN_ERROR;
+}
oprd->data = av_malloc(oprd->length);
-if (!oprd->data)
+if (!oprd->data) {
+av_log(ctx, AV_LOG_ERROR, "Failed to malloc memory for input data\n");
return DNN_ERROR;
+}
input->data = oprd->data;
@@ -91,8 +117,10 @@ static DNNReturnType set_input_output_native(void *model,
DNNData *input, const
network->nb_output = 0;
av_freep(>output_indexes);
network->output_indexes = av_mallocz_array(nb_output,
sizeof(*network->output_indexes));
-
[FFmpeg-devel] [PATCH V7 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 197 +
tests/fate/dnn.mak | 5 +
4 files changed, 204 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..d7c33a0e88
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,197 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.6576588108454774, 0.6063336087333997,
0.6452342242996931, 0.707168
[FFmpeg-devel] [PATCH V7 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension yet.
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 141 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 40 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 37 -
7 files changed, 223 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..d745c35b4a
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,141 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 12;
+
+if (dnn_size > file_size || avgpool_params->kernel_size <= 0 ||
avgpool_params->strides <=0){
+av_freep(_params);
+return 0;
+}
+
+layer->params = avgpool_params;
+layer->input_operand_indexes[0] = (int32_t)avio_rl32(model_file_context);
+layer->output_operand_index = (int32_t)avio_rl32(model_file_context);
+dnn_size += 8;
+
+if (layer->input_operand_indexes[0] >= operands_num ||
layer->output_operand_index >= operands_num) {
+return 0;
+}
+return dnn_size;
+}
+
+int dnn_execute_layer_avg_pool(DnnOperand *operands, const int32_t
*input_operand_indexes,
+ int32_t output_operand_index, const void
*parameters)
+{
+float *output;
+int height_end, width_end, height_radius, width_radius, output_height,
output_width, kernel_area;
+int32_t input_operand_index = input_operand_indexes[0];
+int number = operands[input_operand_index].dims[0];
+int height = operands[input_operand_index].dims[1];
+int width = operands[input_operand_index].dims[2];
+int channel = operands[input_operand_index].dims[3];
+const float *inp
[FFmpeg-devel] [PATCH V6 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
V6:
Fix the issue of make fate failed in V4
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 198 +
tests/fate/dnn.mak | 5 +
4 files changed, 205 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..5fadde1024
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,198 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.657658
[FFmpeg-devel] [PATCH V6 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 141 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 40 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 37 -
7 files changed, 223 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..d745c35b4a
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,141 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 12;
+
+if (dnn_size > file_size || avgpool_params->kernel_size <= 0 ||
avgpool_params->strides <=0){
+av_freep(_params);
+return 0;
+}
+
+layer->params = avgpool_params;
+layer->input_operand_indexes[0] = (i
[FFmpeg-devel] [PATCH V5 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
V5:
Fix the issue V4 make fate failed.
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 202 +
tests/fate/dnn.mak | 5 +
4 files changed, 209 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..1c47f9330d
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,202 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.657658
[FFmpeg-devel] [PATCH V5 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 141 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 40 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 37 -
7 files changed, 223 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..d745c35b4a
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,141 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 12;
+
+if (dnn_size > file_size || avgpool_params->kernel_size <= 0 ||
avgpool_params->strides <=0){
+av_freep(_params);
+return 0;
+}
+
+layer->params = avgpool_params;
+layer->input_operand_indexes[0] = (i
[FFmpeg-devel] [PATCH V4 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 141 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 40 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 37 -
7 files changed, 223 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..d745c35b4a
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,141 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 12;
+
+if (dnn_size > file_size || avgpool_params->kernel_size <= 0 ||
avgpool_params->strides <=0){
+av_freep(_params);
+return 0;
+}
+
+layer->params = avgpool_params;
+layer->input_operand_indexes[0] = (i
[FFmpeg-devel] [PATCH V4 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 202 +
tests/fate/dnn.mak | 5 +
4 files changed, 209 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..1c47f9330d
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,202 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.6576588108454774, 0.6063336087333997,
0.645234
[FFmpeg-devel] [PATCH V3 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 202 +
tests/fate/dnn.mak | 5 +
4 files changed, 209 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..1c47f9330d
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,202 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.6576588108454774, 0.6063336087333997,
0.645234
[FFmpeg-devel] [PATCH V3 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 147 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 35 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 39 -
7 files changed, 226 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..a6ebb0db8f
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->in_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->out_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 20;
+
+if (dnn_size > file_size || avgpool_params->in_channels <= 0 ||
+avgpo
[FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 147 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 35 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 35 -
7 files changed, 222 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..a6ebb0db8f
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->in_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->out_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 20;
+
+if (dnn_size > file_size || avgpool_params->in_channels <= 0 ||
+avgpo
[FFmpeg-devel] [PATCH V2 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 202 +
tests/fate/dnn.mak | 5 +
4 files changed, 209 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..1c47f9330d
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,202 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.6576588108454774, 0.6063336087333997,
0.645234
[FFmpeg-devel] [PATCH 2/2] FATE/dnn: add unit test for dnn avgpool layer
'make fate-dnn-layer-avgpool' to run the test
Signed-off-by: Ting Fu
---
tests/dnn/.gitignore | 1 +
tests/dnn/Makefile | 1 +
tests/dnn/dnn-layer-avgpool-test.c | 202 +
tests/fate/dnn.mak | 5 +
4 files changed, 209 insertions(+)
create mode 100644 tests/dnn/dnn-layer-avgpool-test.c
diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore
index 1fcd2410b4..b847a01177 100644
--- a/tests/dnn/.gitignore
+++ b/tests/dnn/.gitignore
@@ -4,3 +4,4 @@
/dnn-layer-pad-test
/dnn-layer-mathbinary-test
/dnn-layer-mathunary-test
+/dnn-layer-avgpool-test
diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile
index 64591b7851..8afdfab5d3 100644
--- a/tests/dnn/Makefile
+++ b/tests/dnn/Makefile
@@ -4,6 +4,7 @@ DNNTESTPROGS += dnn-layer-depth2space
DNNTESTPROGS += dnn-layer-mathbinary
DNNTESTPROGS += dnn-layer-maximum
DNNTESTPROGS += dnn-layer-mathunary
+DNNTESTPROGS += dnn-layer-avgpool
DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%)
$(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o)
DNNTESTPROGS := $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test$(EXESUF))
diff --git a/tests/dnn/dnn-layer-avgpool-test.c
b/tests/dnn/dnn-layer-avgpool-test.c
new file mode 100644
index 00..1c47f9330d
--- /dev/null
+++ b/tests/dnn/dnn-layer-avgpool-test.c
@@ -0,0 +1,202 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include
+#include //?
+#include "libavfilter/dnn/dnn_backend_native_layer_avgpool.h"
+
+#define EPSON 0.1
+
+static int test_with_same(void)
+{
+// the input data and expected data are generated with below python code.
+/*
+import tensorflow as tf
+import numpy as np
+
+x = tf.placeholder(tf.float32, shape=[1, None, None, 3])
+y = tf.layers.average_pooling2d(x, pool_size=[2,2], strides=[1,1],
padding='VALID')
+data = np.random.rand(1, 5, 6, 3);
+
+sess=tf.Session()
+sess.run(tf.global_variables_initializer())
+
+output = sess.run(y, feed_dict={x: data})
+
+print("input:")
+print(data.shape)
+print(list(data.flatten()))
+
+print("output:")
+print(output.shape)
+print(list(output.flatten()))
+*/
+
+AvgPoolParams params;
+DnnOperand operands[2];
+int32_t input_indexes[1];
+float input[1*5*6*3] = {
+0.7461309859908424, 0.7567538372797069, 0.07662743569678687,
0.8882112610336333, 0.9720443314026668, 0.3337200343220823, 0.4421032129780248,
+0.14940809044964876, 0.6773177061961277, 0.9778844630669781,
0.6522650522626998, 0.0317651530878591, 0.31259897552911364, 0.6235936821891896,
+0.40016094349542775, 0.4599222930032276, 0.7893807222960093,
0.8475986363538283, 0.5058802717647394, 0.7827005363222633, 0.3032188123727916,
+0.8983728631302361, 0.20622408444965523, 0.22966072303869878,
0.09535751273161308, 0.8760709100995375, 0.9982324154558745, 0.7904595468621013,
+0.13883671508879347, 0.9332751439533138, 0.0010861680752152214,
0.3607210449251048, 0.6600652759586171, 0.7629572058138805, 0.29441975810476106,
+0.2683471432889405, 0.22574580829831536, 0.8893251976212904,
0.3907737043801005, 0.6421829842863968, 0.6670373870457297, 0.9383850793160277,
+0.4120458907436003, 0.3589847212711481, 0.48047736550128983,
0.6428192648418949, 0.0313661686292348, 0.429357100401472, 0.5123413386514056,
+0.8492446404097114, 0.9045286128486804, 0.8123708563814285,
0.3943245008451698, 0.9576713003177785, 0.5985610965938726, 0.9350833279543561,
+0.8010079897491659, 0.45882114217642866, 0.35275037908941487,
0.4555844661432271, 0.12352455940255314, 0.37801756635035544,
0.2824056214573083,
+0.6229462823245029, 0.7235305681391472, 0.5408259266122064,
0.12142224381781208, 0.34431198802873686, 0.7112823816321276,
0.6307144385115417,
+0.8136734589018082, 0.842095618140585, 0.8602767724004784,
0.6649236853766185, 0.5184782829419623, 0.9119607270982825, 0.3084111974561645,
+0.39460705638161364, 0.17710447526170836, 0.1715485945814199,
0.17277563576521882, 0.40188232428735704, 0.22847985411491878,
0.4135361701550696,
+0.24621846601980057, 0.6576588108454774, 0.6063336087333997,
0.645234
[FFmpeg-devel] [PATCH 1/2] dnn/native: add native support for avg_pool
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 136 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 35 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 31 +++-
7 files changed, 207 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..f5a3f4a0dc
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,136 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->in_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->out_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 20;
+
+if (dnn_size > file_size || avgpool_params->in_channels <= 0 ||
+avgpool_params->out_channels <= 0 || avgpool_params->kern
[FFmpeg-devel] [PATCH V3] tests/dnn/mathunary: fix the issue of NAN
When one of output[i] & expected_output is NAN, the unit test will always pass.
Signed-off-by: Ting Fu
---
tests/dnn/dnn-layer-mathunary-test.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tests/dnn/dnn-layer-mathunary-test.c
b/tests/dnn/dnn-layer-mathunary-test.c
index 683e623d95..5afc5c157e 100644
--- a/tests/dnn/dnn-layer-mathunary-test.c
+++ b/tests/dnn/dnn-layer-mathunary-test.c
@@ -86,7 +86,10 @@ static int test(DNNMathUnaryOperation op)
output = operands[1].data;
for (int i = 0; i < sizeof(input) / sizeof(float); ++i) {
float expected_output = get_expected(input[i], op);
-if(fabs(output[i] - expected_output) > EPS) {
+int output_nan = isnan(output[i]);
+int expected_nan = isnan(expected_output);
+if ((!output_nan && !expected_nan && fabs(output[i] - expected_output)
> EPS) ||
+(output_nan && !expected_nan) || (!output_nan && expected_nan)) {
printf("at index %d, output: %f, expected_output: %f\n", i,
output[i], expected_output);
av_freep();
return 1;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2] tests/dnn/mathunary: fix the issue of NAN
When one of output[i] & expected_output is NAN, the unit test will always pass.
Signed-off-by: Ting Fu
---
tests/dnn/dnn-layer-mathunary-test.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tests/dnn/dnn-layer-mathunary-test.c
b/tests/dnn/dnn-layer-mathunary-test.c
index 683e623d95..70c6a43f95 100644
--- a/tests/dnn/dnn-layer-mathunary-test.c
+++ b/tests/dnn/dnn-layer-mathunary-test.c
@@ -86,7 +86,8 @@ static int test(DNNMathUnaryOperation op)
output = operands[1].data;
for (int i = 0; i < sizeof(input) / sizeof(float); ++i) {
float expected_output = get_expected(input[i], op);
-if(fabs(output[i] - expected_output) > EPS) {
+if ((!isnan(output[i]) && !isnan(expected_output) && fabs(output[i] -
expected_output) > EPS) ||
+(isnan(output[i]) && !isnan(expected_output)) ||
(!isnan(output[i]) && isnan(expected_output))) {
printf("at index %d, output: %f, expected_output: %f\n", i,
output[i], expected_output);
av_freep();
return 1;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] tests/dnn/mathunary: fix the issue of NAN
When one of output[i] & expected_output is NAN, the unit test will always pass.
Signed-off-by: Ting Fu
---
tests/dnn/dnn-layer-mathunary-test.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tests/dnn/dnn-layer-mathunary-test.c
b/tests/dnn/dnn-layer-mathunary-test.c
index bf77c44bbe..f251447771 100644
--- a/tests/dnn/dnn-layer-mathunary-test.c
+++ b/tests/dnn/dnn-layer-mathunary-test.c
@@ -74,7 +74,8 @@ static int test(DNNMathUnaryOperation op)
output = operands[1].data;
for (int i = 0; i < sizeof(input) / sizeof(float); ++i) {
float expected_output = get_expected(input[i], op);
-if(fabs(output[i] - expected_output) > EPS) {
+if ((isnan(output[i]) ^ isnan(expected_output)) ||
+fabs(output[i] - expected_output) > EPS) {
printf("at index %d, output: %f, expected_output: %f\n", i,
output[i], expected_output);
av_freep();
return 1;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 12/12] dnn-layer-math-unary-test: add unit test for atanh
Signed-off-by: Ting Fu
---
tests/dnn/dnn-layer-mathunary-test.c | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/tests/dnn/dnn-layer-mathunary-test.c
b/tests/dnn/dnn-layer-mathunary-test.c
index 5587e47ad5..683e623d95 100644
--- a/tests/dnn/dnn-layer-mathunary-test.c
+++ b/tests/dnn/dnn-layer-mathunary-test.c
@@ -54,6 +54,8 @@ static float get_expected(float f, DNNMathUnaryOperation op)
return asinh(f);
case DMUO_ACOSH:
return acosh(f);
+case DMUO_ATANH:
+return atanh(f);
default:
av_assert0(!"not supported yet");
return 0.f;
@@ -65,8 +67,8 @@ static int test(DNNMathUnaryOperation op)
DnnLayerMathUnaryParams params;
DnnOperand operands[2];
int32_t input_indexes[1];
-float input[1*1*2*3] = {
--3, 2.5, 2, -2.1, 7.8, 100};
+float input[1*1*3*3] = {
+0.1, 0.5, 0.75, -3, 2.5, 2, -2.1, 7.8, 100};
float *output;
params.un_op = op;
@@ -74,7 +76,7 @@ static int test(DNNMathUnaryOperation op)
operands[0].data = input;
operands[0].dims[0] = 1;
operands[0].dims[1] = 1;
-operands[0].dims[2] = 2;
+operands[0].dims[2] = 3;
operands[0].dims[3] = 3;
operands[1].data = NULL;
@@ -121,5 +123,7 @@ int main(int agrc, char **argv)
return 1;
if (test(DMUO_ACOSH))
return 1;
+if (test(DMUO_ATANH))
+return 1;
return 0;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 08/12] dnn-layer-math-unary-test: add unit test for asinh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 6885b4d318..90fce71a0c 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -50,6 +50,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return cosh(f); case DMUO_TANH: return tanh(f); +case DMUO_ASINH: +return asinh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -113,5 +115,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_TANH)) return 1; +if (test(DMUO_ASINH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 07/12] dnn_backend_native_layer_mathunary: add asinh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index ccdbcc21e0..83df98d0f8 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -116,6 +116,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = tanh(src[i]);
return 0;
+case DMUO_ASINH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = asinh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index ae0c1e1cdd..fbe9af5c7d 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -40,6 +40,7 @@ typedef enum {
DMUO_SINH = 7,
DMUO_COSH = 8,
DMUO_TANH = 9,
+DMUO_ASINH = 10,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index f98a3cae3d..0d756c8109 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index d2753f0af0..3211c13f6d 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 15
+minor = 16
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 11/12] dnn_backend_native_layer_mathunary: add atanh support
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input.jpeg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
please uncomment the part you want to test
x_sinh_1 = tf.sinh(x)
x_out = tf.divide(x_sinh_1, 1.176) # sinh(1.0)
x_cosh_1 = tf.cosh(x)
x_out = tf.divide(x_cosh_1, 1.55) # cosh(1.0)
x_tanh_1 = tf.tanh(x)
x__out = tf.divide(x_tanh_1, 0.77) # tanh(1.0)
x_asinh_1 = tf.asinh(x)
x_out = tf.divide(x_asinh_1, 0.89) # asinh(1.0/1.1)
x_acosh_1 = tf.add(x, 1.1)
x_acosh_2 = tf.acosh(x_acosh_1) # accept (1, inf)
x_out = tf.divide(x_acosh_2, 1.4) # acosh(2.1)
x_atanh_1 = tf.divide(x, 1.1)
x_atanh_2 = tf.atanh(x_atanh_1) # accept (-1, 1)
x_out = tf.divide(x_atanh_2, 1.55) # atanhh(1.0/1.1)
y = tf.identity(x_out, name='dnn_out') #please only preserve the x_out you want
to test
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index b77b84a794..c83d50db64 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -124,6 +124,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = acosh(src[i]);
return 0;
+case DMUO_ATANH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = atanh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index eb30231549..8076356ba4 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -42,6 +42,7 @@ typedef enum {
DMUO_TANH = 9,
DMUO_ASINH = 10,
DMUO_ACOSH = 11,
+DMUO_ATANH = 12,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 1e73e3aefe..85db7bf710 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11,
'Atanh':12}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 8fc3438552..9851d84144 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 17
+minor = 18
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 09/12] dnn_backend_native_layer_mathunary: add acosh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 83df98d0f8..b77b84a794 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -120,6 +120,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = asinh(src[i]);
return 0;
+case DMUO_ACOSH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = acosh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index fbe9af5c7d..eb30231549 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -41,6 +41,7 @@ typedef enum {
DMUO_COSH = 8,
DMUO_TANH = 9,
DMUO_ASINH = 10,
+DMUO_ACOSH = 11,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 0d756c8109..1e73e3aefe 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 3211c13f6d..8fc3438552 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 16
+minor = 17
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 06/12] dnn-layer-math-unary-test: add unit test for tanh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 0280debc0b..6885b4d318 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -48,6 +48,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return sinh(f); case DMUO_COSH: return cosh(f); +case DMUO_TANH: +return tanh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -109,5 +111,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_COSH)) return 1; +if (test(DMUO_TANH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 10/12] dnn-layer-math-unary-test: add unit test for acosh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 90fce71a0c..5587e47ad5 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -52,6 +52,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return tanh(f); case DMUO_ASINH: return asinh(f); +case DMUO_ACOSH: +return acosh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -117,5 +119,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_ASINH)) return 1; +if (test(DMUO_ACOSH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 05/12] dnn_backend_native_layer_mathunary: add tanh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index ddb70996e7..ccdbcc21e0 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -112,6 +112,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = cosh(src[i]);
return 0;
+case DMUO_TANH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = tanh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 5a486b4f5f..ae0c1e1cdd 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -39,6 +39,7 @@ typedef enum {
DMUO_ATAN = 6,
DMUO_SINH = 7,
DMUO_COSH = 8,
+DMUO_TANH = 9,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 96da44c4a8..f98a3cae3d 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index a73f51ba48..d2753f0af0 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 14
+minor = 15
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 04/12] dnn-layer-math-unary-test: add unit test for cosh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index a1ff05e5fb..0280debc0b 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -46,6 +46,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return atan(f); case DMUO_SINH: return sinh(f); +case DMUO_COSH: +return cosh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -105,5 +107,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_SINH)) return 1; +if (test(DMUO_COSH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 02/12] dnn-layer-math-unary-test: add unit test for sinh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index bf77c44bbe..a1ff05e5fb 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -44,6 +44,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return acos(f); case DMUO_ATAN: return atan(f); +case DMUO_SINH: +return sinh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -101,5 +103,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_ATAN)) return 1; +if (test(DMUO_SINH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 01/12] dnn_backend_native_layer_mathunary: add sinh support
Signed-off-by: Ting Fu
---
V2:
Add more test number
Fix incorrect atanh unit test
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 42615c43d5..2630fe07e2 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -104,6 +104,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = atan(src[i]);
return 0;
+case DMUO_SINH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = sinh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 13fa33178a..760930c60e 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -37,6 +37,7 @@ typedef enum {
DMUO_ASIN = 4,
DMUO_ACOS = 5,
DMUO_ATAN = 6,
+DMUO_SINH = 7,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index b90c31c495..6f34a71ab4 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 73cf23bf53..4747f41395 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 12
+minor = 13
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 03/12] dnn_backend_native_layer_mathunary: add cosh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 2630fe07e2..ddb70996e7 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -108,6 +108,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = sinh(src[i]);
return 0;
+case DMUO_COSH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = cosh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 760930c60e..5a486b4f5f 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -38,6 +38,7 @@ typedef enum {
DMUO_ACOS = 5,
DMUO_ATAN = 6,
DMUO_SINH = 7,
+DMUO_COSH = 8,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 6f34a71ab4..96da44c4a8 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 4747f41395..a73f51ba48 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 13
+minor = 14
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 12/12] dnn-layer-math-unary-test: add unit test for atanh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 5587e47ad5..1815f79f34 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -54,6 +54,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return asinh(f); case DMUO_ACOSH: return acosh(f); +case DMUO_ATANH: +return acosh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -121,5 +123,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_ACOSH)) return 1; +if (test(DMUO_ATANH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 11/12] dnn_backend_native_layer_mathunary: add atanh support
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input.jpeg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
please uncomment the part you want to test
x_sinh_1 = tf.sinh(x)
x_out = tf.divide(x_sinh_1, 1.176) # sinh(1.0)
x_cosh_1 = tf.cosh(x)
x_out = tf.divide(x_cosh_1, 1.55) # cosh(1.0)
x_tanh_1 = tf.tanh(x)
x__out = tf.divide(x_tanh_1, 0.77) # tanh(1.0)
x_asinh_1 = tf.asinh(x)
x_out = tf.divide(x_asinh_1, 0.89) # asinh(1.0/1.1)
x_acosh_1 = tf.add(x, 1.1)
x_acosh_2 = tf.acosh(x_acosh_1) # accept (1, inf)
x_out = tf.divide(x_acosh_2, 1.4) # acosh(2.1)
x_atanh_1 = tf.divide(x, 1.1)
x_atanh_2 = tf.atanh(x_atanh_1) # accept (-1, 1)
x_out = tf.divide(x_atanh_2, 1.55) # atanhh(1.0/1.1)
y = tf.identity(x_out, name='dnn_out') #please only preserve the x_out you want
to test
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index b77b84a794..c83d50db64 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -124,6 +124,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = acosh(src[i]);
return 0;
+case DMUO_ATANH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = atanh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index eb30231549..8076356ba4 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -42,6 +42,7 @@ typedef enum {
DMUO_TANH = 9,
DMUO_ASINH = 10,
DMUO_ACOSH = 11,
+DMUO_ATANH = 12,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 1e73e3aefe..85db7bf710 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11,
'Atanh':12}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 8fc3438552..9851d84144 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 17
+minor = 18
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 09/12] dnn_backend_native_layer_mathunary: add acosh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 83df98d0f8..b77b84a794 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -120,6 +120,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = asinh(src[i]);
return 0;
+case DMUO_ACOSH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = acosh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index fbe9af5c7d..eb30231549 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -41,6 +41,7 @@ typedef enum {
DMUO_COSH = 8,
DMUO_TANH = 9,
DMUO_ASINH = 10,
+DMUO_ACOSH = 11,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 0d756c8109..1e73e3aefe 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10, 'Acosh':11}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 3211c13f6d..8fc3438552 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 16
+minor = 17
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 10/12] dnn-layer-math-unary-test: add unit test for acosh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 90fce71a0c..5587e47ad5 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -52,6 +52,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return tanh(f); case DMUO_ASINH: return asinh(f); +case DMUO_ACOSH: +return acosh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -117,5 +119,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_ASINH)) return 1; +if (test(DMUO_ACOSH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 07/12] dnn_backend_native_layer_mathunary: add asinh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index ccdbcc21e0..83df98d0f8 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -116,6 +116,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = tanh(src[i]);
return 0;
+case DMUO_ASINH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = asinh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index ae0c1e1cdd..fbe9af5c7d 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -40,6 +40,7 @@ typedef enum {
DMUO_SINH = 7,
DMUO_COSH = 8,
DMUO_TANH = 9,
+DMUO_ASINH = 10,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index f98a3cae3d..0d756c8109 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index d2753f0af0..3211c13f6d 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 15
+minor = 16
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 06/12] dnn-layer-math-unary-test: add unit test for tanh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index 0280debc0b..6885b4d318 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -48,6 +48,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return sinh(f); case DMUO_COSH: return cosh(f); +case DMUO_TANH: +return tanh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -109,5 +111,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_COSH)) return 1; +if (test(DMUO_TANH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 04/12] dnn-layer-math-unary-test: add unit test for cosh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index a1ff05e5fb..0280debc0b 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -46,6 +46,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return atan(f); case DMUO_SINH: return sinh(f); +case DMUO_COSH: +return cosh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -105,5 +107,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_SINH)) return 1; +if (test(DMUO_COSH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 02/12] dnn-layer-math-unary-test: add unit test for sinh
Signed-off-by: Ting Fu --- tests/dnn/dnn-layer-mathunary-test.c | 4 1 file changed, 4 insertions(+) diff --git a/tests/dnn/dnn-layer-mathunary-test.c b/tests/dnn/dnn-layer-mathunary-test.c index bf77c44bbe..a1ff05e5fb 100644 --- a/tests/dnn/dnn-layer-mathunary-test.c +++ b/tests/dnn/dnn-layer-mathunary-test.c @@ -44,6 +44,8 @@ static float get_expected(float f, DNNMathUnaryOperation op) return acos(f); case DMUO_ATAN: return atan(f); +case DMUO_SINH: +return sinh(f); default: av_assert0(!"not supported yet"); return 0.f; @@ -101,5 +103,7 @@ int main(int agrc, char **argv) return 1; if (test(DMUO_ATAN)) return 1; +if (test(DMUO_SINH)) +return 1; return 0; } -- 2.17.1 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 03/12] dnn_backend_native_layer_mathunary: add cosh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 2630fe07e2..ddb70996e7 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -108,6 +108,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = sinh(src[i]);
return 0;
+case DMUO_COSH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = cosh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 760930c60e..5a486b4f5f 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -38,6 +38,7 @@ typedef enum {
DMUO_ACOS = 5,
DMUO_ATAN = 6,
DMUO_SINH = 7,
+DMUO_COSH = 8,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 6f34a71ab4..96da44c4a8 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 4747f41395..a73f51ba48 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 13
+minor = 14
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 05/12] dnn_backend_native_layer_mathunary: add tanh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index ddb70996e7..ccdbcc21e0 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -112,6 +112,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = cosh(src[i]);
return 0;
+case DMUO_TANH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = tanh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 5a486b4f5f..ae0c1e1cdd 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -39,6 +39,7 @@ typedef enum {
DMUO_ATAN = 6,
DMUO_SINH = 7,
DMUO_COSH = 8,
+DMUO_TANH = 9,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index 96da44c4a8..f98a3cae3d 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index a73f51ba48..d2753f0af0 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 14
+minor = 15
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH 01/12] dnn_backend_native_layer_mathunary: add sinh support
Signed-off-by: Ting Fu
---
libavfilter/dnn/dnn_backend_native_layer_mathunary.c | 4
libavfilter/dnn/dnn_backend_native_layer_mathunary.h | 1 +
tools/python/convert_from_tensorflow.py | 2 +-
tools/python/convert_header.py | 2 +-
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
index 42615c43d5..2630fe07e2 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.c
@@ -104,6 +104,10 @@ int dnn_execute_layer_math_unary(DnnOperand *operands,
const int32_t *input_oper
for (int i = 0; i < dims_count; ++i)
dst[i] = atan(src[i]);
return 0;
+case DMUO_SINH:
+for (int i = 0; i < dims_count; ++i)
+dst[i] = sinh(src[i]);
+return 0;
default:
return -1;
}
diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
index 13fa33178a..760930c60e 100644
--- a/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
+++ b/libavfilter/dnn/dnn_backend_native_layer_mathunary.h
@@ -37,6 +37,7 @@ typedef enum {
DMUO_ASIN = 4,
DMUO_ACOS = 5,
DMUO_ATAN = 6,
+DMUO_SINH = 7,
DMUO_COUNT
} DNNMathUnaryOperation;
diff --git a/tools/python/convert_from_tensorflow.py
b/tools/python/convert_from_tensorflow.py
index b90c31c495..6f34a71ab4 100644
--- a/tools/python/convert_from_tensorflow.py
+++ b/tools/python/convert_from_tensorflow.py
@@ -72,7 +72,7 @@ class TFConverter:
self.conv2d_scopename_inputname_dict = {}
self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3,
'Maximum':4, 'MathBinary':5, 'MathUnary':6}
self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3,
'Minimum':4}
-self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6}
+self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
'Acos':5, 'Atan':6, 'Sinh':7}
self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
self.name_operand_dict = {}
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
index 73cf23bf53..4747f41395 100644
--- a/tools/python/convert_header.py
+++ b/tools/python/convert_header.py
@@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE'
major = 1
# increase minor when we don't have to re-convert the model file
-minor = 12
+minor = 13
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
