Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend
> > Hello, > > > > On Fri, 2 Feb 2024, at 08:26, wenbin.chen-at-intel@ffmpeg.org wrote: > > > +static void infer_completion_callback(void *args) { > > > +THRequestItem *request = (THRequestItem*)args; > > > +LastLevelTaskItem *lltask = request->lltask; > > > +TaskItem *task = lltask->task; > > > +DNNData outputs = { 0 }; > > > +THInferRequest *infer_request = request->infer_request; > > > +THModel *th_model = (THModel *)task->model; > > > +torch::Tensor *output = infer_request->output; > > > + > > > +c10::IntArrayRef sizes = output->sizes(); > > > +assert(sizes.size == 5); > > > > Why 5? > > 5 means 5 channels: [batch_size, frame_number, channel, height, width] Sorry, I mean 5 dimensions. > I only add video SR support, so it only support this type of data for now. > I will change the code to be more easy to read. > > > > > > +outputs.order = DCO_RGB; > > > +outputs.layout = DL_NCHW; > > > +outputs.dims[2] = sizes.at(3); > > > +outputs.dims[3] = sizes.at(4); > > > +outputs.dt = DNN_FLOAT; > > > +outputs.dims[1] = 3; > > > > Why 3? > > It is RGB so the channel is 3, but I should use sizes.at(2) instead of a magic > number. > Thanks for pointing it out. I will update it in patch v3. > > > > > > > -- > > Jean-Baptiste Kempf - President > > +33 672 704 734 > > ___ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend
> Hello, > > On Fri, 2 Feb 2024, at 08:26, wenbin.chen-at-intel@ffmpeg.org wrote: > > +static void infer_completion_callback(void *args) { > > +THRequestItem *request = (THRequestItem*)args; > > +LastLevelTaskItem *lltask = request->lltask; > > +TaskItem *task = lltask->task; > > +DNNData outputs = { 0 }; > > +THInferRequest *infer_request = request->infer_request; > > +THModel *th_model = (THModel *)task->model; > > +torch::Tensor *output = infer_request->output; > > + > > +c10::IntArrayRef sizes = output->sizes(); > > +assert(sizes.size == 5); > > Why 5? 5 means 5 channels: [batch_size, frame_number, channel, height, width] I only add video SR support, so it only support this type of data for now. I will change the code to be more easy to read. > > > +outputs.order = DCO_RGB; > > +outputs.layout = DL_NCHW; > > +outputs.dims[2] = sizes.at(3); > > +outputs.dims[3] = sizes.at(4); > > +outputs.dt = DNN_FLOAT; > > +outputs.dims[1] = 3; > > Why 3? It is RGB so the channel is 3, but I should use sizes.at(2) instead of a magic number. Thanks for pointing it out. I will update it in patch v3. > > > -- > Jean-Baptiste Kempf - President > +33 672 704 734 > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend
Hello, On Fri, 2 Feb 2024, at 08:26, wenbin.chen-at-intel@ffmpeg.org wrote: > +static void infer_completion_callback(void *args) { > +THRequestItem *request = (THRequestItem*)args; > +LastLevelTaskItem *lltask = request->lltask; > +TaskItem *task = lltask->task; > +DNNData outputs = { 0 }; > +THInferRequest *infer_request = request->infer_request; > +THModel *th_model = (THModel *)task->model; > +torch::Tensor *output = infer_request->output; > + > +c10::IntArrayRef sizes = output->sizes(); > +assert(sizes.size == 5); Why 5? > +outputs.order = DCO_RGB; > +outputs.layout = DL_NCHW; > +outputs.dims[2] = sizes.at(3); > +outputs.dims[3] = sizes.at(4); > +outputs.dt = DNN_FLOAT; > +outputs.dims[1] = 3; Why 3? -- Jean-Baptiste Kempf - President +33 672 704 734 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend
> On Feb 1, 2024, at 11:26 PM, wenbin.chen-at-intel@ffmpeg.org wrote: > > From: Wenbin Chen > > PyTorch is an open source machine learning framework that accelerates > the path from research prototyping to production deployment. Official > websit: https://pytorch.org/. We call the C++ library of PyTorch as > LibTorch, the same below. > > To build FFmpeg with LibTorch, please take following steps as reference: > 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/, > please select C++/Java for language, and other options as your need. > 2. unzip the file to your own dir, with command > unzip libtorch-shared-with-deps-latest.zip -d your_dir > 3. export libtorch_root/libtorch/include and > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH > export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH > 4. config FFmpeg with ../configure --enable-libtorch > --extra-cflag=-I/libtorch_root/libtorch/include > --extra-cflag=-I/libtorch_root/libtorch/include/torch/csrc/api/include > --extra-ldflags=-L/libtorch_root/libtorch/lib/ > 5. make > > To run FFmpeg DNN inference with LibTorch backend: > ./ffmpeg -i input.jpg -vf > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg > The LibTorch_model.pt can be generated by Python with torch.jit.script() api. > Please note, torch.jit.trace() is not recommanded, since it does not support > ambiguous input size. > > Signed-off-by: Ting Fu > Signed-off-by: Wenbin Chen Is there any feedback on this patch? It would be great to get this in before 7.0. - Cosmin ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend
From: Wenbin Chen PyTorch is an open source machine learning framework that accelerates the path from research prototyping to production deployment. Official websit: https://pytorch.org/. We call the C++ library of PyTorch as LibTorch, the same below. To build FFmpeg with LibTorch, please take following steps as reference: 1. download LibTorch C++ library in https://pytorch.org/get-started/locally/, please select C++/Java for language, and other options as your need. 2. unzip the file to your own dir, with command unzip libtorch-shared-with-deps-latest.zip -d your_dir 3. export libtorch_root/libtorch/include and libtorch_root/libtorch/include/torch/csrc/api/include to $PATH export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH 4. config FFmpeg with ../configure --enable-libtorch --extra-cflag=-I/libtorch_root/libtorch/include --extra-cflag=-I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=-L/libtorch_root/libtorch/lib/ 5. make To run FFmpeg DNN inference with LibTorch backend: ./ffmpeg -i input.jpg -vf dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y output.jpg The LibTorch_model.pt can be generated by Python with torch.jit.script() api. Please note, torch.jit.trace() is not recommanded, since it does not support ambiguous input size. Signed-off-by: Ting Fu Signed-off-by: Wenbin Chen --- configure | 5 +- libavfilter/dnn/Makefile | 1 + libavfilter/dnn/dnn_backend_torch.cpp | 587 ++ libavfilter/dnn/dnn_interface.c | 5 + libavfilter/dnn_filter_common.c | 15 +- libavfilter/dnn_interface.h | 2 +- libavfilter/vf_dnn_processing.c | 3 + 7 files changed, 614 insertions(+), 4 deletions(-) create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp diff --git a/configure b/configure index 68f675a4bc..bc11172fe4 100755 --- a/configure +++ b/configure @@ -279,6 +279,7 @@ External library support: --enable-libtheora enable Theora encoding via libtheora [no] --enable-libtls enable LibreSSL (via libtls), needed for https support if openssl, gnutls or mbedtls is not used [no] + --enable-libtorchenable Torch as one DNN backend [no] --enable-libtwolame enable MP2 encoding via libtwolame [no] --enable-libuavs3d enable AVS3 decoding via libuavs3d [no] --enable-libv4l2 enable libv4l2/v4l-utils [no] @@ -1901,6 +1902,7 @@ EXTERNAL_LIBRARY_LIST=" libtensorflow libtesseract libtheora +libtorch libtwolame libuavs3d libv4l2 @@ -2776,7 +2778,7 @@ cbs_vp9_select="cbs" deflate_wrapper_deps="zlib" dirac_parse_select="golomb" dovi_rpu_select="golomb" -dnn_suggest="libtensorflow libopenvino" +dnn_suggest="libtensorflow libopenvino libtorch" dnn_deps="avformat swscale" error_resilience_select="me_cmp" evcparse_select="golomb" @@ -6873,6 +6875,7 @@ enabled libtensorflow && require libtensorflow tensorflow/c/c_api.h TF_Versi enabled libtesseract && require_pkg_config libtesseract tesseract tesseract/capi.h TessBaseAPICreate enabled libtheora && require libtheora theora/theoraenc.h th_info_init -ltheoraenc -ltheoradec -logg enabled libtls&& require_pkg_config libtls libtls tls.h tls_configure +enabled libtorch && check_cxxflags -std=c++14 && require_cpp libtorch torch/torch.h "torch::Tensor" -ltorch -lc10 -ltorch_cpu -lstdc++ -lpthread enabled libtwolame&& require libtwolame twolame.h twolame_init -ltwolame && { check_lib libtwolame twolame.h twolame_encode_buffer_float32_interleaved -ltwolame || die "ERROR: libtwolame must be installed and version must be >= 0.3.10"; } diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile index 5d5697ea42..3d09927c98 100644 --- a/libavfilter/dnn/Makefile +++ b/libavfilter/dnn/Makefile @@ -6,5 +6,6 @@ OBJS-$(CONFIG_DNN) += dnn/dnn_backend_common.o DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn/dnn_backend_tf.o DNN-OBJS-$(CONFIG_LIBOPENVINO) += dnn/dnn_backend_openvino.o +DNN-OBJS-$(CONFIG_LIBTORCH) += dnn/dnn_backend_torch.o OBJS-$(CONFIG_DNN) += $(DNN-OBJS-yes) diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp new file mode 100644 index 00..b905c55175 --- /dev/null +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -0,0 +1,587 @@ +/* + * Copyright (c) 2024 + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the