Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Pedro Arthur > Sent: Wednesday, June 12, 2019 3:43 AM > To: FFmpeg development discussions and patches > Subject: Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to > convert TensorFlow model (.pb) to native model (.model) > > Hi, > > Em ter, 11 de jun de 2019 às 05:00, Guo, Yejun > escreveu: > > > > > > there are three options for the place to put these .py scripts. > > 1) at libavfilter/dnn/python/ > > the point is to put all the dnn stuffs together > > 2) at tools/python/ > > the point is that there is already a .py script under tools/ > > 3) create a new project controlled by ffmpeg > > the point is that the python scripts should not be part of ffmpeg source > tree. > > (btw, how to apply such sub project?) > > > I think option (2) is better as it is already there, even if (1) is > more convenient. thanks, there are two comments suggest to put under tools/, I'll put all the python scripts under tools/python if no more other comments. > > > > My idea is that the script generates dnn native model file which is loaded > > by > ffmpeg c code, > > it is better to put the script within the ffmpeg source tree, and all the > > dnn > stuffs would be better to put together, thanks. > > > > anyway, I'm open to any option, just to make the progress continue ... > > > > > > > > ping for review, thanks. > > > > > > Here is my rough plan after this patch. > > > - move dnn relative .h/.c from libavfilter to libavfilter/dnn, it is > > > expected > there > > > will be more files for dnn module (code for both model loading and > execution). > > > - add a layer for padding (tf.pad) for native mode and its fate test. > > > - change the script to add tf.pad support, and so the native model and > > > the tf > > > model of vf_sr will be the same. > > > in current implementation, the two models have a little difference, it > makes > > > the script not a general solution to convert tf model to native model. > > > - add layer maximum and fate test. This layer appears in tf model, but > > > not in > > > native model, of vf_sr. > > > - introduce operand concept in native mode (both execution and model), to > > > support data split and merge/concat in the network, such split/concat is > very > > > common. > > > it also makes possible to reuse memory for the intermediate data as the > > > output of the hidden layers. > > > - tune conv2d layer performance (it is very slow now) or add more layers > > > for > > > native mode. > > > > > > ___ > > > ffmpeg-devel mailing list > > > ffmpeg-devel@ffmpeg.org > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > > > To unsubscribe, visit link above, or email > > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > > ___ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)
Hi, Em ter, 11 de jun de 2019 às 05:00, Guo, Yejun escreveu: > > > there are three options for the place to put these .py scripts. > 1) at libavfilter/dnn/python/ > the point is to put all the dnn stuffs together > 2) at tools/python/ > the point is that there is already a .py script under tools/ > 3) create a new project controlled by ffmpeg > the point is that the python scripts should not be part of ffmpeg source > tree. > (btw, how to apply such sub project?) > I think option (2) is better as it is already there, even if (1) is more convenient. > My idea is that the script generates dnn native model file which is loaded by > ffmpeg c code, > it is better to put the script within the ffmpeg source tree, and all the dnn > stuffs would be better to put together, thanks. > > anyway, I'm open to any option, just to make the progress continue ... > > > > > ping for review, thanks. > > > > Here is my rough plan after this patch. > > - move dnn relative .h/.c from libavfilter to libavfilter/dnn, it is > > expected there > > will be more files for dnn module (code for both model loading and > > execution). > > - add a layer for padding (tf.pad) for native mode and its fate test. > > - change the script to add tf.pad support, and so the native model and the > > tf > > model of vf_sr will be the same. > > in current implementation, the two models have a little difference, it > > makes > > the script not a general solution to convert tf model to native model. > > - add layer maximum and fate test. This layer appears in tf model, but not > > in > > native model, of vf_sr. > > - introduce operand concept in native mode (both execution and model), to > > support data split and merge/concat in the network, such split/concat is > > very > > common. > > it also makes possible to reuse memory for the intermediate data as the > > output of the hidden layers. > > - tune conv2d layer performance (it is very slow now) or add more layers for > > native mode. > > > > ___ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Guo, Yejun > Sent: Monday, June 10, 2019 11:10 AM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to > convert TensorFlow model (.pb) to native model (.model) > > > > > -Original Message- > > From: Guo, Yejun > > Sent: Tuesday, June 04, 2019 3:10 PM > > To: ffmpeg-devel@ffmpeg.org > > Cc: Guo, Yejun > > Subject: [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow > model > > (.pb) to native model (.model) > > > > For example, given TensorFlow model file espcn.pb, > > to generate native model file espcn.model, just run: > > python convert.py espcn.pb > > > > In current implementation, the native model file is generated for > > specific dnn network with hard-code python scripts maintained out of ffmpeg. > > For example, srcnn network used by vf_sr is generated with > > > https://github.com/HighVoltageRocknRoll/sr/blob/master/generate_header_a > > nd_model.py#L85 > > > > In this patch, the script is designed as a general solution which > > converts general TensorFlow model .pb file into .model file. The script > > now has some tricky to be compatible with current implemention, will > > be refined step by step. > > > > The script is also added into ffmpeg source tree. It is expected there > > will be many more patches and community needs the ownership of it. > > > > Another technical direction is to do the conversion in c/c++ code within > > ffmpeg source tree. While .pb file is organized with protocol buffers, > > it is not easy to do such work with tiny c/c++ code, see more discussion > > at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/244496.html. So, > > choose the python script. > > > > Signed-off-by: Guo, Yejun > > --- > > .gitignore| 1 + > > libavfilter/dnn/python/convert.py | 52 ++ > > libavfilter/dnn/python/convert_from_tensorflow.py | 201 > > ++ > > 3 files changed, 254 insertions(+) > > create mode 100644 libavfilter/dnn/python/convert.py > > create mode 100644 libavfilter/dnn/python/convert_from_tensorflow.py there are three options for the place to put these .py scripts. 1) at libavfilter/dnn/python/ the point is to put all the dnn stuffs together 2) at tools/python/ the point is that there is already a .py script under tools/ 3) create a new project controlled by ffmpeg the point is that the python scripts should not be part of ffmpeg source tree. (btw, how to apply such sub project?) My idea is that the script generates dnn native model file which is loaded by ffmpeg c code, it is better to put the script within the ffmpeg source tree, and all the dnn stuffs would be better to put together, thanks. anyway, I'm open to any option, just to make the progress continue ... > > ping for review, thanks. > > Here is my rough plan after this patch. > - move dnn relative .h/.c from libavfilter to libavfilter/dnn, it is expected > there > will be more files for dnn module (code for both model loading and execution). > - add a layer for padding (tf.pad) for native mode and its fate test. > - change the script to add tf.pad support, and so the native model and the tf > model of vf_sr will be the same. > in current implementation, the two models have a little difference, it makes > the script not a general solution to convert tf model to native model. > - add layer maximum and fate test. This layer appears in tf model, but not in > native model, of vf_sr. > - introduce operand concept in native mode (both execution and model), to > support data split and merge/concat in the network, such split/concat is very > common. > it also makes possible to reuse memory for the intermediate data as the > output of the hidden layers. > - tune conv2d layer performance (it is very slow now) or add more layers for > native mode. > > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)
> -Original Message- > From: Guo, Yejun > Sent: Tuesday, June 04, 2019 3:10 PM > To: ffmpeg-devel@ffmpeg.org > Cc: Guo, Yejun > Subject: [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow > model > (.pb) to native model (.model) > > For example, given TensorFlow model file espcn.pb, > to generate native model file espcn.model, just run: > python convert.py espcn.pb > > In current implementation, the native model file is generated for > specific dnn network with hard-code python scripts maintained out of ffmpeg. > For example, srcnn network used by vf_sr is generated with > https://github.com/HighVoltageRocknRoll/sr/blob/master/generate_header_a > nd_model.py#L85 > > In this patch, the script is designed as a general solution which > converts general TensorFlow model .pb file into .model file. The script > now has some tricky to be compatible with current implemention, will > be refined step by step. > > The script is also added into ffmpeg source tree. It is expected there > will be many more patches and community needs the ownership of it. > > Another technical direction is to do the conversion in c/c++ code within > ffmpeg source tree. While .pb file is organized with protocol buffers, > it is not easy to do such work with tiny c/c++ code, see more discussion > at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/244496.html. So, > choose the python script. > > Signed-off-by: Guo, Yejun > --- > .gitignore| 1 + > libavfilter/dnn/python/convert.py | 52 ++ > libavfilter/dnn/python/convert_from_tensorflow.py | 201 > ++ > 3 files changed, 254 insertions(+) > create mode 100644 libavfilter/dnn/python/convert.py > create mode 100644 libavfilter/dnn/python/convert_from_tensorflow.py ping for review, thanks. Here is my rough plan after this patch. - move dnn relative .h/.c from libavfilter to libavfilter/dnn, it is expected there will be more files for dnn module (code for both model loading and execution). - add a layer for padding (tf.pad) for native mode and its fate test. - change the script to add tf.pad support, and so the native model and the tf model of vf_sr will be the same. in current implementation, the two models have a little difference, it makes the script not a general solution to convert tf model to native model. - add layer maximum and fate test. This layer appears in tf model, but not in native model, of vf_sr. - introduce operand concept in native mode (both execution and model), to support data split and merge/concat in the network, such split/concat is very common. it also makes possible to reuse memory for the intermediate data as the output of the hidden layers. - tune conv2d layer performance (it is very slow now) or add more layers for native mode. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 1/2] libavfilter/dnn: add script to convert TensorFlow model (.pb) to native model (.model)
For example, given TensorFlow model file espcn.pb,
to generate native model file espcn.model, just run:
python convert.py espcn.pb
In current implementation, the native model file is generated for
specific dnn network with hard-code python scripts maintained out of ffmpeg.
For example, srcnn network used by vf_sr is generated with
https://github.com/HighVoltageRocknRoll/sr/blob/master/generate_header_and_model.py#L85
In this patch, the script is designed as a general solution which
converts general TensorFlow model .pb file into .model file. The script
now has some tricky to be compatible with current implemention, will
be refined step by step.
The script is also added into ffmpeg source tree. It is expected there
will be many more patches and community needs the ownership of it.
Another technical direction is to do the conversion in c/c++ code within
ffmpeg source tree. While .pb file is organized with protocol buffers,
it is not easy to do such work with tiny c/c++ code, see more discussion
at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/244496.html. So,
choose the python script.
Signed-off-by: Guo, Yejun
---
.gitignore| 1 +
libavfilter/dnn/python/convert.py | 52 ++
libavfilter/dnn/python/convert_from_tensorflow.py | 201 ++
3 files changed, 254 insertions(+)
create mode 100644 libavfilter/dnn/python/convert.py
create mode 100644 libavfilter/dnn/python/convert_from_tensorflow.py
diff --git a/.gitignore b/.gitignore
index 0e57cb0..45420da 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,3 +36,4 @@
/lcov/
/src
/mapfile
+/libavfilter/dnn/python/__pycache__/
diff --git a/libavfilter/dnn/python/convert.py
b/libavfilter/dnn/python/convert.py
new file mode 100644
index 000..662b429
--- /dev/null
+++ b/libavfilter/dnn/python/convert.py
@@ -0,0 +1,52 @@
+# Copyright (c) 2019 Guo Yejun
+#
+# This file is part of FFmpeg.
+#
+# FFmpeg is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# FFmpeg is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with FFmpeg; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
==
+
+# verified with Python 3.5.2 on Ubuntu 16.04
+import argparse
+import os
+from convert_from_tensorflow import *
+
+def get_arguments():
+parser = argparse.ArgumentParser(description='generate native mode model
with weights from deep learning model')
+parser.add_argument('--outdir', type=str, default='./', help='where to put
generated files')
+parser.add_argument('--infmt', type=str, default='tensorflow',
help='format of the deep learning model')
+parser.add_argument('infile', help='path to the deep learning model with
weights')
+
+return parser.parse_args()
+
+def main():
+args = get_arguments()
+
+if not os.path.isfile(args.infile):
+print('the specified input file %s does not exist' % args.infile)
+exit(1)
+
+if not os.path.exists(args.outdir):
+print('create output directory %s' % args.outdir)
+os.mkdir(args.outdir)
+
+basefile = os.path.split(args.infile)[1]
+basefile = os.path.splitext(basefile)[0]
+outfile = os.path.join(args.outdir, basefile) + '.model'
+
+if args.infmt == 'tensorflow':
+convert_from_tensorflow(args.infile, outfile)
+
+if __name__ == '__main__':
+main()
diff --git a/libavfilter/dnn/python/convert_from_tensorflow.py
b/libavfilter/dnn/python/convert_from_tensorflow.py
new file mode 100644
index 000..37049e5
--- /dev/null
+++ b/libavfilter/dnn/python/convert_from_tensorflow.py
@@ -0,0 +1,201 @@
+# Copyright (c) 2019 Guo Yejun
+#
+# This file is part of FFmpeg.
+#
+# FFmpeg is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# FFmpeg is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with FFmpeg; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
