Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Steven Liu > Sent: Friday, May 24, 2019 11:35 PM > To: FFmpeg development discussions and patches > Cc: Steven Liu > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > 在 2019年5月24日，20:34，Pedro Arthur 写 > 道： > > > > Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun > escreveu: > >> > >> > >> > >>>>>>>> Option 2) > >>>>>>>> Write c code in FFmpeg to convert tensorflow file format (format 1) > >>>> directly > >>>>>>> into memory representation (format 3), and so we controls everything > in > >>>>>>> ffmpeg community. And the conversion can be extended to import > more > >>>> file > >>>>>>> formats such as torch, darknet, etc. One example is that OpenCV uses > >>>> this > >>>>>>> method. > >>>>>>>> > >>>>>>>> The in memory representation (format 3) can still be current. > >>>>>>>> > >>>>>>> > >>>>>>> Option 2 would be ideal, as it does not introduce any dependency for > >>>>>>> using the native backend. > >>>>>>> Yet I'm not sure how complex implementing the tf model reader can > >>> be, > >>>>>>> If I remember correctly the student said it was not trivial at the > >>>>>>> time. > >>>>>> > >>>>>> yes, it is not easy, but I think it is worthy to do it. Here is a > >>>>>> reference > >>>> example > >>>>>> for the complexity, see > >>>>>> > >>>> > >>> > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > >>>>>> tf_importer.cpp. > >>>>>> > >>>>>>> > >>>>>>> Is the tf model file stable? if not it will be a maintenance burden to > >>>>>>> keep it working whenever tf releases a new version. This point makes > >>>>>>> me think having control over our file format is good. > >>>>>> > >>>>>> imho, this issue is always there, no matter which method used, unless > our > >>>>>> format could be exported by tensorflow (it has little possibility). > >>>>>> > >>>>>> Whenever tf releases a new version with a new file format, we still > have > >>> to > >>>>>> change the python script in phase 1 (convert tf file model to our > >>>>>> format) > >>>> which > >>>>>> is even an external dependency at > >>>>>> https://github.com/HighVoltageRocknRoll/sr, > >>>>>> > >>>>>> As from effort perspective, the current implementation is better since > >>>> python > >>>>>> script is simpler. But I think we are still worth implementing option > >>>>>> 2 as > >>> the > >>>>>> ideal technical direction. > >>>>> > >>>>> I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, > it > >>> is > >>>> actually > >>>>> not an converter (from tf model to native model), but hard code for > >>>>> given > >>>> models. > >>>>> And the native model is not exactly the same as tf model, it even > changes > >>> the > >>>> behavior > >>>>> of pad parameter of conv layer. > >>>>> > >>>>> If community is open to option 2, I'll try it. > >>>>> > >>>> Option 2 is fine for me. > >>> > >>> that's great, :) > >> > >> looks that option 2 is a bit complex, TF model file is in protocol buffers > (protobuf) format and not easy to parse it with simple c code. > >> > >> Since there is no official c support for protobuf, let's first image how > >> the > work can be done via official c++ support. > >> > >> 1. get protobuf compiler protoc, .h header files and .so library files > (download or build from > https://github.com/protocolbuffers/protobuf/tree/master/src). > >> 2. get tensorflow model's .proto fil
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> 在 2019年5月24日，20:34，Pedro Arthur 写道： > > Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun escreveu: >> >> >> Option 2) Write c code in FFmpeg to convert tensorflow file format (format 1) directly >>> into memory representation (format 3), and so we controls everything in >>> ffmpeg community. And the conversion can be extended to import more file >>> formats such as torch, darknet, etc. One example is that OpenCV uses this >>> method. The in memory representation (format 3) can still be current. >>> >>> Option 2 would be ideal, as it does not introduce any dependency for >>> using the native backend. >>> Yet I'm not sure how complex implementing the tf model reader can >>> be, >>> If I remember correctly the student said it was not trivial at the >>> time. >> >> yes, it is not easy, but I think it is worthy to do it. Here is a >> reference example >> for the complexity, see >> >>> https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ >> tf_importer.cpp. >> >>> >>> Is the tf model file stable? if not it will be a maintenance burden to >>> keep it working whenever tf releases a new version. This point makes >>> me think having control over our file format is good. >> >> imho, this issue is always there, no matter which method used, unless our >> format could be exported by tensorflow (it has little possibility). >> >> Whenever tf releases a new version with a new file format, we still have >>> to >> change the python script in phase 1 (convert tf file model to our format) which >> is even an external dependency at >> https://github.com/HighVoltageRocknRoll/sr, >> >> As from effort perspective, the current implementation is better since python >> script is simpler. But I think we are still worth implementing option 2 >> as >>> the >> ideal technical direction. > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it >>> is actually > not an converter (from tf model to native model), but hard code for given models. > And the native model is not exactly the same as tf model, it even changes >>> the behavior > of pad parameter of conv layer. > > If community is open to option 2, I'll try it. > Option 2 is fine for me. >>> >>> that's great, :) >> >> looks that option 2 is a bit complex, TF model file is in protocol buffers >> (protobuf) format and not easy to parse it with simple c code. >> >> Since there is no official c support for protobuf, let's first image how the >> work can be done via official c++ support. >> >> 1. get protobuf compiler protoc, .h header files and .so library files >> (download or build from >> https://github.com/protocolbuffers/protobuf/tree/master/src). >> 2. get tensorflow model's .proto files from >> https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework. >> 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step >> 1). >> 4. let the generated .cc/.h files be part of ffmpeg source tree, and build >> with protobuf header/library files. >> 5. at run time, the protobuf libraries are invoked. It means that the system >> should have installed protobuf dev package. >> >> furthermore, there is a compatible problem between the protobuf compiler, >> header files and library files. >> So, as a practice to fix it, the method is to make the protobuf source code >> be part of ffmpeg source tree. (it is a common practice, so we can many >> other projects contain the protobuf source code). >> >> I guess the above method is not acceptable in ffmpeg. I would be glad to >> continue if the community embrace this change. :) > Indeed I think it is not acceptable. > >> >> While the current implementation has external dependency, my new suggestion >> is: >> - add a python script under .../libavfilter/dnn/ (all other dnn source >> files will be also moved here later), so ffmpeg has the full control on it. > I'm not sure about the policy on putting secondary scripts with the > main code, but another option is to create a repo controlled by ffmpeg > maybe? > I think this option would also help GSoC students that work with dnn, > so they don't have to depend on previous students maintaining > independent repositories. Yes, I agreed with you. I think this is a better way. maintaining the repositories at one repo controlled by ffmpeg. > >> - it is a script to convert tensorflow model file into native model file. >> (other formats such as caffe, torch can also be supported later if needed) >> >> thanks. >> >> ___ >> ffmpeg-devel mailing list >> ffmpeg-devel@ffmpeg.org >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> >> To unsubscribe, visit link above, or email >>
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun escreveu: > > > > > > > > > > Option 2) > > > > > > > Write c code in FFmpeg to convert tensorflow file format (format > > > > > > > 1) > > > directly > > > > > > into memory representation (format 3), and so we controls > > > > > > everything in > > > > > > ffmpeg community. And the conversion can be extended to import more > > > file > > > > > > formats such as torch, darknet, etc. One example is that OpenCV uses > > > this > > > > > > method. > > > > > > > > > > > > > > The in memory representation (format 3) can still be current. > > > > > > > > > > > > > > > > > > > Option 2 would be ideal, as it does not introduce any dependency for > > > > > > using the native backend. > > > > > > Yet I'm not sure how complex implementing the tf model reader can > > be, > > > > > > If I remember correctly the student said it was not trivial at the > > > > > > time. > > > > > > > > > > yes, it is not easy, but I think it is worthy to do it. Here is a > > > > > reference > > > example > > > > > for the complexity, see > > > > > > > > > > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > > > > > tf_importer.cpp. > > > > > > > > > > > > > > > > > Is the tf model file stable? if not it will be a maintenance burden > > > > > > to > > > > > > keep it working whenever tf releases a new version. This point makes > > > > > > me think having control over our file format is good. > > > > > > > > > > imho, this issue is always there, no matter which method used, unless > > > > > our > > > > > format could be exported by tensorflow (it has little possibility). > > > > > > > > > > Whenever tf releases a new version with a new file format, we still > > > > > have > > to > > > > > change the python script in phase 1 (convert tf file model to our > > > > > format) > > > which > > > > > is even an external dependency at > > > > > https://github.com/HighVoltageRocknRoll/sr, > > > > > > > > > > As from effort perspective, the current implementation is better since > > > python > > > > > script is simpler. But I think we are still worth implementing option > > > > > 2 as > > the > > > > > ideal technical direction. > > > > > > > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, > > > > it > > is > > > actually > > > > not an converter (from tf model to native model), but hard code for > > > > given > > > models. > > > > And the native model is not exactly the same as tf model, it even > > > > changes > > the > > > behavior > > > > of pad parameter of conv layer. > > > > > > > > If community is open to option 2, I'll try it. > > > > > > > Option 2 is fine for me. > > > > that's great, :) > > looks that option 2 is a bit complex, TF model file is in protocol buffers > (protobuf) format and not easy to parse it with simple c code. > > Since there is no official c support for protobuf, let's first image how the > work can be done via official c++ support. > > 1. get protobuf compiler protoc, .h header files and .so library files > (download or build from > https://github.com/protocolbuffers/protobuf/tree/master/src). > 2. get tensorflow model's .proto files from > https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework. > 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step > 1). > 4. let the generated .cc/.h files be part of ffmpeg source tree, and build > with protobuf header/library files. > 5. at run time, the protobuf libraries are invoked. It means that the system > should have installed protobuf dev package. > > furthermore, there is a compatible problem between the protobuf compiler, > header files and library files. > So, as a practice to fix it, the method is to make the protobuf source code > be part of ffmpeg source tree. (it is a common practice, so we can many other > projects contain the protobuf source code). > > I guess the above method is not acceptable in ffmpeg. I would be glad to > continue if the community embrace this change. :) Indeed I think it is not acceptable. > > While the current implementation has external dependency, my new suggestion > is: > - add a python script under .../libavfilter/dnn/ (all other dnn source > files will be also moved here later), so ffmpeg has the full control on it. I'm not sure about the policy on putting secondary scripts with the main code, but another option is to create a repo controlled by ffmpeg maybe? I think this option would also help GSoC students that work with dnn, so they don't have to depend on previous students maintaining independent repositories. > - it is a script to convert tensorflow model file into native model file. > (other formats such as caffe, torch can also be supported later if needed) > > thanks. > > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> > > > > > Option 2) > > > > > > Write c code in FFmpeg to convert tensorflow file format (format 1) > > directly > > > > > into memory representation (format 3), and so we controls everything > > > > > in > > > > > ffmpeg community. And the conversion can be extended to import more > > file > > > > > formats such as torch, darknet, etc. One example is that OpenCV uses > > this > > > > > method. > > > > > > > > > > > > The in memory representation (format 3) can still be current. > > > > > > > > > > > > > > > > Option 2 would be ideal, as it does not introduce any dependency for > > > > > using the native backend. > > > > > Yet I'm not sure how complex implementing the tf model reader can > be, > > > > > If I remember correctly the student said it was not trivial at the > > > > > time. > > > > > > > > yes, it is not easy, but I think it is worthy to do it. Here is a > > > > reference > > example > > > > for the complexity, see > > > > > > > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > > > > tf_importer.cpp. > > > > > > > > > > > > > > Is the tf model file stable? if not it will be a maintenance burden to > > > > > keep it working whenever tf releases a new version. This point makes > > > > > me think having control over our file format is good. > > > > > > > > imho, this issue is always there, no matter which method used, unless > > > > our > > > > format could be exported by tensorflow (it has little possibility). > > > > > > > > Whenever tf releases a new version with a new file format, we still have > to > > > > change the python script in phase 1 (convert tf file model to our > > > > format) > > which > > > > is even an external dependency at > > > > https://github.com/HighVoltageRocknRoll/sr, > > > > > > > > As from effort perspective, the current implementation is better since > > python > > > > script is simpler. But I think we are still worth implementing option 2 > > > > as > the > > > > ideal technical direction. > > > > > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it > is > > actually > > > not an converter (from tf model to native model), but hard code for given > > models. > > > And the native model is not exactly the same as tf model, it even changes > the > > behavior > > > of pad parameter of conv layer. > > > > > > If community is open to option 2, I'll try it. > > > > > Option 2 is fine for me. > > that's great, :) looks that option 2 is a bit complex, TF model file is in protocol buffers (protobuf) format and not easy to parse it with simple c code. Since there is no official c support for protobuf, let's first image how the work can be done via official c++ support. 1. get protobuf compiler protoc, .h header files and .so library files (download or build from https://github.com/protocolbuffers/protobuf/tree/master/src). 2. get tensorflow model's .proto files from https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework. 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step 1). 4. let the generated .cc/.h files be part of ffmpeg source tree, and build with protobuf header/library files. 5. at run time, the protobuf libraries are invoked. It means that the system should have installed protobuf dev package. furthermore, there is a compatible problem between the protobuf compiler, header files and library files. So, as a practice to fix it, the method is to make the protobuf source code be part of ffmpeg source tree. (it is a common practice, so we can many other projects contain the protobuf source code). I guess the above method is not acceptable in ffmpeg. I would be glad to continue if the community embrace this change. :) While the current implementation has external dependency, my new suggestion is: - add a python script under .../libavfilter/dnn/ (all other dnn source files will be also moved here later), so ffmpeg has the full control on it. - it is a script to convert tensorflow model file into native model file. (other formats such as caffe, torch can also be supported later if needed) thanks. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Pedro Arthur > Sent: Saturday, April 27, 2019 3:00 AM > To: FFmpeg development discussions and patches > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > Em sex, 26 de abr de 2019 às 02:41, Guo, Yejun > escreveu: > > > > > > > > > -Original Message- > > > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf > Of > > > Guo, Yejun > > > Sent: Friday, April 19, 2019 11:22 PM > > > To: FFmpeg development discussions and patches > > > > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > > > > > > > > > -Original Message- > > > > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On > Behalf Of > > > > Pedro Arthur > > > > Sent: Friday, April 19, 2019 10:43 PM > > > > To: FFmpeg development discussions and patches > > > > > > > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > > > > Hi, > > > > > > > > Em sex, 19 de abr de 2019 às 05:41, Guo, Yejun > > > > escreveu: > > > > > > > > > > Option 2) > > > > > Write c code in FFmpeg to convert tensorflow file format (format 1) > directly > > > > into memory representation (format 3), and so we controls everything in > > > > ffmpeg community. And the conversion can be extended to import more > file > > > > formats such as torch, darknet, etc. One example is that OpenCV uses > this > > > > method. > > > > > > > > > > The in memory representation (format 3) can still be current. > > > > > > > > > > > > > Option 2 would be ideal, as it does not introduce any dependency for > > > > using the native backend. > > > > Yet I'm not sure how complex implementing the tf model reader can be, > > > > If I remember correctly the student said it was not trivial at the > > > > time. > > > > > > yes, it is not easy, but I think it is worthy to do it. Here is a > > > reference > example > > > for the complexity, see > > > > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > > > tf_importer.cpp. > > > > > > > > > > > Is the tf model file stable? if not it will be a maintenance burden to > > > > keep it working whenever tf releases a new version. This point makes > > > > me think having control over our file format is good. > > > > > > imho, this issue is always there, no matter which method used, unless our > > > format could be exported by tensorflow (it has little possibility). > > > > > > Whenever tf releases a new version with a new file format, we still have > > > to > > > change the python script in phase 1 (convert tf file model to our format) > which > > > is even an external dependency at > > > https://github.com/HighVoltageRocknRoll/sr, > > > > > > As from effort perspective, the current implementation is better since > python > > > script is simpler. But I think we are still worth implementing option 2 > > > as the > > > ideal technical direction. > > > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it is > actually > > not an converter (from tf model to native model), but hard code for given > models. > > And the native model is not exactly the same as tf model, it even changes > > the > behavior > > of pad parameter of conv layer. > > > > If community is open to option 2, I'll try it. > > > Option 2 is fine for me. that's great, :) > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
Em sex, 26 de abr de 2019 às 02:41, Guo, Yejun escreveu: > > > > > -Original Message- > > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > > Guo, Yejun > > Sent: Friday, April 19, 2019 11:22 PM > > To: FFmpeg development discussions and patches > > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > > > > > -Original Message- > > > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > > > Pedro Arthur > > > Sent: Friday, April 19, 2019 10:43 PM > > > To: FFmpeg development discussions and patches > > > > > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > > Hi, > > > > > > Em sex, 19 de abr de 2019 às 05:41, Guo, Yejun > > > escreveu: > > > > > > > > Option 2) > > > > Write c code in FFmpeg to convert tensorflow file format (format 1) > > > > directly > > > into memory representation (format 3), and so we controls everything in > > > ffmpeg community. And the conversion can be extended to import more file > > > formats such as torch, darknet, etc. One example is that OpenCV uses this > > > method. > > > > > > > > The in memory representation (format 3) can still be current. > > > > > > > > > > Option 2 would be ideal, as it does not introduce any dependency for > > > using the native backend. > > > Yet I'm not sure how complex implementing the tf model reader can be, > > > If I remember correctly the student said it was not trivial at the > > > time. > > > > yes, it is not easy, but I think it is worthy to do it. Here is a reference > > example > > for the complexity, see > > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > > tf_importer.cpp. > > > > > > > > Is the tf model file stable? if not it will be a maintenance burden to > > > keep it working whenever tf releases a new version. This point makes > > > me think having control over our file format is good. > > > > imho, this issue is always there, no matter which method used, unless our > > format could be exported by tensorflow (it has little possibility). > > > > Whenever tf releases a new version with a new file format, we still have to > > change the python script in phase 1 (convert tf file model to our format) > > which > > is even an external dependency at > > https://github.com/HighVoltageRocknRoll/sr, > > > > As from effort perspective, the current implementation is better since > > python > > script is simpler. But I think we are still worth implementing option 2 as > > the > > ideal technical direction. > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it is > actually > not an converter (from tf model to native model), but hard code for given > models. > And the native model is not exactly the same as tf model, it even changes the > behavior > of pad parameter of conv layer. > > If community is open to option 2, I'll try it. > Option 2 is fine for me. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Guo, Yejun > Sent: Friday, April 19, 2019 11:22 PM > To: FFmpeg development discussions and patches > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > > -Original Message- > > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > > Pedro Arthur > > Sent: Friday, April 19, 2019 10:43 PM > > To: FFmpeg development discussions and patches > > > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > > > Hi, > > > > Em sex, 19 de abr de 2019 às 05:41, Guo, Yejun > > escreveu: > > > > > > Option 2) > > > Write c code in FFmpeg to convert tensorflow file format (format 1) > > > directly > > into memory representation (format 3), and so we controls everything in > > ffmpeg community. And the conversion can be extended to import more file > > formats such as torch, darknet, etc. One example is that OpenCV uses this > > method. > > > > > > The in memory representation (format 3) can still be current. > > > > > > > Option 2 would be ideal, as it does not introduce any dependency for > > using the native backend. > > Yet I'm not sure how complex implementing the tf model reader can be, > > If I remember correctly the student said it was not trivial at the > > time. > > yes, it is not easy, but I think it is worthy to do it. Here is a reference > example > for the complexity, see > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/ > tf_importer.cpp. > > > > > Is the tf model file stable? if not it will be a maintenance burden to > > keep it working whenever tf releases a new version. This point makes > > me think having control over our file format is good. > > imho, this issue is always there, no matter which method used, unless our > format could be exported by tensorflow (it has little possibility). > > Whenever tf releases a new version with a new file format, we still have to > change the python script in phase 1 (convert tf file model to our format) > which > is even an external dependency at > https://github.com/HighVoltageRocknRoll/sr, > > As from effort perspective, the current implementation is better since python > script is simpler. But I think we are still worth implementing option 2 as the > ideal technical direction. I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it is actually not an converter (from tf model to native model), but hard code for given models. And the native model is not exactly the same as tf model, it even changes the behavior of pad parameter of conv layer. If community is open to option 2, I'll try it. > > > ___ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Pedro Arthur > Sent: Friday, April 19, 2019 10:43 PM > To: FFmpeg development discussions and patches > Subject: Re: [FFmpeg-devel] native mode in FFmpeg DNN module > > Hi, > > Em sex, 19 de abr de 2019 às 05:41, Guo, Yejun > escreveu: > > > > Option 2) > > Write c code in FFmpeg to convert tensorflow file format (format 1) directly > into memory representation (format 3), and so we controls everything in > ffmpeg community. And the conversion can be extended to import more file > formats such as torch, darknet, etc. One example is that OpenCV uses this > method. > > > > The in memory representation (format 3) can still be current. > > > > Option 2 would be ideal, as it does not introduce any dependency for > using the native backend. > Yet I'm not sure how complex implementing the tf model reader can be, > If I remember correctly the student said it was not trivial at the > time. yes, it is not easy, but I think it is worthy to do it. Here is a reference example for the complexity, see https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/tf_importer.cpp. > > Is the tf model file stable? if not it will be a maintenance burden to > keep it working whenever tf releases a new version. This point makes > me think having control over our file format is good. imho, this issue is always there, no matter which method used, unless our format could be exported by tensorflow (it has little possibility). Whenever tf releases a new version with a new file format, we still have to change the python script in phase 1 (convert tf file model to our format) which is even an external dependency at https://github.com/HighVoltageRocknRoll/sr, As from effort perspective, the current implementation is better since python script is simpler. But I think we are still worth implementing option 2 as the ideal technical direction. > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] native mode in FFmpeg DNN module
Hi, Em sex, 19 de abr de 2019 às 05:41, Guo, Yejun escreveu: > > Option 2) > Write c code in FFmpeg to convert tensorflow file format (format 1) directly > into memory representation (format 3), and so we controls everything in > ffmpeg community. And the conversion can be extended to import more file > formats such as torch, darknet, etc. One example is that OpenCV uses this > method. > > The in memory representation (format 3) can still be current. > Option 2 would be ideal, as it does not introduce any dependency for using the native backend. Yet I'm not sure how complex implementing the tf model reader can be, If I remember correctly the student said it was not trivial at the time. Is the tf model file stable? if not it will be a maintenance burden to keep it working whenever tf releases a new version. This point makes me think having control over our file format is good. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] native mode in FFmpeg DNN module
Hi, The DNN module currently supports two backends, tensorflow (dnn_backend_tf.c) and native(dnn_backend_native.c). The native mode has external dependency, imho it's not good and need a change. I think it is still a proper timing for the change, for the limited functionality and performance of current native mode. With current implementation, the native mode involves 3 formats and 3 phases, see below. "Tensorflow model file format (format 1)" ->convert model (phase 1)-> "native model file format (format 2) " ->load model file into memory (phase 2)-> "in memory representation (format 3)" ->inference the model (phase 3)-> We created format 2 and format 3, we write c code for phase 2 and phase 3 within ffmpeg. The phase 1 is written in python to convert from tensorflow model file to native mode file, it is an external dependency at https://github.com/HighVoltageRocknRoll/sr. Once we add anything new in the model, for example, add a new layer, or even just add the padding option for the current Conv layer, we always need to change the external dependency. There will be many many times for such change. I have two options to improve it. Option 1) Use ONNX (https://github.com/onnx/onnx) to replace phase 1, format 2 and format 3. Open Neural Network Exchange (ONNX) provides an open source format for AI models, and the model files are protobuf pb files The advantage is that we don't need to worry about phase 1, format 2 and format 3, but to load the protobuf file into memory (phase 2), google just provides C++ support (without c support), while only C is allowed in FFmpeg, we have to write our c code in ffmpeg to parse/load the protobuf file. Option 2) Write c code in FFmpeg to convert tensorflow file format (format 1) directly into memory representation (format 3), and so we controls everything in ffmpeg community. And the conversion can be extended to import more file formats such as torch, darknet, etc. One example is that OpenCV uses this method. The in memory representation (format 3) can still be current. I personally prefer option 2. Anyway, will be glad to see any better options. I can continue the contribute on this area once the community decides the technical direction. Thanks Yejun ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
