ekalda commented on pull request #8368: URL: https://github.com/apache/tvm/pull/8368#issuecomment-872252986
> > > > This patch adds infrastructure to directly generate TFLite model buffers > > > > by using flatc, the flatbuffers command line tool. This gives us more > > > > freedom in creating the models for testing since we don't have to > > > > rely on any of the converters. > > > > > > > > * Add classes and helper functions to create the model buffer > > > > * Add some convolution tests that test TFLite2 models in int8 > > > > with per channel and per tensor quantization and remove the > > > > orphaned Keras tests > > > > > > > > Co-authored with @NicolaLancellotti > > > > > > > > > Hi @ekalda, I am just unable to see the need for such changes. As per my understanding, TFlite framework behaviour is not something we should control in TVM. > > > Model buffers should be created using standard Apis in TFlite. We should not use a custom one to validate our requirements which may result in failure of complete TFlite frontend Parser. > > > Maybe if you share what was the actual motivation for this change, we can discuss about the solution better. Thanks! > > > > > > Hi @ANSHUMAN87, see that RFC for some more motivation - https://discuss.tvm.apache.org/t/rfc-tflite-frontend-create-models-for-frontend-testing-by-directly-writing-tflite-buffers/9811 > > The gist is that the current converters that convert into TFLite are just not flexible enough when it comes to creating the one operator models with various properties (e.g. different fused activations). We have found that writing buffers directly is the most convenient, fast and debuggable way for consistently generating various one operator models with desired properties. > > As of whether the models created like that are valid TFLite models - since we use the TFLite schema to create the buffers, all the models created this way are valid TFLite models and if TVM frontend fails to parse them, that indicates problem with the TVM's TFLite's frontend parser. > > Also tagging @mbaret @FrozenGene @manupa-arm @anijain2305 @leandron > > Thanks @ekalda for detailed response. > I will go through little deeper about the point you mentioned about flexibility. > > But at high level, I am still not convinced for the approach. Please find below few fruits for thought. > > 1. TFlite was initially not designed to create models with first hand. Now it is coming up with model maker of its own. So the basic idea was convert existing tf and keras models to TFlite and then do certain optimization like layer fusions and post training quantization or fine tuning. And prep for inference. Now if the conversion to TFlite does not support certain features, then definitely there will not be any TFlite models with that feature. > > 2. Hence if you want that feature to be part of TFlite, then that has to be first part of Tensorflow project not TVM. > > 3. As per you mentioned, you need operator with fused activation, then again the questions arise for WHICH one? TFlite, Tensorflow or Keras. if TFlite already supports, then no point of this change. But if Tensorflow or Keras supports, but TFlite does not, then why TVM TFlite frontend needs to support? > > > Hope my points are clear. Please revert back in case anything. Definitely I might be missing the point which you might have observed. Please enlighten me, so that I also can be on same page. TIA! Hi @ANSHUMAN87, thanks for clarifying your concerns! I'll try to respond to your comments and expand on our motivaton for this work. - "As per my understanding, TFlite framework behaviour is not something we should control in TVM." - By writing buffers directly we are not controlling TFLite framework behavior, we are creating test cases that are conformant with TFLite schema and run with the TFLite runtime, which we think TVM should be able to compile. - "We should not use a custom one to validate our requirements which may result in failure of complete TFlite frontend Parser." - We are not creating "custom" operators, we are creating test cases that are conformant with TFLite schema and can be run with the TFLite runtime. If you are concerned about the situations where we have aligned the frontend parser to the test cases made by directly writing buffers and that parser failing to parse models that have been converted from TensorFlow or Keras - that will not happen as long as Tensorflow is consistent with TFLite since the models converted from TensorFlow always obey the TFLite schema. - "So the basic idea was convert existing tf and keras models to TFlite and then do certain optimization like layer fusions and post training quantization or fine tuning." - Yes, all outputs of such optimizations will always be expressible using the TFLite schema and runnable in TFLite runtime, so as a result of that we will only test cases subject to those two constraints. - "Now if the conversion to TFlite does not support certain features, then definitely there will not be any TFlite models with that feature." - Since the conversion is model to model (and not op to op), starting with single operators of TensorFlow/Keras might not cover atomic operators of TFLite that get generated during a model to model converter. The mapping between TensorFlow and TFLite is not strictly one-to-one, it can be both one-to-many and many-to-one (e.g. LSTM). - "But if Tensorflow or Keras supports, but TFlite does not, then why TVM TFlite frontend needs to support?" - Since TVM supports TFLite as an input format (not just TensorFlow and Keras), its vital for TVM project to have full coverage of flatbuffers that would execute on TFLite runtime. If TVM claims that it supports a frontend, it should support the whole specification of that frontend, which in this case is defined by the TFLite schema. I'll also expand a bit on the assumption that the above statements make - that TFLite is defined by what can be converted from TensorFlow or Keras: - Not all operators in TFLite can be created by using TensorFlow APIs. The example of this is TFLite_PostProcess_Detection, which currently gets created in TensorFlow by directly writing a TensorFlow buffer and converting that into TFLite model (it is also currently tested in the frontend by downloading an SSD Mobilenet from the web and chopping off rest of the operators). Other examples are RNN and LSTM. - If we assert that TFLite is defined by not only what we can create with using TensorFlow/Keras APIs but also whatever can be converted from TensorFlow flatbuffers, the space of TFLite we need to support and test now becomes whatever can be written in the TensorFlow buffers. - We currently test the TFLite operators individually but we use the converters. This isn't because we believe the results of the converters define TFLite, but because it has historically been the easiest way to make the TFLite operators. - We want to unit test a component whose interface is a TFLite flatbuffer. We don't want to test the behaviour of the converters because their behaviour is not stable unlike the schema which is strictly defined, versioned and executable via the TFLite runtime. - We are not proposing to change the current frontend testing philosophy, we are proposing a neat way of creating operators instead of having to come up tricks to create operators and it is easier to check whether the operators are indeed what we want them to be. I hope that helps. Let me know if that makes sense! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
