This is an automated email from the ASF dual-hosted git repository.
zhaowu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm-rfcs.git
The following commit(s) were added to refs/heads/main by this push:
new fc5d5ed [RFC][Frontend] Add a PaddlePaddle Frontend (#19)
fc5d5ed is described below
commit fc5d5ed74558058df3391089ba79bc98fc73246e
Author: Jason <[email protected]>
AuthorDate: Fri Aug 13 09:47:06 2021 +0800
[RFC][Frontend] Add a PaddlePaddle Frontend (#19)
* add rfc for paddlepaddle frontend
* add rfc for paddlepaddle frontend
* Update add_paddlepaddle_frontend.md
* Update add_paddlepaddle_frontend.md
* Update add_paddlepaddle_frontend.md
* Update add_paddlepaddle_frontend.md
* Update add_paddlepaddle_frontend.md
* refinement
* refinement
* add demo code load model from dist
Co-authored-by: root <[email protected]>
Co-authored-by: will-jl944 <[email protected]>
---
rfcs/add_paddlepaddle_frontend.md | 114 ++++++++++++++++++++++++++++++++++++++
1 file changed, 114 insertions(+)
diff --git a/rfcs/add_paddlepaddle_frontend.md
b/rfcs/add_paddlepaddle_frontend.md
new file mode 100644
index 0000000..c757516
--- /dev/null
+++ b/rfcs/add_paddlepaddle_frontend.md
@@ -0,0 +1,114 @@
+- Feature Name: add-paddlepaddle-frontend
+- Start Date: 2021-08-05
+- RFC PR: https://github.com/apache/tvm-rfcs/pull/19
+- GitHub Issue: TODO
+
+# Summary
+[summary]: #summary
+
+Add a paddlepaddle frontend, enhancing TVM's compatibility for deep learning
frameworks, which supports PaddlePaddle>=2.0
+
+# Motivation
+[motivation]: #motivation
+
+PaddlePaddle, an independent R&D deep learning platform in China, has been
officially open-sourced to professional communities since 2016. It has been
widely adopted by a wide range of sectors including manufacturing, agriculture,
enterprise service, and so on while serving more than 2.3 million developers.
With such advantages, PaddlePaddle has helped an increasing number of partners
commercialize AI.
+
+Currently, PaddlePaddle has built a prosperous technological ecology, there
are more than 500 models developed by official organization or outside
developers, covering CV/NLP/OCR/Speech, refer to the following links for more
details,
+
+- [PaddlePaddle/models](https://github.com/PaddlePaddle/models)
+- [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)
+- [PaddleClas](https://github.com/PaddlePaddle/PaddleClas)
+- [PaddleSeg](https://github.com/PaddlePaddle/PaddleSeg)
+- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
+- [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP)
+- [DeepSpeech](https://github.com/PaddlePaddle/DeepSpeech)
+
+As of version 2.0, PaddlePaddle supports imperative programming like PyTorch.
Furthermore, a mechanism of `Dynamic to Static` is provided to export a
PaddlePaddle model to graph representation, which is more friendly for
deployment. The following example code shows how to export a PaddlePaddle model,
+
+```
+import paddle
+import paddlehub
+model = hub.Module(name="resnet50_vd_imagenet_ssld")
+input_spec = paddle.static.InputSpec(
+ [1, 3, 224, 224], "float32", "image")
+paddle.jit.save(model, "model/infer", input_spec=[input_spec])
+```
+
+PaddlePaddle's deployment is supported by Paddle Inference/Paddle
Lite/OpenVINO/Tengine/Adlik now. We noticed that there are lots of developers
converting models to ONNX format for the compatibility with TVM, but only a
limited number of models are convertible due to lack of ONNX operators.
+Based on this background, we proposed this RFC to add a PaddlePaddle frontend
for TVM, improving usability for PaddlePaddle users and enhancing the
compatibility between PaddlePaddle and TVM.
+
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+If you dive into the pull request code, there are 2 concepts imported from
PaddlePaddle that you may want to know,
+- `paddle.jit.load`: Recommended API to load an exported inference model, the
type of the return value is `TranslatedLayer`, storing `Program`(similar to
computation graph) and parameters;
+- `paddle.static.load_inference_model`: API compatible with older version
PaddlePaddle models, the type of the return value is `Program`. All the
parameters are saved in `Scope` by default, parameters can be extracted from
the `paddle.fluid.global_scope()`.
+
+Therefore, this RFC will also add a new API to TVM to support PaddlePaddle
models,
+```
+relay.frontend.from_paddle(program_or_layer, shape_dict=None, scope=None)
+```
+- `program_or_layer`: the return value of `paddle.static.load_inference_model`
or `paddle.jit.load`
+- `shape_dict`: optional, input shapes of the model
+- `scope`: optional, which is available only if `model` is loaded using
`paddle.static.load_inference_model`
+
+The following example code shows how to import a PaddlePaddle model,
+```
+import paddle
+from tvm.contrib.download import download_testdata
+from tvm.contrib.tar import untar
+from tvm import relay
+
+# Download PaddlePaddle ResNet50 model
+model_url = 'https://bj.bcebos.com/x2paddle/models/paddle_resnet50.tar'
+model_path = download_testdata(model_url, "paddle_resnet50.tar", "./")
+untar(model_path, "./")
+
+# Load PaddlePaddle model
+model = paddle.jit.load('paddle_resnet50/model')
+
+shape_dict = {'inputs': [1, 3, 224, 224]}
+mod, params = relay.frontend.from_paddle(model, shape_dict=shape_dict)
+```
+
+Errors may happen if there exist some operators in the model that are not
supported by this frontend. If so, details will be printed out.
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+Since this RFC only aims to add a new frontend for converting PaddlePaddle
models to TVM Relay IR, no other features will be affected.
+
+In this proposed RFC, the whole process of PaddlePaddle frontend importing can
be divided into 2 steps:
+- 1. Reading a PaddlePaddle Model: The frontend supports models in
PaddlePaddle's inference model format, which are exported as graph based models
by PaddlePaddle's `Dynamic to Static` mechanism. The model contains 2 files
storing the model structure and parameters respectively. We use
`paddle.jit.load` to load the model files (For the compatibility with versions
of PaddlePaddle below 2.0, `paddle.static.load_inference_model` is also
supported);
+- 2. Operators Conversion: After the exported inference model is loaded, we
will extract its parameters and convert operators one by one. Since all the
operators are transversed according to topological ordering, there's no need to
worry about the order of converting the operators.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+Potential increase in time-cost of unit tests.
+
+# Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+The frontend of PaddlePaddle is similar to ONNX and TensorFlow. We support
model loaded by `paddle.jit.load` and `paddle.static.load_inference_model`. We
considered supporting `paddle.jit.load` only since this API is recommended as
of PaddlePaddle 2.0, but there are lots of users still using older versions.
Thus, supporting `paddle.static.load_inference_model` is still necessary.
+Currently, we have to convert PaddlePaddle models to ONNX format to make them
work with TVM, but only a limited number of models are supported due to the
lack of ONNX operators and the operator differences. With a new PaddlePaddle
frontend, we can support more operators and provide a better experience for TVM
and PaddlePaddle's users.
+
+# Prior art
+[prior-art]: #prior-art
+
+It's the first time we have added a PaddlePaddle frontend to an ML compiler.
+
+# Unresolved questions
+[unresolved-questions]: #unresolved-questions
+
+We will add new unit test cases that rely on PaddlePaddle framework, and this
may increase time-cost of unit tests. If there are any proslems, please let me
know.
+
+# Future possibilities
+[future-possibilities]: #future-possibilities
+
+For this RFC, we have made an established plan,
+
+- About 200 operators will be supported in this quarter, such as
deformable_conv/multiclass_nms
+- Control flow operators will be supported this year, mainly refer to
while_loop/if
+- Quantized model will be supported this year, including quantized model
obtained from `PaddleDetection`/`PaddleClas`/`PaddleSeg`
