zhiics commented on a change in pull request #6395: URL: https://github.com/apache/incubator-tvm/pull/6395#discussion_r500486187
########## File path: docs/deploy/tensorrt.rst ########## @@ -0,0 +1,288 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Relay TensorRT Integration +========================== +**Author**: `Trevor Morris <https://github.com/trevor-m>`_ + +Introduction +------------ + +NVIDIA TensorRT is a library for optimized deep learning inference. This integration will offload as +many operators as possible from Relay to TensorRT, providing a performance boost on NVIDIA GPUs +without the need to tune schedules. + +This guide will demonstrate how to install TensorRT and build TVM with TensorRT BYOC and runtime +enabled. It will also provide example code to compile and run a ResNet-18 model using TensorRT and +how to configure the compilation and runtime settings. Finally, we document the supported operators +and how to extend the integration to support other operators. + +Installing TensorRT +------------------- + +In order to download TensorRT, you will need to create an NVIDIA Developer program account. Please +see NVIDIA's documentation for more info: +https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html. If you have a Jetson device +such as a TX1, TX2, Xavier, or Nano, TensorRT will already be installed on the device via the +JetPack SDK. + +There are two methods to install TensorRT: + +* System install via deb or rpm package. +* Tar file installation. + +With the tar file installation method, you must provide the path of the extracted tar archive to +USE_TENSORT_GRAPH_RUNTIME=/path/to/TensorRT. With the system install method, +USE_TENSORT_GRAPH_RUNTIME=ON will automatically locate your installation. + +Building TVM with TensorRT support +---------------------------------- + +There are two separate build flags for TensorRT integration in TVM: + +* USE_TENSORT=ON/OFF - This flag will enable compiling a TensorRT module, which does not require any Review comment: USE_TENSORRT? ########## File path: docs/deploy/tensorrt.rst ########## @@ -0,0 +1,288 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Relay TensorRT Integration +========================== +**Author**: `Trevor Morris <https://github.com/trevor-m>`_ + +Introduction +------------ + +NVIDIA TensorRT is a library for optimized deep learning inference. This integration will offload as +many operators as possible from Relay to TensorRT, providing a performance boost on NVIDIA GPUs +without the need to tune schedules. + +This guide will demonstrate how to install TensorRT and build TVM with TensorRT BYOC and runtime +enabled. It will also provide example code to compile and run a ResNet-18 model using TensorRT and +how to configure the compilation and runtime settings. Finally, we document the supported operators +and how to extend the integration to support other operators. + +Installing TensorRT +------------------- + +In order to download TensorRT, you will need to create an NVIDIA Developer program account. Please +see NVIDIA's documentation for more info: +https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html. If you have a Jetson device +such as a TX1, TX2, Xavier, or Nano, TensorRT will already be installed on the device via the +JetPack SDK. + +There are two methods to install TensorRT: + +* System install via deb or rpm package. +* Tar file installation. + +With the tar file installation method, you must provide the path of the extracted tar archive to +USE_TENSORT_GRAPH_RUNTIME=/path/to/TensorRT. With the system install method, +USE_TENSORT_GRAPH_RUNTIME=ON will automatically locate your installation. + +Building TVM with TensorRT support +---------------------------------- + +There are two separate build flags for TensorRT integration in TVM: + +* USE_TENSORT=ON/OFF - This flag will enable compiling a TensorRT module, which does not require any +TensorRT library. +* USE_TENSORT_GRAPH_RUNTIME=ON/OFF/path-to-TensorRT - This flag will enable the TensorRT runtime +module. This will build TVM against the TensorRT libraries. Review comment: Should we use USE_TENSORRT_GRAPH_RUNTIME or just USE_TENSORRT_RUNTIME? GRAPH_RUNTIME might be misleading because TVM has a graph runtime ########## File path: src/runtime/contrib/tensorrt/tensorrt_runtime.cc ########## @@ -0,0 +1,312 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file src/runtime/contrib/tensorrt/tensorrt_runtime.cc + * \brief JSON runtime implementation for TensorRT. + */ + +#include <dmlc/parameter.h> +#include <tvm/runtime/ndarray.h> +#include <tvm/runtime/registry.h> + +#include <fstream> + +#include "../../file_util.h" +#include "../json/json_node.h" +#include "../json/json_runtime.h" + +#ifdef TVM_GRAPH_RUNTIME_TENSORRT +#include "NvInfer.h" +#include "tensorrt_builder.h" +#endif + +namespace tvm { +namespace runtime { +namespace contrib { + +using namespace tvm::runtime::json; + +class TensorRTRuntime : public JSONRuntimeBase { + public: + /*! + * \brief The TensorRT runtime module. Deserialize the provided functions + * on creation and store in the layer cache. + * + * \param symbol_name The name of the function. + * \param graph_json serialized JSON representation of a sub-graph. + * \param const_names The names of each constant in the sub-graph. + */ + explicit TensorRTRuntime(const std::string& symbol_name, const std::string& graph_json, + const Array<String>& const_names) + : JSONRuntimeBase(symbol_name, graph_json, const_names), + use_implicit_batch_(true), + max_workspace_size_(size_t(1) << 30) {} + + /*! + * \brief The type key of the module. + * + * \return module type key. + */ + const char* type_key() const override { return "tensorrt"; } + + /*! + * \brief Initialize runtime. Create TensorRT layer from JSON + * representation. + * + * \param consts The constant params from compiled model. + */ + void Init(const Array<NDArray>& consts) override { + CHECK_EQ(consts.size(), const_idx_.size()) + << "The number of input constants must match the number of required."; + LoadGlobalAttributes(); + if (GetCachedEnginesFromDisk()) return; + SetupConstants(consts); + BuildEngine(); + CacheEngineToDisk(); + } + + void LoadGlobalAttributes() { + // These settings are global to the entire subgraph. Codegen will add them as attributes to all + // op nodes. Read from first one. + for (size_t i = 0; i < nodes_.size(); ++i) { + if (nodes_[i].HasAttr("use_implicit_batch") && nodes_[i].HasAttr("max_workspace_size")) { + use_implicit_batch_ = + std::stoi(nodes_[i].GetAttr<std::vector<std::string>>("use_implicit_batch")[0]); + // Allow max_workspace_size to be overridden at runtime. + size_t runtime_max_workspace_size = + dmlc::GetEnv("TVM_TENSORRT_MAX_WORKSPACE_SIZE", size_t(0)); + if (runtime_max_workspace_size != 0) { + max_workspace_size_ = runtime_max_workspace_size; + } else { + max_workspace_size_ = + std::stoul(nodes_[i].GetAttr<std::vector<std::string>>("max_workspace_size")[0]); + } + return; Review comment: no need to have `return` ########## File path: src/runtime/contrib/tensorrt/tensorrt_runtime.cc ########## @@ -0,0 +1,312 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file src/runtime/contrib/tensorrt/tensorrt_runtime.cc + * \brief JSON runtime implementation for TensorRT. + */ + +#include <dmlc/parameter.h> +#include <tvm/runtime/ndarray.h> +#include <tvm/runtime/registry.h> + +#include <fstream> + +#include "../../file_util.h" +#include "../json/json_node.h" +#include "../json/json_runtime.h" + +#ifdef TVM_GRAPH_RUNTIME_TENSORRT +#include "NvInfer.h" +#include "tensorrt_builder.h" +#endif + +namespace tvm { +namespace runtime { +namespace contrib { + +using namespace tvm::runtime::json; + +class TensorRTRuntime : public JSONRuntimeBase { + public: + /*! + * \brief The TensorRT runtime module. Deserialize the provided functions + * on creation and store in the layer cache. + * + * \param symbol_name The name of the function. + * \param graph_json serialized JSON representation of a sub-graph. + * \param const_names The names of each constant in the sub-graph. + */ + explicit TensorRTRuntime(const std::string& symbol_name, const std::string& graph_json, + const Array<String>& const_names) + : JSONRuntimeBase(symbol_name, graph_json, const_names), + use_implicit_batch_(true), + max_workspace_size_(size_t(1) << 30) {} + + /*! + * \brief The type key of the module. + * + * \return module type key. + */ + const char* type_key() const override { return "tensorrt"; } + + /*! + * \brief Initialize runtime. Create TensorRT layer from JSON + * representation. + * + * \param consts The constant params from compiled model. + */ + void Init(const Array<NDArray>& consts) override { + CHECK_EQ(consts.size(), const_idx_.size()) + << "The number of input constants must match the number of required."; + LoadGlobalAttributes(); + if (GetCachedEnginesFromDisk()) return; + SetupConstants(consts); + BuildEngine(); + CacheEngineToDisk(); + } + + void LoadGlobalAttributes() { + // These settings are global to the entire subgraph. Codegen will add them as attributes to all + // op nodes. Read from first one. + for (size_t i = 0; i < nodes_.size(); ++i) { + if (nodes_[i].HasAttr("use_implicit_batch") && nodes_[i].HasAttr("max_workspace_size")) { + use_implicit_batch_ = + std::stoi(nodes_[i].GetAttr<std::vector<std::string>>("use_implicit_batch")[0]); + // Allow max_workspace_size to be overridden at runtime. + size_t runtime_max_workspace_size = + dmlc::GetEnv("TVM_TENSORRT_MAX_WORKSPACE_SIZE", size_t(0)); + if (runtime_max_workspace_size != 0) { + max_workspace_size_ = runtime_max_workspace_size; + } else { + max_workspace_size_ = + std::stoul(nodes_[i].GetAttr<std::vector<std::string>>("max_workspace_size")[0]); + } + return; + } + } + } + +#ifdef TVM_GRAPH_RUNTIME_TENSORRT + /*! \brief Run inference using built engine. */ + void Run() override { + auto& engine_and_context = trt_engine_cache_.at(symbol_name_); + auto engine = engine_and_context.engine; + auto context = engine_and_context.context; + std::vector<void*> bindings(engine->getNbBindings(), nullptr); + + for (size_t i = 0; i < input_nodes_.size(); ++i) { + auto nid = input_nodes_[i]; + if (nodes_[nid].GetOpType() == "input") { + for (size_t j = 0; j < nodes_[nid].GetOpShape().size(); ++j) { + uint32_t eid = EntryID(nid, j); + const std::string name = nodes_[nid].GetOpName() + "_" + std::to_string(j); + int binding_index = engine->getBindingIndex(name.c_str()); + CHECK_NE(binding_index, -1); + bindings[binding_index] = data_entry_[eid]->data; + } + } + } + + for (size_t i = 0; i < outputs_.size(); ++i) { + uint32_t eid = EntryID(outputs_[i]); + const std::string& name = engine_and_context.outputs[i]; + int binding_index = engine->getBindingIndex(name.c_str()); + CHECK_NE(binding_index, -1); + bindings[binding_index] = data_entry_[eid]->data; + } + +#if TRT_VERSION_GE(6, 0, 1) + if (use_implicit_batch_) { + CHECK(context->execute(batch_size_, bindings.data())) << "Running TensorRT failed."; + } else { + CHECK(context->executeV2(bindings.data())) << "Running TensorRT failed."; + } +#else + CHECK(context->execute(batch_size_, bindings.data())) << "Running TensorRT failed."; +#endif + } + + private: + /*! + * \brief Build TensorRT engine from JSON representation. + */ + void BuildEngine() { + DLOG(INFO) << "Building new TensorRT engine for subgraph " << symbol_name_; + const bool use_fp16 = dmlc::GetEnv("TVM_TENSORRT_USE_FP16", false); + batch_size_ = GetBatchSize(); + TensorRTBuilder builder(&logger_, max_workspace_size_, use_implicit_batch_, use_fp16, + batch_size_); + + // Add inputs and constants. + for (size_t i = 0; i < input_nodes_.size(); ++i) { + auto nid = input_nodes_[i]; + const auto& node = nodes_[nid]; + std::string name = node.GetOpName(); + if (node.GetOpType() == "input") { + builder.AddInput(nid, node); + } else { + CHECK_EQ(node.GetOpType(), "const"); + uint32_t eid = EntryID(nid, 0); + builder.AddConstant(nid, data_entry_[eid]); + } + } + + // Add layers. + for (size_t nid = 0; nid < nodes_.size(); ++nid) { + const auto& node = nodes_[nid]; + if (node.GetOpType() != "kernel") continue; + builder.AddLayer(nid, node); + } + + // Add outputs. + for (size_t i = 0; i < outputs_.size(); ++i) { + builder.AddOutput(outputs_[i]); + } + + // Build engine. + trt_engine_cache_[symbol_name_] = builder.BuildEngine(); + DLOG(INFO) << "Finished building TensorRT engine for subgraph " << symbol_name_; + } + + /*! \brief If TVM_TENSORRT_CACHE_DIR is set, will check that directory for + * already built TRT engines and load into trt_engine_cache_ so they don't + * have to be built at first inference. + */ + bool GetCachedEnginesFromDisk() { + std::string cache_dir = dmlc::GetEnv("TVM_TENSORRT_CACHE_DIR", std::string("")); + if (cache_dir.empty()) return false; + std::string key = GetSubgraphKey(); + std::string path = cache_dir + "/" + key + ".plan"; + // Check if engine is in the cache. + std::ifstream infile(path, std::ios::binary); + if (!infile.good()) return false; + DLOG(INFO) << "Loading cached TensorRT engine from " << path; + infile.close(); + std::string serialized_engine; + LoadBinaryFromFile(path, &serialized_engine); + // Deserialize engine + nvinfer1::IRuntime* runtime = nvinfer1::createInferRuntime(logger_); + TensorRTEngineAndContext engine_and_context; + engine_and_context.engine = + runtime->deserializeCudaEngine(&serialized_engine[0], serialized_engine.size(), nullptr); + engine_and_context.context = engine_and_context.engine->createExecutionContext(); + // Load metadata + std::string meta_path = cache_dir + "/" + key + ".meta"; + std::string serialized_meta; + LoadBinaryFromFile(meta_path, &serialized_meta); + std::istringstream is(serialized_meta); + dmlc::JSONReader reader(&is); + dmlc::JSONObjectReadHelper helper; + helper.DeclareField("inputs", &engine_and_context.inputs); + helper.DeclareField("outputs", &engine_and_context.outputs); + helper.ReadAllFields(&reader); + trt_engine_cache_[symbol_name_] = engine_and_context; + return true; + } + + /*! \brief If TVM_TENSORRT_CACHE_DIR is set, will save the engine to that + * directory so it can be loaded later. + */ + void CacheEngineToDisk() { + std::string cache_dir = dmlc::GetEnv("TVM_TENSORRT_CACHE_DIR", std::string("")); + if (cache_dir.empty()) return; + std::string key = GetSubgraphKey(); + std::string path = cache_dir + "/" + key + ".plan"; + DLOG(INFO) << "Caching TensorRT engine to " << path; + // Serialize engine to disk + nvinfer1::IHostMemory* serialized_engine = trt_engine_cache_[symbol_name_].engine->serialize(); + SaveBinaryToFile(path, std::string(static_cast<const char*>(serialized_engine->data()), + serialized_engine->size())); + serialized_engine->destroy(); + // Serialize metadata + std::ostringstream os; + dmlc::JSONWriter writer(&os); + writer.BeginObject(); + writer.WriteObjectKeyValue("inputs", trt_engine_cache_[symbol_name_].inputs); + writer.WriteObjectKeyValue("outputs", trt_engine_cache_[symbol_name_].outputs); + writer.EndObject(); + std::string meta_path = cache_dir + "/" + key + ".meta"; + SaveBinaryToFile(meta_path, os.str()); + } + + std::string GetSubgraphKey() { + // Using this key will only allow a single model per TVM_TENSORRT_CACHE_DIR directory. We could + // use instead use a hash of graph_json and all weights to allow many models in the same Review comment: ```suggestion // instead use a hash of graph_json and all weights to allow many models in the same ``` ########## File path: src/runtime/contrib/tensorrt/tensorrt_utils.h ########## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file runtime/contrib/tensorrt/utils.h + * \brief Helper functions used by TensorRTBuilder or TrtOpConverters. + */ + +#ifndef TVM_RUNTIME_CONTRIB_TENSORRT_TENSORRT_UTILS_H_ +#define TVM_RUNTIME_CONTRIB_TENSORRT_TENSORRT_UTILS_H_ + +#include <string> +#include <vector> + +#include "NvInfer.h" + +// There is a conflict between cpplint and clang-format-10. +// clang-format off +#define TRT_VERSION_GE(major, minor, patch) \ + ((NV_TENSORRT_MAJOR > major) || (NV_TENSORRT_MAJOR == major && NV_TENSORRT_MINOR > minor) || \ + (NV_TENSORRT_MAJOR == major && NV_TENSORRT_MINOR == minor && NV_TENSORRT_PATCH >= patch)) +// clang-format on + +namespace tvm { +namespace runtime { +namespace contrib { + +/*! + * \brief Helper function to convert an vector to TRT Dims. + * \param vec Vector. + * \return TRT Dims. + */ +template <typename T> +inline nvinfer1::Dims VectorToTrtDims(const std::vector<T>& vec) { + nvinfer1::Dims dims; + // Dims(nbDims=0, d[0]=1) is used to represent a scalar in TRT. + dims.d[0] = 1; + dims.nbDims = vec.size(); + for (size_t i = 0; i < vec.size(); ++i) { + dims.d[i] = vec[i]; + } + return dims; +} + +/*! + * \brief Helper function to convert TRT Dims to vector. + * \param vec TRT Dims. + * \return Vector. + */ +inline std::vector<int> TrtDimsToVector(const nvinfer1::Dims& dims) { + return std::vector<int>(dims.d, dims.d + dims.nbDims); +} + +/*! + * \brief Helper function to convert vector to string. + * \param vec Vector. + * \return Vector as a string. + */ +template <typename T> +inline std::string DebugString(const std::vector<T>& vec) { Review comment: maybe we have it, but it seems we don't use it in this PR? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
