guest2180 opened a new issue, #20040: URL: https://github.com/apache/tvm/issues/20040
### Problem Relax TensorRT BYOC builds the TensorRT engine during the first runtime initialization or inference. For YOLO-style models, I observed a cold-start delay of approximately 20–80 seconds, depending on the model complexity and the number of BYOC subgraphs. The model currently produces a single TensorRT subgraph, but other models may be partitioned into multiple subgraphs. This is consistent with TensorRT documentation, which states that engine building can take significant time and is normally treated as an offline step: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html#the-build-phase `TVM_TENSORRT_CACHE_DIR` avoids rebuilding on subsequent runs, but it is not very convenient for deployment: - The first process still pays the complete build cost. - A writable persistent cache directory is required. - `.plan` and `.meta` files must be managed separately. - Cache configuration is process-wide rather than model-local. For comparison, I tested the same YOLO model with TVM native CUDA. Compilation took about 24.2 seconds and export took 3.0 seconds, but these costs occurred offline. In a new process, loading the `.so` took about 319 ms and VM initialization took about 143 ms. There was no additional runtime build delay comparable to TensorRT BYOC. Is there a supported way to perform the expensive TensorRT engine-building work during compilation/export instead of the first runtime initialization? TensorRT plans are device- and version-dependent, but even a target-machine export workflow would be easier to manage than an implicit build during application startup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
