echuraev commented on PR #16061: URL: https://github.com/apache/tvm/pull/16061#issuecomment-1792203999
Here is an example how Nvidia tools can be used to analyze kernels in the models, compiled with TVM. I took the code which was used in [autotvm x86 tutorial](https://tvm.apache.org/docs/tutorial/autotvm_relay_x86.html) for running `ResNet50-v2` model and modified the target to run on `cuda`. After that, I added the option `cuda.kernels_output_dir` to `PassContext` during model compilation: ```python with tvm.transform.PassContext(opt_level=3, config={"cuda.kernels_output_dir": "___tmp_cuda_dumps"}): lib = relay.build(mod, target=target, params=params) ``` After running the script, directory `___tmp_cuda_dumps` will be created and the file with cuda kernels will be stored in this directory. If we want to profile our model, first we can use Nvidia Nsight Systems. Run our model with Nsight Systems profiler: ```bash nsys profile python3 model_run.py ``` After executing this command, a new file `report1.nsys-rep` will be created in a working directory. You can open this report by using Nvidia Nsight Systems. On the screenshot you can see an example of the main window after opening the report:  You can zoom in the window with trace and expand row with `CUDA HW (...)`. Then you can see the sequence of cuda kernels which were executed. After high-level analysis, you might want to analyze a concrete kernel. In such case you can select this kernel and click `Analyze the Selected Kernel with NVIDIA Nsight Compute`:  After that you can start GUI interface of installed NVIDIA Nsight Compute or display a command line for Nsight Compute CLI. I prefer GUI interface because it gives you more tools for analysis.  After selecting GUI interface, an instance of Nsight Compute will be opened. You should click on the `Connect` button and on the opened window it will be necessary to configure your profiling session: 1. Configure your connection to the target machine in case of remote profiling 2. Specify environment variables. Usually, I add the following env variables: ``` PATH=<path_to_anaconda>/envs/<env_name>/bin:/usr/local/cuda/bin:/usr/bin; PYTHONPATH=<path_to_tvm>/python; LD_LIBRARY_PATH=<path_to_tvm>/build ``` Path to anaconda is necessary olny in case if you use specific python environment. All variables should be separated by `;`. 3. In profile configuration part, on the tab `Other` set value of parameter `Import Source` equals to `Yes`. 4. After that you should be able to launch kernel profiling.  When the profiling was finished, you can see the detailed overview of your kernel. This page provides a lot of information about your kernel and hardware utilization:  After switching to the `Source` page, you can see the source code of the kernels and SASS instructions:  You can see on the screenshot that code or our kernel is on line 1329. It is happened because TVM dumps kernels for whole network into a single file and file with sources contains all kernels from the model. But as you can see, near the scroll bar there is a colorized area. This is the area where the executed code is located. So we can easy find necessary kernel in this file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
