gemini-code-assist[bot] commented on code in PR #19606:
URL: https://github.com/apache/tvm/pull/19606#discussion_r3299892977
##########
docs/how_to/tutorials/cross_compilation_and_rpc.py:
##########
@@ -201,6 +200,194 @@
cost = time_f(a, b).mean
print(f"{cost:g} secs/op")
+######################################################################
+# Scale RPC to Shared Devices
+# ---------------------------
+#
+# The direct RPC server used above is the simplest way to run on one remote
+# device. In shared environments, the same compile/upload/run flow is usually
+# kept, but the connection is managed by an RPC tracker and, when needed, an
+# RPC proxy.
+#
+# This setup is useful when:
+#
+# - multiple users or CI jobs share a small number of boards,
+# - devices are registered by key rather than by fixed IP address,
+# - the host cannot directly reach the device because of the network layout, or
+# - the target device only has the minimal runtime stack needed for execution.
+#
+# The pieces fit together as follows:
+#
+# - **RPC server**: runs on the target device and executes uploaded modules.
+# - **RPC tracker**: runs on a host and assigns matching RPC servers to
clients.
+# - **RPC proxy**: forwards traffic when the client cannot connect directly to
+# the RPC server.
+#
+# .. figure::
https://raw.githubusercontent.com/tlc-pack/web-data/main/images/dev/how-to/rpc_system_suggested_arch.svg
+# :align: center
+# :width: 85%
+#
+# In the figure above, machine A connects through the tracker. Machine B runs
+# an RPC proxy because machines C and D are not directly reachable from A. The
+# tracker keeps a queue per RPC key. If a matching server is available, it is
+# assigned to the client; otherwise, the request waits in that key's queue.
+#
+# Start the Tracker and Proxy
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# The tracker and proxy generally run on a host machine, not on the target
+# device. They do not require target-specific drivers.
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_tracker --host RPC_TRACKER_IP --port 9190
--port-end 9191
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_proxy \
+# --host RPC_PROXY_IP \
+# --port 9090 \
+# --port-end 9091 \
+# --tracker RPC_TRACKER_IP:RPC_TRACKER_PORT
+#
+# Replace the host names, ports, and port ranges for your environment. The
+# ``--port-end`` option is useful in CI because it prevents the service from
+# silently choosing an unexpected port.
+#
+# Package a Minimal RPC Runtime
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# If the target can build TVM directly, install TVM on the target and launch
+# the RPC server there. Otherwise, cross-compile the TVM runtime and package
+# it with the Python RPC server.
+#
+# A typical CMake toolchain file for 64-bit ARM Linux looks like this:
+#
+# .. code-block:: cmake
+#
+# set(CMAKE_SYSTEM_NAME Linux)
+# set(root_dir "/XXX/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu")
+#
+# set(CMAKE_C_COMPILER "${root_dir}/bin/aarch64-linux-gnu-gcc")
+# set(CMAKE_CXX_COMPILER "${root_dir}/bin/aarch64-linux-gnu-g++")
+# set(CMAKE_SYSROOT "${root_dir}/aarch64-linux-gnu/libc")
+#
+# set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
+# set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
+#
+# Build the runtime from the TVM repository root. Enable target-specific
+# options such as ``USE_OPENCL`` or vendor runtime support in ``config.cmake``.
+# Build any target-specific runtime libraries that your deployment needs, such
+# as ``tvm_runtime_opencl`` for OpenCL.
+#
+# .. code-block:: shell
+#
+# mkdir cross_build
+# cd cross_build
+# cp ../cmake/config.cmake ./
+#
+# # Enable other options as needed, e.g. USE_OPENCL or vendor runtimes.
+# sed -i "s|USE_LLVM.*)|USE_LLVM OFF)|" config.cmake
+#
+# cmake -DCMAKE_TOOLCHAIN_FILE=/YYY/aarch64-linux-gnu.cmake
-DCMAKE_BUILD_TYPE=Release ..
+# cmake --build . --target runtime -j
+# # Optional example when USE_OPENCL is enabled:
+# # cmake --build . --target tvm_runtime_opencl -j
+# cd ..
+#
+# Then package the Python RPC server with the cross-compiled runtime and copy
+# it to the device.
+#
+# .. code-block:: shell
+#
+# rm -rf tvm_runtime_package
+# mkdir tvm_runtime_package
+# cp -a python tvm_runtime_package/
+# cp cross_build/lib/libtvm_ffi.so tvm_runtime_package/python/tvm/
+# cp cross_build/lib/libtvm_runtime*.so tvm_runtime_package/python/tvm/
Review Comment:
In TVM, compiled libraries are output directly to the build directory (e.g.,
`cross_build/`), not to a `lib/` subdirectory. Additionally, there is no
`libtvm_ffi.so` library in TVM; the runtime library is named
`libtvm_runtime.so`. We should update these commands to copy
`libtvm_runtime.so` from the correct path.
```suggestion
# cp cross_build/libtvm_runtime.so tvm_runtime_package/python/tvm/
```
##########
docs/how_to/tutorials/cross_compilation_and_rpc.py:
##########
@@ -201,6 +200,194 @@
cost = time_f(a, b).mean
print(f"{cost:g} secs/op")
+######################################################################
+# Scale RPC to Shared Devices
+# ---------------------------
+#
+# The direct RPC server used above is the simplest way to run on one remote
+# device. In shared environments, the same compile/upload/run flow is usually
+# kept, but the connection is managed by an RPC tracker and, when needed, an
+# RPC proxy.
+#
+# This setup is useful when:
+#
+# - multiple users or CI jobs share a small number of boards,
+# - devices are registered by key rather than by fixed IP address,
+# - the host cannot directly reach the device because of the network layout, or
+# - the target device only has the minimal runtime stack needed for execution.
+#
+# The pieces fit together as follows:
+#
+# - **RPC server**: runs on the target device and executes uploaded modules.
+# - **RPC tracker**: runs on a host and assigns matching RPC servers to
clients.
+# - **RPC proxy**: forwards traffic when the client cannot connect directly to
+# the RPC server.
+#
+# .. figure::
https://raw.githubusercontent.com/tlc-pack/web-data/main/images/dev/how-to/rpc_system_suggested_arch.svg
+# :align: center
+# :width: 85%
+#
+# In the figure above, machine A connects through the tracker. Machine B runs
+# an RPC proxy because machines C and D are not directly reachable from A. The
+# tracker keeps a queue per RPC key. If a matching server is available, it is
+# assigned to the client; otherwise, the request waits in that key's queue.
+#
+# Start the Tracker and Proxy
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# The tracker and proxy generally run on a host machine, not on the target
+# device. They do not require target-specific drivers.
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_tracker --host RPC_TRACKER_IP --port 9190
--port-end 9191
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_proxy \
+# --host RPC_PROXY_IP \
+# --port 9090 \
+# --port-end 9091 \
+# --tracker RPC_TRACKER_IP:RPC_TRACKER_PORT
+#
+# Replace the host names, ports, and port ranges for your environment. The
+# ``--port-end`` option is useful in CI because it prevents the service from
+# silently choosing an unexpected port.
+#
+# Package a Minimal RPC Runtime
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# If the target can build TVM directly, install TVM on the target and launch
+# the RPC server there. Otherwise, cross-compile the TVM runtime and package
+# it with the Python RPC server.
+#
+# A typical CMake toolchain file for 64-bit ARM Linux looks like this:
+#
+# .. code-block:: cmake
+#
+# set(CMAKE_SYSTEM_NAME Linux)
+# set(root_dir "/XXX/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu")
+#
+# set(CMAKE_C_COMPILER "${root_dir}/bin/aarch64-linux-gnu-gcc")
+# set(CMAKE_CXX_COMPILER "${root_dir}/bin/aarch64-linux-gnu-g++")
+# set(CMAKE_SYSROOT "${root_dir}/aarch64-linux-gnu/libc")
+#
+# set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
+# set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
+#
+# Build the runtime from the TVM repository root. Enable target-specific
+# options such as ``USE_OPENCL`` or vendor runtime support in ``config.cmake``.
+# Build any target-specific runtime libraries that your deployment needs, such
+# as ``tvm_runtime_opencl`` for OpenCL.
Review Comment:
In TVM, OpenCL support is compiled directly into the main
`libtvm_runtime.so` library when `USE_OPENCL` is enabled in `config.cmake`.
There is no separate `tvm_runtime_opencl` library target. We should update this
text to reflect that OpenCL support is built directly into the main runtime
library.
```suggestion
# OpenCL support is built directly into the main runtime library.
```
##########
docs/how_to/tutorials/cross_compilation_and_rpc.py:
##########
@@ -201,6 +200,194 @@
cost = time_f(a, b).mean
print(f"{cost:g} secs/op")
+######################################################################
+# Scale RPC to Shared Devices
+# ---------------------------
+#
+# The direct RPC server used above is the simplest way to run on one remote
+# device. In shared environments, the same compile/upload/run flow is usually
+# kept, but the connection is managed by an RPC tracker and, when needed, an
+# RPC proxy.
+#
+# This setup is useful when:
+#
+# - multiple users or CI jobs share a small number of boards,
+# - devices are registered by key rather than by fixed IP address,
+# - the host cannot directly reach the device because of the network layout, or
+# - the target device only has the minimal runtime stack needed for execution.
+#
+# The pieces fit together as follows:
+#
+# - **RPC server**: runs on the target device and executes uploaded modules.
+# - **RPC tracker**: runs on a host and assigns matching RPC servers to
clients.
+# - **RPC proxy**: forwards traffic when the client cannot connect directly to
+# the RPC server.
+#
+# .. figure::
https://raw.githubusercontent.com/tlc-pack/web-data/main/images/dev/how-to/rpc_system_suggested_arch.svg
+# :align: center
+# :width: 85%
+#
+# In the figure above, machine A connects through the tracker. Machine B runs
+# an RPC proxy because machines C and D are not directly reachable from A. The
+# tracker keeps a queue per RPC key. If a matching server is available, it is
+# assigned to the client; otherwise, the request waits in that key's queue.
+#
+# Start the Tracker and Proxy
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# The tracker and proxy generally run on a host machine, not on the target
+# device. They do not require target-specific drivers.
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_tracker --host RPC_TRACKER_IP --port 9190
--port-end 9191
+#
+# .. code-block:: shell
+#
+# python3 -m tvm.exec.rpc_proxy \
+# --host RPC_PROXY_IP \
+# --port 9090 \
+# --port-end 9091 \
+# --tracker RPC_TRACKER_IP:RPC_TRACKER_PORT
+#
+# Replace the host names, ports, and port ranges for your environment. The
+# ``--port-end`` option is useful in CI because it prevents the service from
+# silently choosing an unexpected port.
+#
+# Package a Minimal RPC Runtime
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# If the target can build TVM directly, install TVM on the target and launch
+# the RPC server there. Otherwise, cross-compile the TVM runtime and package
+# it with the Python RPC server.
+#
+# A typical CMake toolchain file for 64-bit ARM Linux looks like this:
+#
+# .. code-block:: cmake
+#
+# set(CMAKE_SYSTEM_NAME Linux)
+# set(root_dir "/XXX/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu")
+#
+# set(CMAKE_C_COMPILER "${root_dir}/bin/aarch64-linux-gnu-gcc")
+# set(CMAKE_CXX_COMPILER "${root_dir}/bin/aarch64-linux-gnu-g++")
+# set(CMAKE_SYSROOT "${root_dir}/aarch64-linux-gnu/libc")
+#
+# set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
+# set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
+# set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
+#
+# Build the runtime from the TVM repository root. Enable target-specific
+# options such as ``USE_OPENCL`` or vendor runtime support in ``config.cmake``.
+# Build any target-specific runtime libraries that your deployment needs, such
+# as ``tvm_runtime_opencl`` for OpenCL.
+#
+# .. code-block:: shell
+#
+# mkdir cross_build
+# cd cross_build
+# cp ../cmake/config.cmake ./
+#
+# # Enable other options as needed, e.g. USE_OPENCL or vendor runtimes.
+# sed -i "s|USE_LLVM.*)|USE_LLVM OFF)|" config.cmake
+#
+# cmake -DCMAKE_TOOLCHAIN_FILE=/YYY/aarch64-linux-gnu.cmake
-DCMAKE_BUILD_TYPE=Release ..
+# cmake --build . --target runtime -j
+# # Optional example when USE_OPENCL is enabled:
+# # cmake --build . --target tvm_runtime_opencl -j
Review Comment:
Since OpenCL support is built directly into the main `runtime` target, there
is no separate `tvm_runtime_opencl` target in TVM's CMake configuration. We
should update this comment to clarify that OpenCL support is included in the
main runtime target.
```suggestion
# # OpenCL support is included in the main runtime target.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
