tqchen commented on PR #654: URL: https://github.com/apache/tvm-ffi/pull/654#issuecomment-4886156451
Benchmarked the Python test suite locally at exact PR head `e308936185d8accc4ba0cf4465aba2c807bd26ae`. Environment: AMD EPYC 7413 (24 physical / 48 logical CPUs), one NVIDIA H200, driver 595.58.03, Python 3.12.12, pytest 9.1.1, pytest-xdist 3.8.0, and torch 2.12.1+cu130. The configured `-n auto` resolved to **48 workers** (`created: 48/48 workers`). Compared the same checkout and environment: - Serial: `pytest -n 0 tests/python` - Configured parallel: `pytest tests/python` (uses the PR's `addopts = ["-n", "auto"]`) Every reported run completed with the same result: **2333 passed, 7 skipped, 2 xfailed** (2342 items). The seven skips are environment-dependent and identical in both modes: three require CUDA > 13.0, two require Python 3.13+, one requires free-threaded Python, and one requires ROCm. | Cache condition | Repetitions per mode | Serial wall samples | `-n auto` wall samples | Median result | |---|---:|---|---|---| | Native-extension cache warm | 5 | 27.758, 27.591, 26.975, 26.459, 28.811 s | 24.391, 25.917, 24.292, 25.806, 24.497 s | 27.591 -> 24.497 s: **1.126x**, **11.2% lower wall time** | | Fresh native-extension cache per invocation | 3 | 97.253, 96.633, 96.993 s | 39.883, 43.168, 39.811 s | 96.993 -> 39.883 s: **2.432x**, **58.9% lower wall time** | Method and caveats: - Wall time is external shell time, so it includes collection and xdist worker startup. - The warm-cache comparison discarded one warm-up per mode, then used balanced alternating order (AB/BA) for five measurements each. - Each cold-cache invocation received fresh, disjoint `TVM_FFI_CACHE_DIR` and `TORCH_EXTENSIONS_DIR` paths, so neither mode warmed the other. Installed dependencies and OS page caches remained warm; "cold" here specifically means the content-addressed native-extension cache, not a completely cold machine. - The 48-worker run trades resources for latency. Median aggregate user CPU time increased from 64.654 s to 490.378 s (7.58x) in the warm-cache comparison and from 109.493 s to 538.752 s (4.92x) in the cold-cache comparison. - These are single-node results. The exact gain will vary with worker count, CPU allocation, dependency versions, and cache state. On this host, parallel execution has a modest steady-state benefit and a large clean extension-build benefit, while retaining identical test outcomes across all measured runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
