bryancall commented on PR #13545:
URL: https://github.com/apache/trafficserver/pull/13545#issuecomment-5344363665
I benchmarked this migration on dedicated hardware rather than reasoning
about it, because a harness swap of this size deserves numbers. Summary up
front: **Uranium is 7.4x faster at 32-way parallelism, loses no code coverage,
uses less memory, and has fewer flaky tests.** Details and methodology below,
including the parts that do not favour the new harness.
## Methodology
Two hosts, both 32 hardware threads, 30 GB RAM, Fedora, gcc 16.1.1, Proxy
Verifier v3.1.3 on both sides (verified identical, checksum `342286244d...`).
Timing on one host, coverage on the other, and each host ran **both** harnesses
so every comparison is within-host.
- master at `a2ea029215`, this branch at `7fcbc9d69`
- Python pinned identically for every run, dependencies resolved from the
checked-in `tests/uv.lock`
- Timing used Release builds; coverage used separate gcov builds, since
`-O0` plus instrumentation distorts wall clock by about 1.5x
- Coverage counters were reset before each run and **only the integration
suite** was measured. Unit tests are identical on both sides and including them
would dilute exactly the delta in question.
**The proxy binary is the same on both sides.** I checked this rather than
assuming it, because the whole comparison depends on it. After normalizing
embedded build paths, both binaries disassemble to **1,341,244 instructions**
and the instruction streams are identical once addresses are masked. The
residual byte differences are relocation displacements from embedded path and
timestamp strings of different lengths, spread across 1,265 unrelated symbols
in single-byte runs. So every number below is harness overhead, not a proxy
change.
## Performance
Full suite, same host, same binary:
| Workers | AuTest | Uranium | Speedup |
|---|---|---|---|
| 4 | 3230s | 914s | 3.5x |
| 8 | 2254s | 497s | 4.5x |
| 16 | 1749s | 293s | 6.0x |
| 32 | 1591s, 1596s | 215s, 215s, 202s | **7.4x** |
The gap widens with worker count, which points at the mechanism:
| | AuTest | Uranium |
|---|---|---|
| Total work | 4873s of worker time | 3899s of test time |
| Wall clock | 1031s | 215s |
| **Effective parallelism** | **4.7x** | **18.1x** |
AuTest's worker durations at `-j32` were min 113s, median 155s, **max
889s**. One worker set the wall clock while roughly 30 sat idle behind it.
**The cause is that AuTest's load balancing never engages.**
`autest-parallel.py` has an LPT balancer that reads
`<sandbox>/test-timings.json`, but that file is never written: the save at line
1135 is gated on `tests_timed > 0` and the per-worker timing dictionaries come
back empty. Every run logs `Using round-robin partitioning` verbatim, and I
confirmed no timings file exists after seven runs. This is not a cold-start
artifact, it is the steady state. Worth knowing regardless of this PR's
outcome, since it means the current suite is leaving most of the machine idle.
Uranium's per-test distribution for reference: p50 1.13s, p95 9.17s, max
90.6s.
## Code coverage
gcov plus gcovr 8.6, identical flags and exclusions on both sides,
integration suite only:
| | AuTest | Uranium | Delta |
|---|---|---|---|
| Lines | 51.0% (96,576 / 189,482) | **51.1%** (96,784 / 189,482) | +208 |
| Functions | 60.7% (11,376 / 18,726) | **60.8%** (11,392 / 18,726) | +16 |
| Branches | 25.6% (77,497 / 302,664) | 25.6% (77,583 / 302,664) | +86 |
**No coverage is lost.** Uranium is fractionally ahead on all three
measures. This is the number that matters most, since a 7.4x speedup naturally
raises the question of whether the suite is simply doing less. It is not. The
denominators are identical on both sides, which independently corroborates the
binary-equivalence check above.
## Memory
Peak resident memory across the whole process tree:
| Workers | AuTest | Uranium |
|---|---|---|
| 4 | 3.78 GB | 3.66 GB |
| 8 | 5.48 GB | 4.52 GB |
| 16 | 7.47 GB | 5.69 GB |
| 32 | 11.0 to 12.26 GB | 9.64 to 10.03 GB |
Uranium uses less memory at every level, with the gap widening to about 31%
at 16 workers. Neither harness came close to exhausting 30 GB.
Caveat worth stating: these are summed RSS across all processes, which
double counts pages shared between the many concurrent Traffic Server
instances, so treat them as upper bounds. The idle baseline was 0.97 GB, so
roughly 92% of each figure is genuine workload. I am re-measuring with
proportional set size to remove the double counting and will follow up if it
changes the picture.
## Flakiness
Seven AuTest runs and six Uranium runs, across all parallelism levels:
| Harness | Test | Failed in | Verdict |
|---|---|---|---|
| AuTest | `per_client_connection_max` | 7/7 | deterministic |
| AuTest | `cripts` | 7/7 | deterministic |
| AuTest | `h2_malformed_request_logging` | 2/7 | **flaky** |
| AuTest | `session_id` | 1/7 | **flaky** |
| Uranium | `test_all_bespoke_tests_are_available_to_pytest` | 6/6 |
deterministic |
| Uranium | `log_mstsms` | 2/6 | **flaky** |
Two observations that only a repeated run surfaces.
**A flaky test was distorting the timing.** AuTest at `-j32` came in
bimodal: 1031s, 1039s, then 1591s, 1596s. The fast runs are the ones where
`h2_malformed_request_logging` **failed**; the slow ones are where it
**passed**. Its success path costs roughly 550 seconds more than its failure
path. A single measurement would have reported a figure that flattered AuTest
by 35%.
**AuTest does not run a deterministic set of tests.** Executed totals across
runs were 586, 586, 586, 586, 586, 583, 581. At `-j8` three tests silently
disappear and at `-j4` five do, with no diagnostic. Uranium collected exactly
**1207 items in all six runs** with no drift. For a suite whose purpose is
regression detection, that difference matters as much as the speed.
## Test inventory
I mapped all 564 master `*.test.py` files against the branch. 388 matched by
directory and stem; I resolved the remaining 176 individually by reading each
successor rather than inferring from names.
**Exactly one test has no successor:
`tests/gold_tests/cache/cache-write-lock-contention.test.py`.** Severity is low
since it was already `SkipUnless(RUN_CACHE_CONTENTION_TEST=1)`, so CI signal is
unchanged, but the scenario is gone and the gate variable is now dead plumbing:
`tests/tools/uranium/runner.py` still forwards `RUN_CACHE_CONTENTION_TEST` into
the container and nothing reads it. Either restore the scenario or drop the
plumbing.
Everything else is verified consolidation, and the counts hold up: the 18
`tls_hooks` files became a 17-entry parametrization plus one function, the 7
`cont_schedule` files became a 7-entry parametrization, 38 `txn_box` tests
became 35 manifests with three merged pairs, and `cache` grew from 39 to 70. I
also confirmed the master merge converted the two tests #13547 added after my
baseline rather than dropping them.
## Other findings
- **`ci/coverage` cannot collect coverage as written.** It calls the
build-tree `./urtest.sh -n "$NPROCS"` without `--no-run-in-docker`. Since
`choose_docker_mode()` returns `not is_container()`, on any non-container host
that shells out to Docker and discards the gcov-instrumented build the script
just made. I had to bypass this script to get the coverage numbers above.
- **`ci/regression` and `ci/jenkins/bin/autest.sh` no longer exercise the
tree they build.** Source-mode `urtest.sh` always runs its own `cmake --preset
urtest` into `build-urtest-container`, so `$DSTROOT` and `${INSTALL}` are
ignored and the build above the test call is dead weight.
- **`tests/uv.lock` never reaches the build tree.** `tests/CMakeLists.txt`
copies only `pyproject.toml`, but `runner.py` runs `uv --project
<build>/tests`, so the vetted lock is unused and every first run resolves fresh
against PyPI. This bit me concretely: without an explicit pin, `uv` selected
**Python 3.12.12** on a host whose system interpreter is 3.14. Copying
`uv.lock` alongside `pyproject.toml` is a one-line fix and makes runs
reproducible.
- **`test_all_bespoke_tests_are_available_to_pytest` fails
deterministically**, in all six runs. A framework self-check reporting that the
declared inventory does not match what pytest collects is worth resolving
before this lands, since it is the test that would otherwise catch a conversion
gap.
- **The documentation contradicts the code on Docker detection.**
`uranium-tests.en.rst` describes a two-condition rule (container **and** Fedora
44); `choose_docker_mode()` checks only `is_container()`.
- **`if(DEFINED ENABLE_AUTEST)` fires even for `-DENABLE_AUTEST=OFF`**,
which will surprise anyone carrying that flag in a script. The companion guard
for `AUTEST_SANDBOX` / `AUTEST_OPTIONS` / `PYTEST_OPTIONS` sits inside
`if(ENABLE_URTEST)`, so it misses the common case.
- **Replay manifests cannot be marked serial.** `_is_serial_test` matches
only `.py` paths and `ReplayItem.runtest()` always takes the shared lock, so a
`.test.yaml` listed in `serial_tests.txt` would be silently ignored. Latent
today since the file lists only `.py` entries, but it is a trap for whoever
first needs an exclusive replay.
- **Sharding moved into the repository, which the PR body understates.**
`SHARD` and `SHARDCNT` appear nowhere on master; the 1of4 through 4of4 split is
external Jenkins configuration. This PR implements sharding in-repo as a modulo
stripe over sorted node IDs, so any existing per-shard intuition about which
tests land where is void, and `ci/jenkins/bin/autest.sh` now passes no shard
flags at all. Worth confirming the Jenkins job definitions move in lockstep.
## Overall
The performance and determinism case is strong and I would not have
predicted the margin. The two things I would want resolved before this leaves
draft are the `ci/coverage` Docker bug, since it silently disables coverage
collection, and the deterministic
`test_all_bespoke_tests_are_available_to_pytest` failure. The single lost test
and the `uv.lock` plumbing are small and easy.
Happy to share the raw logs, per-test timing data, or the gcovr HTML reports
for either side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
