comphead opened a new issue, #12349: URL: https://github.com/apache/arrow-rs/issues/12349
### Is your feature request related to a problem or challenge? While reviewing https://github.com/apache/arrow-rs/pull/12342, two points came up about where benchmark numbers come from: - Apple Silicon is often not a good indicator of cluster performance, because of its memory bandwidth, larger L1 cache line size, and general architecture. Reasonable x86 targets (`x86-64-v3`, `v4`) should be checked as well. See https://github.com/apache/arrow-rs/pull/12342#issuecomment-7020903901. - Running x86 on ARM through Rosetta does not give representative results, and Rosetta is going away in the next macOS. See https://github.com/apache/arrow-rs/pull/12342#issuecomment-7021812802 and the reply below it. Today the only shared way to get Linux numbers is the benchmark bot (https://github.com/adriangb/datafusion-benchmarking). It works well, but it has some costs: - Every iteration on a benchmark (fixing a filter, adding a case, trying a `RUSTFLAGS` variant) is a PR comment and a queued job. In https://github.com/apache/arrow-rs/pull/12342 a filter like `^cast .*decimal` silently skipped the quoted benchmark names, and that was only found after six runs. - It adds load to a runner that is shared with DataFusion and other projects. - Contributors cannot attach a profiler or look at the generated code on the same setup that produced a regression. In https://github.com/apache/arrow-rs/pull/12342 an x86-64-v3 regression could not be profiled locally. Many contributors already have access to a Linux host (an x86 or Graviton cloud VM, a Linux workstation, or a CI box), but they have to recreate the bot setup by hand, and the result is not comparable to what the bot reports. ### Describe the solution you'd like A small, documented Docker sandbox in the repository (for example `dev/bench/Dockerfile` and `dev/bench/run.sh`) that runs the same comparison the bot runs, for a PR number given by the user, and writes the results to the host. Inputs, as environment variables: - `PR`: the PR number. The script fetches `refs/pull/$PR/head` from `apache/arrow-rs`. - `BASELINE`: optional ref. Defaults to the merge-base of the PR with `main`, as the bot does. - `BENCH`: the criterion bench target, for example `cast_kernels`. - `BENCH_FILTER`: optional criterion filter. - `RUSTFLAGS`: optional, for example `-C target-cpu=x86-64-v3` or the alignment flags the bot uses. Steps inside the container, mirroring `controller/src/runner/bench_arrow.rs` in the bot repository: 1. Clone the baseline and the PR head, run `git submodule update --init`. 2. Build both with `cargo bench --features=arrow,async,test_common,experimental,object_store --bench $BENCH --no-run`. 3. Run the baseline with `-- --save-baseline base` and the PR with `-- --save-baseline changed`. 4. Compare with `critcmp base changed`. The bot uses `critcmp` 0.1.8, so the output table has the same format as the bot comments. Getting results to the host: a bind mount of an output directory, which works the same with Docker, Podman, and Colima. The script writes: - `critcmp.txt`: the comparison table, ready to paste into a PR. - `critcmp.json`: `critcmp --export` output for both baselines, so runs can be compared again later or across machines. - `criterion/`: the criterion HTML reports. - `env.txt`: both SHAs, `rustc -vV`, `RUSTFLAGS`, `uname -a`, and `lscpu`, so a result can be traced to its setup. Named volumes for the cargo registry and the `target` directory keep rebuilds fast. Example: ```bash docker build -t arrow-bench dev/bench docker run --rm \ -v "$PWD/bench-out:/out" \ -v arrow-bench-cargo:/usr/local/cargo/registry \ -v arrow-bench-target:/work/target \ -e PR=12342 -e BENCH=cast_kernels -e BENCH_FILTER=decimal \ -e RUSTFLAGS="-C target-cpu=x86-64-v3" \ arrow-bench ``` An optional `PROFILE=1` mode could install `perf` and write `perf record` output for one filtered benchmark to `/out`, which addresses the "cannot profile x86 locally" case on a Linux host. This needs `--cap-add PERFMON` or `--privileged`. Documentation in `CONTRIBUTING.md` under "Performance Improvements" should be explicit about what the numbers mean: - On an x86 or ARM Linux host, results are close to the bot setup and useful for real comparisons. - On Apple Silicon, Docker runs a native `linux/arm64` VM. That gives Linux, glibc, and the bot build flags, but still Apple CPU caches and memory bandwidth. It is useful for iterating on benchmark code and filters, not for final numbers. - `--platform linux/amd64` on Apple Silicon runs under emulation and is not supported for measurements. - The bot stays the source of truth for final performance evaluation in PRs. ### Describe alternatives you've considered - Keep using only the bot. This works, but has the iteration and load costs above. - Run `cargo bench` on macOS directly. This is fine for quick checks, but it is the setup the review comments above warn against. - Export results with `docker buildx build --output type=local,dest=./out`. This runs the benchmark during the image build, where layer caching can skip a rerun and CPU pinning options are not available. A bind mount on `docker run` is simpler. - Copy results out with `docker cp` from a stopped container. This works, but is an extra step and is easy to forget with `--rm`. - Reuse the bot runner image directly. It is built around the controller, GitHub comment posting, and sharding, so it is heavier than a local tool needs. Sharing the `critcmp` version and the bench flags with it keeps the outputs comparable. ### Additional context - Benchmark bot runner: https://github.com/adriangb/datafusion-benchmarking/blob/main/controller/src/runner/bench_arrow.rs - Bot runner Dockerfile (`rust:1-bookworm`, `critcmp` 0.1.8): https://github.com/adriangb/datafusion-benchmarking/blob/main/runner/Dockerfile - Discussion: https://github.com/apache/arrow-rs/pull/12342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
