mbutrovich commented on code in PR #12351: URL: https://github.com/apache/arrow-rs/pull/12351#discussion_r4645649142
########## dev/bench/README.md: ########## @@ -0,0 +1,53 @@ +<!--- + SPDX-License-Identifier: Apache-2.0 + SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership + SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Benchmark a pull request on Linux + +The `Dockerfile` in this directory benchmarks a pull request against `main`, +the same way as the [benchmark bot], and exports the results to the host. It +uses the same cargo features, thin LTO, and `critcmp` version as the bot, so +the table has the same format as the bot comments. + +[benchmark bot]: https://github.com/adriangb/datafusion-benchmarking + +## Usage + +```bash +docker build dev/bench --progress=plain \ + --build-arg PR=12342 \ + --build-arg BENCH=cast_kernels \ + --build-arg BENCH_FILTER=decimal \ + --output ~/arrow-bench-out/12342 +``` + +- `PR`: pull request number. +- `BENCH`: criterion bench target. +- `BENCH_FILTER`: criterion filter regex. Optional. + +The output directory contains: + +- `critcmp.txt`: the comparison table. `base` is `main` and `changed` is the + pull request. +- `base.json` and `changed.json`: `critcmp --export` output. Compare them + later, or with other runs, with `critcmp base.json changed.json`. +- `criterion/`: the raw criterion data. +- `env.txt`: commits, toolchain, kernel, and CPU. + +Docker caches the build. A later run fetches new commits of `main` and the pull +request and only rebuilds what changed. If nothing changed, Docker returns the +cached results. Add `--no-cache-filter bench` to measure again. +`docker builder prune` removes the cache. + +## Interpreting results + +- The baseline is the latest commit of `main`, while the bot uses the + merge-base. Rebase the pull request to compare the same thing. +- On an x86-64 or arm64 Linux host, the results are comparable to the bot. +- On Apple Silicon, Docker runs a native `linux/arm64` VM. The CPU caches and + memory bandwidth are still those of the Mac, so use it to iterate on + benchmarks and filters, not for final numbers. `--platform linux/amd64` runs + under emulation and is not useful for measurements. Review Comment: The run in the PR description shows that the VM doesn't predict the bot. `cast decimal256 to decimal128 512` is 2.01x in the VM and 1.36x on the bot's arm64 runner. `cast decimal256 to decimal256 512` is 1.43x in the VM and 1.57x on the bot, so the two cases swap order. The A/A run also had one case off by 1.35x. Docker on a Mac runs Linux on the Mac's cores, so the core, caches, and memory bandwidth are still the Mac's. Only libc changes. To iterate on benchmarks and filters, native `cargo bench --save-baseline` runs on the same cores without the VM. What a Mac can check against the bot's CPUs is codegen. `cargo asm --target x86_64-unknown-linux-gnu` (or `aarch64-unknown-linux-gnu`) shows the code the bot runs. #12342 turned on that kind of difference, a branch versus a select. ########## dev/bench/README.md: ########## @@ -0,0 +1,53 @@ +<!--- + SPDX-License-Identifier: Apache-2.0 + SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership + SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Benchmark a pull request on Linux + +The `Dockerfile` in this directory benchmarks a pull request against `main`, +the same way as the [benchmark bot], and exports the results to the host. It +uses the same cargo features, thin LTO, and `critcmp` version as the bot, so +the table has the same format as the bot comments. Review Comment: On a Linux host, what does this give a contributor that the `--save-baseline` workflow in `CONTRIBUTING.md` doesn't? Beyond that workflow, it adds the checkouts, the bot's feature list and thin LTO, and a target directory per side. It also doesn't run "the same way as the benchmark bot" in two ways that matter for reviews: - The baseline is the tip of `main`, while the bot uses the merge-base ([`bench_arrow.rs`](https://github.com/adriangb/datafusion-benchmarking/blob/60a1e0e024fecd0e49a50b5b9d1ce574c5be59e3/controller/src/runner/bench_arrow.rs#L76-L80)). - It has no way to set `RUSTFLAGS`, which is how reviewers ask the bot for x86-64-v3 and aligned-build runs ([example](https://github.com/apache/arrow-rs/pull/12289#issuecomment-6936751501)). The bot's runner image already starts from the same `rust:1-bookworm` and `critcmp` 0.1.8 ([`runner/Dockerfile`](https://github.com/adriangb/datafusion-benchmarking/blob/60a1e0e024fecd0e49a50b5b9d1ce574c5be59e3/runner/Dockerfile#L1-L6)). If a local runner is worth having, should it be a local mode in datafusion-benchmarking? That way there's one implementation instead of two that drift apart. ########## dev/bench/Dockerfile: ########## @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements + +# Benchmarks a pull request against main, the same way as the benchmark bot, +# and exports the results. See README.md in this directory. + +FROM rust:1-bookworm AS critcmp +# The bot uses critcmp 0.1.8, so the table has the same format. +RUN cargo install --locked --version 0.1.8 critcmp + +FROM rust:1-bookworm AS bench +ARG PR +ARG BENCH +ARG BENCH_FILTER= +COPY --from=critcmp /usr/local/cargo/bin/critcmp /usr/local/bin/ +# Docker fetches these again when main or the pull request has new commits. +ADD --keep-git-dir=true https://github.com/apache/arrow-rs.git#main /arrow-rs/main +ADD --keep-git-dir=true https://github.com/apache/arrow-rs.git#pull/${PR}/head /arrow-rs/pr +# Same toolchain, LTO, and features as the bot. The toolchain of the image +# replaces the one in rust-toolchain.toml. +ENV RUSTUP_TOOLCHAIN=${RUST_VERSION} \ + CARGO_PROFILE_BENCH_LTO=thin \ + CARGO_PROFILE_RELEASE_LTO=thin \ + CRITERION_HOME=/results/criterion +RUN --mount=type=cache,target=/usr/local/cargo/registry \ Review Comment: #12349 gives profiling as one reason for this ("regressions cannot be profiled on the same setup"). Here the benchmarks run inside a `RUN` step of `docker build`, and the bench binaries stay in the cache mount, so nothing is left to profile afterward. Profiling the bot's CPUs needs a Linux host of the same machine family. On that host, `perf` can run `cargo bench` directly, and the container adds nothing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
