owaiskazi19 commented on issue #16398:
URL: https://github.com/apache/lucene/issues/16398#issuecomment-6174677241
Hi, I'd be interested in picking this up. OpenSearch already has something
very close to what's described here, so it might be a useful starting point.
On an OpenSearch PR you comment something like `{"run-benchmark-test":
"id_5"}`. A GitHub Actions workflow
([benchmark-pull-request.yml](https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/benchmark-pull-request.yml))
checks the id against a fixed catalog of benchmark profiles
([benchmark-configs.json](https://github.com/opensearch-project/OpenSearch/blob/main/.github/benchmark-configs.json)),
builds the PR, runs the benchmark, and posts the results plus a comparison
against a baseline back on the PR. Here's an example:
[results](https://github.com/opensearch-project/OpenSearch/pull/21607#issuecomment-4452292939)
and [baseline
comparison](https://github.com/opensearch-project/OpenSearch/pull/21607#issuecomment-4452304115).
If the person commenting isn't a maintainer, a maintainer has to approve the
run first, since it runs code from the PR.
For Lucene I'd keep that overall shape but change a couple of things.
Instead of comparing against a stored baseline, I'd build the PR and its
merge-base and benchmark both on the same machine in the same job. `luceneutil`
already supports this kind of comparison (with p-values), and it avoids the "is
this the PR or did the machine/baseline drift" question. I'd also put a fixed
set of profiles behind the
commands, e.g. `/benchmark jmh <filter>` checked against the existing JMH
benchmark names, rather than passing arbitrary arguments through. And I'd leave
the AI-written benchmark idea for later, or have it propose a benchmark as a
commit on the PR that someone reviews before it runs, rather than running
generated code directly.
The tricky part is hardware. I looked at luceneutil's setup: the Wikipedia
line file is ~6 GB compressed (and much larger uncompressed), plus ~13 GB for
vectors, and standard GitHub runners only have around 14 GB of disk. So I think
the realistic split is JMH comparisons on regular GitHub runners first (noisy,
but should catch large regressions), then a small luceneutil tier (~1M docs,
cached) on GitHub runners as a quick check, and full-size corpora and vector
benchmarks only once there's a dedicated machine, e.g. something like what the
nightlies run on.
Before building any of it I would like to do a quick experiment on a fork:
run `luceneutil` on a GitHub runner, benchmark a commit against itself a few
times to see how noisy it really is, and check how long a run takes. If the
noise is too high, that tells us early that the GitHub-runner tier isn't worth
it. I can post the numbers here.
A few questions:
1. Is there any appetite for running luceneutil from Lucene's CI?
2. Does anyone know ASF Infra's current stance on self-hosted runners and on
third-party actions like a manual-approval step? That decides how the third
tier and the approval gate could work.
3. And would a dedicated benchmark machine be possible at all, or should
this stay GitHub-runner only for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
