[
https://issues.apache.org/jira/browse/SOLR-16525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016798#comment-18016798
]
Jason Gerlowski commented on SOLR-16525:
----------------------------------------
Hey all,
Following some of the perf regressions we've hit in recent releases (e.g.
SOLR-17792), I've been thinking about how important it is to resume this
effort. I'd really like to get something landed within the ASF umbrella.
IMO it's important to start small, to increase our odds of landing _something_
within the ASF umbrella that can then be iterated on. Even with a small
starting point through there are a lot of questions around what this should
look like: what perf framework should we use? what should we benchmark? where
will things run? I've offered my best attempt at answering each of these
questions below. Again, the one motivation running through all is: how can we
reduce scope and simplify for our initial pass.
(Questions/answers below are influenced a good bit by recent discussion in our
August "Community Meetup"; thanks to all who took part in the discussion there!)
1. **Where Should It Live?** My vote would be to put the new code in
`solr-sandbox`, at least initially. The code is conceptually distinct from
Solr itself, and something we never intend to ship in the Solr distribution or
a related artifact. Having it in a separate repo makes it easier to run the
latest benchmarking code with varying Solr versions as well. And of course, as
a new initiative the perf-testing code would be somewhat "experimental", which
fits `solr-sandbox` well.
2. **What Framework Should It Use?** There are a number of choices out there:
from the more "homegrown" test-harnesses written by other search projects (e.g.
Mike McCandless's "luceneutil" benchmarks, ElasticSearch's "Rally" and
OpenSearch's fork "OpenSearch Benchmark", Fullstory's solr-bench), to
"off-the-shelf" frameworks like JMeter and Gatling. My inclination would be to
use one of the off-the-shelf options, in order to benefit from the strong
community around those projects. "Gatling" seems like the best choice of the
OTS options, as its community seems more active than JMeter and some others.
3. **What Should It Test?** While there are a lot of parts of Solr where
performance is critical, I'd advocate that we limit our scope initially to
indexing and query performance. Both because this is where most users spent
most of their time, and because indexing/query traffic lends itself to
off-the-shelf frameworks better than, say, node startup/shutdown. Ideally a
first pass would have a single benchmark: something simple like indexing the
wikipedia dataset. Bonus points if we could eventually expand to cover some of
the same benchmarks available at benchmarks.mikemccandless.com
4. **Where Should It Run?** Our project has dedicated "nodes" in the ASF
Jenkins, that are configured to run a single job at a time. So running nightly
perf jobs there seems do-able, in terms of having a "public" running option?
But the benchmarks should be set up to expect solr already started and to
interact using HTTP traffic only. This limits scope and keeps things flexible
so that developers can run against any type of Solr deployment they desire
(single local instance vs. remote multi-node cluster, "bare-metal" vs. VM vs.
docker container vs. Kubernetes, behind a proxy, etc).
Curious what folks think about the "design" outlined in broad strokes by the
questions and answers above. I've taken a crack [at a draft PR in
"solr-sandbox"|https://github.com/apache/solr-sandbox/pull/123] that implements
the approach outlined above. If folks don't have big objections there, we
could look at merging that PR relatively soon!
> Periodic performance benchmarking
> ---------------------------------
>
> Key: SOLR-16525
> URL: https://issues.apache.org/jira/browse/SOLR-16525
> Project: Solr
> Issue Type: Task
> Reporter: Ishan Chattopadhyaya
> Priority: Major
> Attachments: Screenshot from 2023-01-14 17-39-42.png, Screenshot from
> 2023-02-08 04-15-34.png, image-2022-11-09-13-23-21-131.png,
> logs-stress-facets-local.json-b9f5f9ffed8e0c1af679911db6af698d18ab8fe1.zip,
> logs-stress-facets-local.json-ebc823a599160914f7f0f2694f64a53d9cd20268.zip,
> periodic-benchmarks.odp, periodic-benchmarks.pdf
>
>
> This issue is about periodic performance testing of Solr commits against
> pre-written suites of performance tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
