I haven't tried "solr-bench" https://github.com/thesearchstack/solr-bench closely but I sure hope we can rally around something that's pretty good; maybe this is it. I really need to give this one a shot. I've noticed on occasion some of us will throw together dedicated utilities to do benchmarking for a specific matter and then throw it away. Maybe if we have a common benchmarking suite, such efforts will be less effort and can actually be contributed back so that we can potentially monitor the matter.
Whatever we do or not do is imperfect. I hope some "mandate" doesn't stop progress. We don't go changing code just for the heck of it; we do it for a variety of matters. If those things are blocked, we'll be trading the opportunity cost of the change for the performance risk. Each issue is different -- has its own risk-reward trade-off. Just keep this in mind, Ishan. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Tue, Aug 11, 2020 at 8:47 PM Mike Drob <[email protected]> wrote: > Can you give examples of this? I don’t see them in the repo. > > On Tue, Aug 11, 2020 at 4:30 PM Ishan Chattopadhyaya < > [email protected]> wrote: > >> Local mode uses the installed JDK. GCP mode can pick up a JDK url as >> configured. It is just a configuration, one among many, that can be changed >> as per needs of the benchmark. The benchmarks can be used with almost any >> branch (just specify the commit sha in the repository section, or >> alternatively build Solr tgz separately and refer to it in the solr-package >> parameter). >> >> >> On Wed, 12 Aug, 2020, 2:39 am Mike Drob, <[email protected]> wrote: >> >>> Hi Ishan, >>> >>> Thanks for starting this conversation! I think it's important to pay >>> attention to performance, but I also have some concerns with coming out >>> with such a strong mandate. In the repository, I'm looking at how to run in >>> local mode, and see that it looks like it will try to download a jdk from >>> some university website? That seems overly restrictive to me, why can't we >>> use the already installed JDK? >>> >>> Is the benchmark suite designed for master? Or for branch_8x? >>> >>> Mike >>> >>> On Tue, Aug 11, 2020 at 9:04 AM Ishan Chattopadhyaya < >>> [email protected]> wrote: >>> >>>> Hi Everyone! >>>> From now on, I intend to request/nag/demand/veto code changes, which >>>> affect default code paths for most users, be accompanied by performance >>>> testing numbers for it (e.g. [1]). Opt in features are fine, I won't >>>> personally bother about them (but if you'd like to perf test them, it would >>>> set a great precedent anyway). >>>> >>>> I will also work on setting up automated performance and stress testing >>>> [2], but in the absence of that, let us do performance test manually and >>>> report them in the JIRA. Unless we don't hold ourselves to a high >>>> standards, performance will be a joke whereby performance regressions can >>>> creep in without the committer(s) taking any responsibility towards those >>>> users affected by it (SOLR-14665). >>>> >>>> A benchmarking suite that I am working on is at >>>> https://github.com/thesearchstack/solr-bench (SOLR-10317). A stress >>>> test suite is under development (SOLR-13933). If you wish to use either of >>>> these, I shall offer help and support (please ping me on Slack directly or >>>> #solr-dev, or open a Github Issue on that repo). >>>> >>>> Regards, >>>> Ishan >>>> >>>> [1] - >>>> https://issues.apache.org/jira/browse/SOLR-14354?focusedCommentId=17174221&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17174221 >>>> [2] - >>>> https://issues.apache.org/jira/browse/SOLR-14354?focusedCommentId=17174234&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17174234 >>>> (edited) >>>> >>>> >>>>
