I'm working on automating performance testing, details in https://issues.apache.org/jira/browse/SOLR-16525.
Even before I could complete the automation, I observed massive slowdown in restart performance, now attributable to https://issues.apache.org/jira/browse/SOLR-16414. This affected 9.1 release candidate RC1, but is now fixed in 9.1 and 9x branches. However, while performance was back to original levels on 9.1 branch, there was a 80-100% slowdown on the 9x branch even after this fix. Please see: http://mostly.cool/cluster-test.json.html The test is here: https://github.com/fullstorydev/solr-bench/blob/ishan/repeatable-jenkins/suites/cluster-test.json In order to investigate the slowdown, I retroactively applied the patch that fixed the performance problem in SOLR-16414 (removing use of parallelStream) to the intermediate commits and plotted the graph: http://mostly.cool/cluster-test-with-patch.html And now, two more commits with potential slowdowns are observed. Here are the JIRA issues I've opened for both: https://issues.apache.org/jira/browse/SOLR-16530 https://issues.apache.org/jira/browse/SOLR-16531 In a week of working on this automation, I was able to catch 3 slowdowns on the first thing I automated. It might be good to keep this running and test other aspects. Going forward, I'll be automating more performance suites and open blocker JIRA issues on significant performance degradation, whenever observed. I'll make it easy for all of us to add suites to the framework and have their personal branches/PRs tested through this. Please let me know about any thoughts / concerns / suggestions. Thanks, Ishan
