> > Another note here is problems come and go away in unpredictable ways. Before SOLR-14665 I never thought about doing a performance test of creating thousands of collections. The problem here is the same with our tests, even Solr has a huge number of tests, bugs still happen here and there, sometimes they are serious bugs. So no matter how good a performance tool we have (not mentioning that we do not have a consistent, unified way to do that) degrade performance can still happen in unpredictable ways.
Moreover, if a commit changes a particular codepath, like PeerSync class, we do not have an available tool for that. Requiring people to write a tool just to measure their changes to do a simple commit (and likely throw it away) seems a big -1 to me, not to mention that bugs can arise in their performance tools and numbers are hardly to be trusted.
