Good point on the reference_impl branch. Eventually that's the goal, but given there's not a timeline for that to be merged yet I think this is a good stop-gap. It's a few minutes of work to get these PR actions written, so I feel like there is little downside. And we can always remove them when the reference_imp branch gets merged.
This may block my ability to merge any PR whatsoever. The docker tests will not be run on any PRs that don't touch bin/solr, solr/packaging or solr/docker. There are complications around running integration tests in a non-containerized environment as well. And if the docker image tests are failing on the PR, wouldn't you rather know before committing? Even if you don't want to install docker, you can call in someone else that has it to help debug. Much like PRs that affect solr.cmd, for committers without access to a windows machine. On Thu, Sep 17, 2020 at 6:23 PM Ishan Chattopadhyaya < [email protected]> wrote: > > It would be great to run all the tests every time, but clearly that is > too expensive. > > The reference_impl branch requires around 30 seconds to run all solr-core > tests. That's where we should all put our collective efforts. > Also, I have reservations against docker based tests blocking PRs. If I > don't have docker running on my dev machine, I wouldn't be able to make > those tests pass. This may block my ability to merge any PR whatsoever. > Why can't we have integration tests that do not rely on docker? > > On Thu, Sep 17, 2020 at 9:26 PM Houston Putman <[email protected]> > wrote: > >> Thought I'd make this a thread instead of a discussion on a single JIRA >> ticket. >> >> Currently we have gradle precommit run on PRs for master, which is very >> useful and gives people confidence in approving PRs. But precommit is >> obviously not the only thing we care about before committing. It would be >> great to run all the tests every time, but clearly that is too expensive. >> >> In SOLR-14856 <https://issues.apache.org/jira/browse/SOLR-14856>, I >> proposed adding a github action to build and test the solr docker image for >> PRs that affected relevant parts of the repo (solr/docker, solr/bin, >> solr/packaging and solr/contrib/prometheus-exporter/bin). Running the >> docker tests currently takes roughly 12 minutes in the github action, which >> would be costly if it ran on every PR. But when running on the small >> percentage of PRs that affect those code paths, I think the benefit >> outweighs the cost. >> >> Beyond just the docker tests, I think we can leverage this ability for >> other features that are limited to certain code paths. For example running >> tests for contrib modules, testing solr/examples, and many of >> the independent lucene modules. The SolrJ tests just ran in 3 minutes >> locally for me, maybe that'd be a good candidate as well. >> >> Anyways I'm sure there are other good candidates out there, but I just >> wanted to start the discussion and hear other opinions before diving any >> deeper. >> >
