Thanks to Crave! Seems like there's Dual XEON 24 core processors available on the system.
I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5 min. ~/crave run -- './gradlew localSettings && ./gradlew --max-workers=`nproc` -Ptests.jvms=96 test' .... The slowest tests (exceeding 500 ms) during this run: 94.39s BasicDistributedZkTest.test (:solr:core) 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs) 74.73s PeerSyncReplicationTest.test (:solr:core) 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection (:solr:core) 59.15s ReplicationFactorTest.test (:solr:core) 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs) 57.89s TestDistributedSearch.test (:solr:core) 54.44s BasicDistributedZk2Test.test (:solr:core) 52.49s TestStressInPlaceUpdates.stressTest (:solr:core) 51.46s UnloadDistributedZkTest.test (:solr:core) The slowest suites (exceeding 1s) during this run: 188.60s CheckHdfsIndexTest (:solr:modules:hdfs) 184.41s TestRecoveryHdfs (:solr:modules:hdfs) 116.30s SchemaTest (:solr:solrj) 100.00s BasicDistributedZkTest (:solr:core) 94.33s StreamDecoratorTest (:solr:solrj-streaming) 88.09s TestTlogReplica (:solr:core) 83.08s ReindexCollectionTest (:solr:core) 79.25s TestCloudConsistency (:solr:core) 78.69s HdfsDirectoryTest (:solr:modules:hdfs) 76.33s MultiThreadedOCPTest (:solr:core) BUILD SUCCESSFUL in 5m 2s 153 actionable tasks: 55 executed, 98 up-to-date ------------------------------------------------------------------------ Build Successful Total time: 5m25.5s ------------------------------------------------------------------------ On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh <ep...@opensourceconnections.com> wrote: > > Fantastic! > > > On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar <yuvr...@gmail.com> wrote: > > > > Hurray! > > > > On Fri, Jan 27, 2023, 19:27 David Smiley <dsmi...@apache.org> wrote: > > > >> The PR is merged. Henceforth, all java based tests will run on PRs. > >> Also "crave run" now has the defaults so you can run that at your terminal > >> without adding all the extra args :-) > >> > >> I did an experiment[1] wondering how long it would take a typical GitHub > >> Action machine to run all these Solr tests. The answer is 96 minutes. No > >> failures / timeouts; it really took that long. > >> [1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956 > >> > >> ~ David Smiley > >> Apache Lucene/Solr Search Developer > >> http://www.linkedin.com/in/davidwsmiley > >> > >> > >> On Wed, Jan 25, 2023 at 5:22 PM David Smiley <dsmi...@apache.org> wrote: > >> > >>> We haven't been running all our tests in GitHub Actions (i.e. PR > >>> validation) because it was too time consuming to do so. I don't recall > >> how > >>> slow it was when someone last tried; it's probably better now but still > >>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches > >>> SolrJ. > >>> > >>> There's now a PR here to introduce a new GHA that builds on Crave.io on a > >>> beefy machine: https://github.com/apache/solr/pull/1303 The PR > >>> validation took 11 minutes which is similar to the amount of time it > >> took a > >>> GHA to just do precommit checks -- 10 minutes :-) > >>> I think we can remove the SolrJ specific GHA as it'll be redundant. > >>> > >>> Furthermore, anyone can use this to run tests from the convenience of > >> your > >>> laptop at the CLI while you are in the middle of any change (doesn't > >> matter > >>> what you have committed or not, pushed or not). To do so, run: crave run > >>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc` > >>> -Ptests.jvms=48 test' > >>> > >>> Yeah that's long. There is a discussion in JIRA underway that may lead > >> to > >>> eliminating the "localSettings" step if, for example, it's moved to a > >> bash > >>> script executed by the gradle wrapper (my proposal). I should also be > >> able > >>> to configure crave with a default run configuration with this baked in. > >>> I'll post an update when I'm able to do that. > >>> > >>> ~ David Smiley > >>> Apache Lucene/Solr Search Developer > >>> http://www.linkedin.com/in/davidwsmiley > >>> > >> > > _______________________ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com <http://www.opensourceconnections.com/> > | My Free/Busy <http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless of > whether attachments are marked as such. > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
