Updated default command to ./gradlew localSettings && ./gradlew --max-workers=`nproc` -Ptests.jvms=96 test
Thanks, -Uv On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya <ichattopadhy...@gmail.com> wrote: > Thanks to Crave! Seems like there's Dual XEON 24 core processors > available on the system. > > I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5 min. > ~/crave run -- './gradlew localSettings && ./gradlew > --max-workers=`nproc` -Ptests.jvms=96 test' > .... > The slowest tests (exceeding 500 ms) during this run: > 94.39s BasicDistributedZkTest.test (:solr:core) > 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs) > 74.73s PeerSyncReplicationTest.test (:solr:core) > 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection > (:solr:core) > 59.15s ReplicationFactorTest.test (:solr:core) > 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs) > 57.89s TestDistributedSearch.test (:solr:core) > 54.44s BasicDistributedZk2Test.test (:solr:core) > 52.49s TestStressInPlaceUpdates.stressTest (:solr:core) > 51.46s UnloadDistributedZkTest.test (:solr:core) > The slowest suites (exceeding 1s) during this run: > 188.60s CheckHdfsIndexTest (:solr:modules:hdfs) > 184.41s TestRecoveryHdfs (:solr:modules:hdfs) > 116.30s SchemaTest (:solr:solrj) > 100.00s BasicDistributedZkTest (:solr:core) > 94.33s StreamDecoratorTest (:solr:solrj-streaming) > 88.09s TestTlogReplica (:solr:core) > 83.08s ReindexCollectionTest (:solr:core) > 79.25s TestCloudConsistency (:solr:core) > 78.69s HdfsDirectoryTest (:solr:modules:hdfs) > 76.33s MultiThreadedOCPTest (:solr:core) > > BUILD SUCCESSFUL in 5m 2s > 153 actionable tasks: 55 executed, 98 up-to-date > > ------------------------------------------------------------------------ > Build Successful > > Total time: 5m25.5s > ------------------------------------------------------------------------ > > On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh > <ep...@opensourceconnections.com> wrote: > > > > Fantastic! > > > > > On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar <yuvr...@gmail.com> wrote: > > > > > > Hurray! > > > > > > On Fri, Jan 27, 2023, 19:27 David Smiley <dsmi...@apache.org> wrote: > > > > > >> The PR is merged. Henceforth, all java based tests will run on PRs. > > >> Also "crave run" now has the defaults so you can run that at your > > >> terminal > > >> without adding all the extra args :-) > > >> > > >> I did an experiment[1] wondering how long it would take a typical GitHub > > >> Action machine to run all these Solr tests. The answer is 96 minutes. No > > >> failures / timeouts; it really took that long. > > >> [1] > > >> https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956 > > >> > > >> ~ David Smiley > > >> Apache Lucene/Solr Search Developer > > >> http://www.linkedin.com/in/davidwsmiley > > >> > > >> > > >> On Wed, Jan 25, 2023 at 5:22 PM David Smiley <dsmi...@apache.org> wrote: > > >> > > >>> We haven't been running all our tests in GitHub Actions (i.e. PR > > >>> validation) because it was too time consuming to do so. I don't recall > > >> how > > >>> slow it was when someone last tried; it's probably better now but still > > >>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches > > >>> SolrJ. > > >>> > > >>> There's now a PR here to introduce a new GHA that builds on Crave.io on > > >>> a > > >>> beefy machine: https://github.com/apache/solr/pull/1303 The PR > > >>> validation took 11 minutes which is similar to the amount of time it > > >> took a > > >>> GHA to just do precommit checks -- 10 minutes :-) > > >>> I think we can remove the SolrJ specific GHA as it'll be redundant. > > >>> > > >>> Furthermore, anyone can use this to run tests from the convenience of > > >> your > > >>> laptop at the CLI while you are in the middle of any change (doesn't > > >> matter > > >>> what you have committed or not, pushed or not). To do so, run: crave run > > >>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc` > > >>> -Ptests.jvms=48 test' > > >>> > > >>> Yeah that's long. There is a discussion in JIRA underway that may lead > > >> to > > >>> eliminating the "localSettings" step if, for example, it's moved to a > > >> bash > > >>> script executed by the gradle wrapper (my proposal). I should also be > > >> able > > >>> to configure crave with a default run configuration with this baked in. > > >>> I'll post an update when I'm able to do that. > > >>> > > >>> ~ David Smiley > > >>> Apache Lucene/Solr Search Developer > > >>> http://www.linkedin.com/in/davidwsmiley > > >>> > > >> > > > > _______________________ > > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > > http://www.opensourceconnections.com > > <http://www.opensourceconnections.com/> | My Free/Busy > > <http://tinyurl.com/eric-cal> > > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > > <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > > This e-mail and all contents, including attachments, is considered to be > > Company Confidential unless explicitly stated otherwise, regardless of > > whether attachments are marked as such. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org >
