Could an approach of building only the updated parts of the repo help to reduce build times?
For example, changes to the classes under the AWS bundle (and only that bundle) would only need those classes to be built and tested. Where such an approach gets a bit more complex is interdependence between parts of the repo. For example, if nifi-api is updated, should all classes be built and tested? As part of a release, the entire repo could then be built and tested. This approach might help if the majority of repo changes are to individual NARs rather than wider ranging. I'm also assuming this would be possible via GitHub Actions (I have no experience of them, but have implemented this kind of approach in a Drone.io mono-repo previously). Cheers, Chris Sampson On Mon, 19 Apr 2021, 17:16 David Handermann, <[email protected]> wrote: > This background is very helpful to keep in mind when evaluating new and > updated unit tests. There are definitely some expensive tests that could > be streamlined, but introducing a separate version lifecycle for framework > and extensions seems like it is becoming more necessary. Moving to a Java > 11 baseline would also reduce the need to build on multiple versions, but > based on other discussions, it sounds like that is not currently scheduled. > > I have noticed that Windows builds can run for a longer period of time, is > there a reason that the Windows workflow definition does not have the same > timeout setting as the Linux and macOS builds? Introducing one would at > least kill off problematic Windows builds. > > Regards, > David Handermann > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <[email protected]> wrote: > > > Thanks for bringing this up. The most clear next step I can envision > > at this point is that we break up our core framework from our > > extensions. Not obvious how best to break this up but we need to. > > The build times are insane. > > > > Joe > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <[email protected]> > > wrote: > > > > > > As you can probably imagine, it takes a lot of resources in order to CI > > for all the Apache projects. Periodically this becomes an issue, as the > > donated resources from cloud CI providers ( Travis and now GitHub > Actions ) > > end up queuing and delaying builds across Apache projects because of > larger > > projects and their requirements. > > > > > > The discussions center around a few common themes: > > > > > > - the CI requirements of large complex projects dominate the Apache > Queue > > > - how those projects can supplement their builds in a way acceptable to > > ASF INFRA > > > - how those projects can have per project metering such that a project > > can pay for hours over the ’norm’ > > > - how to optimize projects > > > > > > This issue is currently being discussed again on the @builds apache > > list. I’m sending this over to the Nifi Dev community because Nifi > itself > > has been mentioned as one of the top users of GitHub action resources by > > some measure. Many of the other projects are actively taking measures > to > > decrease or optimize their usage, and we should probably think about how > we > > can do so as well. > > > > > > Here is the *current* thread: > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > > < > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > > > > Here is the message where 13 projects are listed > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%[email protected]%3e > > < > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%[email protected]%3E > > > > > > > > > There are many other threads regarding GitHub Action limits and > > resources if you start scrolling back through the archives. > > > > > > I’m posting this to hopefully kick off some discussions. > > >
