Chris, Yeah that would be very helpful. But do you have any idea how that might be achieved in this environment?
Thanks On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson <[email protected]> wrote: > > Could an approach of building only the updated parts of the repo help to > reduce build times? > > For example, changes to the classes under the AWS bundle (and only that > bundle) would only need those classes to be built and tested. > > Where such an approach gets a bit more complex is interdependence between > parts of the repo. For example, if nifi-api is updated, should all classes > be built and tested? > > As part of a release, the entire repo could then be built and tested. > > This approach might help if the majority of repo changes are to individual > NARs rather than wider ranging. I'm also assuming this would be possible > via GitHub Actions (I have no experience of them, but have implemented this > kind of approach in a Drone.io mono-repo previously). > > > Cheers, > > Chris Sampson > > On Mon, 19 Apr 2021, 17:16 David Handermann, <[email protected]> > wrote: > > > This background is very helpful to keep in mind when evaluating new and > > updated unit tests. There are definitely some expensive tests that could > > be streamlined, but introducing a separate version lifecycle for framework > > and extensions seems like it is becoming more necessary. Moving to a Java > > 11 baseline would also reduce the need to build on multiple versions, but > > based on other discussions, it sounds like that is not currently scheduled. > > > > I have noticed that Windows builds can run for a longer period of time, is > > there a reason that the Windows workflow definition does not have the same > > timeout setting as the Linux and macOS builds? Introducing one would at > > least kill off problematic Windows builds. > > > > Regards, > > David Handermann > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <[email protected]> wrote: > > > > > Thanks for bringing this up. The most clear next step I can envision > > > at this point is that we break up our core framework from our > > > extensions. Not obvious how best to break this up but we need to. > > > The build times are insane. > > > > > > Joe > > > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler <[email protected]> > > > wrote: > > > > > > > > As you can probably imagine, it takes a lot of resources in order to CI > > > for all the Apache projects. Periodically this becomes an issue, as the > > > donated resources from cloud CI providers ( Travis and now GitHub > > Actions ) > > > end up queuing and delaying builds across Apache projects because of > > larger > > > projects and their requirements. > > > > > > > > The discussions center around a few common themes: > > > > > > > > - the CI requirements of large complex projects dominate the Apache > > Queue > > > > - how those projects can supplement their builds in a way acceptable to > > > ASF INFRA > > > > - how those projects can have per project metering such that a project > > > can pay for hours over the ’norm’ > > > > - how to optimize projects > > > > > > > > This issue is currently being discussed again on the @builds apache > > > list. I’m sending this over to the Nifi Dev community because Nifi > > itself > > > has been mentioned as one of the top users of GitHub action resources by > > > some measure. Many of the other projects are actively taking measures > > to > > > decrease or optimize their usage, and we should probably think about how > > we > > > can do so as well. > > > > > > > > Here is the *current* thread: > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > > > < > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > > > > > > > Here is the message where 13 projects are listed > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%[email protected]%3e > > > < > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%[email protected]%3E > > > > > > > > > > > > There are many other threads regarding GitHub Action limits and > > > resources if you start scrolling back through the archives. > > > > > > > > I’m posting this to hopefully kick off some discussions. > > > > >
