Chris,, Thanks for the description of the modular approach. Here is one GitHub action that supports conditional execution based on Git changes:
https://github.com/dorny/paths-filter It would take some effort to implement an approach that covers all the bases, but using the Maven also-make and project-list command line options should support building both a module and its dependencies. The list of changes files can be passed to another action that could determine one or more modules to build. Regards, David Handermann On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson <chris.samp...@naimuri.com.invalid> wrote: > Our approach is the use a `git diff` between source and destination > branches when a branch is merged (and the same for on dev branch builds). > > Each component within the repo then checks for whether any files within its > directories were changed (bearing in mind a dev branch may consist of > multiple commits over time, so don't just look at changes in individual > commits). > > I'm not sure if the mechanics for the existing nifi repo though with its > use of: > > * GutHub Actions (can a git diff be performed at the start and then > following actions use the output?) > > * Maven with sub-modules (can they be optionally built based on such > conditions? The GitHub Action could create files depending upon what's been > changed in order to activate profiles maybe, would that work?) > > This is broadly how I've implemented a multi-component build previously, > but not using Maven sub-modules (each component is typically its own docker > image and built separately from others). > > > Cheers, > > Chris Sampson > > On Mon, 19 Apr 2021, 18:35 Joe Witt, <joe.w...@gmail.com> wrote: > > > Chris, > > > > Yeah that would be very helpful. But do you have any idea how that > > might be achieved in this environment? > > > > Thanks > > > > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson > > <chris.samp...@naimuri.com.invalid> wrote: > > > > > > Could an approach of building only the updated parts of the repo help > to > > > reduce build times? > > > > > > For example, changes to the classes under the AWS bundle (and only that > > > bundle) would only need those classes to be built and tested. > > > > > > Where such an approach gets a bit more complex is interdependence > between > > > parts of the repo. For example, if nifi-api is updated, should all > > classes > > > be built and tested? > > > > > > As part of a release, the entire repo could then be built and tested. > > > > > > This approach might help if the majority of repo changes are to > > individual > > > NARs rather than wider ranging. I'm also assuming this would be > possible > > > via GitHub Actions (I have no experience of them, but have implemented > > this > > > kind of approach in a Drone.io mono-repo previously). > > > > > > > > > Cheers, > > > > > > Chris Sampson > > > > > > On Mon, 19 Apr 2021, 17:16 David Handermann, < > exceptionfact...@gmail.com > > > > > > wrote: > > > > > > > This background is very helpful to keep in mind when evaluating new > and > > > > updated unit tests. There are definitely some expensive tests that > > could > > > > be streamlined, but introducing a separate version lifecycle for > > framework > > > > and extensions seems like it is becoming more necessary. Moving to a > > Java > > > > 11 baseline would also reduce the need to build on multiple versions, > > but > > > > based on other discussions, it sounds like that is not currently > > scheduled. > > > > > > > > I have noticed that Windows builds can run for a longer period of > > time, is > > > > there a reason that the Windows workflow definition does not have the > > same > > > > timeout setting as the Linux and macOS builds? Introducing one would > > at > > > > least kill off problematic Windows builds. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <joe.w...@gmail.com> > wrote: > > > > > > > > > Thanks for bringing this up. The most clear next step I can > envision > > > > > at this point is that we break up our core framework from our > > > > > extensions. Not obvious how best to break this up but we need to. > > > > > The build times are insane. > > > > > > > > > > Joe > > > > > > > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler < > ottobackwa...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > As you can probably imagine, it takes a lot of resources in order > > to CI > > > > > for all the Apache projects. Periodically this becomes an issue, > as > > the > > > > > donated resources from cloud CI providers ( Travis and now GitHub > > > > Actions ) > > > > > end up queuing and delaying builds across Apache projects because > of > > > > larger > > > > > projects and their requirements. > > > > > > > > > > > > The discussions center around a few common themes: > > > > > > > > > > > > - the CI requirements of large complex projects dominate the > Apache > > > > Queue > > > > > > - how those projects can supplement their builds in a way > > acceptable to > > > > > ASF INFRA > > > > > > - how those projects can have per project metering such that a > > project > > > > > can pay for hours over the ’norm’ > > > > > > - how to optimize projects > > > > > > > > > > > > This issue is currently being discussed again on the @builds > apache > > > > > list. I’m sending this over to the Nifi Dev community because Nifi > > > > itself > > > > > has been mentioned as one of the top users of GitHub action > > resources by > > > > > some measure. Many of the other projects are actively taking > > measures > > > > to > > > > > decrease or optimize their usage, and we should probably think > about > > how > > > > we > > > > > can do so as well. > > > > > > > > > > > > Here is the *current* thread: > > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > > > > > < > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > > > > > > > > > > > > > Here is the message where 13 projects are listed > > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3e > > > > > < > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3E > > > > > > > > > > > > > > > > > > There are many other threads regarding GitHub Action limits and > > > > > resources if you start scrolling back through the archives. > > > > > > > > > > > > I’m posting this to hopefully kick off some discussions. > > > > > > > > > > > >
