In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would also be a good time to look at more clearly defining the separation between the UI and the framework. Where nifi-api is the contract between the extensions and the framework, the NiFi Rest api is the contract between the UI and framework... These pieces could potentially be built / deployed / updated independently.
On Thu, May 30, 2019 at 11:39 AM Jeff <[email protected]> wrote: > In the same category of challenges that Peter pointed out, it might be > difficult for Travis to build the "framework" and "extensions" projects if > there are changes in a PR that affect both projects. > > Is there a good way in Travis to have the workspace/maven repo shared > between projects in a single build? > > It's probably always in the direction of the extensions project needing > something new to be added to the framework project rather than the other > way around, but it'll be tricky to get that working right in Travis if it's > not possible to set up the Travis build to know it needs to deploy the > framework project artifacts into a maven repo that the extension project > will use. > > One way might be to make sure that changes to the framework project must be > in master before the extensions project can make use of them, but that > would require a "default master" build for the framework project which > builds master after each commit, and deploys the build artifacts to a > persistent maven repo that the extension project builds can access. It > also makes project-spanning change-sets take longer to review and get fully > committed to master. > > On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <[email protected]> > wrote: > > > One more "not awesome" would be that core changes that affect extensions > > will be a little harder to test. If I make a core change that changes the > > signature of an interface/etc... I'll need to do some extra work to make > > sure I don't break extensions that use it. > > > > Still worth it, just one more thing to mention. > > > > -----Original Message----- > > From: Joe Witt <[email protected]> > > Sent: Thursday, May 30, 2019 9:19 AM > > To: [email protected] > > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > > releases > > > > Team, > > > > We've discussed this a bit over the years in various forms but it again > > seems time to progress this topic and enough has changed I think to > warrant > > it. > > > > Tensions: > > 1) Our build times take too long. In travis-ci for instance it takes 40 > > minutes when it works. > > 2) The number of builds we do has increased. We do us/jp/fr builds on > > open and oracle JDKs. That is 6 builds. > > 3) We want to add Java 11 support such that one could build with 8 or 11 > > and the above still apply. The becomes 6 builds. > > 4) With the progress in NiFi registry we can now load artifacts there and > > could pull them into NiFi. And this integration will only get better. > > 5) The NiFi build is too huge and cannot grow any longer or else we > cannot > > upload convenience binaries. > > > > We cannot solve all the things just yet but we can make progress. I > > suggest we split apart the NiFi 'framework/application' in its own > release > > cycle and code repository from the 'nifi extensions' into its own > > repository and release cycle. The NiFi release would still pull in a > > specific set of extension bundles so to our end users at this time there > is > > no change. In the future we could also just stop including the extensions > > in nifi the application and they could be sourced at runtime as needed > from > > the registry (call that a NiFi 2.x thing). > > > > Why does this help? > > - Builds would only take as long as just extensions take or just core/app > > takes. This reduces time for each change cycle and reduces load on > > travis-ci which runs the same tests over and over and over for each pull > > request/push regardless of whether it was an extension or core. > > > > - It moves us toward the direction we're heading anyway whereby > extensions > > can have their own lifecycle from the framework/app itself. > > > > How is this not awesome: > > - Doesn't yet solve for the large builds problem. I think we'll get > there > > with a NiFi 2.x release which fully leverages nifi-registry for retrieval > > of all extensions. > > - Adds another 'thing we need to do a release cycle for'. This is > > generally unpleasant but it is paid for once a release cycle and it does > > allow us to release independently for new cool extensions/fixes apart > from > > the framework itself. > > > > Would be great to hear others thoughts if they too feel it is time to > make > > this happen. > > > > Thanks > > Joe > > >
