I am a strong +1 on the separation and reducing the build time. With that in mind, I think the process I brought up yesterday [1] of signing our artifacts with GPG as part of the Maven build is paramount, because we would now be consuming core code across multiple projects/repositories, so there is even less guarantee the code is coming from “us”.
[1] https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E <https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E> Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 30, 2019, at 9:15 AM, Brandon DeVries <[email protected]> wrote: > > In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would > also be a good time to look at more clearly defining the separation between > the UI and the framework. Where nifi-api is the contract between the > extensions and the framework, the NiFi Rest api is the contract between the > UI and framework... These pieces could potentially be built / deployed / > updated independently. > > On Thu, May 30, 2019 at 11:39 AM Jeff <[email protected]> wrote: > >> In the same category of challenges that Peter pointed out, it might be >> difficult for Travis to build the "framework" and "extensions" projects if >> there are changes in a PR that affect both projects. >> >> Is there a good way in Travis to have the workspace/maven repo shared >> between projects in a single build? >> >> It's probably always in the direction of the extensions project needing >> something new to be added to the framework project rather than the other >> way around, but it'll be tricky to get that working right in Travis if it's >> not possible to set up the Travis build to know it needs to deploy the >> framework project artifacts into a maven repo that the extension project >> will use. >> >> One way might be to make sure that changes to the framework project must be >> in master before the extensions project can make use of them, but that >> would require a "default master" build for the framework project which >> builds master after each commit, and deploys the build artifacts to a >> persistent maven repo that the extension project builds can access. It >> also makes project-spanning change-sets take longer to review and get fully >> committed to master. >> >> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <[email protected]> >> wrote: >> >>> One more "not awesome" would be that core changes that affect extensions >>> will be a little harder to test. If I make a core change that changes the >>> signature of an interface/etc... I'll need to do some extra work to make >>> sure I don't break extensions that use it. >>> >>> Still worth it, just one more thing to mention. >>> >>> -----Original Message----- >>> From: Joe Witt <[email protected]> >>> Sent: Thursday, May 30, 2019 9:19 AM >>> To: [email protected] >>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and >>> releases >>> >>> Team, >>> >>> We've discussed this a bit over the years in various forms but it again >>> seems time to progress this topic and enough has changed I think to >> warrant >>> it. >>> >>> Tensions: >>> 1) Our build times take too long. In travis-ci for instance it takes 40 >>> minutes when it works. >>> 2) The number of builds we do has increased. We do us/jp/fr builds on >>> open and oracle JDKs. That is 6 builds. >>> 3) We want to add Java 11 support such that one could build with 8 or 11 >>> and the above still apply. The becomes 6 builds. >>> 4) With the progress in NiFi registry we can now load artifacts there and >>> could pull them into NiFi. And this integration will only get better. >>> 5) The NiFi build is too huge and cannot grow any longer or else we >> cannot >>> upload convenience binaries. >>> >>> We cannot solve all the things just yet but we can make progress. I >>> suggest we split apart the NiFi 'framework/application' in its own >> release >>> cycle and code repository from the 'nifi extensions' into its own >>> repository and release cycle. The NiFi release would still pull in a >>> specific set of extension bundles so to our end users at this time there >> is >>> no change. In the future we could also just stop including the extensions >>> in nifi the application and they could be sourced at runtime as needed >> from >>> the registry (call that a NiFi 2.x thing). >>> >>> Why does this help? >>> - Builds would only take as long as just extensions take or just core/app >>> takes. This reduces time for each change cycle and reduces load on >>> travis-ci which runs the same tests over and over and over for each pull >>> request/push regardless of whether it was an extension or core. >>> >>> - It moves us toward the direction we're heading anyway whereby >> extensions >>> can have their own lifecycle from the framework/app itself. >>> >>> How is this not awesome: >>> - Doesn't yet solve for the large builds problem. I think we'll get >> there >>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval >>> of all extensions. >>> - Adds another 'thing we need to do a release cycle for'. This is >>> generally unpleasant but it is paid for once a release cycle and it does >>> allow us to release independently for new cool extensions/fixes apart >> from >>> the framework itself. >>> >>> Would be great to hear others thoughts if they too feel it is time to >> make >>> this happen. >>> >>> Thanks >>> Joe >>> >>
