I agree. It's very well thought out. One change to consider is splitting the extensions further into two separate repos. One that would serve as a standard library of sorts for other component developers and another that would include everything else. Things like the Record API would go into the former so that we could have a more conservative release schedule going forward with those components.
On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto <alopre...@apache.org> wrote: > Thanks Kevin, this looks really promising. > > Updating the link here as I think the page may have moved: > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > < > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdo...@apache.org> wrote: > > > > Hi NiFi Dev Community, > > > > Jeff Storck, Bryan Bende, and I have been collaborating back and forth > > on a proposal for how to restructure the NiFi source code into smaller > > Maven projects and repositories based on the discussion that took > > place awhile back on this thread. I'm reviving this older thread in > > order to share that proposal with the community and generate farther > > discussion about at solidifying a destination and a plan for how to > > get there. > > > > Specifically, the proposal we've started working on has three parts: > > > > 1. Goals (more or less a summary of the earlier discussion that took > > place on this thread) > > 2. Proposed end state of the new Maven project and repository structure > > 3. Proposed approach for how to get from where we are today to the > > desired end state > > > > The proposal is on the Apache NiFi Wiki [1], so that we can all > > collaborate on it or leave comments there. > > > > [1] > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > > > Thanks, > > Kevin, Jeff, and Bryan > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdo...@apache.org> wrote: > >> > >> I am also in favor of splitting the nifi maven project up into smaller > >> projects with independent release cycles in order to decouple > >> development at well defined boundaries/interfaces and also to > >> facilitate code reuse. > >> > >> In anticipation of eventually working towards a NiFi 2.0 that > >> introduces bigger changes for developers and users, I've started work > >> on a nifi-commons project in which I've extracted out some of the code > >> that originally got ported from NiFi -> NiFi Registry, and now exists > >> as similar code in both projects, into a standalone modular library. > >> That premilinary work is here on my personal github account for now > >> [1]. > >> > >> So far, it only contains some security code in a submodule, and is a > >> WIP (more work coming when I have time), but the idea is nifi-commons > >> could have several libraries/modules and would be released > >> periodically to use across nifi and registry. If we are talking about > >> spliting the nifi project into framework and extensions, then > >> nifi-commons might be a good home for code that needs to be shared > >> across those two sub projects as well, such as the nifi-api bits Joe > >> mentioned. > >> > >> As part of this larger effort, I would be happy to help get a > >> nifi-commons repository started in Apache where we can move shared > >> code such as nifi-api to prepare for splitting nifi-framework and > >> nifi-extensions. It also occurs to me that if nifi-framework and > >> nifi-extensions are being released independently, nifi-assembly should > >> probably just become a project that pulls in and assembles the latest > >> releases of framework and extensions. > >> > >> Overall, I think this would be beneficial for most of the work going > >> on in Apache NiFi, which would not have to cut across these different > >> project and therefore would be easier to code, test, build, and > >> release. However, the level of difficulty will increase for changes > >> that will need to span multiple projects, though those are fewer in > >> number, so overall I think it would be a net win for the dev > >> community. > >> > >> [1] https://github.com/kevdoran/nifi-commons > >> > >> Thanks, > >> Kevin > >> > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto <alopre...@apache.org> > wrote: > >>> > >>> I am a strong +1 on the separation and reducing the build time. With > that in mind, I think the process I brought up yesterday [1] of signing our > artifacts with GPG as part of the Maven build is paramount, because we > would now be consuming core code across multiple projects/repositories, so > there is even less guarantee the code is coming from “us”. > >>> > >>> [1] > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > < > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > >>> > >>> Andy LoPresto > >>> alopre...@apache.org > >>> alopresto.apa...@gmail.com > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>> > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <b...@jhu.edu> wrote: > >>>> > >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 > would > >>>> also be a good time to look at more clearly defining the separation > between > >>>> the UI and the framework. Where nifi-api is the contract between the > >>>> extensions and the framework, the NiFi Rest api is the contract > between the > >>>> UI and framework... These pieces could potentially be built / > deployed / > >>>> updated independently. > >>>> > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff <jtsw...@gmail.com> wrote: > >>>> > >>>>> In the same category of challenges that Peter pointed out, it might > be > >>>>> difficult for Travis to build the "framework" and "extensions" > projects if > >>>>> there are changes in a PR that affect both projects. > >>>>> > >>>>> Is there a good way in Travis to have the workspace/maven repo shared > >>>>> between projects in a single build? > >>>>> > >>>>> It's probably always in the direction of the extensions project > needing > >>>>> something new to be added to the framework project rather than the > other > >>>>> way around, but it'll be tricky to get that working right in Travis > if it's > >>>>> not possible to set up the Travis build to know it needs to deploy > the > >>>>> framework project artifacts into a maven repo that the extension > project > >>>>> will use. > >>>>> > >>>>> One way might be to make sure that changes to the framework project > must be > >>>>> in master before the extensions project can make use of them, but > that > >>>>> would require a "default master" build for the framework project > which > >>>>> builds master after each commit, and deploys the build artifacts to a > >>>>> persistent maven repo that the extension project builds can access. > It > >>>>> also makes project-spanning change-sets take longer to review and > get fully > >>>>> committed to master. > >>>>> > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > pwi...@micron.com> > >>>>> wrote: > >>>>> > >>>>>> One more "not awesome" would be that core changes that affect > extensions > >>>>>> will be a little harder to test. If I make a core change that > changes the > >>>>>> signature of an interface/etc... I'll need to do some extra work to > make > >>>>>> sure I don't break extensions that use it. > >>>>>> > >>>>>> Still worth it, just one more thing to mention. > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: Joe Witt <joew...@apache.org> > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > >>>>>> To: dev@nifi.apache.org > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > repos and > >>>>>> releases > >>>>>> > >>>>>> Team, > >>>>>> > >>>>>> We've discussed this a bit over the years in various forms but it > again > >>>>>> seems time to progress this topic and enough has changed I think to > >>>>> warrant > >>>>>> it. > >>>>>> > >>>>>> Tensions: > >>>>>> 1) Our build times take too long. In travis-ci for instance it > takes 40 > >>>>>> minutes when it works. > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr builds > on > >>>>>> open and oracle JDKs. That is 6 builds. > >>>>>> 3) We want to add Java 11 support such that one could build with 8 > or 11 > >>>>>> and the above still apply. The becomes 6 builds. > >>>>>> 4) With the progress in NiFi registry we can now load artifacts > there and > >>>>>> could pull them into NiFi. And this integration will only get > better. > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else we > >>>>> cannot > >>>>>> upload convenience binaries. > >>>>>> > >>>>>> We cannot solve all the things just yet but we can make progress. I > >>>>>> suggest we split apart the NiFi 'framework/application' in its own > >>>>> release > >>>>>> cycle and code repository from the 'nifi extensions' into its own > >>>>>> repository and release cycle. The NiFi release would still pull in > a > >>>>>> specific set of extension bundles so to our end users at this time > there > >>>>> is > >>>>>> no change. In the future we could also just stop including the > extensions > >>>>>> in nifi the application and they could be sourced at runtime as > needed > >>>>> from > >>>>>> the registry (call that a NiFi 2.x thing). > >>>>>> > >>>>>> Why does this help? > >>>>>> - Builds would only take as long as just extensions take or just > core/app > >>>>>> takes. This reduces time for each change cycle and reduces load on > >>>>>> travis-ci which runs the same tests over and over and over for each > pull > >>>>>> request/push regardless of whether it was an extension or core. > >>>>>> > >>>>>> - It moves us toward the direction we're heading anyway whereby > >>>>> extensions > >>>>>> can have their own lifecycle from the framework/app itself. > >>>>>> > >>>>>> How is this not awesome: > >>>>>> - Doesn't yet solve for the large builds problem. I think we'll get > >>>>> there > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for > retrieval > >>>>>> of all extensions. > >>>>>> - Adds another 'thing we need to do a release cycle for'. This is > >>>>>> generally unpleasant but it is paid for once a release cycle and it > does > >>>>>> allow us to release independently for new cool extensions/fixes > apart > >>>>> from > >>>>>> the framework itself. > >>>>>> > >>>>>> Would be great to hear others thoughts if they too feel it is time > to > >>>>> make > >>>>>> this happen. > >>>>>> > >>>>>> Thanks > >>>>>> Joe > >>>>>> > >>>>> > >>> > >
