to clarify user management for infra is not a prob. it is an ldap group. repo creation is self service as well amd group access is tied to that.
release artifact is the source we produce. this is typically correlated to a tag of the repo. if we have all source in one repo it isnt clear to me how we can maintain that. in any event im not making a statement of whether to do many repos or not. just correcting some potentially misleading claims. thanks On Fri, Jul 12, 2019, 12:01 PM Adam Taft <[email protected]> wrote: > Just as a point of discussion, I'm not entirely sure that splitting into > multiple physical git repositories is actually adding any value. I think > it's worth consideration that all the (good) changes being proposed are > done under a single mono-repository model. > > If we split into multiple repositories, you have substantially increased > the infra surface area. User account management overhead goes up. Support > from the infra team goes up. JIRA issue management goes up, > misfiled/miscategorized issues become common. It becomes harder for > community members to interact and engage with the project, steeper learning > curve for new contributors. There are more "side channel" conversations and > less transparency into the project as a whole. Git history is much harder > (or impossible) to follow across the entire project. Tracking down bugs and > performing git blame or git bisect becomes hard. > > There's nothing really stopping all of these changes from occurring in the > existing repo, we don't have to have a maven pom.xml in the root of the > project repository. It's much easier for contributors to just clone a > single repository, read the README at the root, and get oriented to the > project layout. Output artifacts can still be versioned differently (api > can have a different version from extensions). "Splitting out" modules can > still happen in the mono-repository. Jenkins and friends can be taught the > project layout. > > tl;dr - The changes being proposed can be done in a single repository. > Splitting into multiple repositories is adding overhead on multiple levels, > which might be a sneaky form of muda. [1] > > Thanks for reading, > Adam > > [1] https://dzone.com/articles/seven-wastes-software > > > On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler <[email protected]> > wrote: > > > I agree that this looks great. I think Mike’s idea is worth considering > as > > well. I would hope, that as part of this effort some thought will be > given > > to enhancing the developer documentation around the modules would be > given > > as well. > > > > > > > > > > On July 10, 2019 at 18:15:21, Mike Thomsen ([email protected]) > wrote: > > > > I agree. It's very well thought out. One change to consider is splitting > > the extensions further into two separate repos. One that would serve as a > > standard library of sorts for other component developers and another that > > would include everything else. Things like the Record API would go into > the > > former so that we could have a more conservative release schedule going > > forward with those components. > > > > On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto <[email protected]> > > wrote: > > > > > Thanks Kevin, this looks really promising. > > > > > > Updating the link here as I think the page may have moved: > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > < > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > Andy LoPresto > > > [email protected] > > > [email protected] > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > On Jul 10, 2019, at 12:08 PM, Kevin Doran <[email protected]> wrote: > > > > > > > > Hi NiFi Dev Community, > > > > > > > > Jeff Storck, Bryan Bende, and I have been collaborating back and > forth > > > > on a proposal for how to restructure the NiFi source code into > smaller > > > > Maven projects and repositories based on the discussion that took > > > > place awhile back on this thread. I'm reviving this older thread in > > > > order to share that proposal with the community and generate farther > > > > discussion about at solidifying a destination and a plan for how to > > > > get there. > > > > > > > > Specifically, the proposal we've started working on has three parts: > > > > > > > > 1. Goals (more or less a summary of the earlier discussion that took > > > > place on this thread) > > > > 2. Proposed end state of the new Maven project and repository > structure > > > > 3. Proposed approach for how to get from where we are today to the > > > > desired end state > > > > > > > > The proposal is on the Apache NiFi Wiki [1], so that we can all > > > > collaborate on it or leave comments there. > > > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > > > > > > > Thanks, > > > > Kevin, Jeff, and Bryan > > > > > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <[email protected]> > wrote: > > > >> > > > >> I am also in favor of splitting the nifi maven project up into > smaller > > > >> projects with independent release cycles in order to decouple > > > >> development at well defined boundaries/interfaces and also to > > > >> facilitate code reuse. > > > >> > > > >> In anticipation of eventually working towards a NiFi 2.0 that > > > >> introduces bigger changes for developers and users, I've started > work > > > >> on a nifi-commons project in which I've extracted out some of the > code > > > >> that originally got ported from NiFi -> NiFi Registry, and now > exists > > > >> as similar code in both projects, into a standalone modular library. > > > >> That premilinary work is here on my personal github account for now > > > >> [1]. > > > >> > > > >> So far, it only contains some security code in a submodule, and is a > > > >> WIP (more work coming when I have time), but the idea is > nifi-commons > > > >> could have several libraries/modules and would be released > > > >> periodically to use across nifi and registry. If we are talking > about > > > >> spliting the nifi project into framework and extensions, then > > > >> nifi-commons might be a good home for code that needs to be shared > > > >> across those two sub projects as well, such as the nifi-api bits Joe > > > >> mentioned. > > > >> > > > >> As part of this larger effort, I would be happy to help get a > > > >> nifi-commons repository started in Apache where we can move shared > > > >> code such as nifi-api to prepare for splitting nifi-framework and > > > >> nifi-extensions. It also occurs to me that if nifi-framework and > > > >> nifi-extensions are being released independently, nifi-assembly > should > > > >> probably just become a project that pulls in and assembles the > latest > > > >> releases of framework and extensions. > > > >> > > > >> Overall, I think this would be beneficial for most of the work going > > > >> on in Apache NiFi, which would not have to cut across these > different > > > >> project and therefore would be easier to code, test, build, and > > > >> release. However, the level of difficulty will increase for changes > > > >> that will need to span multiple projects, though those are fewer in > > > >> number, so overall I think it would be a net win for the dev > > > >> community. > > > >> > > > >> [1] https://github.com/kevdoran/nifi-commons > > > >> > > > >> Thanks, > > > >> Kevin > > > >> > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > [email protected]> > > > wrote: > > > >>> > > > >>> I am a strong +1 on the separation and reducing the build time. > With > > > that in mind, I think the process I brought up yesterday [1] of signing > > our > > > artifacts with GPG as part of the Maven build is paramount, because we > > > would now be consuming core code across multiple projects/repositories, > > so > > > there is even less guarantee the code is coming from “us”. > > > >>> > > > >>> [1] > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > < > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > >>> > > > >>> Andy LoPresto > > > >>> [email protected] > > > >>> [email protected] > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > >>> > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <[email protected]> wrote: > > > >>>> > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi > > 2.0 > > > would > > > >>>> also be a good time to look at more clearly defining the > separation > > > between > > > >>>> the UI and the framework. Where nifi-api is the contract between > the > > > >>>> extensions and the framework, the NiFi Rest api is the contract > > > between the > > > >>>> UI and framework... These pieces could potentially be built / > > > deployed / > > > >>>> updated independently. > > > >>>> > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff <[email protected]> wrote: > > > >>>> > > > >>>>> In the same category of challenges that Peter pointed out, it > might > > > be > > > >>>>> difficult for Travis to build the "framework" and "extensions" > > > projects if > > > >>>>> there are changes in a PR that affect both projects. > > > >>>>> > > > >>>>> Is there a good way in Travis to have the workspace/maven repo > > shared > > > >>>>> between projects in a single build? > > > >>>>> > > > >>>>> It's probably always in the direction of the extensions project > > > needing > > > >>>>> something new to be added to the framework project rather than > the > > > other > > > >>>>> way around, but it'll be tricky to get that working right in > Travis > > > if it's > > > >>>>> not possible to set up the Travis build to know it needs to > deploy > > > the > > > >>>>> framework project artifacts into a maven repo that the extension > > > project > > > >>>>> will use. > > > >>>>> > > > >>>>> One way might be to make sure that changes to the framework > project > > > must be > > > >>>>> in master before the extensions project can make use of them, but > > > that > > > >>>>> would require a "default master" build for the framework project > > > which > > > >>>>> builds master after each commit, and deploys the build artifacts > to > > a > > > >>>>> persistent maven repo that the extension project builds can > access. > > > It > > > >>>>> also makes project-spanning change-sets take longer to review and > > > get fully > > > >>>>> committed to master. > > > >>>>> > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > > [email protected]> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> One more "not awesome" would be that core changes that affect > > > extensions > > > >>>>>> will be a little harder to test. If I make a core change that > > > changes the > > > >>>>>> signature of an interface/etc... I'll need to do some extra work > > to > > > make > > > >>>>>> sure I don't break extensions that use it. > > > >>>>>> > > > >>>>>> Still worth it, just one more thing to mention. > > > >>>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: Joe Witt <[email protected]> > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > >>>>>> To: [email protected] > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > > > repos and > > > >>>>>> releases > > > >>>>>> > > > >>>>>> Team, > > > >>>>>> > > > >>>>>> We've discussed this a bit over the years in various forms but > it > > > again > > > >>>>>> seems time to progress this topic and enough has changed I think > > to > > > >>>>> warrant > > > >>>>>> it. > > > >>>>>> > > > >>>>>> Tensions: > > > >>>>>> 1) Our build times take too long. In travis-ci for instance it > > > takes 40 > > > >>>>>> minutes when it works. > > > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr > builds > > > on > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > >>>>>> 3) We want to add Java 11 support such that one could build > with 8 > > > or 11 > > > >>>>>> and the above still apply. The becomes 6 builds. > > > >>>>>> 4) With the progress in NiFi registry we can now load artifacts > > > there and > > > >>>>>> could pull them into NiFi. And this integration will only get > > > better. > > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else > > we > > > >>>>> cannot > > > >>>>>> upload convenience binaries. > > > >>>>>> > > > >>>>>> We cannot solve all the things just yet but we can make > progress. > > I > > > >>>>>> suggest we split apart the NiFi 'framework/application' in its > own > > > >>>>> release > > > >>>>>> cycle and code repository from the 'nifi extensions' into its > own > > > >>>>>> repository and release cycle. The NiFi release would still pull > in > > > a > > > >>>>>> specific set of extension bundles so to our end users at this > time > > > there > > > >>>>> is > > > >>>>>> no change. In the future we could also just stop including the > > > extensions > > > >>>>>> in nifi the application and they could be sourced at runtime as > > > needed > > > >>>>> from > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > >>>>>> > > > >>>>>> Why does this help? > > > >>>>>> - Builds would only take as long as just extensions take or just > > > core/app > > > >>>>>> takes. This reduces time for each change cycle and reduces load > on > > > >>>>>> travis-ci which runs the same tests over and over and over for > > each > > > pull > > > >>>>>> request/push regardless of whether it was an extension or core. > > > >>>>>> > > > >>>>>> - It moves us toward the direction we're heading anyway whereby > > > >>>>> extensions > > > >>>>>> can have their own lifecycle from the framework/app itself. > > > >>>>>> > > > >>>>>> How is this not awesome: > > > >>>>>> - Doesn't yet solve for the large builds problem. I think we'll > > get > > > >>>>> there > > > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for > > > retrieval > > > >>>>>> of all extensions. > > > >>>>>> - Adds another 'thing we need to do a release cycle for'. This > is > > > >>>>>> generally unpleasant but it is paid for once a release cycle and > > it > > > does > > > >>>>>> allow us to release independently for new cool extensions/fixes > > > apart > > > >>>>> from > > > >>>>>> the framework itself. > > > >>>>>> > > > >>>>>> Would be great to hear others thoughts if they too feel it is > time > > > to > > > >>>>> make > > > >>>>>> this happen. > > > >>>>>> > > > >>>>>> Thanks > > > >>>>>> Joe > > > >>>>>> > > > >>>>> > > > >>> > > > > > > > > >
