I think the concerns around user management are valid, are they not? Overhead in JIRA goes up (assigning rights to users in JIRA is multiplied). Risk to new contributors is high, because each isolated repository has its own life and code contribution styles. Maybe the actual apache infra involvement is low, but the negative effects of community and source code bifurcation goes up.
Tagging in mono-repos is done by prefixing the name of the component in the tag name. Your release sources are still generated from the component folder (not from the root). Modularization (as being proposed) is a good thing, but can be done in a single repository. It's not a requirement to split up the git project to get the benefits of modularization. That's the point I'm hoping is seen in this. On Fri, Jul 12, 2019 at 10:08 AM Joe Witt <joe.w...@gmail.com> wrote: > to clarify user management for infra is not a prob. it is an ldap group. > > repo creation is self service as well amd group access is tied to that. > > release artifact is the source we produce. this is typically correlated to > a tag of the repo. if we have all source in one repo it isnt clear to me > how we can maintain that. > > in any event im not making a statement of whether to do many repos or not. > just correcting some potentially misleading claims. > > thanks > > On Fri, Jul 12, 2019, 12:01 PM Adam Taft <a...@adamtaft.com> wrote: > > > Just as a point of discussion, I'm not entirely sure that splitting into > > multiple physical git repositories is actually adding any value. I think > > it's worth consideration that all the (good) changes being proposed are > > done under a single mono-repository model. > > > > If we split into multiple repositories, you have substantially increased > > the infra surface area. User account management overhead goes up. Support > > from the infra team goes up. JIRA issue management goes up, > > misfiled/miscategorized issues become common. It becomes harder for > > community members to interact and engage with the project, steeper > learning > > curve for new contributors. There are more "side channel" conversations > and > > less transparency into the project as a whole. Git history is much harder > > (or impossible) to follow across the entire project. Tracking down bugs > and > > performing git blame or git bisect becomes hard. > > > > There's nothing really stopping all of these changes from occurring in > the > > existing repo, we don't have to have a maven pom.xml in the root of the > > project repository. It's much easier for contributors to just clone a > > single repository, read the README at the root, and get oriented to the > > project layout. Output artifacts can still be versioned differently (api > > can have a different version from extensions). "Splitting out" modules > can > > still happen in the mono-repository. Jenkins and friends can be taught > the > > project layout. > > > > tl;dr - The changes being proposed can be done in a single repository. > > Splitting into multiple repositories is adding overhead on multiple > levels, > > which might be a sneaky form of muda. [1] > > > > Thanks for reading, > > Adam > > > > [1] https://dzone.com/articles/seven-wastes-software > > > > > > On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > I agree that this looks great. I think Mike’s idea is worth considering > > as > > > well. I would hope, that as part of this effort some thought will be > > given > > > to enhancing the developer documentation around the modules would be > > given > > > as well. > > > > > > > > > > > > > > > On July 10, 2019 at 18:15:21, Mike Thomsen (mikerthom...@gmail.com) > > wrote: > > > > > > I agree. It's very well thought out. One change to consider is > splitting > > > the extensions further into two separate repos. One that would serve > as a > > > standard library of sorts for other component developers and another > that > > > would include everything else. Things like the Record API would go into > > the > > > former so that we could have a more conservative release schedule going > > > forward with those components. > > > > > > On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto <alopre...@apache.org> > > > wrote: > > > > > > > Thanks Kevin, this looks really promising. > > > > > > > > Updating the link here as I think the page may have moved: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > < > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > > > > Andy LoPresto > > > > alopre...@apache.org > > > > alopresto.apa...@gmail.com > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > > On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdo...@apache.org> > wrote: > > > > > > > > > > Hi NiFi Dev Community, > > > > > > > > > > Jeff Storck, Bryan Bende, and I have been collaborating back and > > forth > > > > > on a proposal for how to restructure the NiFi source code into > > smaller > > > > > Maven projects and repositories based on the discussion that took > > > > > place awhile back on this thread. I'm reviving this older thread in > > > > > order to share that proposal with the community and generate > farther > > > > > discussion about at solidifying a destination and a plan for how to > > > > > get there. > > > > > > > > > > Specifically, the proposal we've started working on has three > parts: > > > > > > > > > > 1. Goals (more or less a summary of the earlier discussion that > took > > > > > place on this thread) > > > > > 2. Proposed end state of the new Maven project and repository > > structure > > > > > 3. Proposed approach for how to get from where we are today to the > > > > > desired end state > > > > > > > > > > The proposal is on the Apache NiFi Wiki [1], so that we can all > > > > > collaborate on it or leave comments there. > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > Thanks, > > > > > Kevin, Jeff, and Bryan > > > > > > > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdo...@apache.org> > > wrote: > > > > >> > > > > >> I am also in favor of splitting the nifi maven project up into > > smaller > > > > >> projects with independent release cycles in order to decouple > > > > >> development at well defined boundaries/interfaces and also to > > > > >> facilitate code reuse. > > > > >> > > > > >> In anticipation of eventually working towards a NiFi 2.0 that > > > > >> introduces bigger changes for developers and users, I've started > > work > > > > >> on a nifi-commons project in which I've extracted out some of the > > code > > > > >> that originally got ported from NiFi -> NiFi Registry, and now > > exists > > > > >> as similar code in both projects, into a standalone modular > library. > > > > >> That premilinary work is here on my personal github account for > now > > > > >> [1]. > > > > >> > > > > >> So far, it only contains some security code in a submodule, and > is a > > > > >> WIP (more work coming when I have time), but the idea is > > nifi-commons > > > > >> could have several libraries/modules and would be released > > > > >> periodically to use across nifi and registry. If we are talking > > about > > > > >> spliting the nifi project into framework and extensions, then > > > > >> nifi-commons might be a good home for code that needs to be shared > > > > >> across those two sub projects as well, such as the nifi-api bits > Joe > > > > >> mentioned. > > > > >> > > > > >> As part of this larger effort, I would be happy to help get a > > > > >> nifi-commons repository started in Apache where we can move shared > > > > >> code such as nifi-api to prepare for splitting nifi-framework and > > > > >> nifi-extensions. It also occurs to me that if nifi-framework and > > > > >> nifi-extensions are being released independently, nifi-assembly > > should > > > > >> probably just become a project that pulls in and assembles the > > latest > > > > >> releases of framework and extensions. > > > > >> > > > > >> Overall, I think this would be beneficial for most of the work > going > > > > >> on in Apache NiFi, which would not have to cut across these > > different > > > > >> project and therefore would be easier to code, test, build, and > > > > >> release. However, the level of difficulty will increase for > changes > > > > >> that will need to span multiple projects, though those are fewer > in > > > > >> number, so overall I think it would be a net win for the dev > > > > >> community. > > > > >> > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > >> > > > > >> Thanks, > > > > >> Kevin > > > > >> > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > alopre...@apache.org> > > > > wrote: > > > > >>> > > > > >>> I am a strong +1 on the separation and reducing the build time. > > With > > > > that in mind, I think the process I brought up yesterday [1] of > signing > > > our > > > > artifacts with GPG as part of the Maven build is paramount, because > we > > > > would now be consuming core code across multiple > projects/repositories, > > > so > > > > there is even less guarantee the code is coming from “us”. > > > > >>> > > > > >>> [1] > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > < > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > >>> > > > > >>> Andy LoPresto > > > > >>> alopre...@apache.org > > > > >>> alopresto.apa...@gmail.com > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > EF69 > > > > >>> > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <b...@jhu.edu> > wrote: > > > > >>>> > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > NiFi > > > 2.0 > > > > would > > > > >>>> also be a good time to look at more clearly defining the > > separation > > > > between > > > > >>>> the UI and the framework. Where nifi-api is the contract between > > the > > > > >>>> extensions and the framework, the NiFi Rest api is the contract > > > > between the > > > > >>>> UI and framework... These pieces could potentially be built / > > > > deployed / > > > > >>>> updated independently. > > > > >>>> > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff <jtsw...@gmail.com> > wrote: > > > > >>>> > > > > >>>>> In the same category of challenges that Peter pointed out, it > > might > > > > be > > > > >>>>> difficult for Travis to build the "framework" and "extensions" > > > > projects if > > > > >>>>> there are changes in a PR that affect both projects. > > > > >>>>> > > > > >>>>> Is there a good way in Travis to have the workspace/maven repo > > > shared > > > > >>>>> between projects in a single build? > > > > >>>>> > > > > >>>>> It's probably always in the direction of the extensions project > > > > needing > > > > >>>>> something new to be added to the framework project rather than > > the > > > > other > > > > >>>>> way around, but it'll be tricky to get that working right in > > Travis > > > > if it's > > > > >>>>> not possible to set up the Travis build to know it needs to > > deploy > > > > the > > > > >>>>> framework project artifacts into a maven repo that the > extension > > > > project > > > > >>>>> will use. > > > > >>>>> > > > > >>>>> One way might be to make sure that changes to the framework > > project > > > > must be > > > > >>>>> in master before the extensions project can make use of them, > but > > > > that > > > > >>>>> would require a "default master" build for the framework > project > > > > which > > > > >>>>> builds master after each commit, and deploys the build > artifacts > > to > > > a > > > > >>>>> persistent maven repo that the extension project builds can > > access. > > > > It > > > > >>>>> also makes project-spanning change-sets take longer to review > and > > > > get fully > > > > >>>>> committed to master. > > > > >>>>> > > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > > > pwi...@micron.com> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> One more "not awesome" would be that core changes that affect > > > > extensions > > > > >>>>>> will be a little harder to test. If I make a core change that > > > > changes the > > > > >>>>>> signature of an interface/etc... I'll need to do some extra > work > > > to > > > > make > > > > >>>>>> sure I don't break extensions that use it. > > > > >>>>>> > > > > >>>>>> Still worth it, just one more thing to mention. > > > > >>>>>> > > > > >>>>>> -----Original Message----- > > > > >>>>>> From: Joe Witt <joew...@apache.org> > > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > > >>>>>> To: dev@nifi.apache.org > > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and > extension > > > > repos and > > > > >>>>>> releases > > > > >>>>>> > > > > >>>>>> Team, > > > > >>>>>> > > > > >>>>>> We've discussed this a bit over the years in various forms but > > it > > > > again > > > > >>>>>> seems time to progress this topic and enough has changed I > think > > > to > > > > >>>>> warrant > > > > >>>>>> it. > > > > >>>>>> > > > > >>>>>> Tensions: > > > > >>>>>> 1) Our build times take too long. In travis-ci for instance it > > > > takes 40 > > > > >>>>>> minutes when it works. > > > > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr > > builds > > > > on > > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > > >>>>>> 3) We want to add Java 11 support such that one could build > > with 8 > > > > or 11 > > > > >>>>>> and the above still apply. The becomes 6 builds. > > > > >>>>>> 4) With the progress in NiFi registry we can now load > artifacts > > > > there and > > > > >>>>>> could pull them into NiFi. And this integration will only get > > > > better. > > > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or > else > > > we > > > > >>>>> cannot > > > > >>>>>> upload convenience binaries. > > > > >>>>>> > > > > >>>>>> We cannot solve all the things just yet but we can make > > progress. > > > I > > > > >>>>>> suggest we split apart the NiFi 'framework/application' in its > > own > > > > >>>>> release > > > > >>>>>> cycle and code repository from the 'nifi extensions' into its > > own > > > > >>>>>> repository and release cycle. The NiFi release would still > pull > > in > > > > a > > > > >>>>>> specific set of extension bundles so to our end users at this > > time > > > > there > > > > >>>>> is > > > > >>>>>> no change. In the future we could also just stop including the > > > > extensions > > > > >>>>>> in nifi the application and they could be sourced at runtime > as > > > > needed > > > > >>>>> from > > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > > >>>>>> > > > > >>>>>> Why does this help? > > > > >>>>>> - Builds would only take as long as just extensions take or > just > > > > core/app > > > > >>>>>> takes. This reduces time for each change cycle and reduces > load > > on > > > > >>>>>> travis-ci which runs the same tests over and over and over for > > > each > > > > pull > > > > >>>>>> request/push regardless of whether it was an extension or > core. > > > > >>>>>> > > > > >>>>>> - It moves us toward the direction we're heading anyway > whereby > > > > >>>>> extensions > > > > >>>>>> can have their own lifecycle from the framework/app itself. > > > > >>>>>> > > > > >>>>>> How is this not awesome: > > > > >>>>>> - Doesn't yet solve for the large builds problem. I think > we'll > > > get > > > > >>>>> there > > > > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry > for > > > > retrieval > > > > >>>>>> of all extensions. > > > > >>>>>> - Adds another 'thing we need to do a release cycle for'. This > > is > > > > >>>>>> generally unpleasant but it is paid for once a release cycle > and > > > it > > > > does > > > > >>>>>> allow us to release independently for new cool > extensions/fixes > > > > apart > > > > >>>>> from > > > > >>>>>> the framework itself. > > > > >>>>>> > > > > >>>>>> Would be great to hear others thoughts if they too feel it is > > time > > > > to > > > > >>>>> make > > > > >>>>>> this happen. > > > > >>>>>> > > > > >>>>>> Thanks > > > > >>>>>> Joe > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > > > > > > > > > > >
