Andy - fair points. Note that by definition, the process you describe is harder (requires more maneuvers). Maybe it's warranted/justified for the desired integrity that you are after, but it's most definitely a total sum of work that is greater.
Your registry example is really good. In your example, you are proposing a change to the framework and commons repositories before a change to the registry can be finalized. You'd need the changes to framework and commons to "land" and become released before the final change to the registry was committed. You'd end up with a small release queued up for the framework (whose release cycle is mostly infrequent) and you wouldn't be able to finish the work on the registry changes until that new function was releasable. The ability to mark that JIRA ticket as "closed" is delayed because you are waiting for releases from dependent components. Of course, you can build/test against -SNAPSHOT versions in each of those repositories (which is what Bryan was getting to). But the registry feature itself can't be totally finalized and is waiting on the release cycle of the slowest of the components. There are definitely tradeoffs with this direction. On Fri, Jul 12, 2019 at 12:42 PM Andy LoPresto <alopre...@apache.org> wrote: > I think by definition, a contribution _must_ fit into a single repository. > This will force developers to carefully consider the boundaries between > modules and build clean abstractions. If you are a new contributor, I would > be surprised if you are making a single (logical) contribution that would > span multiple repositories on the first go. I think enforcing clear > divisions is good for both new and experienced contributors. I also think a > change that requires contributions to multiple repositories should be > subdivided into atomic tasks. > > For example, if someone wants to contribute a new feature to nifi-registry > which also requires changes to nifi-commons for the security piece and adds > new behavior to the nifi-framework component to consume new changes from > Registry, in my mind those are actually 3 atomic changes which, while > related and interdependent, can all be contributed as standalone code to > their respective repositories in an ordered fashion. I would prefer this > over one large commit to a single repository which influences behavior in > all three modules and requires one or more reviewers with comprehensive > knowledge over all aspects of the project. > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jul 12, 2019, at 10:49 AM, Adam Taft <a...@adamtaft.com> wrote: > > > > Bryan, > > > > I think both of your points are actually closely related, and they > somewhat > > speak to my thoughts/concerns about splitting the repository. > > > > I would argue that one PR that affects multiple modules in a single > > repository is _easier_ to review than multiple PRs that affect single > > modules. In the split repository model, if a change affects several > > repositories, individual PRs would be issued against each repository. A > > reviewer would not as easily see the context of the changes and may even > > consider them out of order. > > > > In the single repository model, a PR is atomic. There is no race > condition, > > ordering or loss of context across multiple repositories. > > > > This is the concern I was making for new contributors. If your > > contribution doesn't fit neatly into a single repository, then it's quite > > the tough process to communicate and deal with changes. It will > discourage > > new folks from being involved, because the contribution barrier is > raised. > > > > It's ideal that changesets are atomic, but you definitely lose this > > property in a multi-repo scenario. Imagine rolling back a change, for > > example, that spans multiple repositories. > > > > Adam > > > > On Fri, Jul 12, 2019 at 11:27 AM Bryan Bende <bbe...@gmail.com> wrote: > > > >> Two other points to throw out there... > >> > >> 1) I think something to consider is how the management of pull > >> requests would be impacted, since that is the main form of > >> contribution. > >> > >> Separate repos forces pull requests to stay scoped to a given module, > >> making for more straight forward reviews. It also makes it easier to > >> look at a repo and see what work/contributions are still open, > >> although I suppose all the PRs in the nifi repo could be labeled by > >> module and then filtered, but it seems a little more tedious. Just > >> something to think about. > >> > >> > >> 2) We should also consider how we plan to handle changes across modules. > >> > >> As an example, currently we have nifi and nifi-registry in separate > >> repos, and nifi depends on nifi-registry, but nifi master always stays > >> on the last release version of nifi-registry. > >> > >> So if you are working on a change across both projects, the process is > >> something like the following... > >> > >> - Make change in nifi-registry and run a Maven install locally > >> - Change nifi pom to the snapshot version of nifi-registry > >> - Make changes in nifi and stage them in a branch, possibly a draft PR > >> that can't be merged yet > >> - nifi-registry gets released > >> - Put up a PR for the nifi work, bumping the nifi-registry version to > >> released version > >> > >> I have no issue continuing to work like this, as long as we accept > >> that the complexity of these scenarios will increase with more > >> modules. > >> > >> An alternative approach would be to allow master of each module to > >> depend on a snapshot of a dependent module. For example, the nifi PR > >> above could be merged before nifi-registry is ever released. It lets > >> the work proceed instead of letting these draft changes build up, and > >> it forces the dependency chain of releases to occur since now you > >> can't release nifi master until nifi-registry is released. The > >> downside is it requires everyone to locally build all the snapshot > >> modules to get the latest changes, even if they aren't working on > >> those other modules, unless there is a way for them to be provided > >> through an apache infra build process. > >> > >> This second point is less about mono vs multi repo, and more about how > >> to manage development of a change that requires modifying several > >> dependent modules. > >> > >> On Fri, Jul 12, 2019 at 1:05 PM Kevin Doran <kdo...@apache.org> wrote: > >>> > >>> Thanks Adam and Edward! This is exactly the type of discussion I was > >>> hoping a detailed and specific proposal would generate, so thanks for > >>> the input. I'll reply to each of you in turn: > >>> > >>> Adam, > >>> > >>> It is true that a repo-per-project approach is not required. I've > >>> worked on projects that do it both ways and there are advantages to > >>> both. > >>> > >>> Single-repo was considered, but as one of the primary goals is to cut > >>> down on Travis / CI build times, the mutli-repo approach seemed to > >>> have a big advantage. Personally, I've never found a reliable, stable > >>> way to introduce CI builds to a repository with multiple projects that > >>> did not require building all the projects in the repository. It's > >>> possible to try to use commands to determine which files have changed > >>> and infer which project(s) to build from that, but maintaining that > >>> logic can get messy. If the logic is wrong, it's possible a project > >>> that is not built is broken by a PR. Building everything is not an > >>> option for a project our size, as our build already time out today. > >>> Fast, reliable Travis builds with no false positives / negatives is > >>> definitely something NiFi needs, and I think it will be simplest to > >>> get there with a multi-repo approach. > >>> > >>> That said, I agree that the *biggest* win comes from splitting > >>> projects, and that splitting repos is a smaller step. I don't feel > >>> strongly about it and could live with a single repo with multiple > >>> projects (though, for what it's worth, the NiFi umbrella already has > >>> several repositories and I personally don't feel it has been > >>> burdensome). > >>> > >>> And I agree - let's not start splitting JIRA projects. Let's use > >>> components or labels or something to differentiate issues under the > >>> existing NIFI Jira project. > >>> > >>> > >>> Edward, > >>> > >>> Thanks. I totally agree and I know others who feel the same way. > >>> Better defined boundaries and loosely coupled modules is 100% a > >>> long-term goal. I think this project restructuring won't solve the > >>> problem completely (in fact, to your point, it may uncover some > >>> unfortunate tight-coupling that needs to be reworked on the current > >>> master before the split can happen), but I do think it will encourage > >>> developers to more faithfully build to APIs and avoid leaky > >>> abstractions as there will be more hard division points in the code > >>> base. Some of those issues might be able to be addressed immediately. > >>> Others might have to wait for a major version change. > >>> > >>> Thanks, > >>> Kevin > >>> > >>> On Fri, Jul 12, 2019 at 1:04 PM Adam Taft <a...@adamtaft.com> wrote: > >>>> > >>>> To be honest and to your point Joe, the thing that optimizes the RM > >> duties > >>>> should probably be preferred in all of this. There is so much > >> overhead for > >>>> the release manager, that lubricating the RM process probably trumps a > >> lot > >>>> of my concerns. I think there's real concern for making the project > >> harder > >>>> for new contributors. But likewise, that concern should be balanced > >> with > >>>> making the project harder for longtime contributors who have pulled > the > >>>> cart the most. > >>>> > >>>> I was just at least hoping for a discussion on the concept. Thanks as > >>>> always for your leadership and contributions to the nifi community. > >>>> > >>>> On Fri, Jul 12, 2019 at 10:48 AM Joe Witt <joe.w...@gmail.com> wrote: > >>>> > >>>>> Ah I agree the JIRA thing would be too heavy handed. A single JIRA > >> with > >>>>> well defined components tied to 'repos' is good. > >>>>> > >>>>> As far as separate code repos we're talking about different > >> releasable > >>>>> artifacts for which we as a PMC are responsible for the > >> meaning/etc.. As a > >>>>> many time RM I definitely dislike the mono repo construct as I > >> understand > >>>>> it to function. I prefer repos per source release artifact where all > >>>>> source in that artifact is a function of the release. I am ok with > >>>>> different convenience binaries resulting from a single source release > >>>>> artifact though. > >>>>> > >>>>> Thanks > >>>>> > >>>>> On Fri, Jul 12, 2019 at 12:26 PM Adam Taft <a...@adamtaft.com> > >> wrote: > >>>>> > >>>>>> I think the concerns around user management are valid, are they > >> not? > >>>>>> Overhead in JIRA goes up (assigning rights to users in JIRA is > >>>>>> multiplied). Risk to new contributors is high, because each > >> isolated > >>>>>> repository has its own life and code contribution styles. Maybe > >> the > >>>>> actual > >>>>>> apache infra involvement is low, but the negative effects of > >> community > >>>>> and > >>>>>> source code bifurcation goes up. > >>>>>> > >>>>>> Tagging in mono-repos is done by prefixing the name of the > >> component in > >>>>> the > >>>>>> tag name. Your release sources are still generated from the > >> component > >>>>>> folder (not from the root). > >>>>>> > >>>>>> Modularization (as being proposed) is a good thing, but can be > >> done in a > >>>>>> single repository. It's not a requirement to split up the git > >> project to > >>>>>> get the benefits of modularization. That's the point I'm hoping > >> is seen > >>>>> in > >>>>>> this. > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Fri, Jul 12, 2019 at 10:08 AM Joe Witt <joe.w...@gmail.com> > >> wrote: > >>>>>> > >>>>>>> to clarify user management for infra is not a prob. it is an > >> ldap > >>>>> group. > >>>>>>> > >>>>>>> repo creation is self service as well amd group access is tied > >> to that. > >>>>>>> > >>>>>>> release artifact is the source we produce. this is typically > >>>>> correlated > >>>>>> to > >>>>>>> a tag of the repo. if we have all source in one repo it isnt > >> clear to > >>>>> me > >>>>>>> how we can maintain that. > >>>>>>> > >>>>>>> in any event im not making a statement of whether to do many > >> repos or > >>>>>> not. > >>>>>>> just correcting some potentially misleading claims. > >>>>>>> > >>>>>>> thanks > >>>>>>> > >>>>>>> On Fri, Jul 12, 2019, 12:01 PM Adam Taft <a...@adamtaft.com> > >> wrote: > >>>>>>> > >>>>>>>> Just as a point of discussion, I'm not entirely sure that > >> splitting > >>>>>> into > >>>>>>>> multiple physical git repositories is actually adding any > >> value. I > >>>>>> think > >>>>>>>> it's worth consideration that all the (good) changes being > >> proposed > >>>>> are > >>>>>>>> done under a single mono-repository model. > >>>>>>>> > >>>>>>>> If we split into multiple repositories, you have substantially > >>>>>> increased > >>>>>>>> the infra surface area. User account management overhead goes > >> up. > >>>>>> Support > >>>>>>>> from the infra team goes up. JIRA issue management goes up, > >>>>>>>> misfiled/miscategorized issues become common. It becomes > >> harder for > >>>>>>>> community members to interact and engage with the project, > >> steeper > >>>>>>> learning > >>>>>>>> curve for new contributors. There are more "side channel" > >>>>> conversations > >>>>>>> and > >>>>>>>> less transparency into the project as a whole. Git history is > >> much > >>>>>> harder > >>>>>>>> (or impossible) to follow across the entire project. Tracking > >> down > >>>>> bugs > >>>>>>> and > >>>>>>>> performing git blame or git bisect becomes hard. > >>>>>>>> > >>>>>>>> There's nothing really stopping all of these changes from > >> occurring > >>>>> in > >>>>>>> the > >>>>>>>> existing repo, we don't have to have a maven pom.xml in the > >> root of > >>>>> the > >>>>>>>> project repository. It's much easier for contributors to just > >> clone a > >>>>>>>> single repository, read the README at the root, and get > >> oriented to > >>>>> the > >>>>>>>> project layout. Output artifacts can still be versioned > >> differently > >>>>>> (api > >>>>>>>> can have a different version from extensions). "Splitting out" > >>>>> modules > >>>>>>> can > >>>>>>>> still happen in the mono-repository. Jenkins and friends can > >> be > >>>>> taught > >>>>>>> the > >>>>>>>> project layout. > >>>>>>>> > >>>>>>>> tl;dr - The changes being proposed can be done in a single > >>>>> repository. > >>>>>>>> Splitting into multiple repositories is adding overhead on > >> multiple > >>>>>>> levels, > >>>>>>>> which might be a sneaky form of muda. [1] > >>>>>>>> > >>>>>>>> Thanks for reading, > >>>>>>>> Adam > >>>>>>>> > >>>>>>>> [1] https://dzone.com/articles/seven-wastes-software > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler < > >>>>> ottobackwa...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I agree that this looks great. I think Mike’s idea is worth > >>>>>> considering > >>>>>>>> as > >>>>>>>>> well. I would hope, that as part of this effort some thought > >> will > >>>>> be > >>>>>>>> given > >>>>>>>>> to enhancing the developer documentation around the modules > >> would > >>>>> be > >>>>>>>> given > >>>>>>>>> as well. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On July 10, 2019 at 18:15:21, Mike Thomsen ( > >> mikerthom...@gmail.com > >>>>> ) > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> I agree. It's very well thought out. One change to consider > >> is > >>>>>>> splitting > >>>>>>>>> the extensions further into two separate repos. One that > >> would > >>>>> serve > >>>>>>> as a > >>>>>>>>> standard library of sorts for other component developers and > >>>>> another > >>>>>>> that > >>>>>>>>> would include everything else. Things like the Record API > >> would go > >>>>>> into > >>>>>>>> the > >>>>>>>>> former so that we could have a more conservative release > >> schedule > >>>>>> going > >>>>>>>>> forward with those components. > >>>>>>>>> > >>>>>>>>> On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto < > >>>>> alopre...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Thanks Kevin, this looks really promising. > >>>>>>>>>> > >>>>>>>>>> Updating the link here as I think the page may have moved: > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > >>>>>>>>>> < > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Andy LoPresto > >>>>>>>>>> alopre...@apache.org > >>>>>>>>>> alopresto.apa...@gmail.com > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B > >> 2F7D > >>>>> EF69 > >>>>>>>>>> > >>>>>>>>>>> On Jul 10, 2019, at 12:08 PM, Kevin Doran < > >> kdo...@apache.org> > >>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi NiFi Dev Community, > >>>>>>>>>>> > >>>>>>>>>>> Jeff Storck, Bryan Bende, and I have been collaborating > >> back > >>>>> and > >>>>>>>> forth > >>>>>>>>>>> on a proposal for how to restructure the NiFi source > >> code into > >>>>>>>> smaller > >>>>>>>>>>> Maven projects and repositories based on the discussion > >> that > >>>>> took > >>>>>>>>>>> place awhile back on this thread. I'm reviving this older > >>>>> thread > >>>>>> in > >>>>>>>>>>> order to share that proposal with the community and > >> generate > >>>>>>> farther > >>>>>>>>>>> discussion about at solidifying a destination and a plan > >> for > >>>>> how > >>>>>> to > >>>>>>>>>>> get there. > >>>>>>>>>>> > >>>>>>>>>>> Specifically, the proposal we've started working on has > >> three > >>>>>>> parts: > >>>>>>>>>>> > >>>>>>>>>>> 1. Goals (more or less a summary of the earlier > >> discussion that > >>>>>>> took > >>>>>>>>>>> place on this thread) > >>>>>>>>>>> 2. Proposed end state of the new Maven project and > >> repository > >>>>>>>> structure > >>>>>>>>>>> 3. Proposed approach for how to get from where we are > >> today to > >>>>>> the > >>>>>>>>>>> desired end state > >>>>>>>>>>> > >>>>>>>>>>> The proposal is on the Apache NiFi Wiki [1], so that we > >> can all > >>>>>>>>>>> collaborate on it or leave comments there. > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Kevin, Jeff, and Bryan > >>>>>>>>>>> > >>>>>>>>>>> On Thu, May 30, 2019 at 1:31 PM Kevin Doran < > >> kdo...@apache.org > >>>>>> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I am also in favor of splitting the nifi maven project > >> up into > >>>>>>>> smaller > >>>>>>>>>>>> projects with independent release cycles in order to > >> decouple > >>>>>>>>>>>> development at well defined boundaries/interfaces and > >> also to > >>>>>>>>>>>> facilitate code reuse. > >>>>>>>>>>>> > >>>>>>>>>>>> In anticipation of eventually working towards a NiFi > >> 2.0 that > >>>>>>>>>>>> introduces bigger changes for developers and users, I've > >>>>> started > >>>>>>>> work > >>>>>>>>>>>> on a nifi-commons project in which I've extracted out > >> some of > >>>>>> the > >>>>>>>> code > >>>>>>>>>>>> that originally got ported from NiFi -> NiFi Registry, > >> and now > >>>>>>>> exists > >>>>>>>>>>>> as similar code in both projects, into a standalone > >> modular > >>>>>>> library. > >>>>>>>>>>>> That premilinary work is here on my personal github > >> account > >>>>> for > >>>>>>> now > >>>>>>>>>>>> [1]. > >>>>>>>>>>>> > >>>>>>>>>>>> So far, it only contains some security code in a > >> submodule, > >>>>> and > >>>>>>> is a > >>>>>>>>>>>> WIP (more work coming when I have time), but the idea is > >>>>>>>> nifi-commons > >>>>>>>>>>>> could have several libraries/modules and would be > >> released > >>>>>>>>>>>> periodically to use across nifi and registry. If we are > >>>>> talking > >>>>>>>> about > >>>>>>>>>>>> spliting the nifi project into framework and > >> extensions, then > >>>>>>>>>>>> nifi-commons might be a good home for code that needs > >> to be > >>>>>> shared > >>>>>>>>>>>> across those two sub projects as well, such as the > >> nifi-api > >>>>> bits > >>>>>>> Joe > >>>>>>>>>>>> mentioned. > >>>>>>>>>>>> > >>>>>>>>>>>> As part of this larger effort, I would be happy to help > >> get a > >>>>>>>>>>>> nifi-commons repository started in Apache where we can > >> move > >>>>>> shared > >>>>>>>>>>>> code such as nifi-api to prepare for splitting > >> nifi-framework > >>>>>> and > >>>>>>>>>>>> nifi-extensions. It also occurs to me that if > >> nifi-framework > >>>>> and > >>>>>>>>>>>> nifi-extensions are being released independently, > >>>>> nifi-assembly > >>>>>>>> should > >>>>>>>>>>>> probably just become a project that pulls in and > >> assembles the > >>>>>>>> latest > >>>>>>>>>>>> releases of framework and extensions. > >>>>>>>>>>>> > >>>>>>>>>>>> Overall, I think this would be beneficial for most of > >> the work > >>>>>>> going > >>>>>>>>>>>> on in Apache NiFi, which would not have to cut across > >> these > >>>>>>>> different > >>>>>>>>>>>> project and therefore would be easier to code, test, > >> build, > >>>>> and > >>>>>>>>>>>> release. However, the level of difficulty will increase > >> for > >>>>>>> changes > >>>>>>>>>>>> that will need to span multiple projects, though those > >> are > >>>>> fewer > >>>>>>> in > >>>>>>>>>>>> number, so overall I think it would be a net win for > >> the dev > >>>>>>>>>>>> community. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] https://github.com/kevdoran/nifi-commons > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Kevin > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > >>>>>>>> alopre...@apache.org> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am a strong +1 on the separation and reducing the > >> build > >>>>> time. > >>>>>>>> With > >>>>>>>>>> that in mind, I think the process I brought up yesterday > >> [1] of > >>>>>>> signing > >>>>>>>>> our > >>>>>>>>>> artifacts with GPG as part of the Maven build is paramount, > >>>>> because > >>>>>>> we > >>>>>>>>>> would now be consuming core code across multiple > >>>>>>> projects/repositories, > >>>>>>>>> so > >>>>>>>>>> there is even less guarantee the code is coming from “us”. > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > >>>>>>>>>> < > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > >>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Andy LoPresto > >>>>>>>>>>>>> alopre...@apache.org > >>>>>>>>>>>>> alopresto.apa...@gmail.com > >>>>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E > >> F65B 2F7D > >>>>>>> EF69 > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On May 30, 2019, at 9:15 AM, Brandon DeVries < > >> b...@jhu.edu> > >>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In regards to "We 'could' also split out the > >> 'nifi-api'...", > >>>>>>> NiFi > >>>>>>>>> 2.0 > >>>>>>>>>> would > >>>>>>>>>>>>>> also be a good time to look at more clearly defining > >> the > >>>>>>>> separation > >>>>>>>>>> between > >>>>>>>>>>>>>> the UI and the framework. Where nifi-api is the > >> contract > >>>>>> between > >>>>>>>> the > >>>>>>>>>>>>>> extensions and the framework, the NiFi Rest api is the > >>>>>> contract > >>>>>>>>>> between the > >>>>>>>>>>>>>> UI and framework... These pieces could potentially be > >> built > >>>>> / > >>>>>>>>>> deployed / > >>>>>>>>>>>>>> updated independently. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, May 30, 2019 at 11:39 AM Jeff < > >> jtsw...@gmail.com> > >>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In the same category of challenges that Peter > >> pointed out, > >>>>> it > >>>>>>>> might > >>>>>>>>>> be > >>>>>>>>>>>>>>> difficult for Travis to build the "framework" and > >>>>>> "extensions" > >>>>>>>>>> projects if > >>>>>>>>>>>>>>> there are changes in a PR that affect both projects. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Is there a good way in Travis to have the > >> workspace/maven > >>>>>> repo > >>>>>>>>> shared > >>>>>>>>>>>>>>> between projects in a single build? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It's probably always in the direction of the > >> extensions > >>>>>> project > >>>>>>>>>> needing > >>>>>>>>>>>>>>> something new to be added to the framework project > >> rather > >>>>>> than > >>>>>>>> the > >>>>>>>>>> other > >>>>>>>>>>>>>>> way around, but it'll be tricky to get that working > >> right > >>>>> in > >>>>>>>> Travis > >>>>>>>>>> if it's > >>>>>>>>>>>>>>> not possible to set up the Travis build to know it > >> needs to > >>>>>>>> deploy > >>>>>>>>>> the > >>>>>>>>>>>>>>> framework project artifacts into a maven repo that > >> the > >>>>>>> extension > >>>>>>>>>> project > >>>>>>>>>>>>>>> will use. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> One way might be to make sure that changes to the > >> framework > >>>>>>>> project > >>>>>>>>>> must be > >>>>>>>>>>>>>>> in master before the extensions project can make use > >> of > >>>>> them, > >>>>>>> but > >>>>>>>>>> that > >>>>>>>>>>>>>>> would require a "default master" build for the > >> framework > >>>>>>> project > >>>>>>>>>> which > >>>>>>>>>>>>>>> builds master after each commit, and deploys the > >> build > >>>>>>> artifacts > >>>>>>>> to > >>>>>>>>> a > >>>>>>>>>>>>>>> persistent maven repo that the extension project > >> builds can > >>>>>>>> access. > >>>>>>>>>> It > >>>>>>>>>>>>>>> also makes project-spanning change-sets take longer > >> to > >>>>> review > >>>>>>> and > >>>>>>>>>> get fully > >>>>>>>>>>>>>>> committed to master. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks > >> (pwicks) < > >>>>>>>>>> pwi...@micron.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> One more "not awesome" would be that core changes > >> that > >>>>>> affect > >>>>>>>>>> extensions > >>>>>>>>>>>>>>>> will be a little harder to test. If I make a core > >> change > >>>>>> that > >>>>>>>>>> changes the > >>>>>>>>>>>>>>>> signature of an interface/etc... I'll need to do > >> some > >>>>> extra > >>>>>>> work > >>>>>>>>> to > >>>>>>>>>> make > >>>>>>>>>>>>>>>> sure I don't break extensions that use it. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Still worth it, just one more thing to mention. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>>>> From: Joe Witt <joew...@apache.org> > >>>>>>>>>>>>>>>> Sent: Thursday, May 30, 2019 9:19 AM > >>>>>>>>>>>>>>>> To: dev@nifi.apache.org > >>>>>>>>>>>>>>>> Subject: [EXT] [discuss] Splitting NiFi framework > >> and > >>>>>>> extension > >>>>>>>>>> repos and > >>>>>>>>>>>>>>>> releases > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Team, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We've discussed this a bit over the years in > >> various forms > >>>>>> but > >>>>>>>> it > >>>>>>>>>> again > >>>>>>>>>>>>>>>> seems time to progress this topic and enough has > >> changed I > >>>>>>> think > >>>>>>>>> to > >>>>>>>>>>>>>>> warrant > >>>>>>>>>>>>>>>> it. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Tensions: > >>>>>>>>>>>>>>>> 1) Our build times take too long. In travis-ci for > >>>>> instance > >>>>>> it > >>>>>>>>>> takes 40 > >>>>>>>>>>>>>>>> minutes when it works. > >>>>>>>>>>>>>>>> 2) The number of builds we do has increased. We do > >>>>> us/jp/fr > >>>>>>>> builds > >>>>>>>>>> on > >>>>>>>>>>>>>>>> open and oracle JDKs. That is 6 builds. > >>>>>>>>>>>>>>>> 3) We want to add Java 11 support such that one > >> could > >>>>> build > >>>>>>>> with 8 > >>>>>>>>>> or 11 > >>>>>>>>>>>>>>>> and the above still apply. The becomes 6 builds. > >>>>>>>>>>>>>>>> 4) With the progress in NiFi registry we can now > >> load > >>>>>>> artifacts > >>>>>>>>>> there and > >>>>>>>>>>>>>>>> could pull them into NiFi. And this integration > >> will only > >>>>>> get > >>>>>>>>>> better. > >>>>>>>>>>>>>>>> 5) The NiFi build is too huge and cannot grow any > >> longer > >>>>> or > >>>>>>> else > >>>>>>>>> we > >>>>>>>>>>>>>>> cannot > >>>>>>>>>>>>>>>> upload convenience binaries. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We cannot solve all the things just yet but we can > >> make > >>>>>>>> progress. > >>>>>>>>> I > >>>>>>>>>>>>>>>> suggest we split apart the NiFi > >> 'framework/application' in > >>>>>> its > >>>>>>>> own > >>>>>>>>>>>>>>> release > >>>>>>>>>>>>>>>> cycle and code repository from the 'nifi > >> extensions' into > >>>>>> its > >>>>>>>> own > >>>>>>>>>>>>>>>> repository and release cycle. The NiFi release > >> would still > >>>>>>> pull > >>>>>>>> in > >>>>>>>>>> a > >>>>>>>>>>>>>>>> specific set of extension bundles so to our end > >> users at > >>>>>> this > >>>>>>>> time > >>>>>>>>>> there > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>> no change. In the future we could also just stop > >> including > >>>>>> the > >>>>>>>>>> extensions > >>>>>>>>>>>>>>>> in nifi the application and they could be sourced at > >>>>> runtime > >>>>>>> as > >>>>>>>>>> needed > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> the registry (call that a NiFi 2.x thing). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Why does this help? > >>>>>>>>>>>>>>>> - Builds would only take as long as just extensions > >> take > >>>>> or > >>>>>>> just > >>>>>>>>>> core/app > >>>>>>>>>>>>>>>> takes. This reduces time for each change cycle and > >> reduces > >>>>>>> load > >>>>>>>> on > >>>>>>>>>>>>>>>> travis-ci which runs the same tests over and over > >> and over > >>>>>> for > >>>>>>>>> each > >>>>>>>>>> pull > >>>>>>>>>>>>>>>> request/push regardless of whether it was an > >> extension or > >>>>>>> core. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - It moves us toward the direction we're heading > >> anyway > >>>>>>> whereby > >>>>>>>>>>>>>>> extensions > >>>>>>>>>>>>>>>> can have their own lifecycle from the framework/app > >>>>> itself. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> How is this not awesome: > >>>>>>>>>>>>>>>> - Doesn't yet solve for the large builds problem. I > >> think > >>>>>>> we'll > >>>>>>>>> get > >>>>>>>>>>>>>>> there > >>>>>>>>>>>>>>>> with a NiFi 2.x release which fully leverages > >>>>> nifi-registry > >>>>>>> for > >>>>>>>>>> retrieval > >>>>>>>>>>>>>>>> of all extensions. > >>>>>>>>>>>>>>>> - Adds another 'thing we need to do a release cycle > >> for'. > >>>>>> This > >>>>>>>> is > >>>>>>>>>>>>>>>> generally unpleasant but it is paid for once a > >> release > >>>>> cycle > >>>>>>> and > >>>>>>>>> it > >>>>>>>>>> does > >>>>>>>>>>>>>>>> allow us to release independently for new cool > >>>>>>> extensions/fixes > >>>>>>>>>> apart > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> the framework itself. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Would be great to hear others thoughts if they too > >> feel it > >>>>>> is > >>>>>>>> time > >>>>>>>>>> to > >>>>>>>>>>>>>>> make > >>>>>>>>>>>>>>>> this happen. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>> Joe > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > >
