Long term I would also like to see #3 be the solution. I think what Joseph N described could be part of the capabilities of #3.
I would like to add a note of caution with respect to reorganizing and releasing extension bundles separately: - the burden on release manager expands because many more projects have to be released; probably not all on each release cycle but it could still be many - the chance of accidentally forgetting to release a project in a release cycle becomes non-zero - sharing code between projects gets a bit harder because you have to manage releasing projects in a specific order - it becomes harder to find all of the projects that need to change when shared code is added - the simple act of finding code becomes harder ... in which project is that class in? (IDEs like IntelliJ can search in 1 project, but if they search across multiple projects, then I haven't learned how) I used to maintain several nars in separate projects, and recently reorganized them into 1 project (following NiFi's multi-module maven build) and life has become much easier! -- Mike On Fri, Jan 12, 2018 at 4:33 PM, Chris Herrera <[email protected]> wrote: > I very much like the solution proposed by Bryan below. This would allow > for a cleaner docker image as well, while still proving the functionality > as needed. For sure, the extension registry will be great, but in the mean > time this is an adequate mid step. > > Regards, > Chris > > On Jan 12, 2018, 2:52 PM -0600, Bryan Bende <[email protected]>, wrote: > > Long term I'd like to see the extension registry take form and have > > that be the solution (#3). > > > > In the more near term, we could separate all of the NARs, except for > > framework and maybe standard processors & services, into a separate > > git repo. > > > > In that new git repo we could organize things like Joe N just > > described according to some kind of functional grouping. Each of these > > functional bundles could produce its own tar/zip which we can make > > available for download. > > > > That would separate the release cycles between core NiFi and the other > > NARs, and also avoid having any single binary artifact that gets too > > large. > > > > > > > > On Fri, Jan 12, 2018 at 3:43 PM, Joseph Niemiec <[email protected]> > wrote: > > > just a random thought. > > > > > > Drop In Lib packs... All the Hadoop ones in one package for example > that > > > can be added to a slim Nifi install. Another may be for Cloud, or > Database > > > Interactions, Integration (JMS, FTP, etc) of course defining these > groups > > > would be the tricky part... Or perhaps some type of installer which > allows > > > you to elect which packages to download to add to the slim install? > > > > > > > > > On Fri, Jan 12, 2018 at 3:10 PM, Joe Witt <[email protected]> wrote: > > > > > > > Team, > > > > > > > > The NiFi convenience binary (tar.gz/zip) size has grown to 1.1GB now > > > > in the latest release. Apache infra expanded it to 1.6GB allowance > > > > for us but has stated this is the last time. > > > > https://issues.apache.org/jira/browse/INFRA-15816 > > > > > > > > We need consider: > > > > 1) removing old nars/less commonly used nars/or particularly massive > > > > nars from the assembly we distribute by default. Folks can still use > > > > these things if they want just not from our convenience binary > > > > 2) collapsing nars with highly repeating deps > > > > 3) Getting the extension registry baked into the Flow Registry then > > > > moving to separate releases for extension bundles. The main release > > > > then would be just the NiFi framework. > > > > > > > > Any other ideas ? > > > > > > > > I'll plan to start identifying candiates for removal soon. > > > > > > > > Thanks > > > > Joe > > > > > > > > > > > > > > > > -- > > > Joseph >
