I agree... Long term extension registry, short term one repo with different assemblies (e.g. standard, slim, analytic, etc...).
Brandon On Sat, Jan 13, 2018 at 1:35 PM Pierre Villard <[email protected]> wrote: > Option #3 also has my preference. But it's probably a good idea to only > keep one git repo and play with the assembly and Maven profiles for the > releases, no? It'd be certainly easier for release management process. But > this decision could also depend on how the option #3 is going to be > implemented I guess. > > 2018-01-13 6:36 GMT-07:00 Joe Witt <[email protected]>: > > > thanks tony! > > > > On Jan 12, 2018 10:48 PM, "Tony Kurc" <[email protected]> wrote: > > > > > I put some of the data I was working with on the wiki - > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+1.5.0+nar+files > > > > > > On Fri, Jan 12, 2018 at 10:28 PM, Jeremy Dyer <[email protected]> > wrote: > > > > > > > So my favorite option is Bryan’s option number “three” of using the > > > > extension registry. Now my thought is do we really need to add > > complexity > > > > and do anything in the mean time or just focus on that? Meaning we > have > > > > roughly 500mb of available capacity today so why don’t we spend those > > man > > > > hours we would spend on getting the second repo up on the extension > > > > registry instead? > > > > > > > > @Bryan do you have thoughts about the deployment of those bars in the > > > > extension registry? Since we won’t be able to build the release > binary > > > > anymore would we still need to create separate repos for the nars or > > > no?? I > > > > have used the registry a little but I’m not 100% sure on your vision > > for > > > > the nars > > > > > > > > - Jeremy Dyer > > > > > > > > Sent from my iPhone > > > > > > > > > On Jan 12, 2018, at 10:18 PM, Tony Kurc <[email protected]> wrote: > > > > > > > > > > I was looking at nar sizes, and thought some data may be helpful. I > > > used > > > > my recent RC1 verification as a basis for getting file sizes, and > just > > > got > > > > the file size for each file in the assembly named "*.nar". I don't > know > > > > whether the images I pasted in will go through, but I made some > > graphs.b > > > > The first is a histogram of nar file size in buckets of 10MB. The > > second > > > > basically is similar to a cumulative distribution, the x axis is the > > > "rank" > > > > of the nar (smallest to largest), and the y-axis is how what fraction > > of > > > > the all the sizes of the nars together are that rank or lower. In > other > > > > words, on the graph, the dot at 60 and ~27 means that the smallest 60 > > > nars > > > > contribute only ~27% of the total. Of note, the standard and > framework > > > nars > > > > are at 83 and 84. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Fri, Jan 12, 2018 at 5:04 PM, Michael Moser < > [email protected]> > > > > wrote: > > > > >> And of course, as I hit <send> I thought of one more thing. > > > > >> > > > > >> We could keep all of the code in 1 git repo (1 project) but the > > > > >> nifi-assembly part of the build could be broken up to build core > > NiFi > > > > >> separately from the tar/zip functional grouping of other NARs. > > > > >> > > > > >> On Fri, Jan 12, 2018 at 5:01 PM, Michael Moser < > [email protected]> > > > > wrote: > > > > >> > > > > >> > Long term I would also like to see #3 be the solution. I think > > what > > > > >> > Joseph N described could be part of the capabilities of #3. > > > > >> > > > > > >> > I would like to add a note of caution with respect to > reorganizing > > > and > > > > >> > releasing extension bundles separately: > > > > >> > > > > > >> > - the burden on release manager expands because many more > > > projects > > > > >> > have to be released; probably not all on each release cycle > but > > > it > > > > could > > > > >> > still be many > > > > >> > - the chance of accidentally forgetting to release a project > > in a > > > > >> > release cycle becomes non-zero > > > > >> > - sharing code between projects gets a bit harder because you > > > have > > > > to > > > > >> > manage releasing projects in a specific order > > > > >> > - it becomes harder to find all of the projects that need to > > > change > > > > >> > when shared code is added > > > > >> > - the simple act of finding code becomes harder ... in which > > > > project > > > > >> > is that class in? (IDEs like IntelliJ can search in 1 > project, > > > but > > > > if they > > > > >> > search across multiple projects, then I haven't learned how) > > > > >> > > > > > >> > I used to maintain several nars in separate projects, and > recently > > > > >> > reorganized them into 1 project (following NiFi's multi-module > > maven > > > > build) > > > > >> > and life has become much easier! > > > > >> > > > > > >> > -- Mike > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Fri, Jan 12, 2018 at 4:33 PM, Chris Herrera < > > > > [email protected]> > > > > >> > wrote: > > > > >> > > > > > >> >> I very much like the solution proposed by Bryan below. This > would > > > > allow > > > > >> >> for a cleaner docker image as well, while still proving the > > > > functionality > > > > >> >> as needed. For sure, the extension registry will be great, but > in > > > > the mean > > > > >> >> time this is an adequate mid step. > > > > >> >> > > > > >> >> Regards, > > > > >> >> Chris > > > > >> >> > > > > >> >> On Jan 12, 2018, 2:52 PM -0600, Bryan Bende <[email protected] > >, > > > > wrote: > > > > >> >> > Long term I'd like to see the extension registry take form > and > > > have > > > > >> >> > that be the solution (#3). > > > > >> >> > > > > > >> >> > In the more near term, we could separate all of the NARs, > > except > > > > for > > > > >> >> > framework and maybe standard processors & services, into a > > > separate > > > > >> >> > git repo. > > > > >> >> > > > > > >> >> > In that new git repo we could organize things like Joe N just > > > > >> >> > described according to some kind of functional grouping. Each > > of > > > > these > > > > >> >> > functional bundles could produce its own tar/zip which we can > > > make > > > > >> >> > available for download. > > > > >> >> > > > > > >> >> > That would separate the release cycles between core NiFi and > > the > > > > other > > > > >> >> > NARs, and also avoid having any single binary artifact that > > gets > > > > too > > > > >> >> > large. > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > On Fri, Jan 12, 2018 at 3:43 PM, Joseph Niemiec < > > > > [email protected]> > > > > >> >> wrote: > > > > >> >> > > just a random thought. > > > > >> >> > > > > > > >> >> > > Drop In Lib packs... All the Hadoop ones in one package for > > > > example > > > > >> >> that > > > > >> >> > > can be added to a slim Nifi install. Another may be for > > Cloud, > > > or > > > > >> >> Database > > > > >> >> > > Interactions, Integration (JMS, FTP, etc) of course > defining > > > > these > > > > >> >> groups > > > > >> >> > > would be the tricky part... Or perhaps some type of > installer > > > > which > > > > >> >> allows > > > > >> >> > > you to elect which packages to download to add to the slim > > > > install? > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > On Fri, Jan 12, 2018 at 3:10 PM, Joe Witt < > > [email protected]> > > > > wrote: > > > > >> >> > > > > > > >> >> > > > Team, > > > > >> >> > > > > > > > >> >> > > > The NiFi convenience binary (tar.gz/zip) size has grown > to > > > > 1.1GB now > > > > >> >> > > > in the latest release. Apache infra expanded it to 1.6GB > > > > allowance > > > > >> >> > > > for us but has stated this is the last time. > > > > >> >> > > > https://issues.apache.org/jira/browse/INFRA-15816 > > > > >> >> > > > > > > > >> >> > > > We need consider: > > > > >> >> > > > 1) removing old nars/less commonly used nars/or > > particularly > > > > massive > > > > >> >> > > > nars from the assembly we distribute by default. Folks > can > > > > still use > > > > >> >> > > > these things if they want just not from our convenience > > > binary > > > > >> >> > > > 2) collapsing nars with highly repeating deps > > > > >> >> > > > 3) Getting the extension registry baked into the Flow > > > Registry > > > > then > > > > >> >> > > > moving to separate releases for extension bundles. The > main > > > > release > > > > >> >> > > > then would be just the NiFi framework. > > > > >> >> > > > > > > > >> >> > > > Any other ideas ? > > > > >> >> > > > > > > > >> >> > > > I'll plan to start identifying candiates for removal > soon. > > > > >> >> > > > > > > > >> >> > > > Thanks > > > > >> >> > > > Joe > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > -- > > > > >> >> > > Joseph > > > > >> >> > > > > >> > > > > > >> > > > > > > > > > > > > > > > >
