Given the time needed to do the other options "right," the removal of the optional NARs seems most prudent to avoid making the release of 1.10 unduly protracted.
The listing above seems quite reasonable. We could still have these items be built but just not included in the resulting assembly. This would allow excluded NAR artifacts to make their way to/be accessible from repositories (e.g. nifi-media-nar: https://search.maven.org/artifact/org.apache.nifi/nifi-media-nar/1.9.2/nar) . Accordingly, with some notes on migration, it would still be possible for users to customize their distribution with a few curl/wget commands and not be required to build anything. This is likely the experience that extension registry will eventually provide in a more user friendly/integrated manner but helps us bridge the gap in the interim. On Thu, Aug 22, 2019 at 9:15 AM Bryan Bende <[email protected]> wrote: > I think putting all the NARs in an extension registry where users can > pick and choose what they want will probably achieve the same thing. > Either of those options require hosting binaries somewhere and making > them accessible to be downloaded into a base installation. > > For the short term, I think we should re-focus on which NARs to make > optional for 1.10.0. So far proposed options are... > > - nifi-flume-nar (32 MB) > - nifi-other-graph-services-nar (40 MB) > - nifi-kafka-0-8-nar (21 MB) > - nifi-media-nar (63 MB) > > That would give us roughly 150 MB back. > > If we want to keep going, I'm wondering if the Druid NAR would be > another candidate (43 MB). > > On Thu, Aug 22, 2019 at 4:32 AM Edward Armes <[email protected]> > wrote: > > > > Could we potentially look at removing the embedded external dependences > for > > the nars and have them resolve on first start up using an embed manger > like > > what spark jobs and some of the OSGI frameworks do? > > > > Edward > > > > On Thu, 22 Aug 2019, 04:48 Bryan Bende, <[email protected]> wrote: > > > > > I would vote for doing the reorganization after this release because > on its > > > own it’s doesn’t really solver this specific problem. We would be > moving > > > all the NARs to another git repo, but still need a way to distribute > them. > > > Some of the options would be... > > > > > > A) Pick and choose only some of them to be included in the standard > > > distribution (same thing we are doing now and doesn’t require > > > reorganization). > > > > > > B) Create a couple of different NAR assemblies that each on their own > don’t > > > exceed the size limit, and could easily be dropped into the base > > > distribution (In previous discussions no one could agree on how to do > > > this). > > > > > > C) Release each NAR independently to a central extension registry to be > > > easily installed to anyone’s NiFi (ultimate goal but doesn’t exist > yet). > > > > > > On Wed, Aug 21, 2019 at 10:13 PM Jeff <[email protected]> wrote: > > > > > > > Realistically we'll have to do the reorganization and removal of > optional > > > > NARs. Regardless of the reorganization, the convenience binary has > grown > > > > too large. Given that, it probably makes the most sense to pull out > an > > > > initial set of optional NARs from the convenience binary this > release, > > > and > > > > implement the reorganization in the following release. > > > > > > > > On Wed, Aug 21, 2019 at 10:06 PM Mike Thomsen < > [email protected]> > > > > wrote: > > > > > > > > > My impression was that the reorganization initiative has enough > steps > > > > that > > > > > it might actually be wise for us to at least take on a chunk of > them > > > now > > > > so > > > > > we don't end up with a release that suddenly feels to NiFi users > the > > > way > > > > > Java 9 felt to Java 7 and 8 users. > > > > > > > > > > On Wed, Aug 21, 2019 at 9:59 PM Jeff <[email protected]> wrote: > > > > > > > > > > > How motivated could we be to do the reorganization of the NiFi > > > > repository > > > > > > before the 1.10.0 release? It sounds like we have a few paths > to get > > > > the > > > > > > resulting convenience binary down below the size limit. If we > don't > > > do > > > > > the > > > > > > reorganization for this upcoming release, we should make it a top > > > > > priority > > > > > > for the following release. > > > > > > > > > > > > On Wed, Aug 21, 2019 at 6:22 PM Mike Thomsen < > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Another factor on why that would be a good idea: I might soon > have > > > to > > > > > > pivot > > > > > > > and do the R&D on adding dgraph support to graph bundle. So > it's > > > not > > > > > > > altogether unlikely that it might need to be refactored to make > > > room > > > > > for > > > > > > > other graph tech. > > > > > > > > > > > > > > On Wed, Aug 21, 2019 at 6:20 PM Mike Thomsen < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Go ahead and remove the whole graph bundle from the > assembly. I > > > > would > > > > > > > > recommend cutting a release of it separately and putting it > up on > > > > > > > GitHub's > > > > > > > > releases listing if that's a possibility for us/INFRA w/ > GitHub. > > > > Most > > > > > > of > > > > > > > > our potential graph users are savvy enough that if add a few > > > > steps, I > > > > > > > don't > > > > > > > > see it causing any grief on them getting it stood up and > giving > > > us > > > > > > > feedback. > > > > > > > > > > > > > > > > Might be a good idea also to add a "full-build" profile to > the > > > > > assembly > > > > > > > so > > > > > > > > that we can throw the whole kitchen sink into an unofficial > build > > > > if > > > > > we > > > > > > > > build it ourselves for someone else. > > > > > > > > > > > > > > > > On Wed, Aug 21, 2019 at 3:09 PM Joe Witt <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > >> Bryan > > > > > > > >> > > > > > > > >> I agree with all of that. What does that get us to? > > > > > > > >> > > > > > > > >> Thanks > > > > > > > >> > > > > > > > >> On Wed, Aug 21, 2019 at 3:03 PM Bryan Bende < > [email protected]> > > > > > wrote: > > > > > > > >> > > > > > > > >> > I would vote to make nifi-flume-nar optional, and it looks > > > like > > > > > > > >> > nifi-other-graph-services-nar might be new since last > release, > > > > so > > > > > > > >> > since that is in the top 10 and not released yet, it might > > > also > > > > > be a > > > > > > > >> > good candidate (not downplaying the usefulness of > anything in > > > > that > > > > > > > >> > NAR). > > > > > > > >> > > > > > > > > >> > I would also think we could consider the > nifi-kafka-0-8-nar > > > > since > > > > > > > >> > Kafka 0.8 is quite old at this point, and we already have > > > other > > > > > > Kafka > > > > > > > >> > NARs for 0.9, 0.10, 0.11, 1.0, and 2.0. Might even > consider > > > > > > dropping a > > > > > > > >> > few more versions from default assembly. > > > > > > > >> > > > > > > > > >> > On Wed, Aug 21, 2019 at 2:45 PM Aldrin Piri < > > > [email protected]> > > > > > > > wrote: > > > > > > > >> > > > > > > > > > >> > > Hi folks, > > > > > > > >> > > > > > > > > > >> > > Doing a recent PR review and build, it seems that > master has > > > > > > amassed > > > > > > > >> some > > > > > > > >> > > additional size since our 1.9.2 release approaching > 200MB. > > > > > > > >> > > > > > > > > > >> > > Unfortunately, this is problematic and needs to be > addressed > > > > in > > > > > > > >> advance > > > > > > > >> > of > > > > > > > >> > > our 1.10 release. INFRA has been more than helpful > making > > > one > > > > > off > > > > > > > >> > > exceptions [1][2] for the larger assembly to get > published > > > to > > > > > the > > > > > > > ASF > > > > > > > >> > > repository and its associated mirrors, but another > release > > > > that > > > > > is > > > > > > > >> even > > > > > > > >> > > larger is not something we can allow. In a Linux > > > environment, > > > > > the > > > > > > > >> master > > > > > > > >> > > build reports in at 1575671276 which puts us over the > hard > > > > limit > > > > > > > >> > > highlighted in [2]. > > > > > > > >> > > > > > > > > > >> > > We had a prior community discussion [3] about splitting > the > > > > > > > framework > > > > > > > >> and > > > > > > > >> > > extension repos and I am hoping to revive that > discussion, > > > in > > > > > > part. > > > > > > > >> We > > > > > > > >> > > certainly know what our longer term goals and ambitions > are > > > > but > > > > > > > need a > > > > > > > >> > fix > > > > > > > >> > > in the interim. In the current state, we will not be > able > > > to > > > > > make > > > > > > > our > > > > > > > >> > > convenience binaries available at the conclusion of the > > > > release > > > > > > > >> process. > > > > > > > >> > > > > > > > > > >> > > At minimum we should evaluate which bundles are > eligible to > > > > get > > > > > > > >> treated > > > > > > > >> > as > > > > > > > >> > > optional dependencies and only enabled via profile, much > > > like > > > > > the > > > > > > > work > > > > > > > >> > that > > > > > > > >> > > has occurred surrounding some of our other, hefty NARs. > [4] > > > A > > > > > > > listing > > > > > > > >> of > > > > > > > >> > > the top 50 largest NARs, excluding framework and > standard, > > > is > > > > > > > >> available > > > > > > > >> > in > > > > > > > >> > > a gist [5]. The nifi-media-nar looks to be a good > initial > > > > > > candidate > > > > > > > >> for > > > > > > > >> > > exclusion. > > > > > > > >> > > > > > > > > > >> > > Thanks for your consideration! > > > > > > > >> > > > > > > > > > >> > > --aldrin > > > > > > > >> > > > > > > > > > >> > > [1] https://issues.apache.org/jira/browse/INFRA-11252 > > > > > > > >> > > [2] https://issues.apache.org/jira/browse/INFRA-15816 > > > > > > > >> > > [3] > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/939a7630a2e32594cd10444e48b7a1321fd9ce51834d911a8c04b6a9@ > > > > > > > >> > <dev.nifi.apache.org> > > > > > > > >> > > [4] > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/nifi/blob/master/nifi-assembly/pom.xml#L807-L875 > > > > > > > >> > > [5] > > > > > > https://gist.github.com/apiri/4d9a02f9f6b46867b601956df83b6d8c > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > > > >
