I would like to point out that some very common dependencies have been placed into the nifi-standard-services-api-nar. This nar is a Nar-Dependency-Id of several other nars. Many of these other nars also include the same dependencies, which would not be used by the nar class loader.
In particular, nifi-standard-services-api-nar contains nifi-utils, nifi-processor-utils, and nifi-security-utils. It also contains the bcprov-jdk15on and bcpkix-jdk15on jars. Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and more, they also contain these same jars in their bundled-dependencies. Their Nar-Dependency-Id is nifi-standard-services-api-nar. -- Mike On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri <[email protected]> wrote: > I think in light of the fact that the bcprov item was an accidental > inclusion and not explicitly meant to be handled as such in the framework > reorganizing seems like the best option as well. The original intent of > the PR was to minimize that footprint where we were double dipping given > the inclusion of nifi-processor-utils. However, if we manage to shuffle > items around a bit, the net size reduction could ultimately be the same. > Most importantly, from a maintenance perspective, bundles will only include > the library in the NARs that need it which allows that dependency graph to > be significantly more comprehensible than the scope tweaking originally > presented. > > Will this also work to adjust the dependencies of nifi-processor-utils (and > potentially nifi-security-utils)? Currently, the archetype also provides > nifi-processor-utils, which could lead to similar conflicts for folks that > were looking to make use of different versions of JARs within a given NAR. > > I think the listing Joe provided in the ticket is great and we should be > mindful of any deltas while performing each release. For the reasons > highlighted in this chain and on that issue, one jar inclusion could be > quite detrimental and cause some difficult issues to those that have > created extensions. > > On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <[email protected]> wrote: > > > So...I might be a bit of a flip flopper... > > > > Had some off-list discussions with a couple people and they really > > disliked this idea. They, rightfully, pointed out how this > > establishes a problematic precedent, how it is really only being done > > to save space, and that while perhaps not able to pair down to 200MB > > we can likely do more analysis of the dependency graph to improve the > > situation. The real issue that kicked this off was the finding that > > bcprov (and commons-lang3) had made their way into the root of our > > classloader (which impacts all classloaders) and this was introduced > > with the important NIFI-1831 work. So I've created [1] to address > > that and return the application classloader to the minimal set of jars > > necessary which should basically be the runtime application launcher, > > the logging libraries, and the nifi-api. > > > > The ultimate solution to addressing the distribution size concerns > > will be to address it via the extension registry. > > > > [1] https://issues.apache.org/jira/browse/NIFI-2971 > > > > Thanks > > Joe > > > > On Mon, Oct 31, 2016 at 8:51 PM, Andre <[email protected]> wrote: > > > +1 as well. > > > > > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <[email protected]> wrote: > > > > > >> Team, > > >> > > >> In the 1.x line bcprov made its way into the root of the classpath/lib > > >> folder to support the encrypted sensitive properties features. This > > >> can be reworked to isolate it a bit more if we need to. > > >> > > >> However, in [1] it has been observed that we have bcprov dependencies > > >> in about 53 places with each instance taking about 4MB of space for a > > >> total hit of about 200MB of bcprov. > > >> > > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the > > >> core provided list of things right along side the > > >> logback/slf4j/logging interfaces. This means all things will have > > >> access to these going forward and it means it impacts our version > > >> compatibility. Given that it would be a standard/provided thing if we > > >> want to upgrade versions or swap it out with something else we could > > >> break extensions that then chose to depend on it. We are not in > > >> control of the bcprov api just like we're not in control of the > > >> logging APIs. If there was some important security related fix we > > >> needed from bcprov but changing also pulled in api changes for them it > > >> could break our extensions. > > >> > > >> Even with all this said, given the nature, importance, and size > > >> benefit, I am in favor of NIFI-2954. But, would like to highlight > > >> this in case others have perspective they'd like to share. > > >> > > >> [1] https://issues.apache.org/jira/browse/NIFI-2954 > > >> > > >> Thanks > > >> Joe > > >> > > >
