Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Joe Witt]

Mike

Yep - that will need to get addressed/improved as well.  First step is
get the root classpath cleaned up again (NIFI-2971) and the next step
is to understand and address the excessive declaration of dependencies
which leads to duplication on such a scale.

Thanks
Joe

On Tue, Nov 1, 2016 at 12:15 PM, Michael Moser  wrote:
> I would like to point out that some very common dependencies have been
> placed into the nifi-standard-services-api-nar.  This nar is a
> Nar-Dependency-Id of several other nars.  Many of these other nars also
> include the same dependencies, which would not be used by the nar class
> loader.
>
> In particular, nifi-standard-services-api-nar contains nifi-utils,
> nifi-processor-utils, and nifi-security-utils.  It also contains the
> bcprov-jdk15on and bcpkix-jdk15on jars.
>
> Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and
> more, they also contain these same jars in their bundled-dependencies.
> Their Nar-Dependency-Id is nifi-standard-services-api-nar.
>
> -- Mike
>
>
> On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri  wrote:
>
>> I think in light of the fact that the bcprov item was an accidental
>> inclusion and not explicitly meant to be handled as such in the framework
>> reorganizing seems like the best option as well.  The original intent of
>> the PR was to minimize that footprint where we were double dipping given
>> the inclusion of nifi-processor-utils.  However, if we manage to shuffle
>> items around a bit, the net size reduction could ultimately be the same.
>> Most importantly, from a maintenance perspective, bundles will only include
>> the library in the NARs that need it which allows that dependency graph to
>> be significantly more comprehensible than the scope tweaking originally
>> presented.
>>
>> Will this also work to adjust the dependencies of nifi-processor-utils (and
>> potentially nifi-security-utils)?  Currently, the archetype also provides
>> nifi-processor-utils, which could lead to similar conflicts for folks that
>> were looking to make use of different versions of JARs within a given NAR.
>>
>> I think the listing Joe provided in the ticket is great and we should be
>> mindful of any deltas while performing each release.  For the reasons
>> highlighted in this chain and on that issue, one jar inclusion could be
>> quite detrimental and cause some difficult issues to those that have
>> created extensions.
>>
>> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt  wrote:
>>
>> > So...I might be a bit of a flip flopper...
>> >
>> > Had some off-list discussions with a couple people and they really
>> > disliked this idea.  They, rightfully, pointed out how this
>> > establishes a problematic precedent, how it is really only being done
>> > to save space, and that while perhaps not able to pair down to 200MB
>> > we can likely do more analysis of the dependency graph to improve the
>> > situation.  The real issue that kicked this off was the finding that
>> > bcprov (and commons-lang3) had made their way into the root of our
>> > classloader (which impacts all classloaders) and this was introduced
>> > with the important NIFI-1831 work.  So I've created [1] to address
>> > that and return the application classloader to the minimal set of jars
>> > necessary which should basically be the runtime application launcher,
>> > the logging libraries, and the nifi-api.
>> >
>> > The ultimate solution to addressing the distribution size concerns
>> > will be to address it via the extension registry.
>> >
>> > [1] https://issues.apache.org/jira/browse/NIFI-2971
>> >
>> > Thanks
>> > Joe
>> >
>> > On Mon, Oct 31, 2016 at 8:51 PM, Andre  wrote:
>> > > +1 as well.
>> > >
>> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt  wrote:
>> > >
>> > >> Team,
>> > >>
>> > >> In the 1.x line bcprov made its way into the root of the classpath/lib
>> > >> folder to support the encrypted sensitive properties features.  This
>> > >> can be reworked to isolate it a bit more if we need to.
>> > >>
>> > >> However, in [1] it has been observed that we have bcprov dependencies
>> > >> in about 53 places with each instance taking about 4MB of space for a
>> > >> total hit of about 200MB of bcprov.
>> > >>
>> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
>> > >> core provided list of things right along side the
>> > >> logback/slf4j/logging interfaces. This means all things will have
>> > >> access to these going forward and it means it impacts our version
>> > >> compatibility.  Given that it would be a standard/provided thing if we
>> > >> want to upgrade versions or swap it out with something else we could
>> > >> break extensions that then chose to depend on it.  We are not in
>> > >> control of the bcprov api just like we're not in control of the
>> > >> logging APIs.  If there was some important security related fix we
>> > >> needed from 

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Michael Moser]

I would like to point out that some very common dependencies have been
placed into the nifi-standard-services-api-nar.  This nar is a
Nar-Dependency-Id of several other nars.  Many of these other nars also
include the same dependencies, which would not be used by the nar class
loader.

In particular, nifi-standard-services-api-nar contains nifi-utils,
nifi-processor-utils, and nifi-security-utils.  It also contains the
bcprov-jdk15on and bcpkix-jdk15on jars.

Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and
more, they also contain these same jars in their bundled-dependencies.
Their Nar-Dependency-Id is nifi-standard-services-api-nar.

-- Mike


On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri  wrote:

> I think in light of the fact that the bcprov item was an accidental
> inclusion and not explicitly meant to be handled as such in the framework
> reorganizing seems like the best option as well.  The original intent of
> the PR was to minimize that footprint where we were double dipping given
> the inclusion of nifi-processor-utils.  However, if we manage to shuffle
> items around a bit, the net size reduction could ultimately be the same.
> Most importantly, from a maintenance perspective, bundles will only include
> the library in the NARs that need it which allows that dependency graph to
> be significantly more comprehensible than the scope tweaking originally
> presented.
>
> Will this also work to adjust the dependencies of nifi-processor-utils (and
> potentially nifi-security-utils)?  Currently, the archetype also provides
> nifi-processor-utils, which could lead to similar conflicts for folks that
> were looking to make use of different versions of JARs within a given NAR.
>
> I think the listing Joe provided in the ticket is great and we should be
> mindful of any deltas while performing each release.  For the reasons
> highlighted in this chain and on that issue, one jar inclusion could be
> quite detrimental and cause some difficult issues to those that have
> created extensions.
>
> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt  wrote:
>
> > So...I might be a bit of a flip flopper...
> >
> > Had some off-list discussions with a couple people and they really
> > disliked this idea.  They, rightfully, pointed out how this
> > establishes a problematic precedent, how it is really only being done
> > to save space, and that while perhaps not able to pair down to 200MB
> > we can likely do more analysis of the dependency graph to improve the
> > situation.  The real issue that kicked this off was the finding that
> > bcprov (and commons-lang3) had made their way into the root of our
> > classloader (which impacts all classloaders) and this was introduced
> > with the important NIFI-1831 work.  So I've created [1] to address
> > that and return the application classloader to the minimal set of jars
> > necessary which should basically be the runtime application launcher,
> > the logging libraries, and the nifi-api.
> >
> > The ultimate solution to addressing the distribution size concerns
> > will be to address it via the extension registry.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-2971
> >
> > Thanks
> > Joe
> >
> > On Mon, Oct 31, 2016 at 8:51 PM, Andre  wrote:
> > > +1 as well.
> > >
> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt  wrote:
> > >
> > >> Team,
> > >>
> > >> In the 1.x line bcprov made its way into the root of the classpath/lib
> > >> folder to support the encrypted sensitive properties features.  This
> > >> can be reworked to isolate it a bit more if we need to.
> > >>
> > >> However, in [1] it has been observed that we have bcprov dependencies
> > >> in about 53 places with each instance taking about 4MB of space for a
> > >> total hit of about 200MB of bcprov.
> > >>
> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
> > >> core provided list of things right along side the
> > >> logback/slf4j/logging interfaces. This means all things will have
> > >> access to these going forward and it means it impacts our version
> > >> compatibility.  Given that it would be a standard/provided thing if we
> > >> want to upgrade versions or swap it out with something else we could
> > >> break extensions that then chose to depend on it.  We are not in
> > >> control of the bcprov api just like we're not in control of the
> > >> logging APIs.  If there was some important security related fix we
> > >> needed from bcprov but changing also pulled in api changes for them it
> > >> could break our extensions.
> > >>
> > >> Even with all this said, given the nature, importance, and size
> > >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
> > >> this in case others have perspective they'd like to share.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/NIFI-2954
> > >>
> > >> Thanks
> > >> Joe
> > >>
> >
>

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Aldrin Piri]

I think in light of the fact that the bcprov item was an accidental
inclusion and not explicitly meant to be handled as such in the framework
reorganizing seems like the best option as well.  The original intent of
the PR was to minimize that footprint where we were double dipping given
the inclusion of nifi-processor-utils.  However, if we manage to shuffle
items around a bit, the net size reduction could ultimately be the same.
Most importantly, from a maintenance perspective, bundles will only include
the library in the NARs that need it which allows that dependency graph to
be significantly more comprehensible than the scope tweaking originally
presented.

Will this also work to adjust the dependencies of nifi-processor-utils (and
potentially nifi-security-utils)?  Currently, the archetype also provides
nifi-processor-utils, which could lead to similar conflicts for folks that
were looking to make use of different versions of JARs within a given NAR.

I think the listing Joe provided in the ticket is great and we should be
mindful of any deltas while performing each release.  For the reasons
highlighted in this chain and on that issue, one jar inclusion could be
quite detrimental and cause some difficult issues to those that have
created extensions.

On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt  wrote:

> So...I might be a bit of a flip flopper...
>
> Had some off-list discussions with a couple people and they really
> disliked this idea.  They, rightfully, pointed out how this
> establishes a problematic precedent, how it is really only being done
> to save space, and that while perhaps not able to pair down to 200MB
> we can likely do more analysis of the dependency graph to improve the
> situation.  The real issue that kicked this off was the finding that
> bcprov (and commons-lang3) had made their way into the root of our
> classloader (which impacts all classloaders) and this was introduced
> with the important NIFI-1831 work.  So I've created [1] to address
> that and return the application classloader to the minimal set of jars
> necessary which should basically be the runtime application launcher,
> the logging libraries, and the nifi-api.
>
> The ultimate solution to addressing the distribution size concerns
> will be to address it via the extension registry.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2971
>
> Thanks
> Joe
>
> On Mon, Oct 31, 2016 at 8:51 PM, Andre  wrote:
> > +1 as well.
> >
> > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt  wrote:
> >
> >> Team,
> >>
> >> In the 1.x line bcprov made its way into the root of the classpath/lib
> >> folder to support the encrypted sensitive properties features.  This
> >> can be reworked to isolate it a bit more if we need to.
> >>
> >> However, in [1] it has been observed that we have bcprov dependencies
> >> in about 53 places with each instance taking about 4MB of space for a
> >> total hit of about 200MB of bcprov.
> >>
> >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
> >> core provided list of things right along side the
> >> logback/slf4j/logging interfaces. This means all things will have
> >> access to these going forward and it means it impacts our version
> >> compatibility.  Given that it would be a standard/provided thing if we
> >> want to upgrade versions or swap it out with something else we could
> >> break extensions that then chose to depend on it.  We are not in
> >> control of the bcprov api just like we're not in control of the
> >> logging APIs.  If there was some important security related fix we
> >> needed from bcprov but changing also pulled in api changes for them it
> >> could break our extensions.
> >>
> >> Even with all this said, given the nature, importance, and size
> >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
> >> this in case others have perspective they'd like to share.
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-2954
> >>
> >> Thanks
> >> Joe
> >>
>

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply Joe,

As matt suggested we moved to groovy. 

Thanks
Bala



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply matt, 
Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking
more than 2 seconds to execute below high level logic

The logic: 
1) Split data from flowfile
2) Query influxdb with splitted data (influxdb jar file used/external
dependency)
3) write to influxdb
4) write flow file attributes

Without influxdb calling, the code is fast. Can you suggest me about
improving the performance of the below mentioned groovy script



Thanks
Bala




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: FlowFile Package Spec

[Joe Gresock]

The Tar v1 format is actually quite nice for debugging purposes, since you
can actually edit flow files in place on disk with VI, whereas the other
flow file types cannot be easily edited once on disk.  Our team has used
this to quickly tweak and resubmit erroneous flow files that we capture on
disk.

Not that it should not be deprecated, but I thought I'd bring it up since
it might be nice to capture this feature in some other way if it is
deprecated.

On Tue, Nov 1, 2016 at 2:43 AM, Joe Witt  wrote:

> Nick,
>
> They aren't documented anywhere that I know of and were really only
> there for compatibility to versions of NiFi previous to its open
> sourcing.  They should probably be deprecated.
>
> The streamV3 is the latest version and we should probably document
> that.  Were you thinking of using that format to interact with flow
> files outside of NiFi?  Our site-to-site client library would have
> support for that built-in.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 5:42 PM, Nicholas Hughes
>  wrote:
> > Are the FlowFile "package" versions (stream v3, stream v2, TAR v1) used
> in
> > the MergeContent and UnpackContent processors documented somewhere? I
> > checked the Docs and Wiki (briefly) and didn't find anything. Could be my
> > search isn't worded properly. The TAR is pretty straightforward, but I'd
> > like to see something describing the composition of the stream formats.
> >
> > -Nick
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*