Re: Need help with Custom Processor

[Andrew Psaltis]

Hi Pushkar,
The preferred approach is to add the external dependency to the pom file
for your processor. After doing that when you build your nar bundle, the
resulting nar file will contain all of the 3rd party dependencies required
by your processor. Can you double check that you are indeed including it as
a maven dependency?

Thanks,
Andrew


On Tue, Feb 28, 2017 at 12:31 PM, Pushkara R  wrote:

> Hi,
>
> I'm writing a new processor for NiFi which has a dependency on an external
> library. When I build the NAR and add it to the lib folder and restart
> NiFi, the external library classes that my processor depends on fail
> getting loaded with a "ClassNotFound" exception.
>
> How do I go about fixing this issue? Do I add the jar for the external
> library separately in the lib folder of NiFi? Or is there something else to
> it?
>
> Thanks
> Pushkar
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

Need help with Custom Processor

[Pushkara R]

Hi,

I'm writing a new processor for NiFi which has a dependency on an external
library. When I build the NAR and add it to the lib folder and restart
NiFi, the external library classes that my processor depends on fail
getting loaded with a "ClassNotFound" exception.

How do I go about fixing this issue? Do I add the jar for the external
library separately in the lib folder of NiFi? Or is there something else to
it?

Thanks
Pushkar

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

It may already be in the Licensing Guide (
https://nifi.apache.org/licensing-guide.html).  I'll have to read it again
to identify net-new material.  I'll open a PR if there is.

Thanks,

James

On Mon, Feb 27, 2017 at 4:23 PM, Tony Kurc  wrote:

> Joe,
> This is awesome and useful - should this guidance go on the wiki somewhere?
>
> On Mon, Feb 27, 2017 at 4:59 PM, Joe Witt  wrote:
>
> > 1)  All nars once built do need to contain a LICENSE and NOTICE file
> > to cover what ends up in them as an archive of binary dependencies and
> > also it should cover any specific source dependencies they might have
> > (like MIT javascript libs in nifi-web-ui).
> >
> > 2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
> > will be auto generated if not already present anyway.  What we need
> > them to cover is mentioned in #1.
> >
> > Also, for any source or binary dependency in a given nar we must also
> > ensure all source dependencies are captured appropriately in the top
> > level nifi.git/LICENSE & NOTICE and any source & binary dependencies
> > are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.
> >
> > As we progress toward separating the application from the nars by way
> > of some extension registry we'll be able to have a far smaller/simple
> > LICENSE and NOTICE at the top level but the above nar L
> > considerations will still apply per L
> >
> > Does that help at all James?
> >
> > On Mon, Feb 27, 2017 at 4:31 PM, James Wing  wrote:
> > > Thank you both for bringing up this discussion.  I have a few follow-up
> > > questions:
> > >
> > > 1.) Is it true all of the NAR bundles in the NiFi source should have
> > their
> > > own LICENSE and NOTICE files, without exception?  In looking through
> the
> > > source, most nifi-*-nar projects have both files for binary
> dependencies.
> > > I understand these binary dependencies should roll up to nifi-assembly
> > > LICENSE/NOTICE.
> > >
> > > 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle
> > projects
> > > unless they have distinctive source dependency terms that roll up to
> the
> > > root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so
> > far.  I
> > > assume the ASLv2 file headers cover most of the source.
> > >
> > > 3.) Where dependencies also distinguish between their source
> > LICENSE/NOTICE
> > > and binary LICENSE/NOTICE, should we match to our dependency
> > relationship?
> > > For example, a binary dependency would result in the inclusion of the
> > > binary NOTICE terms?
> > >
> > >
> > > Thanks,
> > >
> > > James
> > >
> > > On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri 
> > wrote:
> > >
> > >> Hey Andre,
> > >>
> > >> I may be off, but to help you along, I will give you my take on things
> > to
> > >> the best of my understanding.  If there are any wrong points, I hope
> > >> someone can further clarify.
> > >>
> > >> For your case, it looks like simply you are simply using binary
> > >> dependencies.  For that case, you have to consider where these items
> are
> > >> showing up and how they are released.  Your inclusion of a dependency
> > will
> > >> affect the generated NAR (nifi-irc-processors-nar) and, while it seems
> > to
> > >> be missing in the current PR, the zip and tgz nifi-assembly artifacts.
> > You
> > >> shouldn't need to include it in levels lower than this assuming you
> are
> > >> talking about JARs that compose the overall NAR.  While you are
> linking
> > >> these against dependencies, you are not explicitly bringing them into
> > the
> > >> project through the JARs incorporated in the NAR.
> > >>
> > >> Source inclusions are handled similarly but do go a level deeper as
> they
> > >> are also bundled with the JARs and are present in the root LICENSE and
> > >> NOTICE where applicable.  Again, both are for similar reasons with the
> > >> generated source package and JARs bundling this work including the
> > source.
> > >>
> > >> Do keep in mind the transitive dependencies.  Looking quickly through
> > the
> > >> pom for kitteh, I see usage of some netty libraries as well as
> > mbassador.
> > >> These would presumably also be collected upon building the NAR.
> > >>
> > >> Of course, the docs we have on the site are quite nice if you need
> some
> > >> light reading material ;)  https://nifi.apache.org/
> licensing-guide.html
> > >> Both the guide and the links from it are good information and a nice
> > >> reference to revisit when working through these things.
> > >>
> > >> Let us know if there are additional questions or if some additional
> > >> clarification is needed.  Hopefully my anecdotal thoughts are both
> > somewhat
> > >> helpful and mostly correct!
> > >>
> > >> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami <
> murakam...@icloud.com
> > >
> > >> wrote:
> > >>
> > >> > I just start and I really don’t know much so let see what I can
> learn
> > >> when
> > >> > time pass by 

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[Tony Kurc]

Joe,
This is awesome and useful - should this guidance go on the wiki somewhere?

On Mon, Feb 27, 2017 at 4:59 PM, Joe Witt  wrote:

> 1)  All nars once built do need to contain a LICENSE and NOTICE file
> to cover what ends up in them as an archive of binary dependencies and
> also it should cover any specific source dependencies they might have
> (like MIT javascript libs in nifi-web-ui).
>
> 2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
> will be auto generated if not already present anyway.  What we need
> them to cover is mentioned in #1.
>
> Also, for any source or binary dependency in a given nar we must also
> ensure all source dependencies are captured appropriately in the top
> level nifi.git/LICENSE & NOTICE and any source & binary dependencies
> are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.
>
> As we progress toward separating the application from the nars by way
> of some extension registry we'll be able to have a far smaller/simple
> LICENSE and NOTICE at the top level but the above nar L
> considerations will still apply per L
>
> Does that help at all James?
>
> On Mon, Feb 27, 2017 at 4:31 PM, James Wing  wrote:
> > Thank you both for bringing up this discussion.  I have a few follow-up
> > questions:
> >
> > 1.) Is it true all of the NAR bundles in the NiFi source should have
> their
> > own LICENSE and NOTICE files, without exception?  In looking through the
> > source, most nifi-*-nar projects have both files for binary dependencies.
> > I understand these binary dependencies should roll up to nifi-assembly
> > LICENSE/NOTICE.
> >
> > 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle
> projects
> > unless they have distinctive source dependency terms that roll up to the
> > root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so
> far.  I
> > assume the ASLv2 file headers cover most of the source.
> >
> > 3.) Where dependencies also distinguish between their source
> LICENSE/NOTICE
> > and binary LICENSE/NOTICE, should we match to our dependency
> relationship?
> > For example, a binary dependency would result in the inclusion of the
> > binary NOTICE terms?
> >
> >
> > Thanks,
> >
> > James
> >
> > On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri 
> wrote:
> >
> >> Hey Andre,
> >>
> >> I may be off, but to help you along, I will give you my take on things
> to
> >> the best of my understanding.  If there are any wrong points, I hope
> >> someone can further clarify.
> >>
> >> For your case, it looks like simply you are simply using binary
> >> dependencies.  For that case, you have to consider where these items are
> >> showing up and how they are released.  Your inclusion of a dependency
> will
> >> affect the generated NAR (nifi-irc-processors-nar) and, while it seems
> to
> >> be missing in the current PR, the zip and tgz nifi-assembly artifacts.
> You
> >> shouldn't need to include it in levels lower than this assuming you are
> >> talking about JARs that compose the overall NAR.  While you are linking
> >> these against dependencies, you are not explicitly bringing them into
> the
> >> project through the JARs incorporated in the NAR.
> >>
> >> Source inclusions are handled similarly but do go a level deeper as they
> >> are also bundled with the JARs and are present in the root LICENSE and
> >> NOTICE where applicable.  Again, both are for similar reasons with the
> >> generated source package and JARs bundling this work including the
> source.
> >>
> >> Do keep in mind the transitive dependencies.  Looking quickly through
> the
> >> pom for kitteh, I see usage of some netty libraries as well as
> mbassador.
> >> These would presumably also be collected upon building the NAR.
> >>
> >> Of course, the docs we have on the site are quite nice if you need some
> >> light reading material ;)  https://nifi.apache.org/licensing-guide.html
> >> Both the guide and the links from it are good information and a nice
> >> reference to revisit when working through these things.
> >>
> >> Let us know if there are additional questions or if some additional
> >> clarification is needed.  Hopefully my anecdotal thoughts are both
> somewhat
> >> helpful and mostly correct!
> >>
> >> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami  >
> >> wrote:
> >>
> >> > I just start and I really don’t know much so let see what I can learn
> >> when
> >> > time pass by and hope I can learn as much as you, thank’s
> >> > > On Feb 25, 2017, at 5:12 PM, Andre  wrote:
> >> > >
> >> > > Hi there,
> >> > >
> >> > > Quick question on proper licensing:
> >> > >
> >> > > When bundling Processors, Services and APIs, where should the
> NOTICES
> >> and
> >> > > LICENSES be added to?
> >> > >
> >> > > The PR in question is https://github.com/apache/nifi/pull/1541
> >> > >
> >> > > My current reading is that all NAR levels will have to include the
> >> 

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

Yes, thank you, that does help.  I'm slowly sneaking up on an understanding
of how it works.

James

On Mon, Feb 27, 2017 at 1:59 PM, Joe Witt  wrote:

> 1)  All nars once built do need to contain a LICENSE and NOTICE file
> to cover what ends up in them as an archive of binary dependencies and
> also it should cover any specific source dependencies they might have
> (like MIT javascript libs in nifi-web-ui).
>
> 2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
> will be auto generated if not already present anyway.  What we need
> them to cover is mentioned in #1.
>
> Also, for any source or binary dependency in a given nar we must also
> ensure all source dependencies are captured appropriately in the top
> level nifi.git/LICENSE & NOTICE and any source & binary dependencies
> are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.
>
> As we progress toward separating the application from the nars by way
> of some extension registry we'll be able to have a far smaller/simple
> LICENSE and NOTICE at the top level but the above nar L
> considerations will still apply per L
>
> Does that help at all James?
>
> On Mon, Feb 27, 2017 at 4:31 PM, James Wing  wrote:
> > Thank you both for bringing up this discussion.  I have a few follow-up
> > questions:
> >
> > 1.) Is it true all of the NAR bundles in the NiFi source should have
> their
> > own LICENSE and NOTICE files, without exception?  In looking through the
> > source, most nifi-*-nar projects have both files for binary dependencies.
> > I understand these binary dependencies should roll up to nifi-assembly
> > LICENSE/NOTICE.
> >
> > 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle
> projects
> > unless they have distinctive source dependency terms that roll up to the
> > root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so
> far.  I
> > assume the ASLv2 file headers cover most of the source.
> >
> > 3.) Where dependencies also distinguish between their source
> LICENSE/NOTICE
> > and binary LICENSE/NOTICE, should we match to our dependency
> relationship?
> > For example, a binary dependency would result in the inclusion of the
> > binary NOTICE terms?
> >
> >
> > Thanks,
> >
> > James
> >
> > On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri 
> wrote:
> >
> >> Hey Andre,
> >>
> >> I may be off, but to help you along, I will give you my take on things
> to
> >> the best of my understanding.  If there are any wrong points, I hope
> >> someone can further clarify.
> >>
> >> For your case, it looks like simply you are simply using binary
> >> dependencies.  For that case, you have to consider where these items are
> >> showing up and how they are released.  Your inclusion of a dependency
> will
> >> affect the generated NAR (nifi-irc-processors-nar) and, while it seems
> to
> >> be missing in the current PR, the zip and tgz nifi-assembly artifacts.
> You
> >> shouldn't need to include it in levels lower than this assuming you are
> >> talking about JARs that compose the overall NAR.  While you are linking
> >> these against dependencies, you are not explicitly bringing them into
> the
> >> project through the JARs incorporated in the NAR.
> >>
> >> Source inclusions are handled similarly but do go a level deeper as they
> >> are also bundled with the JARs and are present in the root LICENSE and
> >> NOTICE where applicable.  Again, both are for similar reasons with the
> >> generated source package and JARs bundling this work including the
> source.
> >>
> >> Do keep in mind the transitive dependencies.  Looking quickly through
> the
> >> pom for kitteh, I see usage of some netty libraries as well as
> mbassador.
> >> These would presumably also be collected upon building the NAR.
> >>
> >> Of course, the docs we have on the site are quite nice if you need some
> >> light reading material ;)  https://nifi.apache.org/licensing-guide.html
> >> Both the guide and the links from it are good information and a nice
> >> reference to revisit when working through these things.
> >>
> >> Let us know if there are additional questions or if some additional
> >> clarification is needed.  Hopefully my anecdotal thoughts are both
> somewhat
> >> helpful and mostly correct!
> >>
> >> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami  >
> >> wrote:
> >>
> >> > I just start and I really don’t know much so let see what I can learn
> >> when
> >> > time pass by and hope I can learn as much as you, thank’s
> >> > > On Feb 25, 2017, at 5:12 PM, Andre  wrote:
> >> > >
> >> > > Hi there,
> >> > >
> >> > > Quick question on proper licensing:
> >> > >
> >> > > When bundling Processors, Services and APIs, where should the
> NOTICES
> >> and
> >> > > LICENSES be added to?
> >> > >
> >> > > The PR in question is https://github.com/apache/nifi/pull/1541
> >> > >
> >> > > My current reading is that all NAR levels will have to 

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[Joe Witt]

1)  All nars once built do need to contain a LICENSE and NOTICE file
to cover what ends up in them as an archive of binary dependencies and
also it should cover any specific source dependencies they might have
(like MIT javascript libs in nifi-web-ui).

2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
will be auto generated if not already present anyway.  What we need
them to cover is mentioned in #1.

Also, for any source or binary dependency in a given nar we must also
ensure all source dependencies are captured appropriately in the top
level nifi.git/LICENSE & NOTICE and any source & binary dependencies
are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.

As we progress toward separating the application from the nars by way
of some extension registry we'll be able to have a far smaller/simple
LICENSE and NOTICE at the top level but the above nar L
considerations will still apply per L

Does that help at all James?

On Mon, Feb 27, 2017 at 4:31 PM, James Wing  wrote:
> Thank you both for bringing up this discussion.  I have a few follow-up
> questions:
>
> 1.) Is it true all of the NAR bundles in the NiFi source should have their
> own LICENSE and NOTICE files, without exception?  In looking through the
> source, most nifi-*-nar projects have both files for binary dependencies.
> I understand these binary dependencies should roll up to nifi-assembly
> LICENSE/NOTICE.
>
> 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle projects
> unless they have distinctive source dependency terms that roll up to the
> root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so far.  I
> assume the ASLv2 file headers cover most of the source.
>
> 3.) Where dependencies also distinguish between their source LICENSE/NOTICE
> and binary LICENSE/NOTICE, should we match to our dependency relationship?
> For example, a binary dependency would result in the inclusion of the
> binary NOTICE terms?
>
>
> Thanks,
>
> James
>
> On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri  wrote:
>
>> Hey Andre,
>>
>> I may be off, but to help you along, I will give you my take on things to
>> the best of my understanding.  If there are any wrong points, I hope
>> someone can further clarify.
>>
>> For your case, it looks like simply you are simply using binary
>> dependencies.  For that case, you have to consider where these items are
>> showing up and how they are released.  Your inclusion of a dependency will
>> affect the generated NAR (nifi-irc-processors-nar) and, while it seems to
>> be missing in the current PR, the zip and tgz nifi-assembly artifacts.  You
>> shouldn't need to include it in levels lower than this assuming you are
>> talking about JARs that compose the overall NAR.  While you are linking
>> these against dependencies, you are not explicitly bringing them into the
>> project through the JARs incorporated in the NAR.
>>
>> Source inclusions are handled similarly but do go a level deeper as they
>> are also bundled with the JARs and are present in the root LICENSE and
>> NOTICE where applicable.  Again, both are for similar reasons with the
>> generated source package and JARs bundling this work including the source.
>>
>> Do keep in mind the transitive dependencies.  Looking quickly through the
>> pom for kitteh, I see usage of some netty libraries as well as mbassador.
>> These would presumably also be collected upon building the NAR.
>>
>> Of course, the docs we have on the site are quite nice if you need some
>> light reading material ;)  https://nifi.apache.org/licensing-guide.html
>> Both the guide and the links from it are good information and a nice
>> reference to revisit when working through these things.
>>
>> Let us know if there are additional questions or if some additional
>> clarification is needed.  Hopefully my anecdotal thoughts are both somewhat
>> helpful and mostly correct!
>>
>> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami 
>> wrote:
>>
>> > I just start and I really don’t know much so let see what I can learn
>> when
>> > time pass by and hope I can learn as much as you, thank’s
>> > > On Feb 25, 2017, at 5:12 PM, Andre  wrote:
>> > >
>> > > Hi there,
>> > >
>> > > Quick question on proper licensing:
>> > >
>> > > When bundling Processors, Services and APIs, where should the NOTICES
>> and
>> > > LICENSES be added to?
>> > >
>> > > The PR in question is https://github.com/apache/nifi/pull/1541
>> > >
>> > > My current reading is that all NAR levels will have to include the
>> proper
>> > > references (although I may reduce a bit of the dependencies by
>> excluding
>> > > some of the deeper dependencies, specially at the
>> > > nifi-irc-client-service-api-nar ).
>> > >
>> > > Would you agree?
>> > >
>> > > Cheers
>> >
>> >
>>

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

Thank you both for bringing up this discussion.  I have a few follow-up
questions:

1.) Is it true all of the NAR bundles in the NiFi source should have their
own LICENSE and NOTICE files, without exception?  In looking through the
source, most nifi-*-nar projects have both files for binary dependencies.
I understand these binary dependencies should roll up to nifi-assembly
LICENSE/NOTICE.

2.) Is it true we do not need source LICENSE/NOTICE for nar bundle projects
unless they have distinctive source dependency terms that roll up to the
root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so far.  I
assume the ASLv2 file headers cover most of the source.

3.) Where dependencies also distinguish between their source LICENSE/NOTICE
and binary LICENSE/NOTICE, should we match to our dependency relationship?
For example, a binary dependency would result in the inclusion of the
binary NOTICE terms?


Thanks,

James

On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri  wrote:

> Hey Andre,
>
> I may be off, but to help you along, I will give you my take on things to
> the best of my understanding.  If there are any wrong points, I hope
> someone can further clarify.
>
> For your case, it looks like simply you are simply using binary
> dependencies.  For that case, you have to consider where these items are
> showing up and how they are released.  Your inclusion of a dependency will
> affect the generated NAR (nifi-irc-processors-nar) and, while it seems to
> be missing in the current PR, the zip and tgz nifi-assembly artifacts.  You
> shouldn't need to include it in levels lower than this assuming you are
> talking about JARs that compose the overall NAR.  While you are linking
> these against dependencies, you are not explicitly bringing them into the
> project through the JARs incorporated in the NAR.
>
> Source inclusions are handled similarly but do go a level deeper as they
> are also bundled with the JARs and are present in the root LICENSE and
> NOTICE where applicable.  Again, both are for similar reasons with the
> generated source package and JARs bundling this work including the source.
>
> Do keep in mind the transitive dependencies.  Looking quickly through the
> pom for kitteh, I see usage of some netty libraries as well as mbassador.
> These would presumably also be collected upon building the NAR.
>
> Of course, the docs we have on the site are quite nice if you need some
> light reading material ;)  https://nifi.apache.org/licensing-guide.html
> Both the guide and the links from it are good information and a nice
> reference to revisit when working through these things.
>
> Let us know if there are additional questions or if some additional
> clarification is needed.  Hopefully my anecdotal thoughts are both somewhat
> helpful and mostly correct!
>
> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami 
> wrote:
>
> > I just start and I really don’t know much so let see what I can learn
> when
> > time pass by and hope I can learn as much as you, thank’s
> > > On Feb 25, 2017, at 5:12 PM, Andre  wrote:
> > >
> > > Hi there,
> > >
> > > Quick question on proper licensing:
> > >
> > > When bundling Processors, Services and APIs, where should the NOTICES
> and
> > > LICENSES be added to?
> > >
> > > The PR in question is https://github.com/apache/nifi/pull/1541
> > >
> > > My current reading is that all NAR levels will have to include the
> proper
> > > references (although I may reduce a bit of the dependencies by
> excluding
> > > some of the deeper dependencies, specially at the
> > > nifi-irc-client-service-api-nar ).
> > >
> > > Would you agree?
> > >
> > > Cheers
> >
> >
>

Re: Testing automation, Deployment in NiFi

[Anshuman Ghosh]

Hello Bryan,

Greetings of the day!

Thank you very much for your reply.
I have quickly looked into the documents and we are looking forward for the
deployment stuffs that are in development now.

However it would be great, if I get some more guidance on the REST API part
for testing automation. I am very new in this area so any help would be
really beneficial.

   1. Is it like first we need to create the flow in Web UI and then create
   a script to instantiate/ start it?
   2. I have attached a DataFlow snap here (PFA), so for this what you
   would suggest? Can I create the entire Flow using REST API or how does that
   work?

I know thesse might be basics again but I do not have knowledge on this!
Thanking you in advance.


​
__

*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*


On Fri, Feb 24, 2017 at 5:35 PM, Bryan Bende  wrote:

> Hello,
>
> In terms of automated testing, anything that you can do in the UI can
> also be done via the REST API [1]. So you could write scripts/code to
> create a process group, then instantiate processors, connect them,
> start them, send in known data, and check the output somewhere.
>
> As far as deployment, this is an area that the community is currently
> working to improve [2]. Currently you can use templates as a
> deployment vehicle [3][4], or you can copy the whole flow.xml.gz from
> one environment to the other, assuming you want to completely move the
> whole canvas from one environment to the other (depends on sensitive
> properties encryption key being the same in each environment). This is
> definitely an area that will significantly improve in the future, and
> we are happy to hear and discuss on feedback on the feature proposals.
>
> Thanks,
>
> Bryan
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> [2] https://cwiki.apache.org/confluence/display/NIFI/
> Configuration+Management+of+Flows
> [3] https://github.com/aperepel/nifi-api-deploy
> [4] https://github.com/ijokarumawak/nifi-deploy-process-group
>
> On Fri, Feb 24, 2017 at 7:54 AM, Anshuman Ghosh
>  wrote:
> > Hello team,
> >
> > Greetings of the day!
> >
> > This is Anshu here and we in our organization have started with Apache
> NiFi
> > recently.
> > I must appreciate the features and ease of usage of this tool and the
> > community response, however I have couple of questions since we are in
> the
> > beginning.
> > It would really be great if we can get some guidance on this.
> >
> > *Brief about our use case. *
> > - We need to consume data from one/ more Kafka topics.
> > - Need to transform data on the fly.
> > - Need to write to HDFS finally with some partition logic.
> >
> > *What are we looking for?*
> > - Please guide us on how to automate our end to end testing.
> > - How to handle deployment in NiFi. If we develop and test a flow in
> lower
> > environments, is there a way to package them together and deploy them in
> > higher environment?
> >
> > Thanking you in advance!
> > Should there be any information required from my side, please do let me
> > know.
> >
> >
> >
> > __
> >
> > *Kind Regards,*
> > *Anshuman Ghosh*
> > *Contact - +49 179 9090964*
>

Re: Closing in on a NiFi 0.8.0 release?

[Joe Skora]

Sorry for the confusion regarding 1.0.0-BETA vs 1.0.0, my bad.

As for stability, I don't mean build and test stability but real world
stability feedback that has led to various repository fixes including the
1.x line transition to the schema based provenance and newly refactored
provenance repository.

Again, apologies for the beta confusion.

On Mon, Feb 27, 2017 at 10:24 AM, Joe Witt  wrote:

> Joe
>
> 1.0.0 was not a beta release.
>
> 1.0.0-beta was a beta release.
>
> The intent of the language was we support the old major line for one year
> once there is a major release.  It is of course imperative to respect that
> folks cannot migrate as quickly as we would always like.  But this sort of
> concern is precisely why we have such a document.
>
> I propose we clarify the language to clearly and simply state that once a
> new major release line release has occurred we will support the old release
> line for up to one year.   This does not distinguish between minor or
> incremental.  There is already language for that.
>
> The stability comment is an unclear line to debate.  The act of voting on a
> release is the point at which the community agrees and asserts the fitness
> of a release.  We have no other open and clear mechanism for doing so.
>
> I think the question of whether an 0.8 can happen is no longer tied to the
> recent portions of this thread.  It would need am RM and vote.
>
> As I mentioned the other day I'll go ahead and update the versioning
> guidance barring any objection or alternative proposal.  I'll wait another
> day or so in case you would like to propose alternative language for the
> commitment we make as a community to our users and ourselves.
>
> Thanks
> Joe
>
> On Feb 27, 2017 9:42 AM, "Joe Skora"  wrote:
>
> > I think there's a couple considerations related to continuing the 0.x
> line.
> >
> > First, as JoeW mentioned the Release Line Management page [1] says we
> > support a major release for one year, so we should plan to support 0.7.x
> > for one year from its July 13, 2016 release date [2].
> >
> > Also, since we considered the 1.0.0 release a beta [3], the 0.x line was
> > due any fixes, features, and enhancements through the release of 1.1.0 on
> > November 30th.  So the features and fixes through November 30th should be
> > backported where possible and appropriate and after that any fixes
> relating
> > to security or data loss should be backported for the life of the 0.x
> line.
> >
> > Next, I agree with JoeW that we don't want old lines to be a burden on
> the
> > community, but I suggest we consider the user base and how practical it
> is
> > to expect them to upgrade.  From 0.x to 1.x is a major transition, so I
> > think we should give more time for that transition than we will might for
> > 1.1 to 1.2.
> >
> > Finally, 0.x only recently became stable for my users in the last couple
> of
> > months, based on the 0.7.1 release with patches added for stability and
> > corruption issues that is similar to Brandon's list of outstanding 0.x
> > tickets.  Since the non-beta 1.1.0 release is less than 3 months old and
> > the transition from 0.x to 1.x is so significant, regardless of our
> release
> > policy I would propose we carry on the 0.x line on until 1.x has settled
> > and been shown to be similarly stable.
> >
> > Regards,
> > Joe
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/NIFI/Git+
> > Branching+and+Release+Line+Management
> > [2]
> > https://lists.apache.org/thread.html/46678ade742887c624c14faf16d187
> > e039827121e35d08f468c3cbfe@%3Cdev.nifi.apache.org%3E
> > [3]
> > https://lists.apache.org/thread.html/91a4c272ddf2e80ec2fefd95c2a1df
> > 6c37689f4290aba5bdb81afd35@%3Cusers.nifi.apache.org%3E
> >
> >
> > On Fri, Feb 24, 2017 at 9:58 PM, Brandon DeVries  wrote:
> >
> > > Based on Jira, there are 20 tickets marked as fixed in 0.8.0 and
> nowhere
> > > else in the 0.x line[1].  Highlights from these include:
> > >
> > >- NIFI-2890 - Provenance Repository corruption
> > >- NIFI-2920 - Swapped FlowFiles are not removed from content repo
> > when a
> > >queue is emptied.
> > >- NIFI-3055 - StandardRecordWriter can throw UTFDataFormatException
> > >- NIFI-3424 - CLONE for 0.x - Unable to generate Provenance Event
> > >because FlowFile UUID is not set
> > >- NIFI-3230 - Get/Put JMS broken for simple ActiveMQ SSL URIs
> > >- NIFI-3403 - NPE in InvokeHTTP
> > >- NIFI-3362 - Update FlowConfiguration.xsd TimePeriod to match
> > >FormatUtils
> > >- NIFI-2861 - ControlRate should accept more than one flow file per
> > >execution
> > >- NIFI-3350 - Reduce NiFi startup time by streamlining documentation
> > >extraction
> > >
> > > Are we willing to port all of the tickets from [1] to the 0.7 branch?
> Or
> > > rather, which of them would not make the cut?  There are a couple of
> > things
> > > on the list that seem like new features as opposed 

Re: Closing in on a NiFi 0.8.0 release?

[Joe Witt]

Joe

1.0.0 was not a beta release.

1.0.0-beta was a beta release.

The intent of the language was we support the old major line for one year
once there is a major release.  It is of course imperative to respect that
folks cannot migrate as quickly as we would always like.  But this sort of
concern is precisely why we have such a document.

I propose we clarify the language to clearly and simply state that once a
new major release line release has occurred we will support the old release
line for up to one year.   This does not distinguish between minor or
incremental.  There is already language for that.

The stability comment is an unclear line to debate.  The act of voting on a
release is the point at which the community agrees and asserts the fitness
of a release.  We have no other open and clear mechanism for doing so.

I think the question of whether an 0.8 can happen is no longer tied to the
recent portions of this thread.  It would need am RM and vote.

As I mentioned the other day I'll go ahead and update the versioning
guidance barring any objection or alternative proposal.  I'll wait another
day or so in case you would like to propose alternative language for the
commitment we make as a community to our users and ourselves.

Thanks
Joe

On Feb 27, 2017 9:42 AM, "Joe Skora"  wrote:

> I think there's a couple considerations related to continuing the 0.x line.
>
> First, as JoeW mentioned the Release Line Management page [1] says we
> support a major release for one year, so we should plan to support 0.7.x
> for one year from its July 13, 2016 release date [2].
>
> Also, since we considered the 1.0.0 release a beta [3], the 0.x line was
> due any fixes, features, and enhancements through the release of 1.1.0 on
> November 30th.  So the features and fixes through November 30th should be
> backported where possible and appropriate and after that any fixes relating
> to security or data loss should be backported for the life of the 0.x line.
>
> Next, I agree with JoeW that we don't want old lines to be a burden on the
> community, but I suggest we consider the user base and how practical it is
> to expect them to upgrade.  From 0.x to 1.x is a major transition, so I
> think we should give more time for that transition than we will might for
> 1.1 to 1.2.
>
> Finally, 0.x only recently became stable for my users in the last couple of
> months, based on the 0.7.1 release with patches added for stability and
> corruption issues that is similar to Brandon's list of outstanding 0.x
> tickets.  Since the non-beta 1.1.0 release is less than 3 months old and
> the transition from 0.x to 1.x is so significant, regardless of our release
> policy I would propose we carry on the 0.x line on until 1.x has settled
> and been shown to be similarly stable.
>
> Regards,
> Joe
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Git+
> Branching+and+Release+Line+Management
> [2]
> https://lists.apache.org/thread.html/46678ade742887c624c14faf16d187
> e039827121e35d08f468c3cbfe@%3Cdev.nifi.apache.org%3E
> [3]
> https://lists.apache.org/thread.html/91a4c272ddf2e80ec2fefd95c2a1df
> 6c37689f4290aba5bdb81afd35@%3Cusers.nifi.apache.org%3E
>
>
> On Fri, Feb 24, 2017 at 9:58 PM, Brandon DeVries  wrote:
>
> > Based on Jira, there are 20 tickets marked as fixed in 0.8.0 and nowhere
> > else in the 0.x line[1].  Highlights from these include:
> >
> >- NIFI-2890 - Provenance Repository corruption
> >- NIFI-2920 - Swapped FlowFiles are not removed from content repo
> when a
> >queue is emptied.
> >- NIFI-3055 - StandardRecordWriter can throw UTFDataFormatException
> >- NIFI-3424 - CLONE for 0.x - Unable to generate Provenance Event
> >because FlowFile UUID is not set
> >- NIFI-3230 - Get/Put JMS broken for simple ActiveMQ SSL URIs
> >- NIFI-3403 - NPE in InvokeHTTP
> >- NIFI-3362 - Update FlowConfiguration.xsd TimePeriod to match
> >FormatUtils
> >- NIFI-2861 - ControlRate should accept more than one flow file per
> >execution
> >- NIFI-3350 - Reduce NiFi startup time by streamlining documentation
> >extraction
> >
> > Are we willing to port all of the tickets from [1] to the 0.7 branch?  Or
> > rather, which of them would not make the cut?  There are a couple of
> things
> > on the list that seem like new features as opposed to pure bug fixes...
> > although I suppose the difference between a "bug fix" and an
> "improvement"
> > is somewhat in the eye of the beholder.
> >
> > Ultimately, as long as there's a release covering these issues
> (everything
> > except the NIFI-2991[2] stuff) I don't particularly care what it's
> called.
> > If there are issues left out and I need to run a SNAPSHOT of some sort to
> > get them, then a further 0.x release doesn't help me anyway, and I'll
> > withdraw my suggestion.
> >
> > Brandon
> >
> > [1]
> > https://issues.apache.org/jira/issues/?jql=project%20%
> > 

Re: Closing in on a NiFi 0.8.0 release?

[Bryan Bende]

Joe,

Just wanted to clarify that I don't believe the 1.0.0 release was a BETA.

There was a 1.0.0-BETA release on 8/5 to gather feedback and testing
from the community, and the regular 1.0.0 release on 8/26.

http://central.maven.org/maven2/org/apache/nifi/nifi-assembly/1.0.0-BETA/  (8/5)
http://central.maven.org/maven2/org/apache/nifi/nifi-assembly/1.0.0/ (8/26)

-Bryan


On Mon, Feb 27, 2017 at 9:42 AM, Joe Skora  wrote:
> I think there's a couple considerations related to continuing the 0.x line.
>
> First, as JoeW mentioned the Release Line Management page [1] says we
> support a major release for one year, so we should plan to support 0.7.x
> for one year from its July 13, 2016 release date [2].
>
> Also, since we considered the 1.0.0 release a beta [3], the 0.x line was
> due any fixes, features, and enhancements through the release of 1.1.0 on
> November 30th.  So the features and fixes through November 30th should be
> backported where possible and appropriate and after that any fixes relating
> to security or data loss should be backported for the life of the 0.x line.
>
> Next, I agree with JoeW that we don't want old lines to be a burden on the
> community, but I suggest we consider the user base and how practical it is
> to expect them to upgrade.  From 0.x to 1.x is a major transition, so I
> think we should give more time for that transition than we will might for
> 1.1 to 1.2.
>
> Finally, 0.x only recently became stable for my users in the last couple of
> months, based on the 0.7.1 release with patches added for stability and
> corruption issues that is similar to Brandon's list of outstanding 0.x
> tickets.  Since the non-beta 1.1.0 release is less than 3 months old and
> the transition from 0.x to 1.x is so significant, regardless of our release
> policy I would propose we carry on the 0.x line on until 1.x has settled
> and been shown to be similarly stable.
>
> Regards,
> Joe
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
> [2]
> https://lists.apache.org/thread.html/46678ade742887c624c14faf16d187e039827121e35d08f468c3cbfe@%3Cdev.nifi.apache.org%3E
> [3]
> https://lists.apache.org/thread.html/91a4c272ddf2e80ec2fefd95c2a1df6c37689f4290aba5bdb81afd35@%3Cusers.nifi.apache.org%3E
>
>
> On Fri, Feb 24, 2017 at 9:58 PM, Brandon DeVries  wrote:
>
>> Based on Jira, there are 20 tickets marked as fixed in 0.8.0 and nowhere
>> else in the 0.x line[1].  Highlights from these include:
>>
>>- NIFI-2890 - Provenance Repository corruption
>>- NIFI-2920 - Swapped FlowFiles are not removed from content repo when a
>>queue is emptied.
>>- NIFI-3055 - StandardRecordWriter can throw UTFDataFormatException
>>- NIFI-3424 - CLONE for 0.x - Unable to generate Provenance Event
>>because FlowFile UUID is not set
>>- NIFI-3230 - Get/Put JMS broken for simple ActiveMQ SSL URIs
>>- NIFI-3403 - NPE in InvokeHTTP
>>- NIFI-3362 - Update FlowConfiguration.xsd TimePeriod to match
>>FormatUtils
>>- NIFI-2861 - ControlRate should accept more than one flow file per
>>execution
>>- NIFI-3350 - Reduce NiFi startup time by streamlining documentation
>>extraction
>>
>> Are we willing to port all of the tickets from [1] to the 0.7 branch?  Or
>> rather, which of them would not make the cut?  There are a couple of things
>> on the list that seem like new features as opposed to pure bug fixes...
>> although I suppose the difference between a "bug fix" and an "improvement"
>> is somewhat in the eye of the beholder.
>>
>> Ultimately, as long as there's a release covering these issues (everything
>> except the NIFI-2991[2] stuff) I don't particularly care what it's called.
>> If there are issues left out and I need to run a SNAPSHOT of some sort to
>> get them, then a further 0.x release doesn't help me anyway, and I'll
>> withdraw my suggestion.
>>
>> Brandon
>>
>> [1]
>> https://issues.apache.org/jira/issues/?jql=project%20%
>> 3D%20NIFI%20AND%20fixVersion%20%3D%200.8.0%20and%
>> 20fixVersion%20not%20in%20(0.2.1%2C%200.3.0%2C%200.4.0%2C0.
>> 4.1%2C0.5.0%2C0.5.1%2C0.6.0%2C%200.6.1%2C0.7.0%2C%200.7.1%
>> 2C%200.7.2%2C%200.7.3)%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC
>>
>> [2] https://issues.apache.org/jira/browse/NIFI-2991
>>
>>
>> On Fri, Feb 24, 2017 at 7:49 PM Joe Witt  wrote:
>>
>> > The 0.8 fixes for licensing remove a processor (gettwitter) at this
>> point.
>> > That I feel requires at least minor.  But avoiding that for now and doing
>> > the bug fix things and doing 073 seems legit.
>> >
>> > Will wait and see if anyone else has a different interpretation on the
>> > intent of our one year version guidance and then update the wiki if
>> appears
>> > we have consensus.
>> >
>> > Thanks
>> > Joe
>> >
>> > On Feb 24, 2017 7:19 PM, "Andy LoPresto"  wrote:
>> >
>> > > Especially as nothing that would be going into the 

Re: Closing in on a NiFi 0.8.0 release?

[Joe Skora]

I think there's a couple considerations related to continuing the 0.x line.

First, as JoeW mentioned the Release Line Management page [1] says we
support a major release for one year, so we should plan to support 0.7.x
for one year from its July 13, 2016 release date [2].

Also, since we considered the 1.0.0 release a beta [3], the 0.x line was
due any fixes, features, and enhancements through the release of 1.1.0 on
November 30th.  So the features and fixes through November 30th should be
backported where possible and appropriate and after that any fixes relating
to security or data loss should be backported for the life of the 0.x line.

Next, I agree with JoeW that we don't want old lines to be a burden on the
community, but I suggest we consider the user base and how practical it is
to expect them to upgrade.  From 0.x to 1.x is a major transition, so I
think we should give more time for that transition than we will might for
1.1 to 1.2.

Finally, 0.x only recently became stable for my users in the last couple of
months, based on the 0.7.1 release with patches added for stability and
corruption issues that is similar to Brandon's list of outstanding 0.x
tickets.  Since the non-beta 1.1.0 release is less than 3 months old and
the transition from 0.x to 1.x is so significant, regardless of our release
policy I would propose we carry on the 0.x line on until 1.x has settled
and been shown to be similarly stable.

Regards,
Joe

[1]
https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
[2]
https://lists.apache.org/thread.html/46678ade742887c624c14faf16d187e039827121e35d08f468c3cbfe@%3Cdev.nifi.apache.org%3E
[3]
https://lists.apache.org/thread.html/91a4c272ddf2e80ec2fefd95c2a1df6c37689f4290aba5bdb81afd35@%3Cusers.nifi.apache.org%3E


On Fri, Feb 24, 2017 at 9:58 PM, Brandon DeVries  wrote:

> Based on Jira, there are 20 tickets marked as fixed in 0.8.0 and nowhere
> else in the 0.x line[1].  Highlights from these include:
>
>- NIFI-2890 - Provenance Repository corruption
>- NIFI-2920 - Swapped FlowFiles are not removed from content repo when a
>queue is emptied.
>- NIFI-3055 - StandardRecordWriter can throw UTFDataFormatException
>- NIFI-3424 - CLONE for 0.x - Unable to generate Provenance Event
>because FlowFile UUID is not set
>- NIFI-3230 - Get/Put JMS broken for simple ActiveMQ SSL URIs
>- NIFI-3403 - NPE in InvokeHTTP
>- NIFI-3362 - Update FlowConfiguration.xsd TimePeriod to match
>FormatUtils
>- NIFI-2861 - ControlRate should accept more than one flow file per
>execution
>- NIFI-3350 - Reduce NiFi startup time by streamlining documentation
>extraction
>
> Are we willing to port all of the tickets from [1] to the 0.7 branch?  Or
> rather, which of them would not make the cut?  There are a couple of things
> on the list that seem like new features as opposed to pure bug fixes...
> although I suppose the difference between a "bug fix" and an "improvement"
> is somewhat in the eye of the beholder.
>
> Ultimately, as long as there's a release covering these issues (everything
> except the NIFI-2991[2] stuff) I don't particularly care what it's called.
> If there are issues left out and I need to run a SNAPSHOT of some sort to
> get them, then a further 0.x release doesn't help me anyway, and I'll
> withdraw my suggestion.
>
> Brandon
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20NIFI%20AND%20fixVersion%20%3D%200.8.0%20and%
> 20fixVersion%20not%20in%20(0.2.1%2C%200.3.0%2C%200.4.0%2C0.
> 4.1%2C0.5.0%2C0.5.1%2C0.6.0%2C%200.6.1%2C0.7.0%2C%200.7.1%
> 2C%200.7.2%2C%200.7.3)%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC
>
> [2] https://issues.apache.org/jira/browse/NIFI-2991
>
>
> On Fri, Feb 24, 2017 at 7:49 PM Joe Witt  wrote:
>
> > The 0.8 fixes for licensing remove a processor (gettwitter) at this
> point.
> > That I feel requires at least minor.  But avoiding that for now and doing
> > the bug fix things and doing 073 seems legit.
> >
> > Will wait and see if anyone else has a different interpretation on the
> > intent of our one year version guidance and then update the wiki if
> appears
> > we have consensus.
> >
> > Thanks
> > Joe
> >
> > On Feb 24, 2017 7:19 PM, "Andy LoPresto"  wrote:
> >
> > > Especially as nothing that would be going into the 0.x release is a
> major
> > > feature or changes compatibility (from my understanding), I would +1
> the
> > > 0.7.3 suggestion.
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > *alopresto.apa...@gmail.com *
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Feb 24, 2017, at 2:07 PM, Tony Kurc  wrote:
> > >
> > > I think it is probably worth clarifying the intent of the support
> > language.
> > > I believe the intent was to support 0.x for a year after 1.x was
> > released.

Re: waitting for your hellp

[Andre]

My last message had some residue of copy and paste residues:

Where it reads:

"Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
PutTCP)"

it should read

"Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
InvokeHTTP)"

Cheers


On Mon, Feb 27, 2017 at 10:58 PM, Andre  wrote:

> Hi,
>
> Writing to OpenTSDB is a reasonably common pattern that can be achieved by:
>
> Using OpenTSDB's telnet API together with the PutTCP processor (payload
> content must be processed to the desired format before reaching PutTCP)
>
> or
>
> Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
> (payload content must be processed to the desired format before reaching
> PutTCP)
>
> Hope this help.
>
> Andre
>
> PS-Due to its (higher) number of subscribers, the user mailing list is
> usually a better channel for messages requesting guidance on use cases.
>
> On Mon, Feb 27, 2017 at 6:55 PM, sczylbh  wrote:
>
>> Hi，
>> I want to store the data to opentsdb or redis use nifi,Do you have any
>> good suggestions?
>> or can you develop the processor that can put/get data to/from opentsdb
>> or redis.
>
>
>

Re: waitting for your hellp

[Andre]

Hi,

Writing to OpenTSDB is a reasonably common pattern that can be achieved by:

Using OpenTSDB's telnet API together with the PutTCP processor (payload
content must be processed to the desired format before reaching PutTCP)

or

Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
PutTCP)

Hope this help.

Andre

PS-Due to its (higher) number of subscribers, the user mailing list is
usually a better channel for messages requesting guidance on use cases.

On Mon, Feb 27, 2017 at 6:55 PM, sczylbh  wrote:

> Hi，
> I want to store the data to opentsdb or redis use nifi,Do you have any
> good suggestions?
> or can you develop the processor that can put/get data to/from opentsdb or
> redis.

waitting for your hellp

[sczylbh]

Hi，
I want to store the data to opentsdb or redis use nifi,Do you have any good 
suggestions?
or can you develop the processor that can put/get data to/from opentsdb or 
redis.