from:"Sivaprasanna"

To all,

If anyone who has been given committers privilege but haven't got their
name added to the NiFi project's people page. Let me know, if you want to
have your name added, the setup process is somewhat time consuming and if
you don't want to go through the hassle of setting up a whole lot of things
which you may not be actually using, I can help since I set them up
recently.

Thanks,
Sivaprasanna

On Tue, Jun 5, 2018 at 11:07 PM, Sivaprasanna 
wrote:

> Thank you so much, Aldrin. I was able to deploy it. Having one's name on
> an Apache projects' site is an honor!
>
> Thanks once again.
>
> -
> Sivaprasanna
>
> On Tue, Jun 5, 2018 at 10:00 PM, Sivaprasanna 
> wrote:
>
>> Aldrin,
>>
>> Thanks for the quick response. I'm going to try that now. I'll update
>> this thread if I'm stuck somewhere.
>>
>> Thanks,
>> Sivaprasanna
>>
>> On Tue, Jun 5, 2018 at 9:41 PM, Aldrin Piri  wrote:
>>
>>> Hi Sivaprasanna,
>>>
>>> You will need to perform a deploy via grunt as outlined here:
>>> https://github.com/apache/nifi-site#grunt-tasks
>>>
>>> You can configure your environment via
>>> https://github.com/apache/nifi-site#setting-up-build-environment.
>>>
>>> Let us know if you hit any issues.
>>>
>>> On Tue, Jun 5, 2018 at 12:04 PM, Sivaprasanna >> >
>>> wrote:
>>>
>>> > Hi
>>> >
>>> > I cloned http://git-wip-us.apache.org/repos/asf/nifi-site
>>> > <http://git-wip-us.apache.org/repos/asf/nifi-site/repo> and made a
>>> change
>>> > and committed it (#8bd32db0
>>> > <https://git1-us-west.apache.org/repos/asf?p=nifi-site.git;
>>> > a=commitdiff;h=8bd32db0>).
>>> > Is that the correct way to do it? The reason why I'm asking this is
>>> because
>>> > I see this (https://svn.apache.org/viewvc/nifi/site/trunk/) doesn't
>>> > include
>>> > the commit I made. I'm sorry if I did it the wrong way. Appreciate your
>>> > inputs.
>>> >
>>> > -
>>> > Sivaprasanna
>>> >
>>>
>>
>>
>

Re: Commit to nifi-site

Thank you so much, Aldrin. I was able to deploy it. Having one's name on an
Apache projects' site is an honor!

Thanks once again.

-
Sivaprasanna

On Tue, Jun 5, 2018 at 10:00 PM, Sivaprasanna 
wrote:

> Aldrin,
>
> Thanks for the quick response. I'm going to try that now. I'll update this
> thread if I'm stuck somewhere.
>
> Thanks,
> Sivaprasanna
>
> On Tue, Jun 5, 2018 at 9:41 PM, Aldrin Piri  wrote:
>
>> Hi Sivaprasanna,
>>
>> You will need to perform a deploy via grunt as outlined here:
>> https://github.com/apache/nifi-site#grunt-tasks
>>
>> You can configure your environment via
>> https://github.com/apache/nifi-site#setting-up-build-environment.
>>
>> Let us know if you hit any issues.
>>
>> On Tue, Jun 5, 2018 at 12:04 PM, Sivaprasanna 
>> wrote:
>>
>> > Hi
>> >
>> > I cloned http://git-wip-us.apache.org/repos/asf/nifi-site
>> > <http://git-wip-us.apache.org/repos/asf/nifi-site/repo> and made a
>> change
>> > and committed it (#8bd32db0
>> > <https://git1-us-west.apache.org/repos/asf?p=nifi-site.git;
>> > a=commitdiff;h=8bd32db0>).
>> > Is that the correct way to do it? The reason why I'm asking this is
>> because
>> > I see this (https://svn.apache.org/viewvc/nifi/site/trunk/) doesn't
>> > include
>> > the commit I made. I'm sorry if I did it the wrong way. Appreciate your
>> > inputs.
>> >
>> > -
>> > Sivaprasanna
>> >
>>
>
>

Re: Commit to nifi-site

Aldrin,

Thanks for the quick response. I'm going to try that now. I'll update this
thread if I'm stuck somewhere.

Thanks,
Sivaprasanna

On Tue, Jun 5, 2018 at 9:41 PM, Aldrin Piri  wrote:

> Hi Sivaprasanna,
>
> You will need to perform a deploy via grunt as outlined here:
> https://github.com/apache/nifi-site#grunt-tasks
>
> You can configure your environment via
> https://github.com/apache/nifi-site#setting-up-build-environment.
>
> Let us know if you hit any issues.
>
> On Tue, Jun 5, 2018 at 12:04 PM, Sivaprasanna 
> wrote:
>
> > Hi
> >
> > I cloned http://git-wip-us.apache.org/repos/asf/nifi-site
> > <http://git-wip-us.apache.org/repos/asf/nifi-site/repo> and made a
> change
> > and committed it (#8bd32db0
> > <https://git1-us-west.apache.org/repos/asf?p=nifi-site.git;
> > a=commitdiff;h=8bd32db0>).
> > Is that the correct way to do it? The reason why I'm asking this is
> because
> > I see this (https://svn.apache.org/viewvc/nifi/site/trunk/) doesn't
> > include
> > the commit I made. I'm sorry if I did it the wrong way. Appreciate your
> > inputs.
> >
> > -
> > Sivaprasanna
> >
>

Commit to nifi-site

Hi

I cloned http://git-wip-us.apache.org/repos/asf/nifi-site
<http://git-wip-us.apache.org/repos/asf/nifi-site/repo> and made a change
and committed it (#8bd32db0
<https://git1-us-west.apache.org/repos/asf?p=nifi-site.git;a=commitdiff;h=8bd32db0>).
Is that the correct way to do it? The reason why I'm asking this is because
I see this (https://svn.apache.org/viewvc/nifi/site/trunk/) doesn't include
the commit I made. I'm sorry if I did it the wrong way. Appreciate your
inputs.

-
Sivaprasanna

Re: KerberosProperties.validatePrincipalAndKeytab Error ?

2018-06-04 Thread Sivaprasanna

Jorge,

Both 'Kerberos Principal' abd 'Kerberos Keytab' support NiFi expression
language so ${principal} and ${keytab} is valid here. Can you check if the
property "nifi.kerberos.krb5.file" is set in nifi.properties file? Looks
like this has to be set according to the description of those properties.

-
Sivaprasanna

On Mon, Jun 4, 2018 at 1:27 PM, Jorge Machado  wrote:

> Hi Guys,
>
> I’m facing the issue that I cannot start the DeleteHDFS with the error:
> "Kerberos Principal must be provided when using a secure configuration”
>
> I’m able to reproduce this with this test:
>
> @Test
> public void testKerberosOptionsWithCredentialServices() throws Exception {
> SimpleHadoopProcessor processor = new 
> SimpleHadoopProcessor(kerberosProperties);
> TestRunner runner = TestRunners.newTestRunner(processor);
>
> // initialize the runner with EL for the kerberos properties
> 
> runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, 
> "${variableHadoopConfigResources}");
> runner.setProperty(kerberosProperties.getKerberosPrincipal(), 
> "${variablePrincipal}");
> runner.setProperty(kerberosProperties.getKerberosKeytab(), 
> "${variableKeytab}");
>
> // add variables for all the kerberos properties except for the keytab
> runner.setVariable("variableHadoopConfigResources", 
> "src/test/resources/core-site-security.xml");
> runner.assertValid();
> }
>
> In our case the ${principal} and ${keytab} is coming as a attribute on the
> incomming the Flowfile, the problem is that this validation of the
> attribute is happening before.
> Should this work like this? In all other places if we are using a variable
> it can be evaluate at run time...
>
> Jorge Machado
>
>
>
>
>
>

Re: Put data to Elastic with static settings or index template

2018-05-22 Thread Sivaprasanna

Bobby,

If I'm correct, this setting is done during index creation and the
PutElasticsearch processors doesn't create index. It primarily works with
the assumption that the configured index already exists (people correct me,
if I'm wrong). If that's the case, there is no need to do anything on the
NiFi side. Rather while creating the index through ES APIs, you set the
"static" setting. Hope that helps.

-
Sivaprasanna

On Tue, May 22, 2018 at 8:21 AM, Bobby <bobbyhars...@gmail.com> wrote:

> Hi, when inserting data to elastic using nifi's processor (putElastic), i
> need to apply static setting for the index..like mentioned in
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/index-modules.html
> <https://www.elastic.co/guide/en/elasticsearch/reference/
> current/index-modules.html>
> , this must be applied in index creation..
>
> With the processor, will it be possible to use this utility? I need to do
> this in order to save the space...or in other word, changing the
> compression
> type...
>
> As for last resort, i might need to write custom processor extended from
> putElastic
>
>
> Any suggestion?
>
> Thank you
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: NIFI-5133: Guidance & help with tackling dependency version issues

2018-05-05 Thread Sivaprasanna

I have created a Jira - NIFI-5156
<https://issues.apache.org/jira/browse/NIFI-5156> and made the necessary
changes to the current nifi-gcp-bundle. I have tested the components - GCS
Processors & GCP Credentials Service. They all work as expected. Anyone
want to give it a look and review the changes? PR #2680
<https://github.com/apache/nifi/pull/2680>

-
Sivaprasanna

On Thu, May 3, 2018 at 9:48 PM, Sivaprasanna <sivaprasanna...@gmail.com>
wrote:

> I'm interested although it might consume some time since I don't know how
> big it is going to be. And I suppose it is better to capture it in a
> separate Jira. The summary could be upgrade and refactor GCP processor
> codebase or something like that. We could then make NIFI-5133 a dependent
> of the new Jira. Thoughts?
>
> Meanwhile, anyone who has any other recommendations, feel free to share. :)
>
> -
> Sivaprasanna
>
> On Thu, May 3, 2018 at 9:43 PM, Joe Witt <joe.w...@gmail.com> wrote:
>
>> Sivaprasanna
>>
>> Ok makes sense.  Are you in a position of interest/expertise/time to
>> make those changes as well?
>>
>> Thanks
>>
>> On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <sivaprasanna...@gmail.com>
>> wrote:
>> > Hi. As I had mentioned, upgrading to the latest version of the library
>> is
>> > not as simple as I thought. Google Cloud team introduced many breaking
>> > changes. Many of the APIs (classes & methods) have been
>> > scrapped/replaced/modified/refactored/renamed.
>> >
>> > In short, a simple change of version may demand changes on the
>> processor's
>> > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP)
>> which
>> > may pose backward compatibility issues, I'm afraid.
>> >
>> > Thanks,
>> > Sivaprasanna
>> >
>> > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <joe.w...@gmail.com> wrote:
>> >
>> >> Sivaprasanna
>> >>
>> >> I might not completely follow but is there a 3rd option to upgrade to
>> >> a more recent library and solve the use of the proper jars
>> >> problem(smaller nar)?
>> >>
>> >> Thanks
>> >>
>> >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <
>> sivaprasanna...@gmail.com>
>> >> wrote:
>> >> > Hi
>> >> >
>> >> > I've started the initial works on implementing Google Cloud Pub/Sub
>> >> > processors. The associated Jira ID is NIFI-5133
>> >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to
>> the
>> >> > existing GCP bundle which currently has only the storage processors.
>> Upon
>> >> > some inspection, I noticed the following:
>> >> >
>> >> >- As of now, the bundle uses google-cloud
>> >> ><http://mvnrepository.com/artifact/com.google.cloud/google-cloud>
>> as
>> >> its
>> >> >dependency which is like uber/fat jar that contains most of the
>> Google
>> >> >Cloud's client library SDKs including storage, bigquery, pubsub,
>> etc.
>> >> The
>> >> >main point is it is using a very older version (0.8.0)
>> >> >- I thought of using google-cloud-bom
>> >> ><http://mvnrepository.com/artifact/com.google.cloud/google-
>> cloud-bom>
>> >> in
>> >> >the bundle's POM
>> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/pom.xml>
>> >> >and then use the required artifacts in the processor's POM
>> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
>> >> >The benefit is, it will help us reduce the overall size of the
>> NAR.
>> >> >
>> >> > When I tried to do #2, I realized this is not a simple version change
>> >> but a
>> >> > change that brings backward compatibility issues. Ex: Some APIs used
>> in
>> >> the
>> >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
>> >> > different library. We can do either two things:
>> >> >
>> >> >1. User the Pub/Sub APIs from the older version but the problem is
>> >> it's
>> >> >very old and the problem of upgrading would soon catchup with us.
>> >> >2. Or we can continue to use the older version of
>> google-cloud-storage
>> >> >only for the storage processors and introduce the #2 mentioned
>> above
>> >> but I
>> >> >don't think then the new processors can't properly extend the
>> existing
>> >> >AbstractGCPProcessor
>> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
>> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
>> >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
>> >> >
>> >> >
>> >> > A quick glance on the processor code and the POM would help you
>> >> understand
>> >> > my concern.
>> >> >
>> >> > I'm stuck up here so any help & guidance in this regard is very much
>> >> > appreciated. :)
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Sivaprasanna
>> >>
>>
>
>

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-04 Thread Sivaprasanna

As Bryan mentioned in the actual ‘GetSplunk’ thread, it is not available
but it certainly makes sense to have that feature. If you’re interested in
having that, please raise a Jira at https://issues.apache.org/jira

-
Sivaprasanna

On Fri, 4 May 2018 at 12:56 PM, Brajendra Mishra <
brajendra_mis...@persistent.com> wrote:

> Hi Mike,
>
> Thanks a lot for your valuable inputs.
> We tried GetMongoDB Template which you shared in previous mail with new
> version of Apache NiFi (NiFi-1.6) , Its working fine and we are able to get
> mongoDB data in desired format.
>
> One query: Is such input flow functionality available with 'GetSplunk'
> processor too with NiFI 1.6.0? if yes, please share sample template or if
> not, When can expect such implementation with 'GetSplunk' processor too?
>
> Brajendra Mishra
> Persistent Systems Ltd.
>
> -Original Message-
> From: Mike Thomsen <mikerthom...@gmail.com>
> Sent: Thursday, May 03, 2018 6:56 PM
> To: dev@nifi.apache.org
> Subject: Re: GetMongoDB : How to pass parameters as input to GetMongoDB
> processor
>
> Brajendra,
>
> I would recommend an update to 1.6.0. It'll make your life a lot easier on
> this. I did that patch to GetMongo because I had a client that had an
> explosion of GetMongos due to that inflexibility. With that said, *be aware
> of this bug* in 1.6.0 w/ PutMongo if you use it and upgrade. It is fixed in
> 1.7.0 (still in development):
>
> Migrating from 1.5.0 to 1.6.0
>
>- PutMongo can fail in insert mode. Will be fixed in next release. In
>the mean time you can set query keys for insert even though they'll be
>ignored it should workaround the validation bug.
>
>
> What it means is there is a validator function that is broken in PutMongo
> when one is using the "insert mode" instead of "update mode." You can do a
> work around by putting a dummy value in the "query key" field to make it
> happy.
>
> On Thu, May 3, 2018 at 8:26 AM Pierre Villard <pierre.villard...@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > As Mike said: incoming relationship has been added for NiFi 1.6.0.
> > https://issues.apache.org/jira/browse/NIFI-4827
> >
> > Pierre
> >
> > 2018-05-03 14:09 GMT+02:00 Brajendra Mishra <
> > brajendra_mis...@persistent.com
> > >:
> >
> > > Hi Mike,
> > >
> > > I did attach the same in my previous mail. Well reattaching it again.
> > > Well error is at GetMongoDB Processor and error text is : "Upstream
> > > Connections is invalid because Processor does not allow upstream
> > > connections but currently has 1"
> > >
> > > Brajendra Mishra
> > > Persistent Systems Ltd.
> > >
> > > -Original Message-
> > > From: Mike Thomsen <mikerthom...@gmail.com>
> > > Sent: Thursday, May 03, 2018 5:20 PM
> > > To: dev@nifi.apache.org
> > > Subject: Re: GetMongoDB : How to pass parameters as input to
> > > GetMongoDB processor
> > >
> > > Brajendra,
> > >
> > > Looks like the image didn't make it.
> > >
> > > On Wed, May 2, 2018 at 11:36 PM Brajendra Mishra <
> > > brajendra_mis...@persistent.com> wrote:
> > >
> > > > Hi Mike,
> > > >
> > > > Thanks for responding.
> > > >
> > > > Here, I have attached missing image attachment.
> > > >
> > > >
> > > >
> > > > Brajendra Mishra
> > > >
> > > > Persistent Systems Ltd.
> > > >
> > > >
> > > >
> > > > *From:* Mike Thomsen <mikerthom...@gmail.com>
> > > > *Sent:* Wednesday, May 02, 2018 6:24 PM
> > > >
> > > >
> > > > *To:* dev@nifi.apache.org
> > > > *Subject:* Re: GetMongoDB : How to pass parameters as input to
> > > > GetMongoDB processor
> > > >
> > > >
> > > >
> > > > That might require 1.6.0. Also, your image didn't come through in
> > > > your response to Sivaprasanna so resend that too.
> > > >
> > > > On Wed, May 2, 2018 at 8:37 AM Brajendra Mishra <
> > > > brajendra_mis...@persistent.com> wrote:
> > > >
> > > > Hi Mike,
> > > >
> > > > Thanks a lot for responding.
> > > >
> > > > On your statement
> > > > "That is its new default behavior if you leave the query field
> > > > blank and have an incoming connection from another processor. That
> >

Re: NIFI-5133: Guidance & help with tackling dependency version issues

2018-05-03 Thread Sivaprasanna

I'm interested although it might consume some time since I don't know how
big it is going to be. And I suppose it is better to capture it in a
separate Jira. The summary could be upgrade and refactor GCP processor
codebase or something like that. We could then make NIFI-5133 a dependent
of the new Jira. Thoughts?

Meanwhile, anyone who has any other recommendations, feel free to share. :)

-
Sivaprasanna

On Thu, May 3, 2018 at 9:43 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Sivaprasanna
>
> Ok makes sense.  Are you in a position of interest/expertise/time to
> make those changes as well?
>
> Thanks
>
> On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
> > Hi. As I had mentioned, upgrading to the latest version of the library is
> > not as simple as I thought. Google Cloud team introduced many breaking
> > changes. Many of the APIs (classes & methods) have been
> > scrapped/replaced/modified/refactored/renamed.
> >
> > In short, a simple change of version may demand changes on the
> processor's
> > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP)
> which
> > may pose backward compatibility issues, I'm afraid.
> >
> > Thanks,
> > Sivaprasanna
> >
> > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >
> >> Sivaprasanna
> >>
> >> I might not completely follow but is there a 3rd option to upgrade to
> >> a more recent library and solve the use of the proper jars
> >> problem(smaller nar)?
> >>
> >> Thanks
> >>
> >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <
> sivaprasanna...@gmail.com>
> >> wrote:
> >> > Hi
> >> >
> >> > I've started the initial works on implementing Google Cloud Pub/Sub
> >> > processors. The associated Jira ID is NIFI-5133
> >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to
> the
> >> > existing GCP bundle which currently has only the storage processors.
> Upon
> >> > some inspection, I noticed the following:
> >> >
> >> >- As of now, the bundle uses google-cloud
> >> ><http://mvnrepository.com/artifact/com.google.cloud/google-cloud>
> as
> >> its
> >> >dependency which is like uber/fat jar that contains most of the
> Google
> >> >Cloud's client library SDKs including storage, bigquery, pubsub,
> etc.
> >> The
> >> >main point is it is using a very older version (0.8.0)
> >> >- I thought of using google-cloud-bom
> >> ><http://mvnrepository.com/artifact/com.google.cloud/
> google-cloud-bom>
> >> in
> >> >the bundle's POM
> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/pom.xml>
> >> >and then use the required artifacts in the processor's POM
> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
> >> >The benefit is, it will help us reduce the overall size of the NAR.
> >> >
> >> > When I tried to do #2, I realized this is not a simple version change
> >> but a
> >> > change that brings backward compatibility issues. Ex: Some APIs used
> in
> >> the
> >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
> >> > different library. We can do either two things:
> >> >
> >> >1. User the Pub/Sub APIs from the older version but the problem is
> >> it's
> >> >very old and the problem of upgrading would soon catchup with us.
> >> >2. Or we can continue to use the older version of
> google-cloud-storage
> >> >only for the storage processors and introduce the #2 mentioned
> above
> >> but I
> >> >don't think then the new processors can't properly extend the
> existing
> >> >AbstractGCPProcessor
> >> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
> >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
> >> >
> >> >
> >> > A quick glance on the processor code and the POM would help you
> >> understand
> >> > my concern.
> >> >
> >> > I'm stuck up here so any help & guidance in this regard is very much
> >> > appreciated. :)
> >> >
> >> > Thanks,
> >> >
> >> > Sivaprasanna
> >>
>

Re: NIFI-5133: Guidance & help with tackling dependency version issues

2018-05-03 Thread Sivaprasanna

Hi. As I had mentioned, upgrading to the latest version of the library is
not as simple as I thought. Google Cloud team introduced many breaking
changes. Many of the APIs (classes & methods) have been
scrapped/replaced/modified/refactored/renamed.

In short, a simple change of version may demand changes on the processor's
code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP) which
may pose backward compatibility issues, I'm afraid.

Thanks,
Sivaprasanna

On Thu, May 3, 2018 at 9:26 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Sivaprasanna
>
> I might not completely follow but is there a 3rd option to upgrade to
> a more recent library and solve the use of the proper jars
> problem(smaller nar)?
>
> Thanks
>
> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
> > Hi
> >
> > I've started the initial works on implementing Google Cloud Pub/Sub
> > processors. The associated Jira ID is NIFI-5133
> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
> > existing GCP bundle which currently has only the storage processors. Upon
> > some inspection, I noticed the following:
> >
> >- As of now, the bundle uses google-cloud
> ><http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as
> its
> >dependency which is like uber/fat jar that contains most of the Google
> >Cloud's client library SDKs including storage, bigquery, pubsub, etc.
> The
> >main point is it is using a very older version (0.8.0)
> >- I thought of using google-cloud-bom
> ><http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom>
> in
> >the bundle's POM
> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/pom.xml>
> >and then use the required artifacts in the processor's POM
> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
> >The benefit is, it will help us reduce the overall size of the NAR.
> >
> > When I tried to do #2, I realized this is not a simple version change
> but a
> > change that brings backward compatibility issues. Ex: Some APIs used in
> the
> > older version i.e. 0.8.0 have now been entirely scrapped and moved to
> > different library. We can do either two things:
> >
> >1. User the Pub/Sub APIs from the older version but the problem is
> it's
> >very old and the problem of upgrading would soon catchup with us.
> >2. Or we can continue to use the older version of google-cloud-storage
> >only for the storage processors and introduce the #2 mentioned above
> but I
> >don't think then the new processors can't properly extend the existing
> >AbstractGCPProcessor
> ><https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/
> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.
> >
> >
> > A quick glance on the processor code and the POM would help you
> understand
> > my concern.
> >
> > I'm stuck up here so any help & guidance in this regard is very much
> > appreciated. :)
> >
> > Thanks,
> >
> > Sivaprasanna
>

NIFI-5133: Guidance & help with tackling dependency version issues

2018-05-03 Thread Sivaprasanna

I've started the initial works on implementing Google Cloud Pub/Sub
processors. The associated Jira ID is NIFI-5133
<https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
existing GCP bundle which currently has only the storage processors. Upon
some inspection, I noticed the following:

- As of now, the bundle uses google-cloud
<http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as its
dependency which is like uber/fat jar that contains most of the Google
Cloud's client library SDKs including storage, bigquery, pubsub, etc. The
main point is it is using a very older version (0.8.0)
- I thought of using google-cloud-bom
<http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom> in
the bundle's POM

<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/pom.xml>
and then use the required artifacts in the processor's POM

<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
The benefit is, it will help us reduce the overall size of the NAR.

When I tried to do #2, I realized this is not a simple version change but a
change that brings backward compatibility issues. Ex: Some APIs used in the
older version i.e. 0.8.0 have now been entirely scrapped and moved to
different library. We can do either two things:

1. User the Pub/Sub APIs from the older version but the problem is it's
very old and the problem of upgrading would soon catchup with us.
2. Or we can continue to use the older version of google-cloud-storage
only for the storage processors and introduce the #2 mentioned above but I
don't think then the new processors can't properly extend the existing
AbstractGCPProcessor

<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.

A quick glance on the processor code and the POM would help you understand
my concern.

I'm stuck up here so any help & guidance in this regard is very much
appreciated. :)

Thanks,

Sivaprasanna

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-02 Thread Sivaprasanna

Since I'm not so sure about your exact use case, I have just created a
rough template based on the simple example flow that I had posted earlier
which is GenerateFlowfile -> UpdateAttribute -> GetMongo. I have attached
the template here.

-
Sivaprasanna

On Wed, May 2, 2018 at 2:55 PM, Brajendra Mishra <
brajendra_mis...@persistent.com> wrote:

> Hi Sivaprasanna,
>
> Could you please provide me the sample template for the same, where I can
> pass parameters (and get those parameters' value to process further) to
> GetMongoDB processor?
> It would be a great help for us.
>
> Brajendra Mishra
> Persistent Systems Ltd.
>
> -----Original Message-
> From: Sivaprasanna <sivaprasanna...@gmail.com>
> Sent: Wednesday, May 02, 2018 2:28 PM
> To: dev@nifi.apache.org
> Subject: Re: GetMongoDB : How to pass parameters as input to GetMongoDB
> processor
>
> Hi.
>
> GetMongo can take input. So technically you can use a processor before and
> then connect it  to GetMongo.
>
> A simple example :
> GenerateFlowfile -> UpdateAttribute -> GetMongo
>
> In the UpdateAttribute, you can add attributes for the database and
> collection and then use them in GetMong using NiFi Expression Language.
>
> Let me know, if that doesn’t help.
>
> -
> Sivaprasanna
>
> On Wed, 2 May 2018 at 1:26 PM, Brajendra Mishra <
> brajendra_mis...@persistent.com> wrote:
>
> > Hi Team,
> > We have found there is only 'GetMongoDB' processor to connect and
> > query to MongoDB in Apache NiFi.
> > Hence, we this processor does not take any type or input.
> >
> > Do we have another type to Apache NiFi processor which can take
> > parameters as input (details of MongoDB, query, instance etc.) from
> other processor?
> > If not then please suggest when such type of processor can be expected
> > in upcoming release?
> >
> > Brajendra Mishra
> > Persistent Systems Ltd.
> >
> > DISCLAIMER
> > ==
> > This e-mail may contain privileged and confidential information which
> > is the property of Persistent Systems Ltd. It is intended only for the
> > use of the individual or entity to which it is addressed. If you are
> > not the intended recipient, you are not authorized to read, retain,
> > copy, print, distribute or use this message. If you have received this
> > communication in error, please notify the sender and delete all copies
> of this message.
> > Persistent Systems Ltd. does not accept any liability for virus
> > infected mails.
> >
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>



f6620488-0162-1000-acf2-cb751ab6e617
Hello Mongo


944a5b06-7f3e-383e--
ab629e9a-a9e2-3dcc--
1 GB
1

ab629e9a-a9e2-3dcc--
ec33b40f-e496-3032--
PROCESSOR

0 sec
1

success

ab629e9a-a9e2-3dcc--
c97a1213-a63e-38ea--
PROCESSOR

0


955cdf67-0ef5-32af--
ab629e9a-a9e2-3dcc--
1 GB
1

ab629e9a-a9e2-3dcc--
f9778f4b-ba15-3efb--
PROCESSOR

0 sec
1

success

ab629e9a-a9e2-3dcc--
ec33b40f-e496-3032--
PROCESSOR

0


fe76ca18-692e-3adb--
ab629e9a-a9e2-3dcc--
1 GB
1

502.5
429.0


ab629e9a-a9e2-3dcc--
53ce9b6e-7c8f-358c--
PROCESSOR

0 sec
1

success

ab629e9a-a9e2-3dcc--

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-02 Thread Sivaprasanna

Hi.

GetMongo can take input. So technically you can use a processor before and
then connect it  to GetMongo.

A simple example :
GenerateFlowfile -> UpdateAttribute -> GetMongo

In the UpdateAttribute, you can add attributes for the database and
collection and then use them in GetMong using NiFi Expression Language.

Let me know, if that doesn’t help.

-
Sivaprasanna

On Wed, 2 May 2018 at 1:26 PM, Brajendra Mishra <
brajendra_mis...@persistent.com> wrote:

> Hi Team,
> We have found there is only 'GetMongoDB' processor to connect and query to
> MongoDB in Apache NiFi.
> Hence, we this processor does not take any type or input.
>
> Do we have another type to Apache NiFi processor which can take parameters
> as input (details of MongoDB, query, instance etc.) from other processor?
> If not then please suggest when such type of processor can be expected in
> upcoming release?
>
> Brajendra Mishra
> Persistent Systems Ltd.
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>

Re: RAT errors after pull of master

2018-04-30 Thread Sivaprasanna

This is the maven unable to identify the iml and meta files that Intellij
IDEA generates. Pierre or someone has already shared a way to solve this
earlier in one dev thread. I’ll try to find but you can just search “rat
checks” in th dev threads. You’ll find it.

-
Sivaprasanna

On Mon, 30 Apr 2018 at 7:10 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> Anyone else seeing:
>
> [WARNING] Files with unapproved licenses:
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-client-service-api/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-client-service-api/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-client-service-api/target/maven-archiver/pom.properties
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-client-service-api/target/.plxarc
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-client-service-api/nifi-mongodb-client-service-api.iml
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/inputFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/createdFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/maven-archiver/pom.properties
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/target/.plxarc
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services/nifi-mongodb-services.iml
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/target/.plxarc
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services-bundle.iml
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services-nar/target/.plxarc
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-mongodb-services-bundle/nifi-mongodb-services-nar/nifi-mongodb-services-nar.iml
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/target/maven-archiver/pom.properties
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/target/.plxarc
>
>
> /Users/ottofowler/src/apache/forks/nifi/nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/nifi-elasticsearch-client-service-api.iml
> [INFO] 
>

Re: [DISCUSS] Support for accessing sensitive values safely

2018-04-26 Thread Sivaprasanna

Andy, That's exactly what I had in mind, but without a separate textbox
popping up. What I had originally though, there is a checkbox instead of a
button named "Use Variables". If it is checked, whatever the user types,
AJAX comes in to play and suggests ( auto-completion ) variables that the
user has access to. If the box is not checked, it is going to be a literal
password. But in this case, the normal textbox that pops up when a property
is edited, because it is more of a "textarea" that a textbox and the
suggestion dropdown would feel awkward there. I haven't worked extensively
on UI so not sure about how difficult it is going to be. Definitely UI
folks would pitch in much better ideas.

Thanks,
Sivaprasanna

On Thu, Apr 26, 2018 at 7:25 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Yes, I think a dynamically-populated dropdown is unnecessary, and I think
> there might be a way to do this without requiring complex EL parsing.
>
> For sensitive properties, I imagine a text field as we have now, with a
> button to the right that says “Use Variable”. I do not anticipate having to
> support a combination of variable interpolation and literal text (i.e. no
> one’s password should be “${db.password}12345”), so we only have to support
> the use cases of *literal password entry* OR *variable population*.
>
> If the user clicks “Use Variable”, a dialog would appear which has a text
> field and allows the user to type, and uses an AJAX function to call a
> query API endpoint, which allows them to filter the available variables as
> they type. For example, if the user types “d”, the following list might
> appear:
>
> data.endpoint
> db.url
> db.username
> db.password
>
> As the user continues to type “db.p”, only “db.password” is available. The
> user selects this (thus not allowing true freeform input to the final
> value, only a predetermined list of valid values). Obviously this API would
> need to be protected with proper authorization to ensure that only the
> variables available to that user were exposed, but this would be the case
> on validation as well to ensure the user didn’t manually type in a variable
> reference they were not allowed to use.
>
> I’ve included a couple mockups of how I envision this. I’ve implemented
> the same thing in a Rails app a few years ago, so I know it’s possible, and
> with modern libraries, should not be too difficult. This should also
> demonstrate that people like Rob, Matt, and Scott who actually design the
> UI/UX that goes into NiFi are invaluable.
>
> Current sensitive property entry field
> Proposed “Use Variable” button
>
> “Use Variable” dialog with empty text field
>
> User starts typing, triggering AJAX query
>
> User continues typing, narrowing down result set, and selects it
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 26, 2018, at 9:05 AM, Bryan Bende <bbe...@gmail.com> wrote:
>
> That is a fair point about the list of variables potentially being
> long, we would probably want to get some UI/UX recommendations from
> the folks that have worked in that are the most.
>
> In practice I wonder if it would really be an issue though...
>
> The variables that would be selectable would only be the sensitive
> variables the user has access to, not all variables. I would imagine
> out of hundreds of variables only a handful are sensitive variables.
> It is similar to processor properties in that there are tons of
> properties across all processors, but only a small batch of sensitive
> properties (I just counted 63 references to sensitive(true) in the
> codebase).
>
> The other factor is that it would only be the sensitive variables that
> are in scope to where you are in the data flow, using the same logic
> of how variables are resolved in a hierarchical order. So you would
> only see sensitive variables that are in the current process group or
> a parent group.
>
> As far as knowing if the enter plain password vs. EL, that was the
> reason I suggested selecting a variable from a list. That is the
> mechanism that tells us that a variable was used. The hope was that
> you could enter free form text or select a variable, any time free
> form text was used it would function the way it does today where the
> value is cleared out when saved to registry. If a variable was
> selected then it could keep the variable name.
>
> If we didn't do a list of variables then we would need a checkbox or
> something for the user to indicate whether the sensitive value is a
> literal or expression language.
>
>
> On Thu, Apr 26, 2018 at 5:24 AM, S

Re: [DISCUSS] Support for accessing sensitive values safely

2018-04-26 Thread Sivaprasanna

Initially when I thought of this, I imagined that we can still leverage
`PropertyDescriptor` and just add new method(s) to handle the sensitive
variables. Some thought has to be put on this.

*"It would provide a list of variables that are readable to the current
user and one can be selected"*
I think this might lead to bad UX. When the number of variables grow,
rendering the variables in a drop-down list may not be that good, IMHO.
Maybe we can still stick with textbox and do the following:

   - Once a user enters sensitive variable expression language, the
   framework does a background check if the user is authorized to use this
   variable. If the user doesn't have access, render the same and make the
   component invalid there by preventing it to be enabled/started

Only catch is how are we specifying what the user entered is plain password
and not EL. Either we can go with the assumption that all sensitive values
have to be taken from Variable Registry and only the EL for those variables
will have to be provided as the sensitive property value but this would
restrict the developers' freedom to choose what approach they want. Or we
can tweak the UI and present the option better, like have a checkbox which
when checked, will consider that as an EL and evaluate from the variable
registry and when not checked, assume that the entered password is plain
password and no evaluation needs to happen.

-

Sivaprasanna

On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende <bbe...@gmail.com> wrote:

> The policy model would need more thought, but the point would be that
> a user can select variable references they have been given permission
> to.
>
> In order to configure the processor that is referencing the variable,
> they already need write permissions to that processor, or some parent
> in the hierarchy if no specific policy exists.
>
>
>
> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
> >
> > "It would provide a list of variables that are readable to the current
> user
> > and one can be selected, just like allowable values or controller
> services.”
> >
> > A person may have rights to configure nifi without knowing the “value” of
> > the secure db password ( for example ), but that doesn’t mean they
> > don’t have there rights to reference it.
> >
> >
> >
> > On April 25, 2018 at 14:15:16, Bryan Bende (bbe...@gmail.com) wrote:
> >
> > There is definitely room for improvement here.
> >
> > Keep in mind that often the sensitive information is specific to a
> > given environment. For example you build a flow in dev with your
> > db.password. You don't actually want your dev db password to be
> > propagated to the next environment, but you do want to be able to set
> > a variable placeholder like ${db.password} and leave that placeholder
> > so you can just set that variable in the next environment. So to me
> > the goal here is how to handle secure variables.
> >
> > Andy highlighted many of the issues, my proposal would be the
> following...
> >
> > First, we can introduce a concept of a sensitive variable. This would
> > be something in the UI where a user can indicate a variable is
> > sensitive, maybe a checkbox, and then the framework can store these
> > values encrypted (currently all variable values are stored in plain
> > text because they aren't meant to be sensitive).
> >
> > Second, we can introduce policies on sensitive variables so that we
> > can restrict who can read them elsewhere, just like policies on
> > controller services that determine which controller services show up
> > in the drop down of a processor.
> >
> > Third, we introduce a new kind of PropertyDescriptor that allows
> > selecting a variable from the variable registry rather than free-form
> > expression language. It would provide a list of variables that are
> > readable to the current user and one can be selected, just like
> > allowable values or controller services. Ideally we can have a way to
> > still allow free form values for people who don't want to use
> > variables.
> >
> > Fourth, anytime variables are evaluated from expression language we
> > would prevent evaluating any of these new sensitive variables since we
> > have no way of knowing if a user should have access to it from
> > free-form EL, so they can only be used from the special
> > PropertyDescriptors above.
> >
> > If we put all this in place then when we save flows to the registry,
> > we can leave the variable place-holders in the sensitive properties,
> > and then when you import to the next environment you only need to edit
> > the var

Support for accessing sensitive values safely

Hi

Since flowfile attributes and VariableRegistry is not suitable (not safe,
to be specific), developers have to rely on manually configuring the
sensitive values on the components (Processors & ControllerServices). And
during CI/CD (using flow registry), the sensitive information are dropped
and once imported to the next environment (QA or Prod), the user is
expected to configure the sensitive information again, although for the
first time. How about we introduce sort of a 'vault' that holds sensitive
values which could possibly avoid this unnecessary step completely ?

-
Sivaprasanna

Re: Custom Controller Service

Okay.. but two questions:


   1. We are passing the attribute 'db.id' that means, we'll be using
   'UpdateAttribute' to do add that attribute to flowfile?
   2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we
   need to know before hand, right?

-

Sivaprasanna

On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Charlie,
>
> That is a really nice solution, thanks for sharing.
>
> If we make the API changes in that JIRA I just sent, I could see
> having a new implementation of DBCPService that does something very
> similar.
>
> Basically there could be "DelegatingDBCPService" which still
> implemented the same DBCPService interface but followed a convention
> where it always looked in the attribute map for an attribute called
> "db.id", and then it itself has dynamic properties to define many
> DBCPServices where the property name was the db.id, and it would just
> return a Connection from the one with the given id.
>
> There are definitely other options for how to implement the dynamic
> connection service, but this would be a good one to have.
>
> -Bryan
>
>
> On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer
> <charlie.me...@civitaslearning.com> wrote:
> > Chiming in a bit late on this, but we faced this same issue and got
> around
> > it by implementing a custom controller service which acts as a "router"
> to
> > different dbcp services. It exposes a method which given a uuid, returns
> > back the DBCPservice that corresponds with that uuid if it exists using
> >
> >  DBCPService dbcpService =
> > (DBCPService)
> > getControllerServiceLookup().getControllerService(uuid);
> >
> > From there, we created processors we needed based on the stock ones which
> > relied on our "router" service rather than a single DBCP. Our custom
> > processors read an attribute from incoming flow files, then send that to
> > the router which returns back the connection pool.
> >
> > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote:
> >
> >> Here is a proposal for how to modify the existing API to support both
> >> scenarios:
> >>
> >> https://issues.apache.org/jira/browse/NIFI-5121
> >>
> >> The scope of that ticket would be to make the interface change, and
> >> then update all of NiFi's DB processors to pass in the attribute map.
> >>
> >> Then a separate effort to provide a new service implementation that
> >> used the attribute map to somehow manage multiple connection pools, or
> >> create connections on the fly, or whatever the desired behavior is.
> >>
> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote:
> >> > To Otto's question...
> >> >
> >> > For simplicity sake, there is a new implementation of
> >> > DBCPConnectionPool that behind the scenes has two connection pools,
> >> > one for DB A and one for DB B, it doesn't matter how these are
> >> > configured.
> >> >
> >> > Now a flow file comes into the ExecuteSQL and it goes to
> >> > connectionPoolService.getConnection()...
> >> >
> >> > How does it know which DB to return a connection for?
> >> >
> >> >
> >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <
> sivaprasanna...@gmail.com>
> >> wrote:
> >> >> Option 2 and 3 seem to be a probable approach. However creating
> varying
> >> >> number of connections based on *each* flowfile still sounds to be
> >> >> suboptimal. If the requirement still demands to take that road, then
> >> it’s
> >> >> better to do some prep-work.. as in the list of probable connections
> >> that
> >> >> are required are taken and connection pools are created for them and
> >> then
> >> >> based on the flowfiles (which connection it needs), we use the
> relevant
> >> >> one.
> >> >>
> >> >> Thanks,
> >> >> Sivaprasanna
> >> >>
> >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com>
> wrote:
> >> >>
> >> >>> The issue here is more about the service API and not the
> >> implementations.
> >> >>>
> >> >>> The current API has no way to pass information between the processor
> >> and
> >> >>> service.
> >> >>>
> >> >>> The options boil down to...
>

Re: Is there a configuration to limit the size of nifi's flowfile repository

No, he actually had mentioned “like content repository”. The answer is,
there aren’t any properties that support this, AFAIK. Pierre’s response
pretty much sums up why there aren’t any properties.

Thanks,
Sivaprasanna

On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen <mikerthom...@gmail.com> wrote:

> I have a feeling that what Ben meant was how to limit the content
> repository size.
>
> On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hi Ben,
> >
> > Since the flow file repository contains the information of the flow files
> > currently being processed by NiFi, you don't want to limit that
> repository
> > in size since it would prevent the workflows to create new flow files.
> >
> > Besides this repository is very lightweight, I'm not sure it'd need to be
> > limited in size.
> > Do you have a specific use case in mind?
> >
> > Pierre
> >
> >
> > 2018-04-25 9:15 GMT+02:00 尹文才 <batman...@gmail.com>:
> >
> > > Hi guys, I checked NIFI's system administrator guide trying to find a
> > > configuration item so that the size of the flowfile repository could be
> > > limited similar to the other repositories(e.g. content repository),
> but I
> > > didn't find such configuration items, is there currently any
> > configuration
> > > to limit the flowfile repository's size? thanks.
> > >
> > > Regards,
> > > Ben
> > >
> >
>

Re: Custom Controller Service

Option 2 and 3 seem to be a probable approach. However creating varying
number of connections based on *each* flowfile still sounds to be
suboptimal. If the requirement still demands to take that road, then it’s
better to do some prep-work.. as in the list of probable connections that
are required are taken and connection pools are created for them and then
based on the flowfiles (which connection it needs), we use the relevant
one.

Thanks,
Sivaprasanna

On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote:

> The issue here is more about the service API and not the implementations.
>
> The current API has no way to pass information between the processor and
> service.
>
> The options boil down to...
>
> - Make a new API, but then you need all new processors that use the new API
>
> - Modify the current API to have a new method, but then we are combing two
> concepts into one API and some impls may not implement both
>
> - Modify the processors to use two different service APIs, but enforce that
> only one can be used at a time, so it can have either the original
> connection pool service or can have some new dynamic connection factory,
>  but not both, and then modify all DB processors to have logic to determine
> which service to use.
>
> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Or you could just call every time you needed properties more likely.
> > This would still be custom unless integrated….
> >
> >
> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Can services work with other controller services?
> > Maybe a PropertiesControllerService, FilePropertiesControllerService
> could
> > work with your service?
> > the PCS could fire events on property changes etc.
> >
> >
> >
> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com)
> > wrote:
> >
> > Shot in the dark here, but what you try to do is create a custom
> connection
> > pool service that uses dynamic properties to build a "pool of connection
> > pools." You could then use the property names as hints for where to send
> > the queries.
> >
> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com
> >
> > wrote:
> >
> > > Hi,
> > >
> > > Basically, there are 'n' number of databases that we are dealing with.
> We
> > > need to fetch the data from the source database into HDFS. Now since we
> > are
> > > dealing with many databases, the source database is not static and
> > changes
> > > every now and then. And every time the source database changes we
> > manually
> > > need to change the value for the connection parameters in
> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create
> 'n'
> > > connections for each database, but that is not possible because 'n' is
> a
> > > big number and creating that many connections in DBCPConnectionPool is
> > not
> > > possible. So we were looking for a way where we can specify all the
> > > connection parameters in a file present in our local system and then
> make
> > > the DBCPConnectionPool controller service to read the values from the
> > file.
> > > In that way we can simply change the value in the file present in the
> > local
> > > system. No need to alter anything in the dataflow. But it turns out
> that
> > > FlowFile attributes are not available to the controller services as the
> > > expression language is evaluated at the time of service enable.
> > >
> > > So can you suggest a way where I can achieve my requirement (except
> > > 'variable.registry' ) ? I am looking to develop a custom controller
> > service
> > > that can serve the requirement but how do I make the flowfile
> attributes
> > > available to the service?
> > >
> >
> --
> Sent from Gmail Mobile
>

Re: Nifi Custom Processor Import Error

2018-04-20 Thread Sivaprasanna

You have to build it locally after downloading or cloning.

-
Sivaprasanna

On Fri, 20 Apr 2018 at 4:16 PM, rishabprasad005 <rishabprasad...@gmail.com>
wrote:

> How can we download a maven dependency for nifi having version
> "1.7.0-SNAPSHOT". I tried doing this but gets an error since the latest
> version is 1.6.0.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: No controller service types found that are applicable for this property

2018-04-19 Thread Sivaprasanna

Hi,

Have you added ‘nifi-dbcp-service-api’ as a dependency in the processor’
POM ?

-
Sivaprasanna

On Thu, 19 Apr 2018 at 8:14 PM, Rishab Prasad <rishabprasad...@gmail.com>
wrote:

>
> Hi,
>
> I am new to Apache Nifi. So I have created a new custom processor by copying 
> the source code of *ListDatabaseTable* processor and build it using maven.
>
> Then I copied nar file to lib folder and restarted it. Then I try to 
> configure the property for Database Connection Pooling Service, so I choose 
> **create new service**
>
> option. It throws the following error "*No controller service types found 
> that are applicable for this property"*. Please help me resolving this.
>
>
>
> ‌
>

Re: [EXT] Suggestion: Apache NiFi component enhancement

2018-04-18 Thread Sivaprasanna

Mike,

Thanks for the response and those are valid concerns.. a couple of them I
didn't even think about in the first place. Here are my thoughts:

*   - When searching by component type, will I be able to find components
by both class name and annotation name?*
Yes. That way it adds more value.

*   - Will the flow.xml contain the class name or the annotation name or
both?  If just one, might people look through the flow.xml for the other
 name and not find anything?*
   Since my intention was to limit the effects to just the UI, I dint
think of writing the annotation anywhere other than using it in the DTO so
the flow.xml won't have the annotation is my assumption. Feel free to share
your idea on this.

*   - Would the auto-generated documentation contain both class name and
annotation name?*
Yes. It should.

*- Would this have any impact to the Registry?  Perhaps not yet, but
does it fit with the desire to include component extensions in the
Registry?*
   I don't have a clear idea on this yet.

*- Would we have to modify the Logger messages to include the
annotation name instead of the class name in the logs?*
   I think that would be neat.. having the annotation name in the
logger messages

Having said that, I'm fairly new to contributing to Apache NiFi so I might
be wrong in some things, like with my assumption or something so feel free
to correct me, and others if this interests you, please pitch in your
thoughts.

*-*
Sivaprasanna

On Fri, Apr 13, 2018 at 9:52 PM, Michael Moser <moser...@gmail.com> wrote:

> Hi Sivaprasanna,
>
> After reading your first email, I thought it would be a lot of work for
> little benefit, because I think you would have needed to touch many parts
> of the framework.
>
> After reading your clarification, the scope seems more limited.  It's
> definitely an interesting idea, and here are my thoughts
>
>- When searching by component type, will I be able to find components by
>both class name and annotation name?
>- Will the flow.xml contain the class name or the annotation name or
>both?  If just one, might people look through the flow.xml for the other
>name and not find anything?
>- Would the auto-generated documentation contain both class name and
>annotation name?
>- Would this have any impact to the Registry?  Perhaps not yet, but does
>it fit with the desire to include component extensions in the Registry?
>- Would we have to modify the Logger messages to include the annotation
>name instead of the class name in the logs?
>
> Thank you for your engagement with NiFi and for thinking of ways to improve
> it!
>
> -- Mike
>
>
>
> On Fri, Apr 13, 2018 at 11:50 AM, Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
>
> > Busy week, eh?
> >
> > Anybody is having any concerns, suggestions? Any input is appreciated :)
> >
> > Cheers,
> > Sivaprasanna
> >
> > On Thu, Apr 12, 2018 at 10:14 PM, Sivaprasanna <
> sivaprasanna...@gmail.com>
> > wrote:
> >
> > > No my suggestion was a simpler approach. It just affects only the UI
> > > aspect as my intention is to just override how the 'type' gets rendered
> > in
> > > the UI. For example, a processor's type is set to its canonical class
> > name (
> > > DtoFactory.java#createProcessorDto
> > > <https://github.com/apache/nifi/blob/master/nifi-nar-
> > bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-
> > web-api/src/main/java/org/apache/nifi/web/api/dto/
> DtoFactory.java#L2783>)
> > > but rather than getting the canonical class name, let's just get from
> > some
> > > other method that checks if the new annotation is present, if it is
> > > present, set the value provided in the annotation as the type, if it's
> > not
> > > present set the canonical class name just like how it is now. Again, my
> > > intention is to just affect the UI so as to avoid making unnecessary
> > > complication that could pose some backwards compatibility issue.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Thu, Apr 12, 2018 at 1:35 PM, Peter Wicks (pwicks) <
> pwi...@micron.com
> > >
> > > wrote:
> > >
> > >> I think this is a good idea. But based on your example I think you
> would
> > >> want to provide a primary Type along with a list of Alias types.
> > >> If NiFi starts and it can no longer find a processor by the Type name
> it
> > >> had in the flow.xml it can check he annotations/aliases to see if it's
> > been
> > >> renamed. This would allow for easy renames.
> > >>
> > >

Re: The import org.apache.nifi.dbcp.DBCPService cannot be resolved

2018-04-17 Thread Sivaprasanna

*not the implementation bundle itself

On Tue, Apr 17, 2018 at 6:33 PM, Sivaprasanna <sivaprasanna...@gmail.com>
wrote:

> Yep. That's correct. You have to use the api and the implementation bundle
> itself. Add the following to your processor's POM:
>
> 
> org.apache.nifi
> nifi-dbcp-service-api
> 1.6.0
> 
>
> Change the version if you're trying to deploy your custom processor bundle
> on a different NiFi version.
>
> -
> Sivaprasanna
>
> On Tue, Apr 17, 2018 at 5:31 PM, Mike Thomsen <mikerthom...@gmail.com>
> wrote:
>
>> Did you declare a dependency on nifi-dbcp-service-api? If not, you have to
>> do that and set the scope to "provided.
>>
>> On Tue, Apr 17, 2018 at 7:05 AM Rishab Prasad <rishabprasad...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I am trying to create a custom processor equivalent to that of
>> > ListDatabaseTable. I copied and pasted the source code of
>> > ListDatabaseTableand
>> > used the same for my custom processor. But I am getting the following
>> > error: -
>> >
>> > [image: enter image description here] <https://i.stack.imgur.com/370
>> N8.png
>> > >
>> >
>> > The error says *The import org.apache.nifi.dbcp.DBCPService cannot be
>> > resolved*. I tried downloading this jar file
>> > <
>> > https://mvnrepository.com/artifact/org.apache.nifi/nifi-dbcp
>> -service/1.5.0
>> > >
>> >  for org.apache.nifi.dbcp and added it as an external jar in my build
>> path.
>> > The error still can't be resolved. How can i resolve this error?
>> > ‌
>> >
>>
>
>

Re: The import org.apache.nifi.dbcp.DBCPService cannot be resolved

2018-04-17 Thread Sivaprasanna

Yep. That's correct. You have to use the api and the implementation bundle
itself. Add the following to your processor's POM:


org.apache.nifi
nifi-dbcp-service-api
1.6.0


Change the version if you're trying to deploy your custom processor bundle
on a different NiFi version.

-
Sivaprasanna

On Tue, Apr 17, 2018 at 5:31 PM, Mike Thomsen <mikerthom...@gmail.com>
wrote:

> Did you declare a dependency on nifi-dbcp-service-api? If not, you have to
> do that and set the scope to "provided.
>
> On Tue, Apr 17, 2018 at 7:05 AM Rishab Prasad <rishabprasad...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am trying to create a custom processor equivalent to that of
> > ListDatabaseTable. I copied and pasted the source code of
> > ListDatabaseTableand
> > used the same for my custom processor. But I am getting the following
> > error: -
> >
> > [image: enter image description here] <https://i.stack.imgur.com/
> 370N8.png
> > >
> >
> > The error says *The import org.apache.nifi.dbcp.DBCPService cannot be
> > resolved*. I tried downloading this jar file
> > <
> > https://mvnrepository.com/artifact/org.apache.nifi/nifi-
> dbcp-service/1.5.0
> > >
> >  for org.apache.nifi.dbcp and added it as an external jar in my build
> path.
> > The error still can't be resolved. How can i resolve this error?
> > ‌
> >
>

Re: [EXT] Suggestion: Apache NiFi component enhancement

2018-04-13 Thread Sivaprasanna

Busy week, eh?

Anybody is having any concerns, suggestions? Any input is appreciated :)

Cheers,
Sivaprasanna

On Thu, Apr 12, 2018 at 10:14 PM, Sivaprasanna <sivaprasanna...@gmail.com>
wrote:

> No my suggestion was a simpler approach. It just affects only the UI
> aspect as my intention is to just override how the 'type' gets rendered in
> the UI. For example, a processor's type is set to its canonical class name (
> DtoFactory.java#createProcessorDto
> <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java#L2783>)
> but rather than getting the canonical class name, let's just get from some
> other method that checks if the new annotation is present, if it is
> present, set the value provided in the annotation as the type, if it's not
> present set the canonical class name just like how it is now. Again, my
> intention is to just affect the UI so as to avoid making unnecessary
> complication that could pose some backwards compatibility issue.
>
> -
> Sivaprasanna
>
> On Thu, Apr 12, 2018 at 1:35 PM, Peter Wicks (pwicks) <pwi...@micron.com>
> wrote:
>
>> I think this is a good idea. But based on your example I think you would
>> want to provide a primary Type along with a list of Alias types.
>> If NiFi starts and it can no longer find a processor by the Type name it
>> had in the flow.xml it can check he annotations/aliases to see if it's been
>> renamed. This would allow for easy renames.
>>
>> Example 1: NiFi can no longer find AzureDocumentDBProcessor. Developer
>> renamed it to CosmosDBProcessor. In this case we don't really want the type
>> to still same "DocumentDB", that's just confusing. Also, we might not want
>> the type named CosmosDBProcessor. So we make the Type be something nice,
>> like "Azure Comos DB", then add Aliases for "AzureDocumentDBProcessor" and
>> "CosmosDBProcessor".
>>
>> Next year when Microsoft renames it "CelestialDB" we can rename the
>> processor and add another alias.
>>
>> Something like that?
>>
>> -Original Message-
>> From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
>> Sent: Wednesday, April 11, 2018 23:37
>> To: dev@nifi.apache.org
>> Subject: [EXT] Suggestion: Apache NiFi component enhancement
>>
>> All,
>>
>> Currently the "type" of a component is actually the component's canonical
>> class name which gets rendered in the UI as the class name with the
>> component version. This is good. However I'm thinking it is better to have
>> an annotation which a developer can use to override the component type.
>>
>> How is it used?
>> I think an annotation can be sufficient. The framework checks if the
>> annotation is present or not, if it is present, it uses the name provided
>> there or else it uses the class name like how it is happening.
>>
>> Why and where is it needed?
>>
>>- In scenarios where we devise a new naming convention and want to
>> apply
>>it to older components without breaking backward compatibility
>>- A developer had created a component class with a name but later down
>>the line, the developer or someone else wants to change it to something
>>else, the reason could again be naming convention or just that the new
>> name
>>makes more sense
>>- A component that has been built to work with third party tech, like
>>Azure, MongoDB, S3, Druid processors but the later versions of that
>> tech
>>has been changed to something else by the original creators. (Something
>>similar has happened to Azure's DocumentDB which got later rebranded as
>>Azure CosmosDB). In such cases, without deprecating or rebuilding a new
>>processor, this can be used.
>>
>> Before creating a JIRA, I wanted to get the community's thoughts. Feel
>> free to share your thoughts, concerns. If everything seems fine, I'll start
>> working on the implementation.
>>
>> -
>>
>> Sivaprasanna
>>
>
>

Re: PutHDFS error

2018-04-13 Thread Sivaprasanna

Glad that you have solved it.. If possible can you share how you resolved
it? It might help others who might face this issue in the future.

-
Sivaprasanna

On Fri, 13 Apr 2018 at 7:05 PM, hemamoger <hemamo...@gmail.com> wrote:

> thanks for your suggestions.
> I have got the answer.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: [EXT] Suggestion: Apache NiFi component enhancement

2018-04-12 Thread Sivaprasanna

No my suggestion was a simpler approach. It just affects only the UI aspect
as my intention is to just override how the 'type' gets rendered in the UI.
For example, a processor's type is set to its canonical class name (
DtoFactory.java#createProcessorDto
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/DtoFactory.java#L2783>)
but rather than getting the canonical class name, let's just get from some
other method that checks if the new annotation is present, if it is
present, set the value provided in the annotation as the type, if it's not
present set the canonical class name just like how it is now. Again, my
intention is to just affect the UI so as to avoid making unnecessary
complication that could pose some backwards compatibility issue.

-
Sivaprasanna

On Thu, Apr 12, 2018 at 1:35 PM, Peter Wicks (pwicks) <pwi...@micron.com>
wrote:

> I think this is a good idea. But based on your example I think you would
> want to provide a primary Type along with a list of Alias types.
> If NiFi starts and it can no longer find a processor by the Type name it
> had in the flow.xml it can check he annotations/aliases to see if it's been
> renamed. This would allow for easy renames.
>
> Example 1: NiFi can no longer find AzureDocumentDBProcessor. Developer
> renamed it to CosmosDBProcessor. In this case we don't really want the type
> to still same "DocumentDB", that's just confusing. Also, we might not want
> the type named CosmosDBProcessor. So we make the Type be something nice,
> like "Azure Comos DB", then add Aliases for "AzureDocumentDBProcessor" and
> "CosmosDBProcessor".
>
> Next year when Microsoft renames it "CelestialDB" we can rename the
> processor and add another alias.
>
> Something like that?
>
> -Original Message-
> From: Sivaprasanna [mailto:sivaprasanna...@gmail.com]
> Sent: Wednesday, April 11, 2018 23:37
> To: dev@nifi.apache.org
> Subject: [EXT] Suggestion: Apache NiFi component enhancement
>
> All,
>
> Currently the "type" of a component is actually the component's canonical
> class name which gets rendered in the UI as the class name with the
> component version. This is good. However I'm thinking it is better to have
> an annotation which a developer can use to override the component type.
>
> How is it used?
> I think an annotation can be sufficient. The framework checks if the
> annotation is present or not, if it is present, it uses the name provided
> there or else it uses the class name like how it is happening.
>
> Why and where is it needed?
>
>- In scenarios where we devise a new naming convention and want to apply
>it to older components without breaking backward compatibility
>- A developer had created a component class with a name but later down
>the line, the developer or someone else wants to change it to something
>else, the reason could again be naming convention or just that the new
> name
>makes more sense
>- A component that has been built to work with third party tech, like
>Azure, MongoDB, S3, Druid processors but the later versions of that tech
>has been changed to something else by the original creators. (Something
>similar has happened to Azure's DocumentDB which got later rebranded as
>Azure CosmosDB). In such cases, without deprecating or rebuilding a new
>processor, this can be used.
>
> Before creating a JIRA, I wanted to get the community's thoughts. Feel
> free to share your thoughts, concerns. If everything seems fine, I'll start
> working on the implementation.
>
> -
>
> Sivaprasanna
>

Re: NiFi to S3

2018-04-11 Thread Sivaprasanna

Adrian,

Current version of S3 processors in Apache NiFi don’t have the object
tagging feature. I think it was recently added to the AWS S3 SDK owing to
the new GDPR regulation announced by EU. Please open a JIRA to have this
feature added to the S3 processors.

-
Sivaprasanna

On Thu, 12 Apr 2018 at 6:00 AM, Adrian D <adnov...@gmail.com> wrote:

> Sir/Ma'am
>
> Is it possible to assign S3 object tags in NiFi upon writing? I am trying
> to implement object tagging as the PutS3Object processor is running. The
> closest reference to object tagging I found:
>
> https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
>
> Please let me know. Thank you.
>

Suggestion: Apache NiFi component enhancement

2018-04-11 Thread Sivaprasanna

All,

Currently the "type" of a component is actually the component's canonical
class name which gets rendered in the UI as the class name with the
component version. This is good. However I'm thinking it is better to have
an annotation which a developer can use to override the component type.

How is it used?
I think an annotation can be sufficient. The framework checks if the
annotation is present or not, if it is present, it uses the name provided
there or else it uses the class name like how it is happening.

Why and where is it needed?

   - In scenarios where we devise a new naming convention and want to apply
   it to older components without breaking backward compatibility
   - A developer had created a component class with a name but later down
   the line, the developer or someone else wants to change it to something
   else, the reason could again be naming convention or just that the new name
   makes more sense
   - A component that has been built to work with third party tech, like
   Azure, MongoDB, S3, Druid processors but the later versions of that tech
   has been changed to something else by the original creators. (Something
   similar has happened to Azure's DocumentDB which got later rebranded as
   Azure CosmosDB). In such cases, without deprecating or rebuilding a new
   processor, this can be used.

Before creating a JIRA, I wanted to get the community's thoughts. Feel free
to share your thoughts, concerns. If everything seems fine, I'll start
working on the implementation.

-

Sivaprasanna

Re: PutHDFS error

2018-04-08 Thread Sivaprasanna

1. You have shared screenshot for "hdfs dfs -ls" which will basically list
the contents under /user/root since you're executing that as a root but you
have provided "/root" in PutHDFS so please share the list of contents
available under /root i.e. hdfs dfs -ls /root
2. Have you tried Pierre's approach? That should work all the time since it
appends epoch timestamp to the filename which has no possibilities of
having duplicate.

On Mon, Apr 9, 2018 at 10:30 AM, hemamoger  wrote:

> please check the attached images puthdfs_properties.png
>  t951/puthdfs_properties.png>
> getsftp_confgrn.png
>  t951/getsftp_confgrn.png>
> hadoop_properties.hadoop_properties
>  t951/hadoop_properties.hadoop_properties>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: PutHDFS error

2018-04-06 Thread Sivaprasanna

Did you try the approach that Pierre suggested? That should work in all
cases. Can you please share the screenshot of the processor’s configuration
menu?

Also please make sure whatever path value that you have set for the
‘Directory’ property is prefixed with a ‘/‘. Otherwise it will look for the
relative path with respect to the user i.e. /user/nifi/. If
that’s the case, you maybhave to check that directory to see if a file with
the same name already exists.

-
Sivaprasanna

On Fri, 6 Apr 2018 at 12:33 PM, hemamoger <hemamo...@gmail.com> wrote:

> Yes i tried with replace and append that is throwing error that java is not
> supported something like that.
> But i dont have  that file in hadoop which i am pushing to hdfs. still its
> throwing that error .
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: PutHDFS error

2018-04-06 Thread Sivaprasanna

You can do what Pierre suggested. However, if you don’t want the already
existing file or want to overwrite that with the new file, configure the
PutHDFS processor and set ‘Conflict Resolution Strategy’ to “replace” or
“append” based on your requirement.

-
Sivaprasanna

On Fri, 6 Apr 2018 at 11:57 AM, Pierre Villard <pierre.villard...@gmail.com>
wrote:

> Hi,
>
> This means that the name of the file you're pushing in HDFS already exists
> in the target directory.
> Usually people use an UpdateAttribute processor before the PuHDFS to update
> the attribute 'filename' with a timestamped suffix.
> Something like: filename => ${filname}-${now()}
> To be changed according to your needs.
>
> Thanks,
> Pierre
>
>
> 2018-04-06 8:18 GMT+02:00 hemamoger <hemamo...@gmail.com>:
>
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/file/t951/data1.png
> > >
> >
> >
> >
> > Hi Team,
> >
> > I am getting error like in the above image. I am not sure wheather
> PUTHDFS
> > configuration is proper or not.
> > Can you pls help with the error. and also what are the confiration i need
> > to
> > setup for PUTHDFS processor.
> >
> > Thanks in advance.
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
>

Re: Read processor property in init()

2018-03-29 Thread Sivaprasanna

Yep. That’s correct.

On Thu, 29 Mar 2018 at 6:45 PM, Jeff Zemerick <jzemer...@apache.org> wrote:

> Thanks! Just to confirm, each time the processor is started the
> @OnScheduled annotated method is executed, right?
>
> Jeff
>
>
> On Wed, Mar 28, 2018 at 9:07 AM, Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
>
> > Just to add on top of what Mike said. The @OnScheduled annotation
> indicates
> > that the method that is marked with this annotation will run when a
> > processor is started every time. So basically the setup() will be called
> > and executed everytime the processor is started from the UI.
> >
> > On Wed, 28 Mar 2018 at 6:34 PM, Jeff Zemerick <jzemer...@apache.org>
> > wrote:
> >
> > > I will give that a go. Thanks for the quick answer, Mike!
> > >
> > > On Wed, Mar 28, 2018 at 9:01 AM, Mike Thomsen <mikerthom...@gmail.com>
> > > wrote:
> > > > Just do...
> > > >
> > > > @OnScheduled
> > > > public void setup(ProcessContext context) {
> > > > //Read properties and do setup.
> > > > }
> > > >
> > > > On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick <jzemer...@apache.org
> >
> > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> Is there a recommended method for making user-configurable property
> > > >> values available to a processor's init()? I would like to load a
> large
> > > >> index file but allow the user to specify the index's path. I am
> > > >> guessing that init() is executed too early to read user properties.
> > > >>
> > > >> Thanks for any suggestions.
> > > >>
> > > >> Jeff
> > > >>
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

Thanks, Scott. That helped.

On Thu, 29 Mar 2018 at 10:09 AM, James Wing  wrote:

> +1 (binding).  Ran through the release helper, worked with a test flow.
> Thanks for putting this together.
>
> On Mon, Mar 26, 2018 at 8:34 PM, Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.6.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1123
> >
> > The Git tag is nifi-1.6.0-RC2
> > The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > b5935ec81a7cbc048820781ac62cd96bbea5b232
> >
> > Checksums of nifi-1.6.0-source-release.zip:
> > SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> > SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> > a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 146 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12316020=12342422
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version1.6.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-1.6.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: ListSFTP incoming relationship

Should we really have to have an optional state saving functionality? If
the user is unaware of the implications and proceed to store the state then
what Andrew Grande mentioned will happen - possibilities of never ending
stream of state information being stored. If we still go with the optional
state management approach, documentation have to be clear in explaining the
implications.

Sivaprasanna

On Thu, 29 Mar 2018 at 9:28 AM, scott <tcots8...@gmail.com> wrote:

> Okay. So, a new processor called "ScanSFTP", allow incoming relationship
> where the content of the flow file is replaced with the list of matching
> files from the remote directory, then the list is filtered by the usual
> regex parameters like today. Optional state information is kept to
> additionally filter the list of files older than the newest file
> observed during the last run. Does that sound okay to everyone? If so,
> what's the next step?
>
> Scott
>
>
> On 03/27/2018 06:21 PM, scott wrote:
> >
> > This is a great discussion, and appreciate the interest in my problem.
> > I think there are workarounds if you decide not to store state, but
> > I'd recommend keeping it. I think state should be kept optionally,
> > even turned off by default. Several times I've had issues where the
> > state has cause me to miss files, because files get moved into the
> > source folder out of order, and I've wished I could turn the state
> > feature off.
> >
> > In my current use-case, I would not be frequently, dynamically
> > changing the source directory, though I can see the use-cases where it
> > would be. In my current use-case, I want to use an external database
> > table to control the configuration of all my flows. I do this by first
> > reading the content of the table for this particular flow ID, then
> > assign the result as attributes to the flowfile, essentially creating
> > variables I can use throughout the flow to control its behavior. This
> > works great with flows that initiate with HTTP or SQL, but not
> > ListSFTP or ListFile.
> >
> > Scott
> >
> >
> > On 03/27/2018 02:05 PM, Andy LoPresto wrote:
> >> I think Bryan’s point is a good one and when I first saw this
> >> question (and thought of the previous times it’s been asked), my
> >> initial response is to propose a second processor.
> >>
> >> Something like “ScanSFTP”/“IndexSFTP”/“SnapshotSFTP” which operates
> >> differently from ListSFTP — it does not maintain state, and performs
> >> a one-time tabulation/chronicling of the state of that directory at
> >> the given point in time.
> >>
> >> The responsibility to maintain and compare state across time is no
> >> longer a requirement. There could even be a setting in the processor
> >> to allow for “individual flowfile output” (i.e. act the same as
> >> ListSFTP and output one flowfile per item listed) or “summary
> >> flowfile output” where a single flowfile is generated containing the
> >> directory listing information for all the items there. (Another
> >> option is to output both on two different relationships).
> >>
> >> I think this would enable the types of workflows that users have
> >> asked about in the past without compromising the mechanism by which
> >> List* processors work and adding undue complexity to those processors.
> >>
> >> Absolutely crystal clear documentation (and a standard verb for the
> >> new processor family) would be necessary (not only because these
> >> processor solve different problems, but to avoid a million variants
> >> of “I used ScanSFTP processor and it’s not tracking state”/“How do I
> >> provide a directory in an attribute to ListSFTP” mailing list
> >> questions).
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org <mailto:alopre...@apache.org>
> >> /alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>/
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >>> On Mar 27, 2018, at 8:33 AM, Andrew Grande <apere...@gmail.com
> >>> <mailto:apere...@gmail.com>> wrote:
> >>>
> >>> The key here is that ListXXX processor maintains state. A directory
> >>> is part
> >>> of such state. Allowing arbitrary directories via an expression would
> >>> create never ending stream of new entries in the state storage,
> >>> effectively
> >>> engineering a distributed DoS attack on the NiFi node or shared ZK
> >>> quorum
> >&g

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

- Confirmed hashes and signature

Tried to build in a fresh Ubuntu VM that doesn't have NiFi installed or
configured previously. Build fails at nifi-web-ui module. Going by how it
works for others, it very well could be a isolated problem on my end.
However still wanted to get this posted.

Error snippet:

[ERROR] npm ERR! Error: No dist in undefined package
[ERROR] npm ERR! at next
(/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/lib/cache.js:746:26)
[ERROR] npm ERR! at
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/lib/cache.js:739:5
[ERROR] npm ERR! at RegClient.get_
(/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/node_modules/npm-registry-client/lib/get.js:105:14)

Also attached the error message.

Environment:
Ubuntu 16.04 LTS x64

Thanks,
Sivaprasanna

On Wed, Mar 28, 2018 at 8:31 PM, Kevin Doran <kdo...@apache.org> wrote:

> +1
>
> I ran through the steps in the release helper guide and everything looks
> good.
>
> Thanks for handling RM duties, Joe.
>
> --Kevin
>
> On 3/26/18, 23:34, "Joe Witt" <joew...@apache.org> wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1123
>
> The Git tag is nifi-1.6.0-RC2
> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424
> beb9
> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 146 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>
>
>
INFO] Installed node locally.
[INFO] Installing npm version 1.3.8
[INFO] Unpacking 
/root/.m2/repository/com/github/eirslett/npm/1.3.8/npm-1.3.8.tar.gz into 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules
[INFO] Installed npm locally.
[INFO] 
[INFO] --- frontend-maven-plugin:1.1:npm (npm install) @ nifi-web-ui ---
[DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.1:npm from 
plugin realm ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.1, 
parent: sun.misc.Launcher$AppClassLoader@33909752]
[DEBUG] Configuring mojo 'com.github.eirslett:frontend-maven-plugin:1.1:npm' 
with basic configurator -->
[DEBUG]   (f) arguments = --cache-min Infinity install
[DEBUG]   (f) installDirectory = 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
[DEBUG]   (f) npmInheritsProxyConfigFromMaven = true
[DEBUG]   (f) project = MavenProject: org.apache.nifi:nifi-web-ui:1.6.0 @ 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
[DEBUG]   (f) repositorySystemSession = 
org.eclipse.aether.DefaultRepositorySystemSession@3b0d3a63
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@5568c66f
[DEBUG]   (f) skip = false
[DEBUG]   (f) skipTests = false
[DEBUG]   (f) workingDirectory = 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
[DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.1:npm 
{execution: npm install}
[DEBUG] -- end configuration --
[INFO] Running 'npm --cache

Re: Read processor property in init()

Just to add on top of what Mike said. The @OnScheduled annotation indicates
that the method that is marked with this annotation will run when a
processor is started every time. So basically the setup() will be called
and executed everytime the processor is started from the UI.

On Wed, 28 Mar 2018 at 6:34 PM, Jeff Zemerick  wrote:

> I will give that a go. Thanks for the quick answer, Mike!
>
> On Wed, Mar 28, 2018 at 9:01 AM, Mike Thomsen 
> wrote:
> > Just do...
> >
> > @OnScheduled
> > public void setup(ProcessContext context) {
> > //Read properties and do setup.
> > }
> >
> > On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick 
> wrote:
> >
> >> Hi everyone,
> >>
> >> Is there a recommended method for making user-configurable property
> >> values available to a processor's init()? I would like to load a large
> >> index file but allow the user to specify the index's path. I am
> >> guessing that init() is executed too early to read user properties.
> >>
> >> Thanks for any suggestions.
> >>
> >> Jeff
> >>
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)