Re: NiFi 1.4: PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joe Witt]

Thanks for reporting the issue.  It appears the NPE can occur if zero
records end up getting published and it fails to lookup the schema
because the message tracker is not yet initialized.

So a couple things:
1) Verify the settings being used to lookup the schema.  You can show
the record reader, writer, and schema registry settings being used for
that publisher.
2) You do not need to SplitAvro before sending it via publish kafka
record.  Avoiding doing the split beforehand can result in vastly
superior performance.

Thanks
Joe

On Fri, Aug 11, 2017 at 8:03 PM, mayank rathi  wrote:
> Hello Joe,
>
> JIRA logged.
>
> https://issues.apache.org/jira/browse/NIFI-4290
>
> Thanks!!
>
> On Fri, Aug 11, 2017 at 9:48 PM, Joe Witt  wrote:
>
>> Hello
>>
>> Can you please file a jira and attach the logs to the jira.
>>
>> Thanks
>> Joe
>>
>> On Aug 11, 2017 6:23 PM, "mayank rathi"  wrote:
>>
>> > Attached are the logs after setting processors in Debug mode. Here is the
>> > flow:
>> >
>> > ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
>> >
>> > Thanks!!
>> >
>> > On Fri, Aug 11, 2017 at 9:01 PM, mayank rathi 
>> > wrote:
>> >
>> >> Hello All,
>> >>
>> >> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I
>> am
>> >> using ConfluentSchemaRegistry Controller service and getting below
>> error:
>> >>
>> >> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4]
>> >> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 PublishKafkaRecord_0_10[id=b3c
>> >> 03961-015d-1000-0946-79ccbe2ffbbd] PublishKafkaRecord_0_10[id=b3c
>> >> 03961-015d-1000-0946-79ccbe2ffbbd] failed to process due to
>> >> java.lang.NullPointerException; rolling back session: {}
>> >> java.lang.NullPointerException: null
>> >>
>> >> I do not see any error on Kafka side.
>> >>
>> >> How can I debug and resolve this issue?
>> >>
>> >> Thanks!!
>> >>
>> >> --
>> >> NOTICE: This email message is for the sole use of the intended
>> >> recipient(s) and may contain confidential and privileged information.
>> Any
>> >> unauthorized review, use, disclosure or distribution is prohibited. If
>> you
>> >> are not the intended recipient, please contact the sender by reply email
>> >> and destroy all copies of the original message.
>> >>
>> >
>> >
>> >
>> > --
>> > NOTICE: This email message is for the sole use of the intended
>> > recipient(s) and may contain confidential and privileged information. Any
>> > unauthorized review, use, disclosure or distribution is prohibited. If
>> you
>> > are not the intended recipient, please contact the sender by reply email
>> > and destroy all copies of the original message.
>> >
>>
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended recipient(s)
> and may contain confidential and privileged information. Any unauthorized
> review, use, disclosure or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy
> all copies of the original message.

Re: NiFi 1.4: PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[mayank rathi]

Hello Joe,

JIRA logged.

https://issues.apache.org/jira/browse/NIFI-4290

Thanks!!

On Fri, Aug 11, 2017 at 9:48 PM, Joe Witt  wrote:

> Hello
>
> Can you please file a jira and attach the logs to the jira.
>
> Thanks
> Joe
>
> On Aug 11, 2017 6:23 PM, "mayank rathi"  wrote:
>
> > Attached are the logs after setting processors in Debug mode. Here is the
> > flow:
> >
> > ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> >
> > Thanks!!
> >
> > On Fri, Aug 11, 2017 at 9:01 PM, mayank rathi 
> > wrote:
> >
> >> Hello All,
> >>
> >> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I
> am
> >> using ConfluentSchemaRegistry Controller service and getting below
> error:
> >>
> >> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4]
> >> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 PublishKafkaRecord_0_10[id=b3c
> >> 03961-015d-1000-0946-79ccbe2ffbbd] PublishKafkaRecord_0_10[id=b3c
> >> 03961-015d-1000-0946-79ccbe2ffbbd] failed to process due to
> >> java.lang.NullPointerException; rolling back session: {}
> >> java.lang.NullPointerException: null
> >>
> >> I do not see any error on Kafka side.
> >>
> >> How can I debug and resolve this issue?
> >>
> >> Thanks!!
> >>
> >> --
> >> NOTICE: This email message is for the sole use of the intended
> >> recipient(s) and may contain confidential and privileged information.
> Any
> >> unauthorized review, use, disclosure or distribution is prohibited. If
> you
> >> are not the intended recipient, please contact the sender by reply email
> >> and destroy all copies of the original message.
> >>
> >
> >
> >
> > --
> > NOTICE: This email message is for the sole use of the intended
> > recipient(s) and may contain confidential and privileged information. Any
> > unauthorized review, use, disclosure or distribution is prohibited. If
> you
> > are not the intended recipient, please contact the sender by reply email
> > and destroy all copies of the original message.
> >
>



-- 
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message.

Re: NiFi 1.4: PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joe Witt]

Hello

Can you please file a jira and attach the logs to the jira.

Thanks
Joe

On Aug 11, 2017 6:23 PM, "mayank rathi"  wrote:

> Attached are the logs after setting processors in Debug mode. Here is the
> flow:
>
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
>
> Thanks!!
>
> On Fri, Aug 11, 2017 at 9:01 PM, mayank rathi 
> wrote:
>
>> Hello All,
>>
>> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am
>> using ConfluentSchemaRegistry Controller service and getting below error:
>>
>> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4]
>> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 PublishKafkaRecord_0_10[id=b3c
>> 03961-015d-1000-0946-79ccbe2ffbbd] PublishKafkaRecord_0_10[id=b3c
>> 03961-015d-1000-0946-79ccbe2ffbbd] failed to process due to
>> java.lang.NullPointerException; rolling back session: {}
>> java.lang.NullPointerException: null
>>
>> I do not see any error on Kafka side.
>>
>> How can I debug and resolve this issue?
>>
>> Thanks!!
>>
>> --
>> NOTICE: This email message is for the sole use of the intended
>> recipient(s) and may contain confidential and privileged information. Any
>> unauthorized review, use, disclosure or distribution is prohibited. If you
>> are not the intended recipient, please contact the sender by reply email
>> and destroy all copies of the original message.
>>
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.
>

Re: NiFi 1.4: PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[mayank rathi]

Attached are the logs after setting processors in Debug mode. Here is the
flow:

ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10

Thanks!!

On Fri, Aug 11, 2017 at 9:01 PM, mayank rathi 
wrote:

> Hello All,
>
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am
> using ConfluentSchemaRegistry Controller service and getting below error:
>
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4]
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 PublishKafkaRecord_0_10[id=
> b3c03961-015d-1000-0946-79ccbe2ffbbd] PublishKafkaRecord_0_10[id=
> b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to process due to 
> java.lang.NullPointerException;
> rolling back session: {}
> java.lang.NullPointerException: null
>
> I do not see any error on Kafka side.
>
> How can I debug and resolve this issue?
>
> Thanks!!
>
> --
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.
>



-- 
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message.

NiFi 1.4: PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[mayank rathi]

Hello All,

I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am
using ConfluentSchemaRegistry Controller service and getting below error:

2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4]
o.a.n.p.k.pubsub.PublishKafkaRecord_0_10
PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd]
PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to
process due to java.lang.NullPointerException; rolling back session: {}
java.lang.NullPointerException: null

I do not see any error on Kafka side.

How can I debug and resolve this issue?

Thanks!!

-- 
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message.

Re: how to execute code when processor is stopping

[尹文才]

Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
sure that the OnUnscheduled method was not called too frequently when the
connection is offline.
You guys were right, these sort of logic should not be placed inside the
scheduling methods, I need to refactor my code to place them into onTrigger.

Regards,
Ben

2017-08-12 0:53 GMT+08:00 Matt Burgess :

> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
> (such as database connection retries), I recommend putting the
> connection logic in the onTrigger() method. If you can check
> connectivity, then your onTrigger() would know whether it needs to try
> to reconnect before it does any work. If it tries to reconnect and is
> unsuccessful, you can yield the processor if you want, so as not to
> hammer the DB with connection attempts. The CaptureChangeMySQL
> processor does this, it has a retry loop for trying various nodes in a
> MySQL cluster, but once it's connected, it goes on about its work, and
> if a connection fails, it will retry the connection loop before it
> does any more work. It only uses onTrigger and none of the scheduling
> stuff.
>
> Regards,
> Matt
>
> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende  wrote:
> > Ben,
> >
> > I apologize if I am not understanding the situation, but...
> >
> > In the case where your OnScheduled code is in a retry loop, if someone
> > stops the processor it will call your OnUnscheduled code which will
> > set the flag to bounce out of the loop. This sounds like what you
> > want, right?
> >
> > In the case where OnScheduled times out, the framework is calling
> > OnUnscheduled which would call your code to set the flag, but wouldn't
> > that not matter at this point because you aren't looping anymore
> > anyway?
> >
> > If the framework calls OnScheduled again, your code should set the
> > flag back to whatever it needs to be to start looping again right?
> >
> > An alternative that might avoid some of this would be to lazily
> > initialize the connection in the onTrigger method of the processor.
> >
> > -Bryan
> >
> >
> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才  wrote:
> >> thanks Pierre, my case is that I need to implement a database connection
> >> retry logic inside my OnScheduled method, when the database is not
> >> available I will retry until the connection is back online.
> >> The problem is when the database is offline it will throw timed out
> >> execution exception inside OnScheduled and then call OnUnscheduled. But
> >> when I manually stop the processor the OnUnsheduled
> >> will also get called. I know my logic sounds a little weird but I need
> to
> >> set some flag in the OnUnscheduled method to stop the retry logic inside
> >> OnScheduled in order to be able to stop the processor,
> >> otherwise the processor is not able to be stopped unless I restart the
> >> whole NIFI.
> >>
> >> Regards,
> >> Ben
> >>
> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard  >:
> >>
> >>> Oh OK, get it now!
> >>>
> >>> Not sure what's your use case, but I don't think you can do that
> unless you
> >>> set some information when the process actually executes onTrigger for
> the
> >>> first time and you then check this value in your OnUnscheduled
> annotated
> >>> method.
> >>>
> >>> Pierre
> >>>
> >>> 2017-08-11 10:11 GMT+02:00 尹文才 :
> >>>
> >>> > Hi Pierre, I've checked the developer guide before I sent the email
> and
> >>> > according to the developer guide, the method annotated with
> OnUnScheduled
> >>> > will be called in 2 cases according to my understanding, please
> correct
> >>> me
> >>> > if I'm wrong:
> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
> >>> processor
> >>> > is no longer scheduled to run in this case, and the method will be
> >>> called.
> >>> > 2. when method annotated with OnScheduled throws exceptions, for
> example
> >>> > time out execution exception, the OnUnScheduled method will also be
> >>> called.
> >>> >
> >>> > My question is how to tell the first scenario from the second one?
> >>> Thanks.
> >>> >
> >>> > Regards,
> >>> > Ben
> >>> >
> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
> pierre.villard...@gmail.com>:
> >>> >
> >>> > > Hi Ben,
> >>> > >
> >>> > > You might want to have a look here:
> >>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
> >>> > > guide.html#component-lifecycle
> >>> > >
> >>> > > Pierre
> >>> > >
> >>> > > 2017-08-11 9:06 GMT+02:00 尹文才 :
> >>> > >
> >>> > > > Hi guys, I'm trying to execute some code in my processor when the
> >>> > > processor
> >>> > > > is asked to stop in the NIFI UI by the user, I checked the
> developer
> >>> > > guide
> >>> > > > and only find OnUnscheduled will be called when the processor is
> no
> >>> > long
> >>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
> >>> > called
> >>> > > > after timed out 

Do the ElasticSearch processors not support the JSON DSL?

[Mike Thomsen]

The Query and Scroll ones say they only accept queries with Lucene syntax,
not the JSON DSL. Am I missing something or is there no way to execute a
query defined with the JSON DSL right now?

My client actually really needs this because they do a lot of aggregations,
so I have time to take a stab at it.

Thanks,

Mike

Re: how to execute code when processor is stopping

[Matt Burgess]

I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
(such as database connection retries), I recommend putting the
connection logic in the onTrigger() method. If you can check
connectivity, then your onTrigger() would know whether it needs to try
to reconnect before it does any work. If it tries to reconnect and is
unsuccessful, you can yield the processor if you want, so as not to
hammer the DB with connection attempts. The CaptureChangeMySQL
processor does this, it has a retry loop for trying various nodes in a
MySQL cluster, but once it's connected, it goes on about its work, and
if a connection fails, it will retry the connection loop before it
does any more work. It only uses onTrigger and none of the scheduling
stuff.

Regards,
Matt

On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende  wrote:
> Ben,
>
> I apologize if I am not understanding the situation, but...
>
> In the case where your OnScheduled code is in a retry loop, if someone
> stops the processor it will call your OnUnscheduled code which will
> set the flag to bounce out of the loop. This sounds like what you
> want, right?
>
> In the case where OnScheduled times out, the framework is calling
> OnUnscheduled which would call your code to set the flag, but wouldn't
> that not matter at this point because you aren't looping anymore
> anyway?
>
> If the framework calls OnScheduled again, your code should set the
> flag back to whatever it needs to be to start looping again right?
>
> An alternative that might avoid some of this would be to lazily
> initialize the connection in the onTrigger method of the processor.
>
> -Bryan
>
>
> On Fri, Aug 11, 2017 at 9:16 AM, 尹文才  wrote:
>> thanks Pierre, my case is that I need to implement a database connection
>> retry logic inside my OnScheduled method, when the database is not
>> available I will retry until the connection is back online.
>> The problem is when the database is offline it will throw timed out
>> execution exception inside OnScheduled and then call OnUnscheduled. But
>> when I manually stop the processor the OnUnsheduled
>> will also get called. I know my logic sounds a little weird but I need to
>> set some flag in the OnUnscheduled method to stop the retry logic inside
>> OnScheduled in order to be able to stop the processor,
>> otherwise the processor is not able to be stopped unless I restart the
>> whole NIFI.
>>
>> Regards,
>> Ben
>>
>> 2017-08-11 17:18 GMT+08:00 Pierre Villard :
>>
>>> Oh OK, get it now!
>>>
>>> Not sure what's your use case, but I don't think you can do that unless you
>>> set some information when the process actually executes onTrigger for the
>>> first time and you then check this value in your OnUnscheduled annotated
>>> method.
>>>
>>> Pierre
>>>
>>> 2017-08-11 10:11 GMT+02:00 尹文才 :
>>>
>>> > Hi Pierre, I've checked the developer guide before I sent the email and
>>> > according to the developer guide, the method annotated with OnUnScheduled
>>> > will be called in 2 cases according to my understanding, please correct
>>> me
>>> > if I'm wrong:
>>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>>> processor
>>> > is no longer scheduled to run in this case, and the method will be
>>> called.
>>> > 2. when method annotated with OnScheduled throws exceptions, for example
>>> > time out execution exception, the OnUnScheduled method will also be
>>> called.
>>> >
>>> > My question is how to tell the first scenario from the second one?
>>> Thanks.
>>> >
>>> > Regards,
>>> > Ben
>>> >
>>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard :
>>> >
>>> > > Hi Ben,
>>> > >
>>> > > You might want to have a look here:
>>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>>> > > guide.html#component-lifecycle
>>> > >
>>> > > Pierre
>>> > >
>>> > > 2017-08-11 9:06 GMT+02:00 尹文才 :
>>> > >
>>> > > > Hi guys, I'm trying to execute some code in my processor when the
>>> > > processor
>>> > > > is asked to stop in the NIFI UI by the user, I checked the developer
>>> > > guide
>>> > > > and only find OnUnscheduled will be called when the processor is no
>>> > long
>>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
>>> > called
>>> > > > after timed out executing OnScheduled task. So is there a way to
>>> > execute
>>> > > > some code only when the processor is stopping?
>>> > > >
>>> > > > Regards,
>>> > > > Ben
>>> > > >
>>> > >
>>> >
>>>

Re: how to execute code when processor is stopping

[Bryan Bende]

Ben,

I apologize if I am not understanding the situation, but...

In the case where your OnScheduled code is in a retry loop, if someone
stops the processor it will call your OnUnscheduled code which will
set the flag to bounce out of the loop. This sounds like what you
want, right?

In the case where OnScheduled times out, the framework is calling
OnUnscheduled which would call your code to set the flag, but wouldn't
that not matter at this point because you aren't looping anymore
anyway?

If the framework calls OnScheduled again, your code should set the
flag back to whatever it needs to be to start looping again right?

An alternative that might avoid some of this would be to lazily
initialize the connection in the onTrigger method of the processor.

-Bryan


On Fri, Aug 11, 2017 at 9:16 AM, 尹文才  wrote:
> thanks Pierre, my case is that I need to implement a database connection
> retry logic inside my OnScheduled method, when the database is not
> available I will retry until the connection is back online.
> The problem is when the database is offline it will throw timed out
> execution exception inside OnScheduled and then call OnUnscheduled. But
> when I manually stop the processor the OnUnsheduled
> will also get called. I know my logic sounds a little weird but I need to
> set some flag in the OnUnscheduled method to stop the retry logic inside
> OnScheduled in order to be able to stop the processor,
> otherwise the processor is not able to be stopped unless I restart the
> whole NIFI.
>
> Regards,
> Ben
>
> 2017-08-11 17:18 GMT+08:00 Pierre Villard :
>
>> Oh OK, get it now!
>>
>> Not sure what's your use case, but I don't think you can do that unless you
>> set some information when the process actually executes onTrigger for the
>> first time and you then check this value in your OnUnscheduled annotated
>> method.
>>
>> Pierre
>>
>> 2017-08-11 10:11 GMT+02:00 尹文才 :
>>
>> > Hi Pierre, I've checked the developer guide before I sent the email and
>> > according to the developer guide, the method annotated with OnUnScheduled
>> > will be called in 2 cases according to my understanding, please correct
>> me
>> > if I'm wrong:
>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>> processor
>> > is no longer scheduled to run in this case, and the method will be
>> called.
>> > 2. when method annotated with OnScheduled throws exceptions, for example
>> > time out execution exception, the OnUnScheduled method will also be
>> called.
>> >
>> > My question is how to tell the first scenario from the second one?
>> Thanks.
>> >
>> > Regards,
>> > Ben
>> >
>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard :
>> >
>> > > Hi Ben,
>> > >
>> > > You might want to have a look here:
>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>> > > guide.html#component-lifecycle
>> > >
>> > > Pierre
>> > >
>> > > 2017-08-11 9:06 GMT+02:00 尹文才 :
>> > >
>> > > > Hi guys, I'm trying to execute some code in my processor when the
>> > > processor
>> > > > is asked to stop in the NIFI UI by the user, I checked the developer
>> > > guide
>> > > > and only find OnUnscheduled will be called when the processor is no
>> > long
>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
>> > called
>> > > > after timed out executing OnScheduled task. So is there a way to
>> > execute
>> > > > some code only when the processor is stopping?
>> > > >
>> > > > Regards,
>> > > > Ben
>> > > >
>> > >
>> >
>>

Re: Assigning JIRAs to myself

[Aldrin Piri]

Sure thing, Ben.

Added you to the list of contributors as well.  Please let us know if there
are any issues.

Thanks!

On Fri, Aug 11, 2017 at 9:36 AM, 尹文才  wrote:

> Hi Aldrin, I would also like to be added to the list of NIFI contributors
> to start working on some easy JIRAS, thanks.
>
> Regards,
> Ben
>
> 2017-08-11 20:03 GMT+08:00 Aldrin Piri :
>
> > Hi Arun,
> >
> > I've added you to the list of contributors and you should be able to self
> > assign issues.  Please let us know if you are having any issues.
> >
> > Thanks and look forward to the contributions!
> >
> > On Fri, Aug 11, 2017 at 7:59 AM, Arun Manivannan 
> wrote:
> >
> > > Hi,
> > >
> > > Good morning.
> > >
> > > I would like to try to solve some beginner level issues.
> > >
> > > Have read the developer's guide and the contributor guide.
> > >
> > > How can I assign an issue to myself on the JIRA?  Great work on the
> > > "beginner" tag.  Thanks !
> > >
> > >
> > > Cheers,
> > > Arun
> > >
> >
>

Re: Assigning JIRAs to myself

[尹文才]

Hi Aldrin, I would also like to be added to the list of NIFI contributors
to start working on some easy JIRAS, thanks.

Regards,
Ben

2017-08-11 20:03 GMT+08:00 Aldrin Piri :

> Hi Arun,
>
> I've added you to the list of contributors and you should be able to self
> assign issues.  Please let us know if you are having any issues.
>
> Thanks and look forward to the contributions!
>
> On Fri, Aug 11, 2017 at 7:59 AM, Arun Manivannan  wrote:
>
> > Hi,
> >
> > Good morning.
> >
> > I would like to try to solve some beginner level issues.
> >
> > Have read the developer's guide and the contributor guide.
> >
> > How can I assign an issue to myself on the JIRA?  Great work on the
> > "beginner" tag.  Thanks !
> >
> >
> > Cheers,
> > Arun
> >
>

Re: Interested in adding two new features to GetMongo

[Mike Thomsen]

We've done a lot with Solr and ElasticSearch, so that was the impetus
behind that change. It seemed like a real waste to have to throw another
processor in there just to make sane date fields (and others in some cases)
that those processors could handle.

On Fri, Aug 11, 2017 at 8:20 AM, Mike Thomsen 
wrote:

> I added this new feature lately based on some needs that came up with a
> client:
>
> https://github.com/apache/nifi/pull/2063
>
> It uses Jackson to serialize the Mongo document instead of the Mongo
> Document class's toJson() which produces "extended JSON." From what I could
> tell with our client's data and other testing, it works just fine.
> Difference is mainly like this:
>
> Default:
>
> {
> "someTimeStamp": {
> "$date": 123456
> }
> }
>
> If you select "clean JSON:"
>
> {
> "someTimeStamp": "2017-08-11T08:18:15Z"
> }
>
> It should do the same for doubles and longs which have their own weird
> representation that makes cleanup necessary for any other JSON-based
> processor.
>
> On Thu, Aug 10, 2017 at 10:01 PM, Joe Witt  wrote:
>
>> Team,
>>
>> Is there anyone else familiar with Mongo that could discuss this with
>> Mike?
>>
>> Thanks
>> Joe
>>
>> On Fri, Aug 4, 2017 at 8:56 AM, Mike Thomsen 
>> wrote:
>> > 1. Add the ability to run aggregations to GetMongo.
>> > 2. Add the ability to get the query from a flowfile.
>> >
>> > I know you can't do #2 right now, but was wondering if there was an
>> > existing way to invoke the aggregation pipeline that I'm missing before
>> I
>> > dive into that.
>> >
>> > Thanks,
>> >
>> > Mike
>>
>
>

Re: Interested in adding two new features to GetMongo

[Mike Thomsen]

I added this new feature lately based on some needs that came up with a
client:

https://github.com/apache/nifi/pull/2063

It uses Jackson to serialize the Mongo document instead of the Mongo
Document class's toJson() which produces "extended JSON." From what I could
tell with our client's data and other testing, it works just fine.
Difference is mainly like this:

Default:

{
"someTimeStamp": {
"$date": 123456
}
}

If you select "clean JSON:"

{
"someTimeStamp": "2017-08-11T08:18:15Z"
}

It should do the same for doubles and longs which have their own weird
representation that makes cleanup necessary for any other JSON-based
processor.

On Thu, Aug 10, 2017 at 10:01 PM, Joe Witt  wrote:

> Team,
>
> Is there anyone else familiar with Mongo that could discuss this with Mike?
>
> Thanks
> Joe
>
> On Fri, Aug 4, 2017 at 8:56 AM, Mike Thomsen 
> wrote:
> > 1. Add the ability to run aggregations to GetMongo.
> > 2. Add the ability to get the query from a flowfile.
> >
> > I know you can't do #2 right now, but was wondering if there was an
> > existing way to invoke the aggregation pipeline that I'm missing before I
> > dive into that.
> >
> > Thanks,
> >
> > Mike
>

Re: Assigning JIRAs to myself

[Aldrin Piri]

Hi Arun,

I've added you to the list of contributors and you should be able to self
assign issues.  Please let us know if you are having any issues.

Thanks and look forward to the contributions!

On Fri, Aug 11, 2017 at 7:59 AM, Arun Manivannan  wrote:

> Hi,
>
> Good morning.
>
> I would like to try to solve some beginner level issues.
>
> Have read the developer's guide and the contributor guide.
>
> How can I assign an issue to myself on the JIRA?  Great work on the
> "beginner" tag.  Thanks !
>
>
> Cheers,
> Arun
>

Assigning JIRAs to myself

[Arun Manivannan]

Hi,

Good morning.

I would like to try to solve some beginner level issues.

Have read the developer's guide and the contributor guide.

How can I assign an issue to myself on the JIRA?  Great work on the
"beginner" tag.  Thanks !


Cheers,
Arun

Re: how to execute code when processor is stopping

[Pierre Villard]

Oh OK, get it now!

Not sure what's your use case, but I don't think you can do that unless you
set some information when the process actually executes onTrigger for the
first time and you then check this value in your OnUnscheduled annotated
method.

Pierre

2017-08-11 10:11 GMT+02:00 尹文才 :

> Hi Pierre, I've checked the developer guide before I sent the email and
> according to the developer guide, the method annotated with OnUnScheduled
> will be called in 2 cases according to my understanding, please correct me
> if I'm wrong:
> 1. when user tries to stop the processor in the NIFI UI, thus the processor
> is no longer scheduled to run in this case, and the method will be called.
> 2. when method annotated with OnScheduled throws exceptions, for example
> time out execution exception, the OnUnScheduled method will also be called.
>
> My question is how to tell the first scenario from the second one? Thanks.
>
> Regards,
> Ben
>
> 2017-08-11 15:51 GMT+08:00 Pierre Villard :
>
> > Hi Ben,
> >
> > You might want to have a look here:
> > https://nifi.apache.org/docs/nifi-docs/html/developer-
> > guide.html#component-lifecycle
> >
> > Pierre
> >
> > 2017-08-11 9:06 GMT+02:00 尹文才 :
> >
> > > Hi guys, I'm trying to execute some code in my processor when the
> > processor
> > > is asked to stop in the NIFI UI by the user, I checked the developer
> > guide
> > > and only find OnUnscheduled will be called when the processor is no
> long
> > > scheduled to run. I've tested this OnUnscheduled, it will also be
> called
> > > after timed out executing OnScheduled task. So is there a way to
> execute
> > > some code only when the processor is stopping?
> > >
> > > Regards,
> > > Ben
> > >
> >
>

Re: how to execute code when processor is stopping

[尹文才]

Hi Pierre, I've checked the developer guide before I sent the email and
according to the developer guide, the method annotated with OnUnScheduled
will be called in 2 cases according to my understanding, please correct me
if I'm wrong:
1. when user tries to stop the processor in the NIFI UI, thus the processor
is no longer scheduled to run in this case, and the method will be called.
2. when method annotated with OnScheduled throws exceptions, for example
time out execution exception, the OnUnScheduled method will also be called.

My question is how to tell the first scenario from the second one? Thanks.

Regards,
Ben

2017-08-11 15:51 GMT+08:00 Pierre Villard :

> Hi Ben,
>
> You might want to have a look here:
> https://nifi.apache.org/docs/nifi-docs/html/developer-
> guide.html#component-lifecycle
>
> Pierre
>
> 2017-08-11 9:06 GMT+02:00 尹文才 :
>
> > Hi guys, I'm trying to execute some code in my processor when the
> processor
> > is asked to stop in the NIFI UI by the user, I checked the developer
> guide
> > and only find OnUnscheduled will be called when the processor is no long
> > scheduled to run. I've tested this OnUnscheduled, it will also be called
> > after timed out executing OnScheduled task. So is there a way to execute
> > some code only when the processor is stopping?
> >
> > Regards,
> > Ben
> >
>

Re: how to execute code when processor is stopping

[Pierre Villard]

Hi Ben,

You might want to have a look here:
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#component-lifecycle

Pierre

2017-08-11 9:06 GMT+02:00 尹文才 :

> Hi guys, I'm trying to execute some code in my processor when the processor
> is asked to stop in the NIFI UI by the user, I checked the developer guide
> and only find OnUnscheduled will be called when the processor is no long
> scheduled to run. I've tested this OnUnscheduled, it will also be called
> after timed out executing OnScheduled task. So is there a way to execute
> some code only when the processor is stopping?
>
> Regards,
> Ben
>

how to execute code when processor is stopping

[尹文才]

Hi guys, I'm trying to execute some code in my processor when the processor
is asked to stop in the NIFI UI by the user, I checked the developer guide
and only find OnUnscheduled will be called when the processor is no long
scheduled to run. I've tested this OnUnscheduled, it will also be called
after timed out executing OnScheduled task. So is there a way to execute
some code only when the processor is stopping?

Regards,
Ben

Re: get controller service's configuration

[尹文才]

Thanks for your explanation Bryan, it seems that I could only get the DBCP
controller service's driver name after something like getDrivername() is
added into the current DBCPService interface.

Regards,
Ben

2017-08-10 22:01 GMT+08:00 Bryan Bende :

> The way controller services are setup you have the following...
>
> - DBCPService interface (provides getConnection()) extends
> ControllerService interface (empty interface to indicate it is a CS)
> - DBCPConnectionPool extends AbstractControllerService implements
> DBCPService
> - Processor XYZ depends on DBCPService interface
>
> The DBCPService interface is the common point between the processor
> and the implementations. The processor XYZ classpath only knows about
> the DBCPService interface, it doesn't know anything about the classes
> that implement it... there could actually be several implementations
> in different NARs, but it is up to the framework to provide access to
> these.
>
> Since the processor only depends on the interface, which in this case
> only exposes getConnection(), you can't really assume the service has
> certain properties because DBCPConnectionPool.DB_DRIVERNAME is
> specific to the DBCPConnectionPool implementation... another
> implementation may not have that property, or may call it something
> different. The interface would have to provide getDriverName() so that
> each implementation could provide that.
>
> -Bryan
>
>
> On Thu, Aug 10, 2017 at 4:33 AM, 尹文才  wrote:
> > Thanks Andy, I've tried your approach, in my case the controller service
> is
> > a DBCPConnectionPool and when I tried to get driver class name property
> > through context.getProperty(DBCPConnectionPool.DB_
> DRIVERNAME).getValue(),
> > but I the value is null. The AbstractControllerService class does have a
> > method getConfigurationContext() to get configuration context, but the
> > method is protected. So I still didn't find a feasible way to get the
> > controller service's properties.
> >
> > Regards,
> > Ben
> >
> > 2017-08-10 12:18 GMT+08:00 Andy LoPresto :
> >
> >> You can get the current property values of a controller service from the
> >> processor by using the ProcessContext object. For example, in GetHTTP
> [1],
> >> in the @OnScheduled method, you could do:
> >>
> >> context.getControllerServiceLookup().getControllerService("my-
> >> controller-service-id”);
> >>
> >> context.getProperty("controller-service-property-name");
> >> context.getProperty(SomeControllerService.
> CONSTANT_PROPERTY_DESCRIPTOR);
> >>
> >> I forget if context.getProperty() will give the controller service
> >> properties as well as the processor properties. If it doesn’t, you can
> cast
> >> the retrieved ControllerService into AbstractControllerService or the
> >> concrete class and access available properties directly from the
> >> encapsulated ConfigurationContext.
> >>
> >> [1] https://github.com/apache/nifi/blob/master/nifi-nar-
> >> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> >> main/java/org/apache/nifi/processors/standard/GetHTTP.java#L295
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> *alopresto.apa...@gmail.com *
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Aug 9, 2017, at 6:57 PM, 尹文才  wrote:
> >>
> >> Thanks Koji, I checked the link you provided and I think getting a
> >> DataSource is no different than getting the DBCP service(they could just
> >> get the connection). Actually I was trying to get the configured driver
> >> class to check the database type.
> >>
> >> Regards,
> >> Ben
> >>
> >> 2017-08-10 9:29 GMT+08:00 Koji Kawamura :
> >>
> >> Hi Ben,
> >>
> >> I'm not aware of ways to obtain configurations of a controller from a
> >> processor. Those should be encapsulated inside a controller service.
> >> If you'd like to create DataSource instance instead of just obtaining
> >> a connection, this discussion might be helpful:
> >> https://github.com/apache/nifi/pull/1417
> >>
> >> Although I would not recommend, if you really need to obtain all
> >> configurations, you can do so by calling NiFi REST API from your
> >> processor.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Thu, Aug 10, 2017 at 10:09 AM, 尹文才  wrote:
> >>
> >> Hi guys, I have a customized processor with a DBCP controller service as
> >>
> >> a
> >>
> >> property. I could get the DBCP controller service in my code, but does
> >> anyone know how to obtain all the configurations of the DBCP controller
> >> service in java code(e.g. Database Connection URL, Database Driver
> >> Location, etc) Thanks.
> >>
> >> Regards,
> >> Ben
> >>
> >>
> >>
> >>
>

Re: Using GregorianCalendar in apache nifi

[Andrew Psaltis]

Hi,
I'm not 100% sure I understand what your trying to accomplish.  However, if
you are wanting to use the Gregorian package from a script that is executed
with the executescript processor then please take a look at the
ExecuteScript cookbook, specifically part 3 [1].

Thanks,
Andrew

[1]
https://community.hortonworks.com/articles/77739/executescript-cookbook-part-3.html

On Fri, Aug 11, 2017 at 14:48 sally  wrote:

> Is  it possible to download apache-nifi open source project from  github
> and
> then add "import java.util.GregorianCallendar" inside ExecuteScript.java
> code and then deploy project  I  have tried it  but it throws errors what
> should i do?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Using-GregorianCalendar-in-apache-nifi-tp16629p16633.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

Re: Using GregorianCalendar in apache nifi

[Sandish Kumar HN]

Can you please path the error?

On 11 August 2017 at 12:07, sally  wrote:

> Is  it possible to download apache-nifi open source project from  github
> and
> then add "import java.util.GregorianCallendar" inside ExecuteScript.java
> code and then deploy project  I  have tried it  but it throws errors what
> should i do?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Using-GregorianCalendar-in-
> apache-nifi-tp16629p16633.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>



-- 

Thanks,
Regards,
SandishKumar HN

Re: Using GregorianCalendar in apache nifi

[sally]

Is  it possible to download apache-nifi open source project from  github and
then add "import java.util.GregorianCallendar" inside ExecuteScript.java
code and then deploy project  I  have tried it  but it throws errors what
should i do?



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Using-GregorianCalendar-in-apache-nifi-tp16629p16633.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: using avro flume source in nifi

[Chris Herssens]

Hello Joe,

Currently we use Flume for sending avro message to our big data platform.
Now I want to replace flume with NIfi. Since the sender is a  flume agent,
I need to use the flume source.
If I configure this sourcs for receiving netcat, I can access the tcp port.
Using avro as type doesn't work. If I set "run schedule" ti 60 sec, then it
work, I can telnet to the configured tcp port.

Regards,

Chris

On Fri, Aug 11, 2017 at 4:03 AM, Joe Witt  wrote:

> Chris,
>
> I dont believe there are many in the community here familiar enough
> with Flume so that might make getting help here a bit tougher.  Are
> you wanting NiFi to act as a TCP socket listener on a given port
> receiving Avro objects?
>
> Thanks
>
> On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
>  wrote:
> > Hello All,
> >
> > I want to configure a flume source processor using avro as flume source.
> > When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> > connect to the source port.
> >
> > telnet localhost 5151 gives connection refused.
> >
> > The configuration is as follows
> >
> > type = avro
> >
> > agent name = a1
> >
> > source name = batchSource
> >
> > Flume configuration
> >
> >a1.sources.batchSource.type = avro
> >
> >a1.sources.batchSource.bind = 0.0.0.0
> >
> >a1.sources.batchSource.port=5151
> >
> >
> > Using netcat as flume source works.
>