Hi
Thanks for your speedy reply.
I have just done a retest and changed the code to handle exceptions by calling
context.yield
and I now get, the following stack trace forever repeating, still my custom
plugin is never yielded.
What I am trying to develop is a stable nifi plugin that does not consume
resource when an exception occurs - Can you help with that also?
015-11-17 16:26:50,476 INFO [NiFi Web Server-26]
c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, version
'Jersey: 1.19 02/11/2015 03:25 AM'
2015-11-17 16:27:08,462 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@54c2b852 // Another save pending =
false
2015-11-17 16:27:17,676 INFO [pool-35-thread-4]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
PutFile[id=5135b041-66ff-43ab-9e71-f1848dcef4f6] to run with 1 threads
2015-11-17 16:27:17,676 INFO [pool-35-thread-2]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
GetFile[id=2886b8ff-1f82-4aa3-b42d-1f4f4c9f03f8] to run with 1 threads
2015-11-17 16:27:17,676 INFO [pool-35-thread-3]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] to run
with 1 threads
2015-11-17 16:27:17,691 INFO [Timer-Driven Process Thread-2]
o.a.nifi.processors.standard.GetFile
GetFile[id=2886b8ff-1f82-4aa3-b42d-1f4f4c9f03f8] added
StandardFlowFileRecord[uuid=6a1800eb-03ca-4aad-9e2c-f96df41252e6,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1447777637680-1, container=default,
section=1], offset=0,
length=15658],offset=0,name=162700882642440.json,size=15658] to flow
2015-11-17 16:27:17,743 ERROR [Timer-Driven Process Thread-5]
c.b.n.p.SampleErrorHandlerProcessor
[SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]] ---
Encountered process exception
org.apache.nifi.processor.exception.FlowFileHandlingException: null is not
known in this session (StandardProcessSession[id=35])
at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:2345)
~[nifi-framework-core-0.3.0.jar:0.3.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1773)
~[nifi-framework-core-0.3.0.jar:0.3.0]
at
com.barclays.nifi.processor.SampleErrorHandlerProcessor.doTrigger(SampleErrorHandlerProcessor.scala:49)
[poc-1.0-SNAPSHOT-nifiPlugins.nar-unpacked/:na]
at
com.barclays.nifi.processor.ErrorHandlerAdapterProcessor$class.onTrigger(ErrorHandlerAdapterProcessor.scala:24)
~[poc-1.0-SNAPSHOT-nifiPlugins.nar-unpacked/:na]
at
com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:26)
[poc-1.0-SNAPSHOT-nifiPlugins.nar-unpacked/:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-0.3.0.jar:0.3.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
[nifi-framework-core-0.3.0.jar:0.3.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
[nifi-framework-core-0.3.0.jar:0.3.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
[nifi-framework-core-0.3.0.jar:0.3.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
[nifi-framework-core-0.3.0.jar:0.3.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_51]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_51]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2015-11-17 16:27:17,743 ERROR [Timer-Driven Process Thread-5]
c.b.n.p.SampleErrorHandlerProcessor
SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] failed to
process due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=6a1800eb-03ca-4aad-9e2c-f96df41252e6,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1447777637680-1, container=default,
section=1], offset=0,
length=15658],offset=0,name=162700882642440.json,size=15658] is not known in
this session
> On 17 Nov 2015, at 16:19, Mark Payne <[email protected]> wrote:
>
> Michael,
>
> Hello. First, thanks for your interest in NiFi! I'm very happy to see that
> you're already starting to dig in pretty deep here!
>
> I admit that the wording here can be a little confusing. Let me try to
> explain what's going on here and how yielding
> works. When you are developing a Processor in NiFi, especially if reaching
> out to some external resource,
> a common occurrence is to realize that there is no work that can really be
> done at this point in time (for
> example, because the remote resource is not accessible). In that case, the
> processor can call ProcessContext.yield()
> and as a result, the framework will stop scheduling that processor to run for
> the configured amount of time, so that
> we aren't wasting resources when we know there is nothing that we can
> accomplish. This is where the configured
> Yield Duration comes in.
>
> There is, however, a concept of "administratively yielding" a processor. The
> idea here is that the Processor throws
> some unexpected exception. Since the Exception was not of type
> ProcessException, the framework considers this
> a bug and will perform an administrative yield to avoid continually
> triggering a processor that may be in a bad state.
>
> Perhaps using the term "administratively yielding" was a bad idea, as it can
> clearly lead to confusion.
> If you feel that something is not behaving properly, though, or you believe
> that the user experience can be improved
> in some way, please do let us know, as we strive to continually improve the
> user experience.
>
> Thanks
> -Mark
>
>
>> On Nov 17, 2015, at 10:34 AM, Michael De Courci <[email protected]> wrote:
>>
>> FYI
>>
>>> Begin forwarded message:
>>>
>>> From: Michael De Courci <[email protected]>
>>> Date: 16 November 2015 at 15:41:27 GMT
>>> To: [email protected], [email protected], Kefah Issa
>>> <[email protected]>
>>> Subject: NiFI Custom Plugin - Yield Duration Ignored
>>>
>>> I wrote a NAR custom plugin to verify NoFi Exception handling. - version;
>>> nifi-0.3.0
>>>
>>> The plugin was written to always through a RunTimeException for any
>>> exception including ProcessException.
>>>
>>> I created a simple flow; GetFile->MyPlugin->PutFile
>>>
>>> and set the Yield duration to 60 seconds.
>>>
>>> The log shows that the yield duration is always 1 sec
>>>
>>> I debugged the NiFi framework code and see that the yield duration always
>>> defaults to 1 sec;
>>> See;
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>>>
>>> Setting the duration updates the class
>>> org.apache.nifi.controller.StandardProcessorNode
>>> but this does not communicate the change to TimerDrivenSchedulingAgent
>>>
>>> Evidence from log;
>>>
>>> 80eb-552a-471c-8ffd-f2509d71fd2e] failed to process session due to
>>> java.lang.RuntimeException: java.lang.RuntimeException
>>>
>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
>>> c.b.n.p.SampleErrorHandlerProcessor
>>> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
>>> Processor Administratively Yielded for 1 sec due to processing failure
>>>
>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
>>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>>> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] due to
>>> uncaught Exception: java.lang.RuntimeException
>>>
>>> 2015-11-15 16:17:58,419 WARN [Timer-Driven Process Thread-7]
>>> o.a.n.c.t.ContinuallyRunProcessorTask
>>>
>>> java.lang.RuntimeException: null
>>>
>>> at
>>> com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:103)
>>> ~[na:na]
>>>
>>>
>>>
>>> With Kind Regards
>>> Michael de Courci
>>
>
