[
https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283953#comment-15283953
]
Joseph Witt commented on NIFI-1754:
-----------------------------------
In response to feedback about the new api method getUnacknowledgedFlowFiles, on
the Github PR [~jskora] wrote:
{quote}
The root problem is tracking "Administratively Yielding" messages back to a
flow file, so it lists the unacknowledged flow files since committed and
transferred files were already processed. Looking at where the
"Administratively Yielding" messages come from in EventDrivenSchedulingAgent
and ContinuallyRunProcessTask, the framework doesn't have visibility into the
active processor that threw the uncaught exception. This implementation
populates the log message without taxing the framework or requiring support in
the processors, but it can still be overridden for customization by individual
processors.
Providing flow files objects instead of a log message would expose internals
between framework layers and complicate api and implementation module
dependencies in the build. The upper level framework delegates flow file
handling further down, so this approach similarly delegates the log message
creation with minimal risk and impact on the framework.
{quote}
Let's address this first statement as it appears to be close to a requirement.
It says
{quote}
The root problem is tracking "Administratively Yielding" messages back to a
flow file, so it lists the unacknowledged flow files since committed and
transferred files were already processed.
{quote}
The concept of a rollback exists to return the state of the flow back to where
it was before a given session was created or last committed with the notable
exception of any provenance events which are deemed to need to be persisted
because they represent things which happened and cannot be rolled back and
therefore must be tracked (such as giving data to another system). The primary
case where rollback can occur is when the framework catches an unhandled
exception from the processor which clearly indicates a programming error with
that processor or some faulty system state which caused it (like an out of
memory exception). Rollback could also occur due to a failure on commit or due
to the programmer intentionally calling rollback. In the event of rollback due
to unhandled exceptions then the framework will also cause an 'administrative
yield' which is simply there to protect the framework from what is very
obviously a programming error in that particular processor. This is really the
case we're trying to solve here i believe so let's restate the requirement...
_As a developer of a processor in a NiFi flow I need detailed information about
the state of the processor at the time some unexpected exception occurred. The
state of the processor includes the detailed stacktrace and what flowfile(s) it
might have been operating on at that time. As an administrator of a NiFi flow
I also need to be able to gather this information so I can provide it to the
developer so they can fix the processor._
An important thing to keep in mind here is we are only talking about how to get
developers better information so they can fix their processor.
Next statement
{quote}
Looking at where the "Administratively Yielding" messages come from in
EventDrivenSchedulingAgent and ContinuallyRunProcessTask, the framework doesn't
have visibility into the active processor that threw the uncaught exception.
{quote}
It does have the ProcessorNode and ProcessContext objects so it knows precisely
which component the uncaught exception came from. What it does not have today
is the details of which flow files were involved in the process session at the
time of the uncaught exception. I'll comment on that when we get to the part
about leaking information across layers.
Next statement
{quote}
This implementation populates the log message without taxing the framework or
requiring support in the processors, but it can still be overridden for
customization by individual processors.
{quote}
It appears the log message is generated by the ProcessSession implementation.
With that in mind I don't understand how processors can override its behavior.
Next statement
{quote}
Providing flow files objects instead of a log message would expose internals
between framework layers and complicate api and implementation module
dependencies in the build.
{quote}
I agree this is not ideal. I'd prefer we avoid adding the String method as-is
in the patch and I'd also like to withdraw my suggestion to do this flow file
thing too. Instead what I recommend is that this patch simply focus in on the
simplest possible step we can take to honor the requirement as stated above and
to the original spirit of the ticket.
To that end I propose the change be as simple as in the event rollback is
called on the standard process session that it logs details about the state of
that process session at DEBUG level. Specifically it would log the sorts of
things you have already identified in your method. However, what I'm
suggesting requires no API changes, no changes to FlowFile attributes, etc..
It just simply makes available more details in the logs to help figure out
where the programming error lives in that processor.
I think over time we could do more as I was implying on my comments when you
filed the ticket. But we would need to really discuss those goals and how it
would fit into the overall user experience.
Does that sound fair?
Thanks
Joe
> Rollback log messages should include the flowfile filename and UUID to assist
> in flow management.
> -------------------------------------------------------------------------------------------------
>
> Key: NIFI-1754
> URL: https://issues.apache.org/jira/browse/NIFI-1754
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.0.0, 0.7.0
> Reporter: Joe Skora
> Assignee: Joe Skora
>
> When a processor calls rollback on a flowfile, the filename and UUID of the
> flowfile should be included in log entry to make it easier for the offending
> files to be found and fixed or removed from the flow by the dataflow managers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
