[
https://issues.apache.org/jira/browse/NIFI-4745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333655#comment-16333655
]
Martin Mucha commented on NIFI-4745:
------------------------------------
Hi!
I closed this pull request, because I found siteToSiteProvenanceReporting
assuming this could solve this issue, but it seems it can't. Therefore I'd like
to push this issue further. I can do the coding, please just discuss it first.
You're correct, that ValidateRecord can get input of multiple errors, thus
error output would be also array. It might be fine to have just one error
message for some, but not for all. Therefore it could make sense to control it.
Actually I'm not sure, if it's actually best to do 'joining' errors into one
record as a default strategy.
So lets discuss the change:
• I'd propose to have variable to set attribute with error description on flow
file. If one-per-record is fine, this is sufficient.
• however sometimes it's not sufficient, thus another boolean property to
control record creation. If false, indifidual FlowFiles with individual error
description will be emited to invalid relationship.
Btw. IIUC current implementation blocks processing, until all records are done.
Only after that new records are routed out. If that's correct, I'd also propose
one improvement. There are createIfNecessary and other methods written on idea
of building record per partes, making the method big, parameter list also big
etc. We can do categorization valid/invalid and store lists of values
(record/flowfile) to route to specific relationship, and only after all records
are categorized, we will transform them (in separate smaller method) into
result which should be routed. I'm talking about it, since this refactoring
would allow way simpler enabling routing out individual failures. Another
benefit of that would could be faster throughput - we do not need for all
records being processed to see first results.
Please advice how we can move this further.
Thanks!
> Emit validation failure description in attribute from ValidateRecord processor
> ------------------------------------------------------------------------------
>
> Key: NIFI-4745
> URL: https://issues.apache.org/jira/browse/NIFI-4745
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.5.0
> Reporter: Martin Mucha
> Priority: Minor
>
> We need to pass description of validation failure further in
> processing chain, and eventually pass it back to calling system.
> Therefore having failure description logged in logs and issued as provenance
> route event is not sufficient for us.
> It should be easy to emit same data, which are being sent in provenance route
> event, from ValidateRecord as new attribute.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
