We have a similar need, and I was thinking about this situation not long ago because we spent a lot of cycles debugging why our record sets fail (we handle lots of very low quality data).
I'd like to propose two options here to help debug: 1. Optional feature to add the stacktrace to the flowfile as an attribute. 2. Optional feature to keep chugging through the record set, logging every error message to an output flowfile that would go to an auto-terminated by default relationship called error_messages For the second scenario, I envision someone being able to wire up the flow to chuck the record set into something like Mongo or Elasticsearch in one index and the error_messages into another. On Fri, Aug 16, 2019 at 9:59 AM Phillip Grenier <[email protected]> wrote: > Would it be reasonable to add the details of the failures to the flow file > attributes? I know they exist on the provenance event, but that can not be > persisted easily to file for later analysis and correction. This also seems > harder to locate for non-developers and flow designers. > > Thanks, > > Phillip >
