Good morning everyone, I discovered the following issue which is bugging me lately: If I connect the output of a MergeRecord processor to the input of a PutDatabaseRecord processor, sometimes a FlowFile makes the latter fail with FlowFileHandlingException: FlowFile already marked for transfer. The issue is that when this happens, the FlowFile is not routed to the failed relationship, but rather the whole processor remain stuck, the queue must be manually emptied, losing all content.
I created this issue with further details: https://issues.apache.org/jira/browse/NIFI-8130 I will start looking at the processor code, but I am not quite sure about what a good solution would be: - A naive yet effective approach would be to just catch such exceptions and route the Flow File to the failed relationship. This would prevent the issue of having to manually empty the full queue, yet 1 FlowFile or 1000 FlowFile, it's still data loss. - A more advanced approach would be to find out exactly what causes this issue to raise and prevent it, but the random nature of this error makes it very hard to investigate it. I have no idea if this might be related, but here's an example of Processor configuration which raises the problem: The flow is the following: I consume a bunch of records (each record is a json) from a Kafka queue, I merge those records in bins, and then I execute a DB transaction for each bin (the DB I'm using is Vertica, which supports a jdbc driver, therefore at NiFi eyes it's treated as a plain relational DB) [image: Screenshot 2021-01-11 at 11.38.45.png] Merge Record configuration: [image: image.png] PutDatabaseRecord configuration: [image: image.png] Any suggestion is appreciated, thanks. Regards, Alessandro D'Armiento
