Deepak Reddy Chirthani created NIFI-9295:
--------------------------------------------
Summary: Issue with PutMongo(and PutMongoRecord)
Key: NIFI-9295
URL: https://issues.apache.org/jira/browse/NIFI-9295
Project: Apache NiFi
Issue Type: Improvement
Components: Configuration
Affects Versions: 1.11.4
Reporter: Deepak Reddy Chirthani
Attachments: image-2021-10-12-12-19-08-085.png
We are having an issue with PutMongoRecord Processor(same for PutMongo as well)
which is routing the flowfiles to the failure relationship when the Mongo
database(or the server hosting the db) is down. We have a set-up which looks
like below on a high-level overview.
!image-2021-10-12-12-19-08-085.png!
As you can clearly see, all the failure relationships of different processors
in the dataflow are connected to an Error Handling Process Group. Now, when the
Mongodb is down, the PutMongoRecord processor is routing all the flowfiles to
the failure relationship. Now, I know that a processor could route a flowfile
to the failure relationship for multitude of reasons and I believe
PutMongoRecord routing the flowfiles to failure when the database is down is
not ideal for our scenario. Our dataflow is real-time streaming and when mongo
database is down and flowfiles get routed to failure relationship, we get
hundreds of emails because of the PutEmail processor in the Error Handling
Process Group. We configured the Error Handling process group to check if we
get any erroneous data which is against our schema or whatever maybe the reason.
Is there any logic you guys help us in this scenario?
I also observed that when we load the data to a RDBMS, lets say SQL Server,
using the PutSql processor, it doesn’t route the flowfiles to the failure
relationship when the sql server is down. I remember the flowfiles were held up
in the connector above it until the db is up and running. I also noticed that
PutSql has retry and PutMongo(and Record) doesn’t have any retry.
Thanks in Advance. Please comment if you guys have any questions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
