Hi Deepak, Currently, in case of any error, MongoDB processors transfer flowfiles to the failure relationship. I believe it is a good idea to enhance the error handling and either rollback if the processor is unable to connect to the database or move the flowfiles to a retry relationship. I'm not sure whether there are any technical limitations, however I'd suggest opening a ticket on NiFi JIRA <https://issues.apache.org/jira/projects/NIFI/summary>.
Kind Regards, Lehel Boér Chirthani, Deepak Reddy <[email protected]> ezt írta (időpont: 2021. okt. 10., V, 21:11): > Hi Nifi Devs, > > > > We are having an issue with PutMongoRecord Processor(same for PutMongo as > well) which is routing the flowfiles to the failure relationship when the > Mongo database(or the server hosting the db) is down. We have a set-up > which looks like below on a high-level overview. > > > > > > As you can clearly see, all the failure relationships of different > processors in the dataflow are connected to an Error Handling Process > Group. Now, when the Mongodb is down, the PutMongoRecord processor is > routing all the flowfiles to the failure relationship. Now, I know that a > processor could route a flowfile to the failure relationship for multitude > of reasons and I believe PutMongoRecord routing the flowfiles to failure > when the database is down is not ideal for our scenario. Our dataflow is > real-time streaming and when mongo database is down and flowfiles get > routed to failure relationship, we get hundreds of emails because of the > PutEmail processor in the Error Handling Process Group. We configured the > Error Handling process group to check if we get any erroneous data which is > against our schema or whatever maybe the reason. > > Is there any logic you guys help us in this scenario? > > I also observed that when we load the data to a RDBMS, lets say SQL > Server, using the PutSql processor, it doesn’t route the flowfiles to the > failure relationship when the sql server is down. I remember the flowfiles > were held up in the connector above it until the db is up and running. I > also noticed that PutSql has retry and PutMongo(and Record) doesn’t have > any retry. > > Please help us what could be best thing for us to as this is critical to > us. > > Thanks in Advance. Please reach out to me if you have any questions. > > Thanks and Regards > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. >
