Dweep, The difference that NiFi offers between Auto Acknowledge and Client Acknowledge is that Auto Acknowledge is much faster but at the expense of possibly losing messages (if it is acknowledge, then NiFi is shutdown before persisting the data). With Client Acknowledge, the ConsumeAMQP Processor will persist the data to its repositories and only then acknowledge the message. As a result, if NiFi is restarted, the data will not be lost but could perhaps be duplicated, if the data is persisted and then NiFi restarts before the message is acknowledged. But on restart, NiFi will have the data wherever it was in the flow and automatically resume processing it.
There is also an effort underway known as Stateless NiFi [1], which would offer semantics more like what you're proposing, where the entire flow would be executed and only then the message would be acknowledged. This would require that the flow be stored in the NiFi Registry and then Stateless NiFi could be launched. But this effort is still pretty well in its infancy and as such doesn't have quite the level of documentation that you may be accustomed to with NiFi yet. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-stateless/README.md On Nov 22, 2019, at 4:10 AM, Dweep Sharma <[email protected]<mailto:[email protected]>> wrote: Hi, I want to use the ConsumeAMQ processor to consume events from rabbitmq. The second step is to make an API call to acquire more data and finally merge contents and store it as Parquet in S3. My question is, since there are so many steps after the consumeAMQ, how can i acknowledge the message only after all these steps are complete to prevent any loss of data The AutoAck false property will not acknowledge the message by default but when does the message get dequeued if this is set to false? -Dweep ::DISCLAIMER:: ---------------------------------------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com<http://redBus.com>. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com<http://redBus.com>. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.<http://redbus.in/>com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.
