Hi Craig, First the generic stuff...
according to the tests I made, no flows are lost when a machine is removed from the cluster. They seem to be requeued. However, I only tested with a very basic flow and not with my whole flow which involves a lot of things. Basically, I used a GenerateFlow to generate some data and a dummy Python process to do something with it. The queue between the two processors was configured to do load balancing using a round robin. I must admit that I haven't look if the item was requeued and dispatched to another node. The output of the python module was split between success and failure and no single flow reached the failure state. then to AWS specific stuff... I had to script a few things to cleanup within the two minutes warning AWS is giving me. Since I am using spot instances, I know the instance will not come back so I had to automate the clean up of the cluster by using an API call to remove the machine from the cluster. In order to remove the machine from the cluster, I need to stop Nifi first and then remove the machine through a call to the API on a second node. I am still polishing the script to accomplish this. I may share it once it is working as expected in case someone else has this issue. Let me know if you need more details about anything... ________________________________ From: Craig Knell <[email protected]> Sent: Wednesday, August 28, 2019 6:52 PM To: [email protected] <[email protected]> Subject: Re: clean shutdown Hi Jean-Sebastien, I’d be interested to hear how this performs Best regards Craig On 28 Aug 2019, at 22:28, Jean-Sebastien Vachon <[email protected]<mailto:[email protected]>> wrote: Hi Pierre, thanks for your input. I am already intercepting AWS termination notification so I will add a few steps and see how it reacts Thanks again ________________________________ From: Pierre Villard <[email protected]<mailto:[email protected]>> Sent: Wednesday, August 28, 2019 4:17 AM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: clean shutdown Hi Jean-Sebastien, When you stop NiFi, by default, it will try to gracefully stop everything in 10 seconds, and if not all components are nicely stopped after that, it will force shut down the NiFi process. This is configured with "nifi.flowcontroller.graceful.shutdown.period" in nifi.properties file. If you have processors/CS that might take longer to stop gracefully (because of connections to external systems for instance), you could increase this value. I'm not very familiar with AWS spot instances but I'd try to catch the spot notification event to stop the NiFi service on the host before the instance is stopped/killed. Pierre Le mar. 27 août 2019 à 20:05, Jean-Sebastien Vachon <[email protected]<mailto:[email protected]>> a écrit : Hi everybody, I am working with AWS spot instances and one thing that is giving me a hard time is to perform a clean (and quick) shutdown of Nifi in order to prevent data loss. AWS will give you about two minutes to clean up everything before the machine is actually shutdown. Is there a way to stop/kill all processes running on the host without loosing anything? It is fine if all the flowfiles being processed are simply requeued. Would simply killing the processes achieve this? (I doubt so)... would it be better to fetch a list of running processors and terminate them using Nifi's API? All ideas and thoughts are welcome thanks
