Hi, I'd recommend looking at the NiFi Registry which has been built for this purpose. https://nifi.apache.org/docs/nifi-registry-docs/index.html
Hope this helps, Pierre Le jeu. 5 mars 2020 à 18:38, [email protected] < [email protected]> a écrit : > > We have come across an issue around updating existing flows that are > currently running in production. > > > 1. We can stop the flows and deploy the entire new flow with all of the > custom processors and all other changes. > * Issues > > i. > Data can be lost during the time of deployment > > ii. How > will this affect the provenance data – can we reattach to existing data > with no issues, do we need to keep the flow and content data also and > reattach that data to the newly installed instances > > iii. For a > cluster how will this affect the zookeeper if at all > > iv. State > between processors, will removing the state have any unforeseen issues > > * Solutions for Streaming Flows & some batch flows > > i. Add > a buffer or Queue using Kafka or some other queue that will allow stopping > of the data into Nifi while we make the changes to the live flow – works as > if we had a connection at the start of every flow > > 1. We can do a live flow deployment using the nifi registry and api and > cli, by upgrading each modified part of the flow piece by piece, we can > stop the queue before the first change and make changes and restart the > flow, this will avoid the loss of data that we will see with the first > method > * Issues > > i. > Custom controller service – nypiapi has some tools around starting and > stopping the controllers and services will this address the service name > issues when changing a controller > > ii. This > can get very complicates when the flow has multiple changes that are needed > > iii. There > will be much more places where we can have a failure in this automated > system > > iv. Are > there any flow changes that we cannot accomplish on a live flow? > > * Pros > > i. The > validation that occurs when using the registry to deploy into a live flow > will stop the deployment if this is a breaking change, hopefully that would > help avoid issues with the deployments > > I was hoping that there are some best practices around updating flows that > are in production, or what has worked for others and what did not. > > > ________________________________ > > The information contained in this message is intended only for the > recipient, and may be a confidential attorney-client communication or may > otherwise be privileged and confidential and protected from disclosure. If > the reader of this message is not the intended recipient, or an employee or > agent responsible for delivering this message to the intended recipient, > please be aware that any dissemination or copying of this communication is > strictly prohibited. If you have received this communication in error, > please immediately notify us by replying to the message and deleting it > from your computer. S&P Global Inc. reserves the right, subject to > applicable local law, to monitor, review and process the content of any > electronic message or information sent to or from S&P Global Inc. e-mail > addresses without informing the sender or recipient of the message. By > sending electronic message or information to S&P Global Inc. e-mail > addresses you, as the sender, are consenting to S&P Global Inc. processing > any of your personal data therein. >
