We have come across an issue around updating existing flows that are currently
running in production.
1. We can stop the flows and deploy the entire new flow with all of the
custom processors and all other changes.
* Issues
i. Data can
be lost during the time of deployment
ii. How will
this affect the provenance data – can we reattach to existing data with no
issues, do we need to keep the flow and content data also and reattach that
data to the newly installed instances
iii. For a
cluster how will this affect the zookeeper if at all
iv. State
between processors, will removing the state have any unforeseen issues
* Solutions for Streaming Flows & some batch flows
i. Add a
buffer or Queue using Kafka or some other queue that will allow stopping of the
data into Nifi while we make the changes to the live flow – works as if we had
a connection at the start of every flow
1. We can do a live flow deployment using the nifi registry and api and cli,
by upgrading each modified part of the flow piece by piece, we can stop the
queue before the first change and make changes and restart the flow, this will
avoid the loss of data that we will see with the first method
* Issues
i. Custom
controller service – nypiapi has some tools around starting and stopping the
controllers and services will this address the service name issues when
changing a controller
ii. This can
get very complicates when the flow has multiple changes that are needed
iii. There will
be much more places where we can have a failure in this automated system
iv. Are there
any flow changes that we cannot accomplish on a live flow?
* Pros
i. The
validation that occurs when using the registry to deploy into a live flow will
stop the deployment if this is a breaking change, hopefully that would help
avoid issues with the deployments
I was hoping that there are some best practices around updating flows that are
in production, or what has worked for others and what did not.
________________________________
The information contained in this message is intended only for the recipient,
and may be a confidential attorney-client communication or may otherwise be
privileged and confidential and protected from disclosure. If the reader of
this message is not the intended recipient, or an employee or agent responsible
for delivering this message to the intended recipient, please be aware that any
dissemination or copying of this communication is strictly prohibited. If you
have received this communication in error, please immediately notify us by
replying to the message and deleting it from your computer. S&P Global Inc.
reserves the right, subject to applicable local law, to monitor, review and
process the content of any electronic message or information sent to or from
S&P Global Inc. e-mail addresses without informing the sender or recipient of
the message. By sending electronic message or information to S&P Global Inc.
e-mail addresses you, as the sender, are consenting to S&P Global Inc.
processing any of your personal data therein.
