Jeremy, Sorry to hear you are having difficulties upgrading. I’ll try to answer your questions as best I can, but welcome others joining the thread.
It sounds like this is less of an in-place upgrade and more of a cluster migration since you are installing the software on new nodes and moving the flow there. If you do not also move the various repositories, you effectively have a new fresh install and have just copied the flow definition. The Apache NiFi wiki does provide Migration Guidance [1] and this document is updated with best practices/standard operating procedure with each release. It is prepared by the developers who have tested the upgrade process, and hopefully is helpful to end users doing the same. Perhaps even more relevant is the Upgrade Guide [2], which has step-by-step instructions for configuring an instance to be easily upgradable, and how to perform the upgrades in-place quickly and repeatably. To “upgrade” to a new node and maintain the existing functionality and configurations, you would want to copy additional files: * conf/authorizers.xml — contains the user/group providers and access policy providers (separated during 1.3.0) which control permissions in the application * conf/bootstrap.conf — contains the master key for encryption and other NiFi runtime arguments * conf/logback.xml — defines log levels * conf/login-identity-providers.xml — defines LDAP or Kerberos connection details * conf/nifi.properties — global property definition for the application * conf/state-management.xml — defines state management configuration (especially important for a cluster) * conf/zookeeper.properties — defines ZK configurations and addresses (especially important for a cluster) [Optional] * flowfile_repository/ — copy this to allow flowfile replay and history * content_repository/ — copy this to allow flowfile replay and history * provenance_repository/ — copy this to maintain the history of the provenance records I’m not quite sure what you mean by “keeping the state of my consumers” — is this specifically a ConsumeKafka processor or just source processors in general? If you upgrade in-place or copy the various configs and repositories mentioned above, when the app starts again, the component state should be as you left it (barring code changes to the component in the new release). Hopefully those instructions and the documents linked below will help you. We definitely want to make upgrading a seamless experience because we are constantly working to bring new features and optimizations into the app, and supporting legacy versions is an added cost. We are not looking for version fragmentation here. Any suggestions you have for improving the upgrade process and our communication around those instructions is welcome. Thanks. [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance <https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance> [2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi <https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi> Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 29, 2017, at 4:15 PM, Jeremy Farbota <[email protected]> wrote: > > We are attempting to upgrade and we're having issues: > > We created new nodes with the new version (1.3.0). We stopped all flowed and > ensured nothing is in queue. We copied the flow.xml.gz file to the new node. > > Are following the right protocol? Should we be copying the users.xml file as > well? > > Is there a way to migrate to the new version while keeping the state of my > consumers? > > It seems like I have to build a cluster and recreate the user permissions and > everything from scratch. Is there a workaround? > > Is there a wiki or any notes about how to upgrade a cluster? > > Last time I posted this there was a reply with a link to a thread that does > not provide any additional info about how to bring the cluster up with the > new version. > > > Jeremy Farbota > Software Engineer, Data > Payoff, Inc. > > [email protected] <mailto:[email protected]> > (217) 898-8110 <tel:+2178988110> > On Thu, Jun 15, 2017 at 12:23 PM, Jeremy Farbota <[email protected] > <mailto:[email protected]>> wrote: > We're preparing to upgrade our cluster from 1.0.0 to 1.3.0. > > We're using external zookeeper. > > We're wondering if we can simply expand the cluster with 1.3.0 machines and > then turn off the other ones to keep processes running. Are there issues with > attempting that? Is there an upgrade guide someone for clusters on Wiki? > > We have kafka consumers and other maintenance processes that are running in > production so we'd like to make the change without messing with the state of > those consumers if possible. > > Kindly, > > > > Jeremy Farbota > Software Engineer, Data > Payoff, Inc. > > [email protected] <mailto:[email protected]> > (217) 898-8110 <tel:+2178988110>
signature.asc
Description: Message signed with OpenPGP using GPGMail
