Andy, Thanks a ton. Huge help. We will report back.
[image: Payoff, Inc.] *Jeremy Farbota* Software Engineer, Data Payoff, Inc. [email protected] (217) 898-8110 <+2178988110> On Thu, Jun 29, 2017 at 5:20 PM, Andy LoPresto <[email protected]> wrote: > I believe I missed a crucial file — if you are using the default > file-based authorizer provided by NiFi, you will also want to copy > conf/authorized-users.xml which defines the various users and their access > control policies. Sorry about that, the instance I referenced when typing > wasn’t using it so I forgot. > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 29, 2017, at 5:17 PM, Andy LoPresto <[email protected]> wrote: > > Jeremy, > > Sorry to hear you are having difficulties upgrading. I’ll try to answer > your questions as best I can, but welcome others joining the thread. > > It sounds like this is less of an in-place upgrade and more of a cluster > migration since you are installing the software on new nodes and moving the > flow there. If you do not also move the various repositories, you > effectively have a new fresh install and have just copied the flow > definition. > > The Apache NiFi wiki does provide Migration Guidance [1] and this document > is updated with best practices/standard operating procedure with each > release. It is prepared by the developers who have tested the upgrade > process, and hopefully is helpful to end users doing the same. Perhaps even > more relevant is the Upgrade Guide [2], which has step-by-step instructions > for configuring an instance to be easily upgradable, and how to perform the > upgrades in-place quickly and repeatably. > > To “upgrade” to a new node and maintain the existing functionality and > configurations, you would want to copy additional files: > > * conf/authorizers.xml — contains the user/group providers and access > policy providers (separated during 1.3.0) which control permissions in the > application > * conf/bootstrap.conf — contains the master key for encryption and other > NiFi runtime arguments > * conf/logback.xml — defines log levels > * conf/login-identity-providers.xml — defines LDAP or Kerberos connection > details > * conf/nifi.properties — global property definition for the application > * conf/state-management.xml — defines state management configuration > (especially important for a cluster) > * conf/zookeeper.properties — defines ZK configurations and addresses > (especially important for a cluster) > > [Optional] > > * flowfile_repository/ — copy this to allow flowfile replay and history > * content_repository/ — copy this to allow flowfile replay and history > * provenance_repository/ — copy this to maintain the history of the > provenance records > > I’m not quite sure what you mean by “keeping the state of my consumers” — > is this specifically a ConsumeKafka processor or just source processors in > general? If you upgrade in-place or copy the various configs and > repositories mentioned above, when the app starts again, the component > state should be as you left it (barring code changes to the component in > the new release). > > Hopefully those instructions and the documents linked below will help you. > We definitely want to make upgrading a seamless experience because we are > constantly working to bring new features and optimizations into the app, > and supporting legacy versions is an added cost. We are not looking for > version fragmentation here. Any suggestions you have for improving the > upgrade process and our communication around those instructions is welcome. > Thanks. > > [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance > [2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi > > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 29, 2017, at 4:15 PM, Jeremy Farbota <[email protected]> wrote: > > We are attempting to upgrade and we're having issues: > > We created new nodes with the new version (1.3.0). We stopped all flowed > and ensured nothing is in queue. We copied the flow.xml.gz file to the new > node. > > Are following the right protocol? Should we be copying the users.xml file > as well? > > Is there a way to migrate to the new version while keeping the state of my > consumers? > > It seems like I have to build a cluster and recreate the user permissions > and everything from scratch. Is there a workaround? > > Is there a wiki or any notes about how to upgrade a cluster? > > Last time I posted this there was a reply with a link to a thread that > does not provide any additional info about how to bring the cluster up with > the new version. > > [image: Payoff, Inc.] > *Jeremy Farbota* > Software Engineer, Data > Payoff, Inc. > > [email protected] > (217) 898-8110 <+2178988110> > > On Thu, Jun 15, 2017 at 12:23 PM, Jeremy Farbota <[email protected]> > wrote: > >> We're preparing to upgrade our cluster from 1.0.0 to 1.3.0. >> >> We're using external zookeeper. >> >> We're wondering if we can simply expand the cluster with 1.3.0 machines >> and then turn off the other ones to keep processes running. Are there >> issues with attempting that? Is there an upgrade guide someone for clusters >> on Wiki? >> >> We have kafka consumers and other maintenance processes that are running >> in production so we'd like to make the change without messing with the >> state of those consumers if possible. >> >> Kindly, >> >> >> [image: Payoff, Inc.] >> *Jeremy Farbota* >> Software Engineer, Data >> Payoff, Inc. >> >> [email protected] >> (217) 898-8110 <+2178988110> >> > > > >
