Thanks for the great summarization Andy. Just wanted to clarify one small detail. The default file-based authorizer will use conf/authorizations.xml though this value is configurable in conf/authorizers.xml. In the 0.x, when NiFi used role-based authorization the file was called conf/authorized-users.xml.
Matt On Thu, Jun 29, 2017 at 11:28 PM, Jeremy Farbota <[email protected]> wrote: > Andy, > > Thanks a ton. Huge help. We will report back. > > [image: Payoff, Inc.] > *Jeremy Farbota* > Software Engineer, Data > Payoff, Inc. > > [email protected] > (217) 898-8110 <+2178988110> > > On Thu, Jun 29, 2017 at 5:20 PM, Andy LoPresto <[email protected]> > wrote: > >> I believe I missed a crucial file — if you are using the default >> file-based authorizer provided by NiFi, you will also want to copy >> conf/authorized-users.xml which defines the various users and their access >> control policies. Sorry about that, the instance I referenced when typing >> wasn’t using it so I forgot. >> >> Andy LoPresto >> [email protected] >> *[email protected] <[email protected]>* >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jun 29, 2017, at 5:17 PM, Andy LoPresto <[email protected]> wrote: >> >> Jeremy, >> >> Sorry to hear you are having difficulties upgrading. I’ll try to answer >> your questions as best I can, but welcome others joining the thread. >> >> It sounds like this is less of an in-place upgrade and more of a cluster >> migration since you are installing the software on new nodes and moving the >> flow there. If you do not also move the various repositories, you >> effectively have a new fresh install and have just copied the flow >> definition. >> >> The Apache NiFi wiki does provide Migration Guidance [1] and this >> document is updated with best practices/standard operating procedure with >> each release. It is prepared by the developers who have tested the upgrade >> process, and hopefully is helpful to end users doing the same. Perhaps even >> more relevant is the Upgrade Guide [2], which has step-by-step instructions >> for configuring an instance to be easily upgradable, and how to perform the >> upgrades in-place quickly and repeatably. >> >> To “upgrade” to a new node and maintain the existing functionality and >> configurations, you would want to copy additional files: >> >> * conf/authorizers.xml — contains the user/group providers and access >> policy providers (separated during 1.3.0) which control permissions in the >> application >> * conf/bootstrap.conf — contains the master key for encryption and other >> NiFi runtime arguments >> * conf/logback.xml — defines log levels >> * conf/login-identity-providers.xml — defines LDAP or Kerberos >> connection details >> * conf/nifi.properties — global property definition for the application >> * conf/state-management.xml — defines state management configuration >> (especially important for a cluster) >> * conf/zookeeper.properties — defines ZK configurations and addresses >> (especially important for a cluster) >> >> [Optional] >> >> * flowfile_repository/ — copy this to allow flowfile replay and history >> * content_repository/ — copy this to allow flowfile replay and history >> * provenance_repository/ — copy this to maintain the history of the >> provenance records >> >> I’m not quite sure what you mean by “keeping the state of my consumers” — >> is this specifically a ConsumeKafka processor or just source processors in >> general? If you upgrade in-place or copy the various configs and >> repositories mentioned above, when the app starts again, the component >> state should be as you left it (barring code changes to the component in >> the new release). >> >> Hopefully those instructions and the documents linked below will help >> you. We definitely want to make upgrading a seamless experience because we >> are constantly working to bring new features and optimizations into the >> app, and supporting legacy versions is an added cost. We are not looking >> for version fragmentation here. Any suggestions you have for improving the >> upgrade process and our communication around those instructions is welcome. >> Thanks. >> >> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance >> [2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi >> >> >> Andy LoPresto >> [email protected] >> *[email protected] <[email protected]>* >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jun 29, 2017, at 4:15 PM, Jeremy Farbota <[email protected]> wrote: >> >> We are attempting to upgrade and we're having issues: >> >> We created new nodes with the new version (1.3.0). We stopped all flowed >> and ensured nothing is in queue. We copied the flow.xml.gz file to the new >> node. >> >> Are following the right protocol? Should we be copying the users.xml file >> as well? >> >> Is there a way to migrate to the new version while keeping the state of >> my consumers? >> >> It seems like I have to build a cluster and recreate the user permissions >> and everything from scratch. Is there a workaround? >> >> Is there a wiki or any notes about how to upgrade a cluster? >> >> Last time I posted this there was a reply with a link to a thread that >> does not provide any additional info about how to bring the cluster up with >> the new version. >> >> [image: Payoff, Inc.] >> *Jeremy Farbota* >> Software Engineer, Data >> Payoff, Inc. >> >> [email protected] >> (217) 898-8110 <+2178988110> >> >> On Thu, Jun 15, 2017 at 12:23 PM, Jeremy Farbota <[email protected]> >> wrote: >> >>> We're preparing to upgrade our cluster from 1.0.0 to 1.3.0. >>> >>> We're using external zookeeper. >>> >>> We're wondering if we can simply expand the cluster with 1.3.0 machines >>> and then turn off the other ones to keep processes running. Are there >>> issues with attempting that? Is there an upgrade guide someone for clusters >>> on Wiki? >>> >>> We have kafka consumers and other maintenance processes that are running >>> in production so we'd like to make the change without messing with the >>> state of those consumers if possible. >>> >>> Kindly, >>> >>> >>> [image: Payoff, Inc.] >>> *Jeremy Farbota* >>> Software Engineer, Data >>> Payoff, Inc. >>> >>> [email protected] >>> (217) 898-8110 <+2178988110> >>> >> >> >> >> >
