Modified: nifi/site/trunk/docs/nifi-docs/html/administration-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/administration-guide.html?rev=1771892&r1=1771891&r2=1771892&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/administration-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/administration-guide.html Tue Nov 29 12:03:34 2016 @@ -455,82 +455,83 @@ body.book #toc,body.book #preamble,body. <div id="toc" class="toc"> <div id="toctitle">Table of Contents</div> <ul class="sectlevel1"> -<li><a href="administration-guide.html#system-requirements">System Requirements</a></li> -<li><a href="administration-guide.html#how-to-install-and-start-nifi">How to install and start NiFi</a></li> -<li><a href="administration-guide.html#configuration-best-practices">Configuration Best Practices</a></li> -<li><a href="administration-guide.html#security-configuration">Security Configuration</a> +<li><a href="#system-requirements">System Requirements</a></li> +<li><a href="#how-to-install-and-start-nifi">How to install and start NiFi</a></li> +<li><a href="#configuration-best-practices">Configuration Best Practices</a></li> +<li><a href="#security-configuration">Security Configuration</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#tls-generation-toolkit">TLS Generation Toolkit</a></li> +<li><a href="#tls-generation-toolkit">TLS Generation Toolkit</a></li> </ul> </li> -<li><a href="administration-guide.html#user-authentication">User Authentication</a> +<li><a href="#user-authentication">User Authentication</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#lightweight-directory-access-protocol-ldap">Lightweight Directory Access Protocol (LDAP)</a></li> -<li><a href="administration-guide.html#kerberos_login_identity_provider">Kerberos</a></li> +<li><a href="#lightweight-directory-access-protocol-ldap">Lightweight Directory Access Protocol (LDAP)</a></li> +<li><a href="#kerberos_login_identity_provider">Kerberos</a></li> </ul> </li> -<li><a href="administration-guide.html#multi-tenant-authorization">Multi-Tenant Authorization</a> +<li><a href="#multi-tenant-authorization">Multi-Tenant Authorization</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#authorizer-configuration">Authorizer Configuration</a></li> -<li><a href="administration-guide.html#authorizers-setup">Authorizers.xml Setup</a></li> -<li><a href="administration-guide.html#config-users-access-policies">Configuring Users & Access Policies</a></li> +<li><a href="#authorizer-configuration">Authorizer Configuration</a></li> +<li><a href="#authorizers-setup">Authorizers.xml Setup</a></li> +<li><a href="#config-users-access-policies">Configuring Users & Access Policies</a></li> </ul> </li> -<li><a href="administration-guide.html#encryption">Encryption Configuration</a> +<li><a href="#encryption">Encryption Configuration</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#key-derivation-functions">Key Derivation Functions</a></li> -<li><a href="administration-guide.html#salt-and-iv-encoding">Salt and IV Encoding</a></li> -<li><a href="administration-guide.html#java-cryptography-extension-jce-limited-strength-jurisdiction-policies">Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies</a></li> -<li><a href="administration-guide.html#allow-insecure-cryptographic-modes">Allow Insecure Cryptographic Modes</a></li> +<li><a href="#key-derivation-functions">Key Derivation Functions</a></li> +<li><a href="#salt-and-iv-encoding">Salt and IV Encoding</a></li> +<li><a href="#java-cryptography-extension-jce-limited-strength-jurisdiction-policies">Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies</a></li> +<li><a href="#allow-insecure-cryptographic-modes">Allow Insecure Cryptographic Modes</a></li> </ul> </li> -<li><a href="administration-guide.html#encrypted-passwords-in-configuration-files">Encrypted Passwords in Configuration Files</a> +<li><a href="#encrypted-passwords-in-configuration-files">Encrypted Passwords in Configuration Files</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#encrypt-config_tool">Encrypt-Config Tool</a></li> -<li><a href="administration-guide.html#encrypt-config_password">Password Key Derivation</a></li> -<li><a href="administration-guide.html#encrypt-config_secure_prompt">Secure Prompt</a></li> +<li><a href="#encrypt-config_tool">Encrypt-Config Tool</a></li> +<li><a href="#encrypt-config_password">Password Key Derivation</a></li> +<li><a href="#encrypt-config_secure_prompt">Secure Prompt</a></li> </ul> </li> -<li><a href="administration-guide.html#clustering">Clustering Configuration</a></li> -<li><a href="administration-guide.html#state_management">State Management</a> +<li><a href="#clustering">Clustering Configuration</a></li> +<li><a href="#state_management">State Management</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#state_providers">Configuring State Providers</a></li> -<li><a href="administration-guide.html#embedded_zookeeper">Embedded ZooKeeper Server</a></li> -<li><a href="administration-guide.html#zk_access_control">ZooKeeper Access Control</a></li> -<li><a href="administration-guide.html#securing_zookeeper">Securing ZooKeeper</a></li> +<li><a href="#state_providers">Configuring State Providers</a></li> +<li><a href="#embedded_zookeeper">Embedded ZooKeeper Server</a></li> +<li><a href="#zk_access_control">ZooKeeper Access Control</a></li> +<li><a href="#securing_zookeeper">Securing ZooKeeper</a></li> +<li><a href="#zookeeper_migrator">ZooKeeper Migrator</a></li> </ul> </li> -<li><a href="administration-guide.html#bootstrap_properties">Bootstrap Properties</a></li> -<li><a href="administration-guide.html#notification_services">Notification Services</a></li> -<li><a href="administration-guide.html#kerberos_service">Kerberos Service</a> +<li><a href="#bootstrap_properties">Bootstrap Properties</a></li> +<li><a href="#notification_services">Notification Services</a></li> +<li><a href="#kerberos_service">Kerberos Service</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#kerberos_service_notes">Notes</a></li> +<li><a href="#kerberos_service_notes">Notes</a></li> </ul> </li> -<li><a href="administration-guide.html#system_properties">System Properties</a> +<li><a href="#system_properties">System Properties</a> <ul class="sectlevel2"> -<li><a href="administration-guide.html#core-properties-br">Core Properties<br></a></li> -<li><a href="administration-guide.html#state-management-br">State Management<br></a></li> -<li><a href="administration-guide.html#h2-settings">H2 Settings</a></li> -<li><a href="administration-guide.html#flowfile-repository">FlowFile Repository</a></li> -<li><a href="administration-guide.html#swap-management">Swap Management</a></li> -<li><a href="administration-guide.html#content-repository">Content Repository</a></li> -<li><a href="administration-guide.html#file-system-content-repository-properties">File System Content Repository Properties</a></li> -<li><a href="administration-guide.html#volatile-content-repository-properties">Volatile Content Repository Properties</a></li> -<li><a href="administration-guide.html#provenance-repository">Provenance Repository</a></li> -<li><a href="administration-guide.html#persistent-provenance-repository-properties">Persistent Provenance Repository Properties</a></li> -<li><a href="administration-guide.html#volatile-provenance-repository-properties">Volatile Provenance Repository Properties</a></li> -<li><a href="administration-guide.html#component-status-repository">Component Status Repository</a></li> -<li><a href="administration-guide.html#site_to_site_properties">Site to Site Properties</a></li> -<li><a href="administration-guide.html#web-properties">Web Properties</a></li> -<li><a href="administration-guide.html#security-properties">Security Properties</a></li> -<li><a href="administration-guide.html#identity-mapping-properties">Identity Mapping Properties</a></li> -<li><a href="administration-guide.html#cluster-common-properties">Cluster Common Properties</a></li> -<li><a href="administration-guide.html#cluster-node-properties">Cluster Node Properties</a></li> -<li><a href="administration-guide.html#claim_management">Claim Management</a></li> -<li><a href="administration-guide.html#zookeeper-properties">ZooKeeper Properties</a></li> -<li><a href="administration-guide.html#kerberos_properties">Kerberos Properties</a></li> -<li><a href="administration-guide.html#custom_properties">Custom Properties</a></li> +<li><a href="#core-properties-br">Core Properties<br></a></li> +<li><a href="#state-management-br">State Management<br></a></li> +<li><a href="#h2-settings">H2 Settings</a></li> +<li><a href="#flowfile-repository">FlowFile Repository</a></li> +<li><a href="#swap-management">Swap Management</a></li> +<li><a href="#content-repository">Content Repository</a></li> +<li><a href="#file-system-content-repository-properties">File System Content Repository Properties</a></li> +<li><a href="#volatile-content-repository-properties">Volatile Content Repository Properties</a></li> +<li><a href="#provenance-repository">Provenance Repository</a></li> +<li><a href="#persistent-provenance-repository-properties">Persistent Provenance Repository Properties</a></li> +<li><a href="#volatile-provenance-repository-properties">Volatile Provenance Repository Properties</a></li> +<li><a href="#component-status-repository">Component Status Repository</a></li> +<li><a href="#site_to_site_properties">Site to Site Properties</a></li> +<li><a href="#web-properties">Web Properties</a></li> +<li><a href="#security-properties">Security Properties</a></li> +<li><a href="#identity-mapping-properties">Identity Mapping Properties</a></li> +<li><a href="#cluster-common-properties">Cluster Common Properties</a></li> +<li><a href="#cluster-node-properties">Cluster Node Properties</a></li> +<li><a href="#claim_management">Claim Management</a></li> +<li><a href="#zookeeper-properties">ZooKeeper Properties</a></li> +<li><a href="#kerberos_properties">Kerberos Properties</a></li> +<li><a href="#custom_properties">Custom Properties</a></li> </ul> </li> </ul> @@ -538,10 +539,10 @@ body.book #toc,body.book #preamble,body. </div> <div id="content"> <div class="sect1"> -<h2 id="system-requirements"><a class="anchor" href="administration-guide.html#system-requirements"></a>System Requirements</h2> +<h2 id="system-requirements"><a class="anchor" href="#system-requirements"></a>System Requirements</h2> <div class="sectionbody"> <div class="paragraph"> -<p>Apache NiFi can run on something as simple as a laptop, but it can also be clustered across many enterprise-class servers. Therefore, the amount of hardware and memory needed will depend on the size and nature of the dataflow involved. The data is stored on disk while NiFi is processing it. So NiFi needs to have sufficient disk space allocated for its various repositories, particularly the content repository, flowfile repository, and provenance repository (see the <a href="administration-guide.html#system_properties">System Properties</a> section for more information about these repositories). NiFi has the following minimum system requirements:</p> +<p>Apache NiFi can run on something as simple as a laptop, but it can also be clustered across many enterprise-class servers. Therefore, the amount of hardware and memory needed will depend on the size and nature of the dataflow involved. The data is stored on disk while NiFi is processing it. So NiFi needs to have sufficient disk space allocated for its various repositories, particularly the content repository, flowfile repository, and provenance repository (see the <a href="#system_properties">System Properties</a> section for more information about these repositories). NiFi has the following minimum system requirements:</p> </div> <div class="ulist"> <ul> @@ -589,12 +590,12 @@ body.book #toc,body.book #preamble,body. </ul> </div> <div class="paragraph"> -<p><strong>Note</strong> Under sustained and extremely high throughput the CodeCache settings may need to be tuned to avoid sudden performance loss. See the <a href="administration-guide.html#bootstrap_properties">Bootstrap Properties</a> section for more information.</p> +<p><strong>Note</strong> Under sustained and extremely high throughput the CodeCache settings may need to be tuned to avoid sudden performance loss. See the <a href="#bootstrap_properties">Bootstrap Properties</a> section for more information.</p> </div> </div> </div> <div class="sect1"> -<h2 id="how-to-install-and-start-nifi"><a class="anchor" href="administration-guide.html#how-to-install-and-start-nifi"></a>How to install and start NiFi</h2> +<h2 id="how-to-install-and-start-nifi"><a class="anchor" href="#how-to-install-and-start-nifi"></a>How to install and start NiFi</h2> <div class="sectionbody"> <div class="ulist"> <ul> @@ -610,7 +611,7 @@ body.book #toc,body.book #preamble,body. <div class="ulist"> <ul> <li> -<p>At a minimum, we recommend editing the <em>nifi.properties</em> file and entering a password for the nifi.sensitive.props.key (see <a href="administration-guide.html#system_properties">System Properties</a> below)</p> +<p>At a minimum, we recommend editing the <em>nifi.properties</em> file and entering a password for the nifi.sensitive.props.key (see <a href="#system_properties">System Properties</a> below)</p> </li> </ul> </div> @@ -665,7 +666,7 @@ body.book #toc,body.book #preamble,body. <div class="ulist"> <ul> <li> -<p>At a minimum, we recommend editing the <em>nifi.properties</em> file and entering a password for the nifi.sensitive.props.key (see <a href="administration-guide.html#system_properties">System Properties</a> below)</p> +<p>At a minimum, we recommend editing the <em>nifi.properties</em> file and entering a password for the nifi.sensitive.props.key (see <a href="#system_properties">System Properties</a> below)</p> </li> </ul> </div> @@ -677,12 +678,6 @@ body.book #toc,body.book #preamble,body. <p>Double-click run-nifi.bat. This runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi</p> </li> <li> -<p>Alternatively, to start NiFi in the background, double-click start-nifi.bat</p> -</li> -<li> -<p>To stop NiFi running in the background, double-click stop-nifi.bat</p> -</li> -<li> <p>To see the current status of NiFi, double-click status-nifi.bat</p> </li> </ul> @@ -719,12 +714,12 @@ body.book #toc,body.book #preamble,body. </ul> </div> <div class="paragraph"> -<p>See the <a href="administration-guide.html#system_properties">System Properties</a> section of this guide for more information about configuring NiFi repositories and configuration files.</p> +<p>See the <a href="#system_properties">System Properties</a> section of this guide for more information about configuring NiFi repositories and configuration files.</p> </div> </div> </div> <div class="sect1"> -<h2 id="configuration-best-practices"><a class="anchor" href="administration-guide.html#configuration-best-practices"></a>Configuration Best Practices</h2> +<h2 id="configuration-best-practices"><a class="anchor" href="#configuration-best-practices"></a>Configuration Best Practices</h2> <div class="sectionbody"> <div class="admonitionblock note"> <table> @@ -829,7 +824,7 @@ and for the partition(s) of interest add </div> </div> <div class="sect1"> -<h2 id="security-configuration"><a class="anchor" href="administration-guide.html#security-configuration"></a>Security Configuration</h2> +<h2 id="security-configuration"><a class="anchor" href="#security-configuration"></a>Security Configuration</h2> <div class="sectionbody"> <div class="paragraph"> <p>NiFi provides several different configuration options for security purposes. The most important properties are those under the @@ -862,7 +857,7 @@ and for the partition(s) of interest add </tr> <tr> <td class="tableblock halign-left valign-top"><p class="tableblock"><code>nifi.security.keystoreType</code></p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">The type of Keystore. Must be either <code>PKCS12</code> or <code>JKS</code>.</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">The type of Keystore. Must be either <code>PKCS12</code> or <code>JKS</code>. JKS is the preferred type, PKCS12 files will be loaded with BouncyCastle provider.</p></td> </tr> <tr> <td class="tableblock halign-left valign-top"><p class="tableblock"><code>nifi.security.keystorePasswd</code></p></td> @@ -879,7 +874,7 @@ and for the partition(s) of interest add </tr> <tr> <td class="tableblock halign-left valign-top"><p class="tableblock"><code>nifi.security.truststoreType</code></p></td> -<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Truststore. Must be either <code>PKCS12</code> or <code>JKS</code>.</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Truststore. Must be either <code>PKCS12</code> or <code>JKS</code>. JKS is the preferred type, PKCS12 files will be loaded with BouncyCastle provider.</p></td> </tr> <tr> <td class="tableblock halign-left valign-top"><p class="tableblock"><code>nifi.security.truststorePasswd</code></p></td> @@ -917,11 +912,14 @@ the web server will REQUIRE certificate accomplished by setting the <code>nifi.remote.input.secure</code> and <code>nifi.cluster.protocol.is.secure</code> properties, respectively, to <code>true</code>.</p> </div> <div class="sect2"> -<h3 id="tls-generation-toolkit"><a class="anchor" href="administration-guide.html#tls-generation-toolkit"></a>TLS Generation Toolkit</h3> +<h3 id="tls-generation-toolkit"><a class="anchor" href="#tls-generation-toolkit"></a>TLS Generation Toolkit</h3> <div class="paragraph"> <p>In order to facilitate the secure setup of NiFi, you can use the <code>tls-toolkit</code> command line utility to automatically generate the required keystores, truststore, and relevant configuration files. This is especially useful for securing multiple NiFi nodes, which can be a tedious and error-prone process.</p> </div> <div class="paragraph"> +<p>Note: JKS keystores and truststores are recommended for NiFi. This tool allows the specification of other keystore types on the command line but will ignore a type of PKCS12 for use as the truststore because that format has some compatibility issues between BouncyCastle and Oracle implementations.</p> +</div> +<div class="paragraph"> <p>The <code>tls-toolkit</code> command line tool has two primary modes of operation:</p> </div> <div class="olist arabic"> @@ -935,7 +933,7 @@ accomplished by setting the <code>nifi.r </ol> </div> <div class="sect3"> -<h4 id="standalone"><a class="anchor" href="administration-guide.html#standalone"></a>Standalone</h4> +<h4 id="standalone"><a class="anchor" href="#standalone"></a>Standalone</h4> <div class="paragraph"> <p>Standalone mode is invoked by running <code>./bin/tls-toolkit.sh standalone -h</code> which prints the usage information along with descriptions of options that can be specified.</p> </div> @@ -1000,12 +998,12 @@ accomplished by setting the <code>nifi.r </div> </div> <div class="sect3"> -<h4 id="client-server"><a class="anchor" href="administration-guide.html#client-server"></a>Client/Server</h4> +<h4 id="client-server"><a class="anchor" href="#client-server"></a>Client/Server</h4> <div class="paragraph"> <p>Client/Server mode relies on a long-running Certificate Authority (CA) to issue certificates. The CA can be stopped when youâre not bringing nodes online.</p> </div> <div class="sect4"> -<h5 id="server"><a class="anchor" href="administration-guide.html#server"></a>Server</h5> +<h5 id="server"><a class="anchor" href="#server"></a>Server</h5> <div class="paragraph"> <p>The CA server is invoked by running <code>./bin/tls-toolkit server -h</code> prints the usage information along with descriptions of options that can be specified.</p> </div> @@ -1030,7 +1028,7 @@ accomplished by setting the <code>nifi.r </div> </div> <div class="sect4"> -<h5 id="client"><a class="anchor" href="administration-guide.html#client"></a>Client</h5> +<h5 id="client"><a class="anchor" href="#client"></a>Client</h5> <div class="paragraph"> <p>The client can be used to request new Certificates from the CA. The client utility generates a keypair and Certificate Signing Request (CSR) and sends the CSR to the Certificate Authority. The client is invoked by running <code>./bin/tls-toolkit.sh client -h</code> which prints the usage information along with descriptions of options that can be specified.</p> </div> @@ -1068,7 +1066,7 @@ accomplished by setting the <code>nifi.r </div> </div> <div class="sect1"> -<h2 id="user-authentication"><a class="anchor" href="administration-guide.html#user-authentication"></a>User Authentication</h2> +<h2 id="user-authentication"><a class="anchor" href="#user-authentication"></a>User Authentication</h2> <div class="sectionbody"> <div class="paragraph"> <p>NiFi supports user authentication via client certificates or via username/password. Username/password authentication is performed by a <em>Login Identity @@ -1080,13 +1078,13 @@ to use is configured in two properties i The <code>nifi.security.user.login.identity.provider</code> property indicates which of the configured Login Identity Provider should be used. If this property is not configured, NiFi will not support username/password authentication and will require client certificates for authenticating users over HTTPS. By default, this property is not configured meaning that username/password must be -explicity enabled.</p> +explicitly enabled.</p> </div> <div class="paragraph"> <p>NiFi does not perform user authentication over HTTP. Using HTTP all users will be granted all roles.</p> </div> <div class="sect2"> -<h3 id="lightweight-directory-access-protocol-ldap"><a class="anchor" href="administration-guide.html#lightweight-directory-access-protocol-ldap"></a>Lightweight Directory Access Protocol (LDAP)</h3> +<h3 id="lightweight-directory-access-protocol-ldap"><a class="anchor" href="#lightweight-directory-access-protocol-ldap"></a>Lightweight Directory Access Protocol (LDAP)</h3> <div class="paragraph"> <p>Below is an example and description of configuring a Login Identity Provider that integrates with a Directory Server to authenticate users.</p> </div> @@ -1118,6 +1116,7 @@ explicity enabled.</p> <property name="User Search Base"></property> <property name="User Search Filter"></property> + <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider></pre> </div> @@ -1220,11 +1219,16 @@ explicity enabled.</p> <td class="tableblock halign-left valign-top"><p class="tableblock"><code>User Search Filter</code></p></td> <td class="tableblock halign-left valign-top"><p class="tableblock">Filter for searching for users against the <em>User Search Base</em>. (i.e. sAMAccountName={0}). The user specified name is inserted into <em>{0}</em>.</p></td> </tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Identity Strategy</code></p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Strategy to identify users. Possible values are USE_DN and USE_USERNAME. The default functionality if this property is missing is USE_DN in order to retain backward +compatibility. USE_DN will use the full DN of the user entry if possible. USE_USERNAME will use the username the user logged in with.</p></td> +</tr> </tbody> </table> </div> <div class="sect2"> -<h3 id="kerberos_login_identity_provider"><a class="anchor" href="administration-guide.html#kerberos_login_identity_provider"></a>Kerberos</h3> +<h3 id="kerberos_login_identity_provider"><a class="anchor" href="#kerberos_login_identity_provider"></a>Kerberos</h3> <div class="paragraph"> <p>Below is an example and description of configuring a Login Identity Provider that integrates with a Kerberos Key Distribution Center (KDC) to authenticate users.</p> </div> @@ -1276,13 +1280,13 @@ explicity enabled.</p> </tbody> </table> <div class="paragraph"> -<p>See also <a href="administration-guide.html#kerberos_service">Kerberos Service</a> to allow single sign-on access via client Kerberos tickets.</p> +<p>See also <a href="#kerberos_service">Kerberos Service</a> to allow single sign-on access via client Kerberos tickets.</p> </div> </div> </div> </div> <div class="sect1"> -<h2 id="multi-tenant-authorization"><a class="anchor" href="administration-guide.html#multi-tenant-authorization"></a>Multi-Tenant Authorization</h2> +<h2 id="multi-tenant-authorization"><a class="anchor" href="#multi-tenant-authorization"></a>Multi-Tenant Authorization</h2> <div class="sectionbody"> <div class="paragraph"> <p>After you have configured NiFi to run securely and with an authentication mechanism, you must configure who has access to the system, and the level of their access. @@ -1291,7 +1295,7 @@ parts of the dataflow, with varying leve user has privileges to perform that action. These privileges are defined by policies that you can apply system-wide or to individual components.</p> </div> <div class="sect2"> -<h3 id="authorizer-configuration"><a class="anchor" href="administration-guide.html#authorizer-configuration"></a>Authorizer Configuration</h3> +<h3 id="authorizer-configuration"><a class="anchor" href="#authorizer-configuration"></a>Authorizer Configuration</h3> <div class="paragraph"> <p>An <em>authorizer</em> grants users the privileges to manage users and policies by creating preliminary authorizations at startup.</p> </div> @@ -1310,7 +1314,7 @@ user has privileges to perform that acti </div> </div> <div class="sect2"> -<h3 id="authorizers-setup"><a class="anchor" href="administration-guide.html#authorizers-setup"></a>Authorizers.xml Setup</h3> +<h3 id="authorizers-setup"><a class="anchor" href="#authorizers-setup"></a>Authorizers.xml Setup</h3> <div class="paragraph"> <p>The <em>authorizers.xml</em> file is used to define and configure available authorizers. The default authorizer is the FileAuthorizer, however, you can develop additional authorizers as extensions. The FileAuthorizer has the following properties:</p> </div> @@ -1334,7 +1338,7 @@ user has privileges to perform that acti </ul> </div> <div class="sect3"> -<h4 id="initial-admin-identity"><a class="anchor" href="administration-guide.html#initial-admin-identity"></a>Initial Admin Identity (New NiFi Instance)</h4> +<h4 id="initial-admin-identity"><a class="anchor" href="#initial-admin-identity"></a>Initial Admin Identity (New NiFi Instance)</h4> <div class="paragraph"> <p>If you are setting up a secured NiFi instance for the first time, you must manually designate an âInitial Admin Identityâ in the <em>authorizers.xml</em> file. This initial admin user is granted access to the UI and given the ability to create additional users, groups, and policies. The value of this property could be a DN (when using certificates or LDAP) or a Kerberos principal. If you are the NiFi administrator, add yourself as the âInitial Admin Identityâ.</p> </div> @@ -1359,7 +1363,7 @@ user has privileges to perform that acti </div> </div> <div class="paragraph"> -<p>Here is a example Kerberos entry using the name John Smith and realm <code>NIFI.APACHE.ORG</code>:</p> +<p>Here is an example Kerberos entry using the name John Smith and realm <code>NIFI.APACHE.ORG</code>:</p> </div> <div class="listingblock"> <div class="content"> @@ -1379,7 +1383,7 @@ user has privileges to perform that acti </div> </div> <div class="paragraph"> -<p>After you have edited and saved the <em>authorizers.xml</em> file, restart NiFi. The âInitial Admin Identityâ user and administrative policies are added to the <em>authorizations.xml</em> file during restart. Once NiFi starts, the âInitial Admin Identityâ user is able to access the UI and begin managing users, groups, and policies.</p> +<p>After you have edited and saved the <em>authorizers.xml</em> file, restart NiFi. The âInitial Admin Identityâ user and administrative policies are added to the <em>users.xml</em> and <em>authorizations.xml</em> files during restart. Once NiFi starts, the âInitial Admin Identityâ user is able to access the UI and begin managing users, groups, and policies.</p> </div> <div class="admonitionblock note"> <table> @@ -1395,7 +1399,7 @@ For a brand new secure flow, providing t </div> </div> <div class="sect3"> -<h4 id="legacy-authorized-users"><a class="anchor" href="administration-guide.html#legacy-authorized-users"></a>Legacy Authorized Users (NiFi Instance Upgrade)</h4> +<h4 id="legacy-authorized-users"><a class="anchor" href="#legacy-authorized-users"></a>Legacy Authorized Users (NiFi Instance Upgrade)</h4> <div class="paragraph"> <p>If you are upgrading from a 0.x NiFi instance, you can convert your previously configured users and roles to the multi-tenant authorization model. In the <em>authorizers.xml</em> file, specify the location of your existing <em>authorized-users.xml</em> file in the âLegacy Authorized Users Fileâ property.</p> </div> @@ -1417,7 +1421,7 @@ For a brand new secure flow, providing t </div> </div> <div class="paragraph"> -<p>After you have edited and saved the <em>authorizers.xml</em> file, restart NiFi. Users and roles from the <em>authorized-users.xml</em> file are converted and added as identities and policies in the <em>authorizations.xml</em> file. Once the application starts, users who previously had a legacy Administrator role can access the UI and begin managing users, groups, and policies.</p> +<p>After you have edited and saved the <em>authorizers.xml</em> file, restart NiFi. Users and roles from the <em>authorized-users.xml</em> file are converted and added as identities and policies in the <em>users.xml</em> and <em>authorizations.xml</em> files. Once the application starts, users who previously had a legacy Administrator role can access the UI and begin managing users, groups, and policies.</p> </div> <div class="paragraph"> <p>Here is a summary of policies assigned to each legacy role if the NiFi instance has an existing flow.xml.gz:</p> @@ -1544,6 +1548,15 @@ For a brand new secure flow, providing t <td class="tableblock halign-center valign-top"></td> </tr> <tr> +<td class="tableblock halign-right valign-top"><p class="tableblock"><strong>access restricted components</strong></p></td> +<td class="tableblock halign-center valign-top"></td> +<td class="tableblock halign-center valign-top"><p class="tableblock"><strong>*</strong></p></td> +<td class="tableblock halign-center valign-top"></td> +<td class="tableblock halign-center valign-top"></td> +<td class="tableblock halign-center valign-top"></td> +<td class="tableblock halign-center valign-top"></td> +</tr> +<tr> <td class="tableblock halign-right valign-top"><p class="tableblock"><strong>view the data</strong></p></td> <td class="tableblock halign-center valign-top"></td> <td class="tableblock halign-center valign-top"><p class="tableblock"><strong>*</strong></p></td> @@ -1582,7 +1595,7 @@ For a brand new secure flow, providing t </tbody> </table> <div class="paragraph"> -<p>For details on the policies in the table, see <a href="administration-guide.html#access-policies">Access Policies</a>.</p> +<p>For details on the policies in the table, see <a href="#access-policies">Access Policies</a>.</p> </div> <div class="admonitionblock note"> <table> @@ -1610,7 +1623,7 @@ Do not manually edit the <em>authorizati </div> </div> <div class="sect3"> -<h4 id="cluster-node-identities"><a class="anchor" href="administration-guide.html#cluster-node-identities"></a>Cluster Node Identities</h4> +<h4 id="cluster-node-identities"><a class="anchor" href="#cluster-node-identities"></a>Cluster Node Identities</h4> <div class="paragraph"> <p>If you are running NiFi in a clustered environment, you must specify the identities for each node. The authorization policies required for the nodes to communicate are created during startup.</p> </div> @@ -1656,7 +1669,7 @@ In a cluster, all nodes must have the sa </div> </div> <div class="sect2"> -<h3 id="config-users-access-policies"><a class="anchor" href="administration-guide.html#config-users-access-policies"></a>Configuring Users & Access Policies</h3> +<h3 id="config-users-access-policies"><a class="anchor" href="#config-users-access-policies"></a>Configuring Users & Access Policies</h3> <div class="paragraph"> <p>This section describes:</p> </div> @@ -1680,34 +1693,34 @@ In a cluster, all nodes must have the sa <i class="fa icon-note" title="Note"></i> </td> <td class="content"> -Instructions requiring interaction with the UI assume the application is being accessed by User1, a user with administrator privileges, such as the âInitial Admin Identityâ user or a converted legacy admin user (see <a href="administration-guide.html#authorizers-setup">Authorizers.xml Setup</a>). +Instructions requiring interaction with the UI assume the application is being accessed by User1, a user with administrator privileges, such as the âInitial Admin Identityâ user or a converted legacy admin user (see <a href="#authorizers-setup">Authorizers.xml Setup</a>). </td> </tr> </table> </div> <div class="sect3"> -<h4 id="creating-users-groups"><a class="anchor" href="administration-guide.html#creating-users-groups"></a>Creating Users and Groups</h4> +<h4 id="creating-users-groups"><a class="anchor" href="#creating-users-groups"></a>Creating Users and Groups</h4> <div class="paragraph"> <p>From the UI, select âUsersâ from the Global Menu. This opens a dialog to create and manage users and groups.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/nifi-users-dialog.png" alt="NiFi Users Dialog"></span></p> +<p><span class="image"><img src="./images/nifi-users-dialog.png" alt="NiFi Users Dialog"></span></p> </div> <div class="paragraph"> -<p>Click the Add icon (<span class="image"><img src="images/iconAddUser.png" alt="Add User Icon"></span>). To create a user, enter the <em>Identity</em> information relevant to the authentication method chosen to secure your NiFi instance. Click OK.</p> +<p>Click the Add icon (<span class="image"><img src="./images/iconAddUser.png" alt="Add User Icon"></span>). To create a user, enter the <em>Identity</em> information relevant to the authentication method chosen to secure your NiFi instance. Click OK.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user-creation-dialog.png" alt="User Creation Dialog"></span></p> +<p><span class="image"><img src="./images/user-creation-dialog.png" alt="User Creation Dialog"></span></p> </div> <div class="paragraph"> <p>To create a group, select the âGroupâ radio button, enter the name of the group and select the users to be included in the group. Click OK.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/group-creation-dialog.png" alt="Group Creation Dialog"></span></p> +<p><span class="image"><img src="./images/group-creation-dialog.png" alt="Group Creation Dialog"></span></p> </div> </div> <div class="sect3"> -<h4 id="access-policies"><a class="anchor" href="administration-guide.html#access-policies"></a>Access Policies</h4> +<h4 id="access-policies"><a class="anchor" href="#access-policies"></a>Access Policies</h4> <div class="paragraph"> <p>You can manage the ability for users and groups to view or modify NiFi resources using <em>access policies</em>. There are two types of access policies that can be applied to a resource:</p> </div> @@ -1725,7 +1738,7 @@ Instructions requiring interaction with <p>You can create and apply access policies on both global and component levels.</p> </div> <div class="sect4"> -<h5 id="global-access-policies"><a class="anchor" href="administration-guide.html#global-access-policies"></a>Global Access Policies</h5> +<h5 id="global-access-policies"><a class="anchor" href="#global-access-policies"></a>Global Access Policies</h5> <div class="paragraph"> <p>Global access policies govern the following system level authorizations:</p> </div> @@ -1759,6 +1772,11 @@ Instructions requiring interaction with <td class="tableblock halign-left valign-top"><p class="tableblock">Data Provenance</p></td> </tr> <tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">access restricted components</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to create/modify restricted components assuming otherwise sufficient permissions</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">N/A</p></td> +</tr> +<tr> <td class="tableblock halign-left valign-top"><p class="tableblock">access all policies</p></td> <td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view/modify the policies for all components</p></td> <td class="tableblock halign-left valign-top"><p class="tableblock">Policies</p></td> @@ -1792,7 +1810,7 @@ Instructions requiring interaction with </table> </div> <div class="sect4"> -<h5 id="component-level-access-policies"><a class="anchor" href="administration-guide.html#component-level-access-policies"></a>Component Level Access Policies</h5> +<h5 id="component-level-access-policies"><a class="anchor" href="#component-level-access-policies"></a>Component Level Access Policies</h5> <div class="paragraph"> <p>Component level access policies govern the following component level authorizations:</p> </div> @@ -1849,19 +1867,19 @@ Instructions requiring interaction with <i class="fa icon-note" title="Note"></i> </td> <td class="content"> -You can apply access policies to all component types except connections. Connection authorizations are inferred by the individual access policies on the source and destination components of the connection, as well as the access policy of the process group containing the components. This is discussed in more detail in the <a href="administration-guide.html#creating-a-connection">Creating a Connection</a> and <a href="administration-guide.html#editing-a-connection">Editing a Connection</a> examples below. +You can apply access policies to all component types except connections. Connection authorizations are inferred by the individual access policies on the source and destination components of the connection, as well as the access policy of the process group containing the components. This is discussed in more detail in the <a href="#creating-a-connection">Creating a Connection</a> and <a href="#editing-a-connection">Editing a Connection</a> examples below. </td> </tr> </table> </div> </div> <div class="sect4"> -<h5 id="access-policy-inheritance"><a class="anchor" href="administration-guide.html#access-policy-inheritance"></a>Access Policy Inheritance</h5> +<h5 id="access-policy-inheritance"><a class="anchor" href="#access-policy-inheritance"></a>Access Policy Inheritance</h5> <div class="paragraph"> <p>An administrator does not need to manually create policies for every component in the dataflow. To reduce the amount of time admins spend on authorization management, policies are inherited from parent resource to child resource. For example, if a user is given access to view and modify a process group, that user can also view and modify the components in the process group. Policy inheritance enables an administrator to assign policies at one time and have the policies apply throughout the entire dataflow.</p> </div> <div class="paragraph"> -<p>You can override an inherited policy (as described in the <a href="administration-guide.html#moving-a-processor">Moving a Processor</a> example below). Overriding a policy removes the inherited policy, breaking the chain of inheritance from parent to child, and creates a replacement policy to add users as desired. Inherited policies and their users can be restored by deleting the replacement policy.</p> +<p>You can override an inherited policy (as described in the <a href="#moving-a-processor">Moving a Processor</a> example below). Overriding a policy removes the inherited policy, breaking the chain of inheritance from parent to child, and creates a replacement policy to add users as desired. Inherited policies and their users can be restored by deleting the replacement policy.</p> </div> <div class="admonitionblock note"> <table> @@ -1875,10 +1893,22 @@ You can apply access policies to all com </tr> </table> </div> +<div class="admonitionblock note"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-note" title="Note"></i> +</td> +<td class="content"> +You cannot modify the users/groups on an inherited policy. Users and groups can only be added or removed from a parent policy or an override policy. +</td> +</tr> +</table> +</div> </div> </div> <div class="sect3"> -<h4 id="access-policy-config-examples"><a class="anchor" href="administration-guide.html#access-policy-config-examples"></a>Access Policy Configuration Examples</h4> +<h4 id="access-policy-config-examples"><a class="anchor" href="#access-policy-config-examples"></a>Access Policy Configuration Examples</h4> <div class="paragraph"> <p>The most effective way to understand how to create and apply access policies is to walk through some common examples. The following scenarios assume User1 is an administrator and User2 is a newly added user that has only been given access to the UI.</p> </div> @@ -1886,13 +1916,13 @@ You can apply access policies to all com <p>Letâs begin with two processors on the canvas as our starting point: GenerateFlowFile and LogAttribute.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/access-policy-config-start.png" alt="Access Policy Config Start"></span></p> +<p><span class="image"><img src="./images/access-policy-config-start.png" alt="Access Policy Config Start"></span></p> </div> <div class="paragraph"> <p>User1 can add components to the dataflow and is able to move, edit and connect all processors. The details and properties of the root process group and processors are visible to User1.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user1-full-access.png" alt="User1 Full Access"></span></p> +<p><span class="image"><img src="./images/user1-full-access.png" alt="User1 Full Access"></span></p> </div> <div class="paragraph"> <p>User1 wants to maintain their current privileges to the dataflow and its components.</p> @@ -1901,10 +1931,10 @@ You can apply access policies to all com <p>User2 is unable to add components to the dataflow or move, edit, or connect components. The details and properties of the root process group and processors are hidden from User2.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-restricted-access.png" alt="User2 Restricted Access"></span></p> +<p><span class="image"><img src="./images/user2-restricted-access.png" alt="User2 Restricted Access"></span></p> </div> <div class="sect4"> -<h5 id="moving-a-processor"><a class="anchor" href="administration-guide.html#moving-a-processor"></a>Moving a Processor</h5> +<h5 id="moving-a-processor"><a class="anchor" href="#moving-a-processor"></a>Moving a Processor</h5> <div class="paragraph"> <p>To allow User2 to move the GenerateFlowFile processor in the dataflow and only that processor, User1 performs the following steps:</p> </div> @@ -1914,33 +1944,43 @@ You can apply access policies to all com <p>Select the GenerateFlowFile processor so that it is highlighted.</p> </li> <li> -<p>Select the Access Policies icon (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> +<p>Select the Access Policies icon (<span class="image"><img src="./images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> </li> <li> <p>Select âmodify the componentâ from the policy drop-down. -<span class="image"><img src="images/processor-modify-policy.png" alt="Processor Modify Policy"></span> +<span class="image"><img src="./images/processor-modify-policy.png" alt="Processor Modify Policy"></span> The âmodify the componentâ policy that currently exists on the processor (child) is the âmodify the componentâ policy inherited from the root process group (parent) on which User1 has privileges.</p> </li> <li> -<p>Select the Override link in the policy inheritance message to create a replacement policy.</p> +<p>Select the Override link in the policy inheritance message. When creating the replacement policy, you are given a choice to override with a copy of the inherited policy or an empty policy.</p> </li> +</ol> +</div> +<div class="paragraph"> +<p><span class="image"><img src="./images/override_policy_copy_empty.png" alt="Create Override Policy"></span></p> +</div> +<div class="paragraph"> +<p>Select the Override button to create a copy.</p> +</div> +<div class="olist arabic"> +<ol class="arabic" start="5"> <li> -<p>On the replacement policy that is created, select the Add User icon (<span class="image"><img src="images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User1 in the User Identity field and select OK. Select the Add User icon again, find or enter User2 and select OK.</p> +<p>On the replacement policy that is created, select the Add User icon (<span class="image"><img src="./images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 in the User Identity field and select OK.</p> </li> </ol> </div> <div class="paragraph"> -<p><span class="image"><img src="images/processor-replacement-modify-policy.png" alt="Processor Replacement Modify Policy"></span></p> +<p><span class="image"><img src="./images/processor-replacement-modify-policy.png" alt="Processor Replacement Modify Policy"></span></p> </div> <div class="paragraph"> <p>With these changes, User1 maintains the ability to move both processors on the canvas. User2 can now move the GenerateFlowFile processor but cannot move the LogAttribute processor.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-moved-processor.png" alt="User2 Moved Processor"></span></p> +<p><span class="image"><img src="./images/user2-moved-processor.png" alt="User2 Moved Processor"></span></p> </div> </div> <div class="sect4"> -<h5 id="editing-a-processor"><a class="anchor" href="administration-guide.html#editing-a-processor"></a>Editing a Processor</h5> +<h5 id="editing-a-processor"><a class="anchor" href="#editing-a-processor"></a>Editing a Processor</h5> <div class="paragraph"> <p>In the âMoving a Processorâ example above, User2 was added to the âmodify the componentâ policy for GenerateFlowFile. Without the ability to view the processor properties, User2 is unable to modify the processorâs configuration. In order to edit a component, a user must be on both the âview the componentâ and âmodify the componentâ policies. To implement this, User1 performs the following steps:</p> </div> @@ -1950,44 +1990,44 @@ The âmodify the componentâ pol <p>Select the GenerateFlowFile processor.</p> </li> <li> -<p>Select the Access Policies icon (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> +<p>Select the Access Policies icon (<span class="image"><img src="./images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> </li> <li> <p>Select "view the componentâ from the policy drop-down. -<span class="image"><img src="images/processor-view-policy.png" alt="Processor View Policy"></span> +<span class="image"><img src="./images/processor-view-policy.png" alt="Processor View Policy"></span> The view the componentâ policy that currently exists on the processor (child) is the "view the componentâ policy inherited from the root process group (parent) on which User1 has privileges.</p> </li> <li> -<p>Select the Override link in the policy inheritance message to create a replacement policy.</p> +<p>Select the Override link in the policy inheritance message, keep the default of Copy policy and select the Override button.</p> </li> <li> -<p>On the replacement policy that is created, select the Add User icon (<span class="image"><img src="images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User1 in the User Identity field and select OK. Select the Add User icon again, find or enter User2 and select OK.</p> +<p>On the override policy that is created, select the Add User icon (<span class="image"><img src="./images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 in the User Identity field and select OK.</p> </li> </ol> </div> <div class="paragraph"> -<p><span class="image"><img src="images/processor-replacement-view-policy.png" alt="Processor Replacement View Policy"></span></p> +<p><span class="image"><img src="./images/processor-replacement-view-policy.png" alt="Processor Replacement View Policy"></span></p> </div> <div class="paragraph"> <p>With these changes, User1 maintains the ability to view and edit the processors on the canvas. User2 can now view and edit the GenerateFlowFile processor.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-edit-processor.png" alt="User2 Edit Processor"></span></p> +<p><span class="image"><img src="./images/user2-edit-processor.png" alt="User2 Edit Processor"></span></p> </div> </div> <div class="sect4"> -<h5 id="creating-a-connection"><a class="anchor" href="administration-guide.html#creating-a-connection"></a>Creating a Connection</h5> +<h5 id="creating-a-connection"><a class="anchor" href="#creating-a-connection"></a>Creating a Connection</h5> <div class="paragraph"> <p>With the access policies configured as discussed in the previous two examples, User1 is able to connect GenerateFlowFile to LogAttribute:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user1-create-connection.png" alt="User1 Create Connection"></span></p> +<p><span class="image"><img src="./images/user1-create-connection.png" alt="User1 Create Connection"></span></p> </div> <div class="paragraph"> <p>User2 cannot make the connection:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-no-connection.png" alt="User2 No Connection"></span></p> +<p><span class="image"><img src="./images/user2-no-connection.png" alt="User2 No Connection"></span></p> </div> <div class="paragraph"> <p>This is because:</p> @@ -1995,10 +2035,10 @@ The view the componentâ policy that <div class="ulist"> <ul> <li> -<p>User2 does not have modify access on the process group and is therefore not able to create a connection.</p> +<p>User2 does not have modify access on the process group.</p> </li> <li> -<p>Even though User2 has view and modify access to the source component (GenerateFlowFile), User2 does not have any access policy on the destination component (LogAttribute).</p> +<p>Even though User2 has view and modify access to the source component (GenerateFlowFile), User2 does not have an access policy on the destination component (LogAttribute).</p> </li> </ul> </div> @@ -2011,56 +2051,55 @@ The view the componentâ policy that <p>Select the root process group. The Operate palette is updated with details for the root process group.</p> </li> <li> -<p>Select the Access Policies icon (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> +<p>Select the Access Policies icon (<span class="image"><img src="./images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette and the Access Policies dialog opens.</p> </li> <li> <p>Select "modify the componentâ from the policy drop-down. -<span class="image"><img src="images/process-group-modify-policy.png" alt="Process Group Modify Policy"></span> -[start=4]</p> +<span class="image"><img src="./images/process-group-modify-policy.png" alt="Process Group Modify Policy"></span></p> </li> <li> -<p>Select the Add User icon (<span class="image"><img src="images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 and select OK.</p> +<p>Select the Add User icon (<span class="image"><img src="./images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 and select OK.</p> </li> </ol> </div> <div class="paragraph"> -<p><span class="image"><img src="images/process-group-modify-policy-add-user2.png" alt="Process Group Modify Policy Add User2"></span></p> +<p><span class="image"><img src="./images/process-group-modify-policy-add-user2.png" alt="Process Group Modify Policy Add User2"></span></p> </div> <div class="paragraph"> -<p>By adding User2 to the âmodify the componentâ policy on the process group, User2 is added to the âmodify the componentâ policy on the LogAttribute processor by policy inheritance. To confirm this, highlight the LogAttribute processor and select the Access Policies icon (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette:</p> +<p>By adding User2 to the âmodify the componentâ policy on the process group, User2 is added to the âmodify the componentâ policy on the LogAttribute processor by policy inheritance. To confirm this, highlight the LogAttribute processor and select the Access Policies icon (<span class="image"><img src="./images/iconAccessPolicies.png" alt="Access Policies Icon"></span>) from the Operate palette:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/processor-inherited-modify-policy.png" alt="User2 Inherited Edit Processor"></span></p> +<p><span class="image"><img src="./images/processor-inherited-modify-policy.png" alt="User2 Inherited Edit Processor"></span></p> </div> <div class="paragraph"> <p>With these changes, User2 can now connect the GenerateFlowFile processor to the LogAttribute processor.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-can-connect.png" alt="User2 Can Connect"></span></p> +<p><span class="image"><img src="./images/user2-can-connect.png" alt="User2 Can Connect"></span></p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-connected-processors.png" alt="User2 Connected Processors"></span></p> +<p><span class="image"><img src="./images/user2-connected-processors.png" alt="User2 Connected Processors"></span></p> </div> </div> <div class="sect4"> -<h5 id="editing-a-connection"><a class="anchor" href="administration-guide.html#editing-a-connection"></a>Editing a Connection</h5> +<h5 id="editing-a-connection"><a class="anchor" href="#editing-a-connection"></a>Editing a Connection</h5> <div class="paragraph"> <p>Assume User1 or User2 adds a ReplaceText processor to the root process group:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/replacetext-processor-added.png" alt="ReplaceText Processor Added"></span></p> +<p><span class="image"><img src="./images/replacetext-processor-added.png" alt="ReplaceText Processor Added"></span></p> </div> <div class="paragraph"> <p>User1 can select and change the existing connection (between GenerateFlowFile to LogAttribute) to now connect GenerateFlowFile to ReplaceText:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user1-edit-connection.png" alt="User1 Edit Connection"></span></p> +<p><span class="image"><img src="./images/user1-edit-connection.png" alt="User1 Edit Connection"></span></p> </div> <div class="paragraph"> <p>User 2 is unable to perform this action.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-no-edit-connection.png" alt="User2 No Edit Connection"></span></p> +<p><span class="image"><img src="./images/user2-no-edit-connection.png" alt="User2 No Edit Connection"></span></p> </div> <div class="paragraph"> <p>To allow User2 to connect GenerateFlowFile to ReplaceText, as User1:</p> @@ -2071,26 +2110,25 @@ The view the componentâ policy that <p>Select the root process group. The Operate palette is updated with details for the root process group.</p> </li> <li> -<p>Select the Access Policies icon (<span class="image"><img src="images/iconAccessPolicies.png" alt="Access Policies Icon"></span>).</p> +<p>Select the Access Policies icon (<span class="image"><img src="./images/iconAccessPolicies.png" alt="Access Policies Icon"></span>).</p> </li> <li> <p>Select "view the componentâ from the policy drop-down. -<span class="image"><img src="images/process-group-view-policy.png" alt="Process Group View Policy"></span> -[start=4]</p> +<span class="image"><img src="./images/process-group-view-policy.png" alt="Process Group View Policy"></span></p> </li> <li> -<p>Select the Add User icon (<span class="image"><img src="images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 and select OK.</p> +<p>Select the Add User icon (<span class="image"><img src="./images/iconAddUser.png" alt="Add User Icon"></span>). Find or enter User2 and select OK.</p> </li> </ol> </div> <div class="paragraph"> -<p><span class="image"><img src="images/process-group-view-policy-add-user2.png" alt="Process Group View Policy Add User2"></span></p> +<p><span class="image"><img src="./images/process-group-view-policy-add-user2.png" alt="Process Group View Policy Add User2"></span></p> </div> <div class="paragraph"> <p>Being added to both the view and modify policies for the process group, User2 can now connect the GenerateFlowFile processor to the ReplaceText processor.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/user2-edit-connection.png" alt="User2 Edit Connection"></span></p> +<p><span class="image"><img src="./images/user2-edit-connection.png" alt="User2 Edit Connection"></span></p> </div> </div> </div> @@ -2098,7 +2136,7 @@ The view the componentâ policy that </div> </div> <div class="sect1"> -<h2 id="encryption"><a class="anchor" href="administration-guide.html#encryption"></a>Encryption Configuration</h2> +<h2 id="encryption"><a class="anchor" href="#encryption"></a>Encryption Configuration</h2> <div class="sectionbody"> <div class="paragraph"> <p>This section provides an overview of the capabilities of NiFi to encrypt and decrypt data.</p> @@ -2107,7 +2145,7 @@ The view the componentâ policy that <p>The <code>EncryptContent</code> processor allows for the encryption and decryption of data, both internal to NiFi and integrated with external systems, such as <code>openssl</code> and other data sources and consumers.</p> </div> <div class="sect2"> -<h3 id="key-derivation-functions"><a class="anchor" href="administration-guide.html#key-derivation-functions"></a>Key Derivation Functions</h3> +<h3 id="key-derivation-functions"><a class="anchor" href="#key-derivation-functions"></a>Key Derivation Functions</h3> <div class="paragraph"> <p>Key Derivation Functions (KDF) are mechanisms by which human-readable information, usually a password or other secret information, is translated into a cryptographic key suitable for data protection. For further information, read <a href="https://en.wikipedia.org/wiki/Key_derivation_function";>the Wikipedia entry on Key Derivation Functions</a>. Currently, KDFs are ingested by <code>CipherProvider</code> implementations and return a fully-initialized <code>Cipher</code> object to be used for encryption or decryption. Due to the use of a <code>CipherProviderFactory</code>, the KDFs are not customizable at this time. Future enhancements will include the ability to provide custom cost parameters to the KDF at initialization time. As a work-around, <code>CipherProvider</code> instances can be initialized with custom cost parameters in the constructor but this is not currently supported by the <code>CipherProviderFactory</code>. @@ -2253,7 +2291,7 @@ Here are the KDFs currently supported by </ul> </div> <div class="sect3"> -<h4 id="additional-resources"><a class="anchor" href="administration-guide.html#additional-resources"></a>Additional Resources</h4> +<h4 id="additional-resources"><a class="anchor" href="#additional-resources"></a>Additional Resources</h4> <div class="ulist"> <ul> <li> @@ -2297,49 +2335,49 @@ Here are the KDFs currently supported by </div> </div> <div class="sect2"> -<h3 id="salt-and-iv-encoding"><a class="anchor" href="administration-guide.html#salt-and-iv-encoding"></a>Salt and IV Encoding</h3> +<h3 id="salt-and-iv-encoding"><a class="anchor" href="#salt-and-iv-encoding"></a>Salt and IV Encoding</h3> <div class="paragraph"> -<p>Initially, the <code>EncryptContent</code> processor had a single method of deriving the encryption key from a user-provided password. This is now referred to as <code>NiFiLegacy</code> mode, effectively <code>MD5 digest, 1000 iterations</code>. In v0.4.0, another method of deriving the key, <code>OpenSSL PKCS#5 v1.5 EVP_BytesToKey</code> was added for compatibility with content encrypted outside of NiFi using the <code>openssl</code> command-line tool. Both of these <a href="administration-guide.html#key-derivation-functions">Key Derivation Functions</a> (KDF) had hard-coded digest functions and iteration counts, and the salt format was also hard-coded. With v0.5.0, additional KDFs are introduced with variable iteration counts, work factors, and salt formats. In addition, <em>raw keyed encryption</em> was also introduced. This required the capacity to encode arbitrary salts and Initialization Vectors (IV) into the cipher stream in order to be recovered by NiFi or a follow-on sys tem to decrypt these messages.</p> +<p>Initially, the <code>EncryptContent</code> processor had a single method of deriving the encryption key from a user-provided password. This is now referred to as <code>NiFiLegacy</code> mode, effectively <code>MD5 digest, 1000 iterations</code>. In v0.4.0, another method of deriving the key, <code>OpenSSL PKCS#5 v1.5 EVP_BytesToKey</code> was added for compatibility with content encrypted outside of NiFi using the <code>openssl</code> command-line tool. Both of these <a href="#key-derivation-functions">Key Derivation Functions</a> (KDF) had hard-coded digest functions and iteration counts, and the salt format was also hard-coded. With v0.5.0, additional KDFs are introduced with variable iteration counts, work factors, and salt formats. In addition, <em>raw keyed encryption</em> was also introduced. This required the capacity to encode arbitrary salts and Initialization Vectors (IV) into the cipher stream in order to be recovered by NiFi or a follow-on system to decrypt these mess ages.</p> </div> <div class="paragraph"> <p>For the existing KDFs, the salt format has not changed.</p> </div> <div class="sect3"> -<h4 id="nifi-legacy"><a class="anchor" href="administration-guide.html#nifi-legacy"></a>NiFi Legacy</h4> +<h4 id="nifi-legacy"><a class="anchor" href="#nifi-legacy"></a>NiFi Legacy</h4> <div class="paragraph"> <p>The first 8 or 16 bytes of the input are the salt. The salt length is determined based on the selected algorithm’s cipher block length. If the cipher block size cannot be determined (such as with a stream cipher like <code>RC4</code>), the default value of 8 bytes is used. On decryption, the salt is read in and combined with the password to derive the encryption key and IV.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/nifi-legacy-salt.png" alt="NiFi Legacy Salt Encoding"></span></p> +<p><span class="image"><img src="./images/nifi-legacy-salt.png" alt="NiFi Legacy Salt Encoding"></span></p> </div> </div> <div class="sect3"> -<h4 id="openssl-pkcs-5-v1-5-evp_bytestokey"><a class="anchor" href="administration-guide.html#openssl-pkcs-5-v1-5-evp_bytestokey"></a>OpenSSL PKCS#5 v1.5 EVP_BytesToKey</h4> +<h4 id="openssl-pkcs-5-v1-5-evp_bytestokey"><a class="anchor" href="#openssl-pkcs-5-v1-5-evp_bytestokey"></a>OpenSSL PKCS#5 v1.5 EVP_BytesToKey</h4> <div class="paragraph"> <p>OpenSSL allows for salted or unsalted key derivation. <em>*Unsalted key derivation is a security risk and is not recommended.*</em> If a salt is present, the first 8 bytes of the input are the ASCII string "<code>Salted__</code>" (<code>0x53 61 6C 74 65 64 5F 5F</code>) and the next 8 bytes are the ASCII-encoded salt. On decryption, the salt is read in and combined with the password to derive the encryption key and IV. If there is no salt header, the entire input is considered to be the cipher text.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/openssl-salt.png" alt="OpenSSL Salt Encoding"></span></p> +<p><span class="image"><img src="./images/openssl-salt.png" alt="OpenSSL Salt Encoding"></span></p> </div> <div class="paragraph"> <p>For new KDFs, each of which allow for non-deterministic IVs, the IV must be stored alongside the cipher text. This is not a vulnerability, as the IV is not required to be secret, but simply to be unique for messages encrypted using the same key to reduce the success of cryptographic attacks. For these KDFs, the output consists of the salt, followed by the salt delimiter, UTF-8 string "<code>NiFiSALT</code>" (<code>0x4E 69 46 69 53 41 4C 54</code>) and then the IV, followed by the IV delimiter, UTF-8 string "<code>NiFiIV</code>" (<code>0x4E 69 46 69 49 56</code>), followed by the cipher text.</p> </div> </div> <div class="sect3"> -<h4 id="bcrypt-scrypt-pbkdf2"><a class="anchor" href="administration-guide.html#bcrypt-scrypt-pbkdf2"></a>Bcrypt, Scrypt, PBKDF2</h4> +<h4 id="bcrypt-scrypt-pbkdf2"><a class="anchor" href="#bcrypt-scrypt-pbkdf2"></a>Bcrypt, Scrypt, PBKDF2</h4> <div class="paragraph"> -<p><span class="image"><img src="images/bcrypt-salt.png" alt="Bcrypt Salt & IV Encoding"></span></p> +<p><span class="image"><img src="./images/bcrypt-salt.png" alt="Bcrypt Salt & IV Encoding"></span></p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/scrypt-salt.png" alt="Scrypt Salt & IV Encoding"></span></p> +<p><span class="image"><img src="./images/scrypt-salt.png" alt="Scrypt Salt & IV Encoding"></span></p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/pbkdf2-salt.png" alt="PBKDF2 Salt & IV Encoding"></span></p> +<p><span class="image"><img src="./images/pbkdf2-salt.png" alt="PBKDF2 Salt & IV Encoding"></span></p> </div> </div> </div> <div class="sect2"> -<h3 id="java-cryptography-extension-jce-limited-strength-jurisdiction-policies"><a class="anchor" href="administration-guide.html#java-cryptography-extension-jce-limited-strength-jurisdiction-policies"></a>Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies</h3> +<h3 id="java-cryptography-extension-jce-limited-strength-jurisdiction-policies"><a class="anchor" href="#java-cryptography-extension-jce-limited-strength-jurisdiction-policies"></a>Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies</h3> <div class="paragraph"> <p>Because of US export regulations, default JVMs have <a href="http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#importlimits";>limits imposed on the strength of cryptographic operations</a> available to them. For example, AES operations are limited to <code>128 bit keys</code> by default. While <code>AES-128</code> is cryptographically safe, this can have unintended consequences, specifically on Password-based Encryption (PBE).</p> </div> @@ -2446,12 +2484,12 @@ Here are the KDFs currently supported by </table> </div> <div class="sect2"> -<h3 id="allow-insecure-cryptographic-modes"><a class="anchor" href="administration-guide.html#allow-insecure-cryptographic-modes"></a>Allow Insecure Cryptographic Modes</h3> +<h3 id="allow-insecure-cryptographic-modes"><a class="anchor" href="#allow-insecure-cryptographic-modes"></a>Allow Insecure Cryptographic Modes</h3> <div class="paragraph"> <p>By default, the <code>Allow Insecure Cryptographic Modes</code> property in <code>EncryptContent</code> processor settings is set to <code>not-allowed</code>. This means that if a password of fewer than <code>10</code> characters is provided, a validation error will occur. 10 characters is a conservative estimate and does not take into consideration full entropy calculations, patterns, etc.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/allow-weak-crypto.png" alt="Allow Insecure Cryptographic Modes" width="940"></span></p> +<p><span class="image"><img src="./images/allow-weak-crypto.png" alt="Allow Insecure Cryptographic Modes" width="940"></span></p> </div> <div class="paragraph"> <p>On a JVM with limited strength cryptography, some PBE algorithms limit the maximum password length to 7, and in this case it will not be possible to provide a "safe" password. It is recommended to install the JCE Unlimited Strength Jurisdiction Policy files for the JVM to mitigate this issue.</p> @@ -2488,7 +2526,7 @@ Here are the KDFs currently supported by </div> </div> <div class="sect1"> -<h2 id="encrypted-passwords-in-configuration-files"><a class="anchor" href="administration-guide.html#encrypted-passwords-in-configuration-files"></a>Encrypted Passwords in Configuration Files</h2> +<h2 id="encrypted-passwords-in-configuration-files"><a class="anchor" href="#encrypted-passwords-in-configuration-files"></a>Encrypted Passwords in Configuration Files</h2> <div class="sectionbody"> <div class="paragraph"> <p>In order to facilitate the secure setup of NiFi, you can use the <code>encrypt-config</code> command line utility to encrypt raw configuration values that NiFi decrypts in memory on startup. This extensible protection scheme transparently allows NiFi to use raw values in operation, while protecting them at rest. In the future, hardware security modules (HSM) and external secure storage mechanisms will be integrated, but for now, an AES encryption provider is the default implementation.</p> @@ -2500,7 +2538,7 @@ Here are the KDFs currently supported by <p>If no administrator action is taken, the configuration values remain unencrypted.</p> </div> <div class="sect2"> -<h3 id="encrypt-config_tool"><a class="anchor" href="administration-guide.html#encrypt-config_tool"></a>Encrypt-Config Tool</h3> +<h3 id="encrypt-config_tool"><a class="anchor" href="#encrypt-config_tool"></a>Encrypt-Config Tool</h3> <div class="paragraph"> <p>The <code>encrypt-config</code> command line tool (invoked as <code>./bin/encrypt-config.sh</code> or <code>bin\encrypt-config.bat</code>) reads from a <em>nifi.properties</em> file with plaintext sensitive configuration values, prompts for a master password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new <em>nifi.properties</em> file if specified.</p> </div> @@ -2513,28 +2551,61 @@ Here are the KDFs currently supported by <div class="ulist"> <ul> <li> -<p><code>-b,--bootstrapConf <arg></code> The bootstrap.conf file to persist master key</p> +<p><code>-A</code>,<code>--newFlowAlgorithm <arg></code> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz</p> </li> <li> -<p><code>-h,--help</code> Prints this usage message</p> +<p><code>-b</code>,<code>--bootstrapConf <arg></code> The bootstrap.conf file to persist master key</p> </li> <li> -<p><code>-k,--key <arg></code> The raw hexadecimal key to use to encrypt the sensitive properties (the key can be entered with spaces or <em>-</em> delimiters to assist manual entry — these will be ignored)</p> +<p><code>-e</code>,<code>--oldKey <arg></code> The old raw hexadecimal key to use during key migration</p> </li> <li> -<p><code>-n,--niFiProperties <arg></code> The <em>nifi.properties</em> file containing unprotected config values (will be overwritten by default unless <code>-o</code> is provided)</p> +<p><code>-f</code>,<code>--flowXml <arg></code> The flow.xml.gz file currently protected with old password (will be overwritten)</p> </li> <li> -<p><code>-o,--outputNiFiProperties <arg></code> The destination <em>nifi.properties</em> file containing protected config values (will not modify input <em>nifi.properties</em>)</p> +<p><code>-g</code>,<code>--outputFlowXml <arg></code> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)</p> </li> <li> -<p><code>-p,--password <arg></code> The password from which to derive the key to use to encrypt the sensitive properties</p> +<p><code>-h</code>,<code>--help</code> Prints this usage message</p> </li> <li> -<p><code>-r,--useRawKey</code> If provided, the secure console will prompt for the raw key value in hexadecimal form</p> +<p><code>-i</code>,<code>--outputLoginIdentityProviders <arg></code> The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)</p> </li> <li> -<p><code>-v,--verbose</code> Sets verbose mode (default false)</p> +<p><code>-k</code>,<code>--key <arg></code> The raw hexadecimal key to use to encrypt the sensitive properties</p> +</li> +<li> +<p><code>-l</code>,<code>--loginIdentityProviders <arg></code> The login-identity-providers.xml file containing unprotected config values (will be overwritten)</p> +</li> +<li> +<p><code>-m</code>,<code>--migrate</code> If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with a new key</p> +</li> +<li> +<p><code>-n</code>,<code>--niFiProperties <arg></code> The nifi.properties file containing unprotected config values (will be overwritten)</p> +</li> +<li> +<p><code>-o</code>,<code>--outputNiFiProperties <arg></code> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)</p> +</li> +<li> +<p><code>-p</code>,<code>--password <arg></code> The password from which to derive the key to use to encrypt the sensitive properties</p> +</li> +<li> +<p><code>-P</code>,<code>--newFlowProvider <arg></code> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz</p> +</li> +<li> +<p><code>-r</code>,<code>--useRawKey</code> If provided, the secure console will prompt for the raw key value in hexadecimal form</p> +</li> +<li> +<p><code>-s</code>,<code>--propsKey <arg></code> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz</p> +</li> +<li> +<p><code>-v</code>,<code>--verbose</code> Sets verbose mode (default false)</p> +</li> +<li> +<p><code>-w</code>,<code>--oldPassword <arg></code> The old password from which to derive the key during migration</p> +</li> +<li> +<p><code>-x</code>,<code>--encryptFlowXmlOnly</code> If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or login-identity-providers.xml files will not be modified</p> </li> </ul> </div> @@ -2607,9 +2678,47 @@ nifi.bootstrap.sensitive.key=0123456789A <div class="paragraph"> <p>If the <em>nifi.properties</em> file already has valid protected values, those property values are not modified by the tool.</p> </div> +<div class="paragraph"> +<p>When applied to <em>login-identity-providers.xml</em>, the property elements are updated with an <code>encryption</code> attribute:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre><!-- LDAP Provider --> +<provider> + <identifier>ldap-provider</identifier> + <class>org.apache.nifi.ldap.LdapProvider</class> + <property name="Authentication Strategy">START_TLS</property> + <property name="Manager DN">someuser</property> + <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property> + <property name="TLS - Keystore"></property> + <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property> + <property name="TLS - Keystore Type"></property> + ... + </provider></pre> +</div> +</div> +<div class="paragraph"> +<p>In order to change the key used to encrypt the sensitive values, indicate <strong>migration mode</strong> using the <code>-m</code> or <code>--migrate</code> flag, provide the new key or password using the <code>-k</code> or <code>-p</code> flags as usual, and provide the existing key or password using <code>-e</code> or <code>-w</code> respectively. This will allow the toolkit to decrypt the existing values and re-encrypt them, and update <code>bootstrap.conf</code> with the new key. Only one of the key or password needs to be specified for each phase (old vs. new), and any combination is sufficient:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>old key → new key</p> +</li> +<li> +<p>old key → new password</p> +</li> +<li> +<p>old password → new key</p> +</li> +<li> +<p>old password → new password</p> +</li> +</ul> +</div> </div> <div class="sect2"> -<h3 id="encrypt-config_password"><a class="anchor" href="administration-guide.html#encrypt-config_password"></a>Password Key Derivation</h3> +<h3 id="encrypt-config_password"><a class="anchor" href="#encrypt-config_password"></a>Password Key Derivation</h3> <div class="paragraph"> <p>Instead of providing a 32 or 64 character raw hexadecimal key, you can provide a password from which the key will be derived. As of 1.0.0, the password must be at least 12 characters, and the key will be derived using <code>SCrypt</code> with the parameters:</p> </div> @@ -2652,7 +2761,7 @@ While fixed salts are counter to best pr </div> </div> <div class="sect2"> -<h3 id="encrypt-config_secure_prompt"><a class="anchor" href="administration-guide.html#encrypt-config_secure_prompt"></a>Secure Prompt</h3> +<h3 id="encrypt-config_secure_prompt"><a class="anchor" href="#encrypt-config_secure_prompt"></a>Secure Prompt</h3> <div class="paragraph"> <p>If you prefer not to provide the password or raw key in the command-line invocation of the tool, leaving these arguments absent will prompt a secure console read of the password (by default) or raw key (if the <code>-r</code> flag is provided at invocation).</p> </div> @@ -2660,7 +2769,7 @@ While fixed salts are counter to best pr </div> </div> <div class="sect1"> -<h2 id="clustering"><a class="anchor" href="administration-guide.html#clustering"></a>Clustering Configuration</h2> +<h2 id="clustering"><a class="anchor" href="#clustering"></a>Clustering Configuration</h2> <div class="sectionbody"> <div class="paragraph"> <p>This section provides a quick overview of NiFi Clustering and instructions on how to set up a basic cluster. @@ -2668,7 +2777,7 @@ In the future, we hope to provide supple </div> <div class="imageblock"> <div class="content"> -<img src="images/zero-master-cluster-http-access.png" alt="NiFi Cluster HTTP Access"> +<img src="./images/zero-master-cluster-http-access.png" alt="NiFi Cluster HTTP Access"> </div> </div> <div class="paragraph"> @@ -2702,7 +2811,7 @@ of the cluster. Through the single inter <p><strong>Terminology</strong><br></p> </div> <div class="paragraph"> -<p><strong>NiFi Cluster Coordinator</strong>: A NiFi Cluster Cluster Coordinator is the node in a NiFI cluster that is responsible for carrying out +<p><strong>NiFi Cluster Coordinator</strong>: A NiFi Cluster Cluster Coordinator is the node in a NiFi cluster that is responsible for carrying out tasks to manage which nodes are allowed in the cluster and providing the most up-to-date flow to newly joining nodes. When a DataFlow Manager manages a dataflow in a cluster, they are able to do so through the User Interface of any node in the cluster. Any change made is then replicated to all nodes in the cluster.</p> @@ -2732,7 +2841,7 @@ It just depends on the resources availab which let the Coordinator know they are still connected to the cluster and working properly. By default, the nodes emit heartbeats every 5 seconds, and if the Cluster Coordinator does not receive a heartbeat from a node within 40 seconds, it disconnects the node due to "lack of heartbeat". (The 5-second setting is configurable in the <em>nifi.properties</em> file. -See the <a href="administration-guide.html#system_properties">System Properties</a> section of this document for more information.) The reason that the Cluster Coordinator +See the <a href="#system_properties">System Properties</a> section of this document for more information.) The reason that the Cluster Coordinator disconnects the node is because the Coordinator needs to ensure that every node in the cluster is in sync, and if a node is not heard from regularly, the Coordinator cannot be sure it is still in sync with the rest of the cluster. If, after 40 seconds, the node does send a new heartbeat, the Coordinator will automatically request that the node re-join the cluster, @@ -2770,6 +2879,22 @@ In this case, they DFM may elect to remo the node cannot be rejoined to the cluster until it has been restarted.</p> </div> <div class="paragraph"> +<p><strong>Flow Election</strong><br> +When a cluster first starts up, NiFi must determine which of the nodes have the +"correct" version of the flow. This is done by voting on the flows that each of the nodes has. When a node +attempts to connect to a cluster, it provides a copy of its local flow to the Cluster Coordinator. If no flow +has yet been elected the "correct" flow, the node’s flow is compared to each of the other Nodes' flows. If another +Node’s flow matches this one, a vote is cast for this flow. If no other Node has reported the same flow yet, this +flow will be added to the pool of possibly elected flows with one vote. After +some amount of time has elapsed (configured by setting the <code>nifi.cluster.flow.election.max.wait.time</code> property) or +some number of Nodes have cast votes (configured by setting the <code>nifi.cluster.flow.election.max.candidates</code> property), +a flow is elected to be the "correct" copy of the flow. All nodes that have incompatible flows are then disconnected +from the cluster while those with compatible flows inherit the cluster’s flow. Election is performed according to +the "popular vote" with the caveat that the winner will never be an "empty flow" unless all flows are empty. This +allows an administrator to remove a node’s <code>flow.xml.gz</code> file and restart the node, knowing that the node’s flow will +not be voted to be the "correct" flow unless no other flow is found.</p> +</div> +<div class="paragraph"> <p><strong>Basic Cluster Setup</strong><br></p> </div> <div class="paragraph"> @@ -2777,7 +2902,7 @@ the node cannot be rejoined to the clust </div> <div class="paragraph"> <p>For each instance, certain properties in the <em>nifi.properties</em> file will need to be updated. In particular, the Web and Clustering properties -should be evaluated for your situation and adjusted accordingly. All the properties are described in the <a href="administration-guide.html#system_properties">System Properties</a> section of this +should be evaluated for your situation and adjusted accordingly. All the properties are described in the <a href="#system_properties">System Properties</a> section of this guide; however, in this section, we will focus on the minimum properties that must be set for a simple cluster.</p> </div> <div class="paragraph"> @@ -2796,7 +2921,7 @@ Also, consider whether you need to set t <li> <p>Under the <em>State Management section</em>, set the <code>nifi.state.management.provider.cluster</code> property to the identifier of the Cluster State Provider. Ensure that the Cluster State Provider has been -configured in the <em>state-management.xml</em> file. See <a href="administration-guide.html#state_providers">Configuring State Providers</a> for more information.</p> +configured in the <em>state-management.xml</em> file. See <a href="#state_providers">Configuring State Providers</a> for more information.</p> </li> <li> <p>Under <em>Cluster Node</em> Properties, set the following:</p> @@ -2828,8 +2953,15 @@ the NiFi instance attempts to join is de that is specified.</p> </li> <li> -<p>nifi.cluster.request.replication.claim.timeout - Specifies how long a component can be <em>locked</em> during a request replication -before the lock expires and is automatically unlocked. See <a href="administration-guide.html#claim_management">Claim Management</a> for more information.</p> +<p>nifi.cluster.flow.election.max.wait.time - Specifies the amount of time to wait before electing a Flow as the "correct" Flow. +If the number of Nodes that have voted is equal to the number specified by the <code>nifi.cluster.flow.election.max.candidates</code> +property, the cluster will not wait this long. The default is 5 minutes. Note that the time starts as soon as the first vote +is cast.</p> +</li> +<li> +<p>nifi.cluster.flow.election.max.candidates - Specifies the number of Nodes required in the cluster to cause early election +of Flows. This allows the Nodes in the cluster to avoid having to wait a long time before starting processing if we reach +at least this number of nodes in the cluster.</p> </li> </ul> </div> @@ -2841,7 +2973,7 @@ before the lock expires and is automatic one of the nodes, and the User Interface should look similar to the following:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="images/ncm.png" alt="Clustered User Interface"></span></p> +<p><span class="image"><img src="./images/ncm.png" alt="Clustered User Interface"></span></p> </div> <div class="paragraph"> <p><strong>Troubleshooting</strong></p> @@ -2861,7 +2993,7 @@ set the level="DEBUG" in the following l </div> </div> <div class="sect1"> -<h2 id="state_management"><a class="anchor" href="administration-guide.html#state_management"></a>State Management</h2> +<h2 id="state_management"><a class="anchor" href="#state_management"></a>State Management</h2> <div class="sectionbody"> <div class="paragraph"> <p>NiFi provides a mechanism for Processors, Reporting Tasks, Controller Services, and the framework itself to persist state. This @@ -2870,7 +3002,7 @@ a Processor to store some piece of infor in the cluster. This allows one node to pick up where another node left off, or to coordinate across all of the nodes in a cluster.</p> </div> <div class="sect2"> -<h3 id="state_providers"><a class="anchor" href="administration-guide.html#state_providers"></a>Configuring State Providers</h3> +<h3 id="state_providers"><a class="anchor" href="#state_providers"></a>Configuring State Providers</h3> <div class="paragraph"> <p>When a component decides to store or retrieve state, it does so by providing a "Scope" - either Node-local or Cluster-wide. The mechanism that is used to store and retrieve this state is then determined based on this Scope, as well as the configured State @@ -2924,7 +3056,7 @@ my-zk-server1:2181,my-zk-server2:2181,my <p>When adding data to ZooKeeper, there are two options for Access Control: <code>Open</code> and <code>CreatorOnly</code>. If the <code>Access Control</code> property is set to <code>Open</code>, then anyone is allowed to log into ZooKeeper and have full permissions to see, change, delete, or administer the data. If <code>CreatorOnly</code> is specified, then only the user that created the data is allowed to read, change, delete, or administer the data. -In order to use the <code>CreatorOnly</code> option, NiFi must provide some form of authentication. See the <a href="administration-guide.html#zk_access_control">ZooKeeper Access Control</a> +In order to use the <code>CreatorOnly</code> option, NiFi must provide some form of authentication. See the <a href="#zk_access_control">ZooKeeper Access Control</a> section below for more information on how to configure authentication.</p> </div> <div class="paragraph">
[... 694 lines stripped ...]
