Repository: nifi Updated Branches: refs/heads/master 94d097400 -> 59970344f
NIFI-4791 Made external links open in new windows and fixed some broken links Signed-off-by: Pierre Villard <[email protected]> This closes #2413. Project: http://git-wip-us.apache.org/repos/asf/nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/59970344 Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/59970344 Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/59970344 Branch: refs/heads/master Commit: 59970344fe992c8a2bc3d173cf3683d8130bfdc6 Parents: 94d0974 Author: Andrew Lim <[email protected]> Authored: Thu Jan 18 16:36:54 2018 -0500 Committer: Pierre Villard <[email protected]> Committed: Sat Jan 20 15:15:48 2018 +0100 ---------------------------------------------------------------------- .../src/main/asciidoc/administration-guide.adoc | 62 ++++++++++---------- .../src/main/asciidoc/developer-guide.adoc | 29 ++++----- .../asciidoc/expression-language-guide.adoc | 9 +-- .../src/main/asciidoc/getting-started.adoc | 11 ++-- nifi-docs/src/main/asciidoc/nifi-in-depth.adoc | 7 ++- nifi-docs/src/main/asciidoc/overview.adoc | 20 ++++--- nifi-docs/src/main/asciidoc/user-guide.adoc | 19 +++--- 7 files changed, 81 insertions(+), 76 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/administration-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index abe520f..81a667a 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -17,6 +17,7 @@ = NiFi System Administrator's Guide Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: == System Requirements Apache NiFi can run on something as simple as a laptop, but it can also be clustered across many enterprise-class servers. Therefore, the amount of hardware and memory needed will depend on the size and nature of the dataflow involved. The data is stored on disk while NiFi is processing it. So NiFi needs to have sufficient disk space allocated for its various repositories, particularly the content repository, flowfile repository, and provenance repository (see the <<system_properties>> section for more information about these repositories). NiFi has the following minimum system requirements: @@ -273,7 +274,7 @@ For a client certificate that can be easily imported into the browser, specify: [[user_authentication]] == User Authentication -NiFi supports user authentication via client certificates, via username/password, via Apache Knox, or via OpenId Connect (http://openid.net/connect). +NiFi supports user authentication via client certificates, via username/password, via Apache Knox, or via link:http://openid.net/connect[OpenId Connect^]. Username/password authentication is performed by a 'Login Identity Provider'. The Login Identity Provider is a pluggable mechanism for authenticating users via their username/password. Which Login Identity Provider to use is configured in the _nifi.properties_ file. @@ -292,7 +293,7 @@ token during authentication. NOTE: NiFi can only be configured for username/password, OpenId Connect, or Apache Knox at a given time. It does not support running each of these concurrently. NiFi will require client certificates for authenticating users over HTTPS if none of these are configured. -A secured instance of NiFi cannot be accessed anonymously unless configured to use an LDAP or Kerberos Login Identity Provider, which in turn must be configured to explicitly allow anonymous access. Anonymous access is not currently possible by the default FileAuthorizer (see <<authorizer-configuration>>), but is a future effort (https://issues.apache.org/jira/browse/NIFI-2730[NIFI-2730]). +A secured instance of NiFi cannot be accessed anonymously unless configured to use an LDAP or Kerberos Login Identity Provider, which in turn must be configured to explicitly allow anonymous access. Anonymous access is not currently possible by the default FileAuthorizer (see <<authorizer-configuration>>), but is a future effort (link:https://issues.apache.org/jira/browse/NIFI-2730[NIFI-2730^]). NOTE: NiFi does not perform user authentication over HTTP. Using HTTP, all users will be granted all roles. @@ -404,7 +405,7 @@ To enable authentication via OpenId Connect the following properties must be con [options="header,footer"] |================================================================================================================================================== | Property Name | Description -|`nifi.security.user.oidc.discovery.url` | The discovery URL for the desired OpenId Connect Provider (http://openid.net/specs/openid-connect-discovery-1_0.html). +|`nifi.security.user.oidc.discovery.url` | The discovery URL for the desired OpenId Connect Provider (link:http://openid.net/specs/openid-connect-discovery-1_0.html[http://openid.net/specs/openid-connect-discovery-1_0.html^]). |`nifi.security.user.oidc.connect.timeout` | Connect timeout when communicating with the OpenId Connect Provider. |`nifi.security.user.oidc.read.timeout` | Read timeout when communicating with the OpenId Connect Provider. |`nifi.security.user.oidc.client.id` | The client id for NiFi after registration with the OpenId Connect Provider. @@ -1264,7 +1265,7 @@ The `EncryptContent` processor allows for the encryption and decryption of data, [[key-derivation-functions]] === Key Derivation Functions -Key Derivation Functions (KDF) are mechanisms by which human-readable information, usually a password or other secret information, is translated into a cryptographic key suitable for data protection. For further information, read https://en.wikipedia.org/wiki/Key_derivation_function[the Wikipedia entry on Key Derivation Functions]. +Key Derivation Functions (KDF) are mechanisms by which human-readable information, usually a password or other secret information, is translated into a cryptographic key suitable for data protection. For further information, read the link:https://en.wikipedia.org/wiki/Key_derivation_function[Wikipedia entry on Key Derivation Functions^]. Currently, KDFs are ingested by `CipherProvider` implementations and return a fully-initialized `Cipher` object to be used for encryption or decryption. Due to the use of a `CipherProviderFactory`, the KDFs are not customizable at this time. Future enhancements will include the ability to provide custom cost parameters to the KDF at initialization time. As a work-around, `CipherProvider` instances can be initialized with custom cost parameters in the constructor but this is not currently supported by the `CipherProviderFactory`. Here are the KDFs currently supported by NiFi (primarily in the `EncryptContent` processor for password-based encryption (PBE)) and relevant notes: @@ -1276,16 +1277,16 @@ Here are the KDFs currently supported by NiFi (primarily in the `EncryptContent` ** This KDF is provided for compatibility with data encrypted using OpenSSL's default PBE, known as `EVP_BytesToKey`. This is a single iteration of MD5 over the concatenation of the password and 8 bytes of random ASCII salt. OpenSSL recommends using `PBKDF2` for key derivation but does not expose the library method necessary to the command-line tool, so this KDF is still the de facto default for command-line encryption. * Bcrypt ** This KDF was added in v0.5.0. -** https://en.wikipedia.org/wiki/Bcrypt[Bcrypt] is an adaptive function based on the https://en.wikipedia.org/wiki/Blowfish_(cipher)[Blowfish] cipher. This KDF is strongly recommended as it automatically incorporates a random 16 byte salt, configurable cost parameter (or "work factor"), and is hardened against brute-force attacks using https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units[GPGPU] (which share memory between cores) by requiring access to "large" blocks of memory during the key derivation. It is less resistant to https://en.wikipedia.org/wiki/Field-programmable_gate_array[FPGA] brute-force attacks where the gate arrays have access to individual embedded RAM blocks. +** link:https://en.wikipedia.org/wiki/Bcrypt[Bcrypt^] is an adaptive function based on the link:https://en.wikipedia.org/wiki/Blowfish_(cipher)[Blowfish^] cipher. This KDF is strongly recommended as it automatically incorporates a random 16 byte salt, configurable cost parameter (or "work factor"), and is hardened against brute-force attacks using link:https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units[GPGPU^] (which share memory between cores) by requiring access to "large" blocks of memory during the key derivation. It is less resistant to link:https://en.wikipedia.org/wiki/Field-programmable_gate_array[FPGA^] brute-force attacks where the gate arrays have access to individual embedded RAM blocks. ** Because the length of a Bcrypt-derived key is always 184 bits, the complete output is then fed to a `SHA-512` digest and truncated to the desired key length. This provides the benefit of the avalanche effect on the formatted input. ** The recommended minimum work factor is 12 (2^12^ key derivation rounds) (as of 2/1/2016 on commodity hardware) and should be increased to the threshold at which legitimate systems will encounter detrimental delays (see schedule below or use `BcryptCipherProviderGroovyTest#testDefaultConstructorShouldProvideStrongWorkFactor()` to calculate safe minimums). ** The salt format is `$2a$10$ABCDEFGHIJKLMNOPQRSTUV`. The salt is delimited by `$` and the three sections are as follows: -*** `2a` - the version of the format. An extensive explanation can be found http://blog.ircmaxell.com/2012/12/seven-ways-to-screw-up-bcrypt.html[here]. NiFi currently uses `2a` for all salts generated internally. +*** `2a` - the version of the format. An extensive explanation can be found link:http://blog.ircmaxell.com/2012/12/seven-ways-to-screw-up-bcrypt.html[here^]. NiFi currently uses `2a` for all salts generated internally. *** `10` - the work factor. This is actually the log~2~ value, so the total iteration count would be 2^10^ in this case. *** `ABCDEFGHIJKLMNOPQRSTUV` - the 22 character, Base64-encoded, unpadded, raw salt value. This decodes to a 16 byte salt used in the key derivation. * Scrypt ** This KDF was added in v0.5.0. -** https://en.wikipedia.org/wiki/Scrypt[Scrypt] is an adaptive function designed in response to `bcrypt`. This KDF is recommended as it requires relatively large amounts of memory for each derivation, making it resistant to hardware brute-force attacks. +** link:https://en.wikipedia.org/wiki/Scrypt[Scrypt^] is an adaptive function designed in response to `bcrypt`. This KDF is recommended as it requires relatively large amounts of memory for each derivation, making it resistant to hardware brute-force attacks. ** The recommended minimum cost is `N`=2^14^, `r`=8, `p`=1 (as of 2/1/2016 on commodity hardware) and should be increased to the threshold at which legitimate systems will encounter detrimental delays (see schedule below or use `ScryptCipherProviderGroovyTest#testDefaultConstructorShouldProvideStrongParameters()` to calculate safe minimums). ** The salt format is `$s0$e0101$ABCDEFGHIJKLMNOPQRSTUV`. The salt is delimited by `$` and the three sections are as follows: *** `s0` - the version of the format. NiFi currently uses `s0` for all salts generated internally. @@ -1294,7 +1295,7 @@ Here are the KDFs currently supported by NiFi (primarily in the `EncryptContent` *** `ABCDEFGHIJKLMNOPQRSTUV` - the 12-44 character, Base64-encoded, unpadded, raw salt value. This decodes to a 8-32 byte salt used in the key derivation. * PBKDF2 ** This KDF was added in v0.5.0. -** https://en.wikipedia.org/wiki/PBKDF2[Password-Based Key Derivation Function 2] is an adaptive derivation function which uses an internal pseudorandom function (PRF) and iterates it many times over a password and salt (at least 16 bytes). +** link:https://en.wikipedia.org/wiki/PBKDF2[Password-Based Key Derivation Function 2^] is an adaptive derivation function which uses an internal pseudorandom function (PRF) and iterates it many times over a password and salt (at least 16 bytes). ** The PRF is recommended to be `HMAC/SHA-256` or `HMAC/SHA-512`. The use of an HMAC cryptographic hash function mitigates a length extension attack. ** The recommended minimum number of iterations is 160,000 (as of 2/1/2016 on commodity hardware). This number should be doubled every two years (see schedule below or use `PBKDF2CipherProviderGroovyTest#testDefaultConstructorShouldProvideStrongIterationCount()` to calculate safe minimums). ** This KDF is not memory-hard (can be parallelized massively with commodity hardware) but is still recommended as sufficient by http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf[NIST SP 800-132 (PDF)] and many cryptographers (when used with a proper iteration count and HMAC cryptographic hash function). @@ -1304,18 +1305,18 @@ Here are the KDFs currently supported by NiFi (primarily in the `EncryptContent` ==== Additional Resources -* http://stackoverflow.com/a/30308723/70465[Explanation of optimal scrypt cost parameters and relationships] -* http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf[NIST Special Publication 800-132] -* https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet#Work_Factor[OWASP Password Storage Work Factor Calculations] -* http://security.stackexchange.com/a/3993/16485[PBKDF2 rounds calculations] -* http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scrypt.html[Scrypt as KDF vs password storage vulnerabilities] -* http://security.stackexchange.com/a/26253/16485[Scrypt vs. Bcrypt (as of 2010)] -* http://security.stackexchange.com/a/6415/16485[Bcrypt vs PBKDF2] -* http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/[Choosing a work factor for Bcrypt] -* https://docs.spring.io/spring-security/site/docs/current/apidocs/org/springframework/security/crypto/bcrypt/BCrypt.html[Spring Security Bcrypt] -* https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html[OpenSSL EVP BytesToKey PKCS#1v1.5] -* https://wiki.openssl.org/index.php/Manual:PKCS5_PBKDF2_HMAC(3)[OpenSSL PBKDF2 KDF] -* http://security.stackexchange.com/a/29139/16485[OpenSSL KDF flaws description] +* link:http://stackoverflow.com/a/30308723/70465[Explanation of optimal scrypt cost parameters and relationships^] +* link:http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf[NIST Special Publication 800-132^] +* link:https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet#Work_Factor[OWASP Password Storage Work Factor Calculations^] +* link:http://security.stackexchange.com/a/3993/16485[PBKDF2 rounds calculations^] +* link:http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scrypt.html[Scrypt as KDF vs password storage vulnerabilities^] +* link:http://security.stackexchange.com/a/26253/16485[Scrypt vs. Bcrypt (as of 2010)^] +* link:http://security.stackexchange.com/a/6415/16485[Bcrypt vs PBKDF2^] +* link:http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/[Choosing a work factor for Bcrypt^] +* link:https://docs.spring.io/spring-security/site/docs/current/apidocs/org/springframework/security/crypto/bcrypt/BCrypt.html[Spring Security Bcrypt^] +* link:https://www.openssl.org/docs/man1.1.0/crypto/EVP_BytesToKey.html[OpenSSL EVP BytesToKey PKCS#1v1.5^] +* link:https://wiki.openssl.org/index.php/Manual:PKCS5_PBKDF2_HMAC(3)[OpenSSL PBKDF2 KDF^] +* link:http://security.stackexchange.com/a/29139/16485[OpenSSL KDF flaws description^] === Salt and IV Encoding @@ -1347,7 +1348,7 @@ image:pbkdf2-salt.png["PBKDF2 Salt & IV Encoding"] === Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies -Because of US export regulations, default JVMs have http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#importlimits[limits imposed on the strength of cryptographic operations] available to them. For example, AES operations are limited to `128 bit keys` by default. While `AES-128` is cryptographically safe, this can have unintended consequences, specifically on Password-based Encryption (PBE). +Because of US export regulations, default JVMs have link:http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#importlimits[limits imposed on the strength of cryptographic operations^] available to them. For example, AES operations are limited to `128 bit keys` by default. While `AES-128` is cryptographically safe, this can have unintended consequences, specifically on Password-based Encryption (PBE). PBE is the process of deriving a cryptographic key for encryption or decryption from _user-provided secret material_, usually a password. Rather than a human remembering a (random-appearing) 32 or 64 character hexadecimal string, a password or passphrase is used. @@ -1426,7 +1427,7 @@ image:allow-weak-crypto.png["Allow Insecure Cryptographic Modes", width=940] On a JVM with limited strength cryptography, some PBE algorithms limit the maximum password length to 7, and in this case it will not be possible to provide a "safe" password. It is recommended to install the JCE Unlimited Strength Jurisdiction Policy files for the JVM to mitigate this issue. -* http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html[JCE Unlimited Strength Jurisdiction Policy files for Java 8] +* link:http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html[JCE Unlimited Strength Jurisdiction Policy files for Java 8^] If on a system where the unlimited strength policies cannot be installed, it is recommended to switch to an algorithm that supports longer passwords (see table above). @@ -1436,7 +1437,7 @@ If on a system where the unlimited strength policies cannot be installed, it is If it is not possible to install the unlimited strength jurisdiction policies, the `Allow Weak Crypto` setting can be changed to `allowed`, but *this is _not_ recommended*. Changing this setting explicitly acknowledges the inherent risk in using weak cryptographic configurations. ===================== -It is preferable to request upstream/downstream systems to switch to https://cwiki.apache.org/confluence/display/NIFI/Encryption+Information[keyed encryption] or use a "strong" https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations[Key Derivation Function (KDF) supported by NiFi]. +It is preferable to request upstream/downstream systems to switch to link:https://cwiki.apache.org/confluence/display/NIFI/Encryption+Information[keyed encryption^] or use a "strong" link:https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations[Key Derivation Function (KDF) supported by NiFi^]. == Encrypted Passwords in Configuration Files @@ -2126,7 +2127,7 @@ echo 2 > state/zookeeper/myid And so on. For more information on the properties used to administer ZooKeeper, see the -link:https://zookeeper.apache.org/doc/current/zookeeperAdmin.html[ZooKeeper Admin Guide]. +link:https://zookeeper.apache.org/doc/current/zookeeperAdmin.html[ZooKeeper Admin Guide^]. For information on securing the embedded ZooKeeper Server, see the <<securing_zookeeper>> section below. @@ -2160,8 +2161,7 @@ In order to secure the communications, we need to ensure that both the client an NiFi ZooKeeper client and embedded ZooKeeper server to use Kerberos are provided below. If Kerberos is not already setup in your environment, you can find information on installing and setting up a Kerberos Server at -https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_] -. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running. +link:https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html^]. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running. Note, the following procedures for kerberizing an Embedded ZooKeeper server in your NiFi Node and kerberizing a ZooKeeper NiFi client will require that Kerberos client libraries be installed. This is accomplished in Fedora-based Linux distributions via: @@ -2335,8 +2335,8 @@ Failure to do so, may result in errors similar to the following: [source] 2016-01-08 16:08:57,888 ERROR [pool-26-thread-1-SendThread(localhost:2181)] o.a.zookeeper.client.ZooKeeperSaslClient An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. -If there are problems communicating or authenticating with Kerberos, -link:http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html[this Troubleshooting Guide] may be of value. +If there are problems communicating or authenticating with Kerberos, this +link:http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html[Troubleshooting Guide^] may be of value. One of the most important notes in the above Troubleshooting guide is the mechanism for turning on Debug output for Kerberos. This is done by setting the `sun.security.krb5.debug` environment variable. @@ -2363,7 +2363,7 @@ For example, you may want to use the ZooKeeper Migrator when you are: * Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x The NiFi ZooKeeper Migrator is part of the NiFi Toolkit and is downloaded separately from the -link:https://nifi.apache.org/download.html[Apache NiFi download page]. +link:https://nifi.apache.org/download.html[Apache NiFi download page^]. [[zk_migrator_command_line_parameters]] ==== zk-migrator.sh Command Line Parameters @@ -2676,9 +2676,9 @@ See <<kerberos_properties>> for complete documentation. === Notes * Kerberos is case-sensitive in many places and the error messages (or lack thereof) may not be sufficiently explanatory. Check the case sensitivity of the service principal in your configuration files. Convention is `HTTP/fully.qualified.domain@REALM`. -* Browsers have varying levels of restriction when dealing with SPNEGO negotiations. Some will provide the local Kerberos ticket to any domain that requests it, while others whitelist the trusted domains. See link:http://docs.spring.io/autorepo/docs/spring-security-kerberos/1.0.2.BUILD-SNAPSHOT/reference/htmlsingle/#browserspnegoconfig[Spring Security Kerberos - Reference Documentation: Appendix E. Configure browsers for SPNEGO Negotiation] for common browsers. +* Browsers have varying levels of restriction when dealing with SPNEGO negotiations. Some will provide the local Kerberos ticket to any domain that requests it, while others whitelist the trusted domains. See link:http://docs.spring.io/autorepo/docs/spring-security-kerberos/1.0.2.BUILD-SNAPSHOT/reference/htmlsingle/#browserspnegoconfig[Spring Security Kerberos - Reference Documentation: Appendix E. Configure browsers for SPNEGO Negotiation^] for common browsers. * Some browsers (legacy IE) do not support recent encryption algorithms such as AES, and are restricted to legacy algorithms (DES). This should be noted when generating keytabs. -* The KDC must be configured and a service principal defined for NiFi and a keytab exported. Comprehensive instructions for Kerberos server configuration and administration are beyond the scope of this document (see link:http://web.mit.edu/kerberos/krb5-current/doc/admin/index.html[MIT Kerberos Admin Guide]), but an example is below: +* The KDC must be configured and a service principal defined for NiFi and a keytab exported. Comprehensive instructions for Kerberos server configuration and administration are beyond the scope of this document (see link:http://web.mit.edu/kerberos/krb5-current/doc/admin/index.html[MIT Kerberos Admin Guide^]), but an example is below: Adding a service principal for a server at `nifi.nifi.apache.org` and exporting the keytab from the KDC: http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/developer-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/developer-guide.adoc b/nifi-docs/src/main/asciidoc/developer-guide.adoc index b4aa227..3d820d8 100644 --- a/nifi-docs/src/main/asciidoc/developer-guide.adoc +++ b/nifi-docs/src/main/asciidoc/developer-guide.adoc @@ -17,6 +17,7 @@ = NiFi Developer's Guide Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: == Introduction @@ -569,21 +570,21 @@ The NiFi API provides notification support through use of Java Annotations. The `org.apache.nifi.annotations.notification` package contains several annotations for notification management. The following annotations may be applied to Java methods in a NiFi component to -indicate to the framework when the methods should be called. For the -discussion of Component Notification, we will define a NiFi component +indicate to the framework when the methods should be called. For the +discussion of Component Notification, we will define a NiFi component as a *Processor*, *Controller Service*, or *Reporting Task*. ==== @OnPrimaryNodeStateChange -The `@OnPrimaryNodeStateChange` annotation causes a method to be invoked -as soon as the state of the Primary Node in a cluster has changed. -Methods with this annotation should take either no arguments or one -argument of type `PrimaryNodeState`. The `PrimaryNodeState` provides -context about what changed so that the component can take appropriate -action. The `PrimaryNodeState` enumerator has two possible values: +The `@OnPrimaryNodeStateChange` annotation causes a method to be invoked +as soon as the state of the Primary Node in a cluster has changed. +Methods with this annotation should take either no arguments or one +argument of type `PrimaryNodeState`. The `PrimaryNodeState` provides +context about what changed so that the component can take appropriate +action. The `PrimaryNodeState` enumerator has two possible values: `ELECTED_PRIMARY_NODE` (the node receiving this -state has been elected the Primary Node of the NiFi cluster), or -`PRIMARY_NODE_REVOKED` (the node receiving this state was the Primary +state has been elected the Primary Node of the NiFi cluster), or +`PRIMARY_NODE_REVOKED` (the node receiving this state was the Primary Node but has now had its Primary Node role revoked). [[restricted]] @@ -1977,7 +1978,7 @@ There are two UI extension points that are available in NiFi: - Custom Processor UIs - Content Viewers -Custom UIs can be created to provide configuration options beyond the standard property/value tables available in most processor settings. Examples of processors with Custom UIs are link:https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-update-attribute-bundle[UpdateAttribute] and link:https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-bundle[JoltTransformJSON]. +Custom UIs can be created to provide configuration options beyond the standard property/value tables available in most processor settings. Examples of processors with Custom UIs are link:https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-update-attribute-bundle[UpdateAttribute^] and link:https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-bundle[JoltTransformJSON^]. Content Viewers can be created to extend the types of data that can be viewed within NiFi. NiFi comes with NARs in the lib directory which contain content viewers for data types such as csv, xml, avro, json (standard-nar) and image types such as png, jpeg and gif (media-nar). @@ -2539,18 +2540,18 @@ The back end of Apache NiFi is written in Java. The web tier makes use of JAX-RS used to provide a user interface. We depend on several third-party JavaScript libraries, including D3 and JQuery, among others. We make use of Apache Maven for our builds and Git for our version control system. -Documentation is created in link:http://asciidoctor.org[AsciiDoc]. +Documentation is created in link:http://asciidoctor.org[AsciiDoc^]. === Where to Start? -link:http://issues.apache.org/jira/browse/NIFI[NiFi's JIRA page] can be used to find tickets that are tagged as "beginner", +link:http://issues.apache.org/jira/browse/NIFI[NiFi's JIRA page^] can be used to find tickets that are tagged as "beginner", or you can dig into any of the tickets for creating Processors. Processors should be self-contained and not rely on other outside components (except for Controller Services), so they make for excellent starting points for new NiFi developers to get started. This exposes the developer to the NiFi API and is the most extensible part of the dataflow system. System-level and overview documentation is located in '<code checkout location>/nifi/nifi-docs/src/main/asciidoc'. -Tools available to facilitate documentation generation are available at link:http://asciidoctor.org/docs/editing-asciidoc-with-live-preview/[Editing AsciiDoc with Live Preview]. +Tools available to facilitate documentation generation are available at link:http://asciidoctor.org/docs/editing-asciidoc-with-live-preview/[Editing AsciiDoc with Live Preview^]. === Supplying a contribution http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/expression-language-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/expression-language-guide.adoc b/nifi-docs/src/main/asciidoc/expression-language-guide.adoc index ca999e8..e494550 100644 --- a/nifi-docs/src/main/asciidoc/expression-language-guide.adoc +++ b/nifi-docs/src/main/asciidoc/expression-language-guide.adoc @@ -17,6 +17,7 @@ = Apache NiFi Expression Language Guide Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: [[overview]] == Overview @@ -191,7 +192,7 @@ Language supports four different data types: - *Decimal*: A Decimal is a numeric value that can support decimals and larger values with minimal loss of precision. More precisely it is a double-precision 64-bit IEEE 754 floating point. Due to this minimal loss of precision this data type should not be used for very precise values, such as currency. For more documentation on the range of values stored in this data type - refer to this https://docs.oracle.com/javase/specs/jls/se7/html/jls-4.html#jls-4.2.3[link]. The following are some examples of the forms of + refer to this link:https://docs.oracle.com/javase/specs/jls/se7/html/jls-4.html#jls-4.2.3[link^]. The following are some examples of the forms of literal decimals that are supported in expression language (the "E" can also be lower-case): * 1.1 @@ -262,7 +263,7 @@ subject exists and `false` otherwise.# [.function] === isEmpty -*Description*: [.description]#The `isEmpty` function returns `true` if the Subject is null, does not contain any characters +*Description*: [.description]#The `isEmpty` function returns `true` if the Subject is null, does not contain any characters or contains only white-space (new line, carriage return, space, tab), `false` otherwise.# *Subject Type*: [.subject]#String# @@ -1716,7 +1717,7 @@ must be valid for the base being converted from. For example converting "0xFF" f [.function] === math -*Description*: [.description]#ADVANCED FEATURE. This expression is designed to be used by advanced users only. It utilizes Java Reflection to run arbitrary java.lang.Math static methods. The exact API will depend on the version of Java you are running. The Java 8 API can be found here: https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html +*Description*: [.description]#ADVANCED FEATURE. This expression is designed to be used by advanced users only. It utilizes Java Reflection to run arbitrary java.lang.Math static methods. The exact API will depend on the version of Java you are running. The Java 8 API can be found here: link:https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html[https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html^] + In order to run the correct method, the parameter types must be correct. The Expression Language "Number" (whole number) type is interpreted as a Java "long". The "Decimal" type is interpreted as a Java "double". Running the desired method may require calling "toNumber()" or "toDecimal()" in order to "cast" the value to the desired type. This also is important to remember when cascading "math()" calls since the return type depends on the method that was run.# @@ -1751,7 +1752,7 @@ In order to run the correct method, the parameter types must be correct. The Exp *Description*: [.description]#Formats a number as a date/time according to the format specified by the argument. The argument must be a String that is a valid Java SimpleDateFormat format. The Subject is expected to be a Number that - represents the number of milliseconds since Midnight GMT on January 1, 1970. The number will be evaluated using the local + represents the number of milliseconds since Midnight GMT on January 1, 1970. The number will be evaluated using the local time zone unless specified in the second optional argument.# *Subject Type*: [.subject]#Number# http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/getting-started.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/getting-started.adoc b/nifi-docs/src/main/asciidoc/getting-started.adoc index b9a6130..bcb0d5e 100644 --- a/nifi-docs/src/main/asciidoc/getting-started.adoc +++ b/nifi-docs/src/main/asciidoc/getting-started.adoc @@ -17,6 +17,7 @@ = Getting Started with Apache NiFi Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: == Who is This Guide For? @@ -57,7 +58,7 @@ dataflows. == Downloading and Installing NiFi -NiFi can be downloaded from the link:http://nifi.apache.org/download.html[NiFi Downloads Page]. There are two packaging options +NiFi can be downloaded from the link:http://nifi.apache.org/download.html[NiFi Downloads Page^]. There are two packaging options available: a "tarball" that is tailored more to Linux and a zip file that is more applicable for Windows users. Mac OS X users may also use the tarball or can install via Homebrew. @@ -111,7 +112,7 @@ and `sudo service nifi stop`. Additionally, the running status can be checked vi == I Started NiFi. Now What? Now that NiFi has been started, we can bring up the User Interface (UI) in order to create and monitor our dataflow. -To get started, open a web browser and navigate to `http://localhost:8080/nifi`. The port can be changed by +To get started, open a web browser and navigate to link:http://localhost:8080/nifi[`http://localhost:8080/nifi`^]. The port can be changed by editing the `nifi.properties` file in the NiFi `conf` directory, but the default port is 8080. This will bring up the User Interface, which at this point is a blank canvas for orchestrating a dataflow: @@ -741,14 +742,14 @@ addition to this Getting Started Guide: - link:developer-guide.html[Developer's Guide] - While not an exhaustive guide to All Things NiFi Development, this guide does provide a comprehensive overview of the different API's available and how they should be used. In addition, it provides Best Practices for developing NiFi components and common Processor idioms to help aid in understanding the logic behind many of the existing NiFi components. -- link:https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide[Contributor's Guide] - A guide for explaining how to contribute +- link:https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide[Contributor's Guide^] - A guide for explaining how to contribute work back to the Apache NiFi community so that others can make use of it. Several blog postings have also been added to the Apache NiFi blog site: -link:https://blogs.apache.org/nifi/[https://blogs.apache.org/nifi/] +link:https://blogs.apache.org/nifi/[https://blogs.apache.org/nifi/^] In addition to the blog and guides provided here, you can browse the different -link:https://nifi.apache.org/mailing_lists.html[NiFi Mailing Lists] or send an e-mail to one of the mailing lists at +link:https://nifi.apache.org/mailing_lists.html[NiFi Mailing Lists^] or send an e-mail to one of the mailing lists at link:mailto:[email protected][[email protected]] or link:mailto:[email protected][[email protected]]. http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/nifi-in-depth.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/nifi-in-depth.adoc b/nifi-docs/src/main/asciidoc/nifi-in-depth.adoc index a1d32d3..d003b51 100644 --- a/nifi-docs/src/main/asciidoc/nifi-in-depth.adoc +++ b/nifi-docs/src/main/asciidoc/nifi-in-depth.adoc @@ -17,6 +17,7 @@ = Apache NiFi In Depth Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: == Intro This advanced level document is aimed at providing an in-depth look at the implementation and design decisions of NiFi. It assumes the reader has read enough of the other documentation to know the basics of NiFi. @@ -49,7 +50,7 @@ NiFi protects against hardware and system failures by keeping a record of what w If the Node was in the middle of writing content when it went down, nothing is corrupted, thanks to the Copy On Write (mentioned below) and Immutability (mentioned above) paradigms. Since FlowFile transactions never modify the original content (pointed to by the content pointer), the original is safe. When NiFi goes down, the write claim for the change is orphaned and then cleaned up by the background garbage collection. This provides a ârollbackâ to the last known stable state. -The Node then restores its state from the FlowFile. For a more in-depth, step-by-step explanation of the process, see this link: https://cwiki.apache.org/confluence/display/NIFI/NiFi%27s+Write-Ahead+Log+Implementation . +The Node then restores its state from the FlowFile. For a more in-depth, step-by-step explanation of the process, see link:https://cwiki.apache.org/confluence/display/NIFI/NiFi%27s+Write-Ahead+Log+Implementation[NiFi's Write-Ahead Log Implementation^]. This setup, in terms of transactional units of work, allows NiFi to be very resilient in the face of adversity, ensuring that even if NiFi is suddenly killed, it can pick back up without any loss of data. @@ -84,7 +85,7 @@ Keep in mind, though, that since Provenance is not copying the content in the Co NOTE: Since provenance events are snapshots of the FlowFile, as it exists in the current flow, changes to the flow may impact the ability to replay provenance events later on. For example, if a Connection is deleted from the flow, the data cannot be replayed from that point in the flow, since there is now nowhere to enqueue the data for processing. -For a look at the design decisions behind the Provenance Repository check out this link: https://cwiki.apache.org/confluence/display/NIFI/Persistent+Provenance+Repository+Design +For a look at the design decisions behind the Provenance Repository check out link:https://cwiki.apache.org/confluence/display/NIFI/Persistent+Provenance+Repository+Design[Persistent Provenance Repository Design^]. ==== Deeper View: Provenance Log Files Each provenance event has two maps, one for the attributes before the event and one for the updated attribute values. In general, provenance events don't store the updated values of the attributes as they existed when the event was emitted, but instead, the attribute values when the session is committed. The events are cached and saved until the session is committed and once the session is committed the events are emitted with the attributes associated with the FlowFile when the session is committed. The exception to this rule is the "SEND" event, in which case the event contains the attributes as they existed when the event was emitted. This is done because if the attributes themselves were also sent, it is important to have an accurate account of exactly what information was sent. @@ -106,7 +107,7 @@ It is considered a best practice to analyze the contents of a FlowFile as few ti == Life of a FlowFile -To better understand how the repos interact with one another, the underlying functionality of NiFi, and the life of a FlowFile; this next section will include examples of a FlowFile at different points in a real flow. The flow is a template called "WebCrawler.xml" and is available here: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates. +To better understand how the repos interact with one another, the underlying functionality of NiFi, and the life of a FlowFile; this next section will include examples of a FlowFile at different points in a real flow. The flow is a template called "WebCrawler.xml" and is available here: link:https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates[https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates^]. At a high level, this template reaches out to a seed URL configured in the GetHTTP processor, then analyzes the response using the RouteText processor to find instances of a keyword (in this case "nifi"), and potential URLs to hit. Then InvokeHTTP executes a HTTP Get request using the URLs found in the original seed web page. The response is routed based on the status code attribute and only 200-202 status codes are routed back to the original RouteText processor for analysis. http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/overview.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/overview.adoc b/nifi-docs/src/main/asciidoc/overview.adoc index f5f2e31..594d9a3 100644 --- a/nifi-docs/src/main/asciidoc/overview.adoc +++ b/nifi-docs/src/main/asciidoc/overview.adoc @@ -17,6 +17,7 @@ = Apache NiFi Overview Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org +:linkattrs: == What is Apache NiFi? Put simply NiFi was built to automate the flow of data between systems. While @@ -272,18 +273,19 @@ Flexible Scaling Model:: help. Use of NiFi's 'site-to-site' feature is also very effective as it is a protocol that allows NiFi and a client (including another NiFi cluster) to talk to each other, share information about loading, and to exchange data on specific authorized ports. Scale-up & down;; - NiFi is also designed to scale-up and down in a very flexible manner. In terms of increasing throughput from the standpoint of the NiFi framework, it is possible to increase the number of concurrent tasks on the processor under the Scheduling tab when configuring. This allows more processes to execute simultaneously, providing greater throughput. On the other side of the spectrum, you can perfectly scale NiFi down to be suitable to run on edge devices where a small footprint is desired due to limited hardware resources. To specifically solve the first mile data collection challenge and edge use cases, you can find more details here: https://cwiki.apache.org/confluence/display/NIFI/MiNiFi regarding a child project effort of Apache NiFi, MiNiFi (pronounced "minify", [min-uh-fahy]). + NiFi is also designed to scale-up and down in a very flexible manner. In terms of increasing throughput from the standpoint of the NiFi framework, it is possible to increase the number of concurrent tasks on the processor under the Scheduling tab when configuring. This allows more processes to execute simultaneously, providing greater throughput. On the other side of the spectrum, you can perfectly scale NiFi down to be suitable to run on edge devices where a small footprint is desired due to limited hardware resources. To specifically solve the first mile data collection challenge and edge use cases, you can find more details here: link:https://cwiki.apache.org/confluence/display/NIFI/MiNiFi[https://cwiki.apache.org/confluence/display/NIFI/MiNiFi^] regarding a child project effort of Apache NiFi, MiNiFi (pronounced "minify", [min-uh-fahy]). == References [bibliography] -- [[[eip]]] Gregor Hohpe. Enterprise Integration Patterns [online]. Retrieved: 27 Dec 2014, from: http://www.enterpriseintegrationpatterns.com/ -- [[[soa]]] Wikipedia. Service Oriented Architecture [online]. Retrieved: 27 Dec 2014, from: http://en.wikipedia.org/wiki/Service-oriented_architecture -- [[[api]]] Eric Savitz. Welcome to the API Economy [online]. Forbes.com. Retrieved: 27 Dec 2014, from: http://www.forbes.com/sites/ciocentral/2012/08/29/welcome-to-the-api-economy/ -- [[[api2]]] Adam Duvander. The rise of the API economy and consumer-led ecosystems [online]. thenextweb.com. Retrieved: 27 Dec 2014, from: http://thenextweb.com/dd/2014/03/28/api-economy/ -- [[[iot]]] Wikipedia. Internet of Things [online]. Retrieved: 27 Dec 2014, from: http://en.wikipedia.org/wiki/Internet_of_Things -- [[[bigdata]]] Wikipedia. Big Data [online]. Retrieved: 27 Dec 2014, from: http://en.wikipedia.org/wiki/Big_data -- [[[fbp]]] Wikipedia. Flow Based Programming [online]. Retrieved: 28 Dec 2014, from: http://en.wikipedia.org/wiki/Flow-based_programming#Concepts -- [[[seda]]] Matt Welsh. Harvard. SEDA: An Architecture for Highly Concurrent Server Applications [online]. Retrieved: 28 Dec 2014, from: http://www.eecs.harvard.edu/~mdw/proj/seda/ +- [[[eip]]] Gregor Hohpe. Enterprise Integration Patterns [online]. Retrieved: 27 Dec 2014, from: link:http://www.enterpriseintegrationpatterns.com[http://www.enterpriseintegrationpatterns.com^] +- [[[soa]]] Wikipedia. Service Oriented Architecture [online]. Retrieved: 27 Dec 2014, from: link:http://en.wikipedia.org/wiki/Service-oriented_architecture[http://en.wikipedia.org/wiki/Service-oriented_architecture^] +- [[[api]]] Eric Savitz. Welcome to the API Economy [online]. Forbes.com. Retrieved: 27 Dec 2014, from: link:http://www.forbes.com/sites/ciocentral/2012/08/29/welcome-to-the-api-economy[http://www.forbes.com/sites/ciocentral/2012/08/29/welcome-to-the-api-economy^] +- [[[api2]]] Adam Duvander. The rise of the API economy and consumer-led ecosystems [online]. thenextweb.com. Retrieved: 27 Dec 2014, from: link:http://thenextweb.com/dd/2014/03/28/api-economy[http://thenextweb.com/dd/2014/03/28/api-economy^] +- [[[iot]]] Wikipedia. Internet of Things [online]. Retrieved: 27 Dec 2014, from: link:http://en.wikipedia.org/wiki/Internet_of_Things[http://en.wikipedia.org/wiki/Internet_of_Things^] +- [[[bigdata]]] Wikipedia. Big Data [online]. Retrieved: 27 Dec 2014, from: link:http://en.wikipedia.org/wiki/Big_data[http://en.wikipedia.org/wiki/Big_data^] +- [[[fbp]]] Wikipedia. Flow Based Programming [online]. Retrieved: 28 Dec 2014, from: link:http://en.wikipedia.org/wiki/Flow-based_programming#Concepts[http://en.wikipedia.org/wiki/Flow-based_programming#Concepts^] +- [[[seda]]] Matt Welsh. Berkeley. SEDA: An Architecture for Well-Conditioned, +Scalable Internet Services [online]. Retrieved: 18 Jan 2018, from: link:http://www.mdw.la/papers/seda-sosp01.pdf[http://www.mdw.la/papers/seda-sosp01.pdf^] http://git-wip-us.apache.org/repos/asf/nifi/blob/59970344/nifi-docs/src/main/asciidoc/user-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/user-guide.adoc b/nifi-docs/src/main/asciidoc/user-guide.adoc index 1a3877e..7b414cf 100644 --- a/nifi-docs/src/main/asciidoc/user-guide.adoc +++ b/nifi-docs/src/main/asciidoc/user-guide.adoc @@ -17,7 +17,7 @@ = Apache NiFi User Guide Apache NiFi Team <[email protected]> :homepage: http://nifi.apache.org - +:linkattrs: == Introduction Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports @@ -147,7 +147,7 @@ When a DFM navigates to the UI for the first time, a blank canvas is provided on image::nifi-toolbar-components.png["NiFi Components Toolbar"] The Components Toolbar runs across the top left portion of your screen. It consists of the components you can drag onto the -canvas to build your dataflow. Each component is described in more detail in link:building-dataflow.html[Building a Dataflow]. +canvas to build your dataflow. Each component is described in more detail in <<building-dataflow>>. The Status Bar is under the Components Toolbar. The Status bar provides information about the number of threads that are currently active in the flow, the amount of data that currently exists in the flow, how many Remote Process Groups exist @@ -628,7 +628,7 @@ For example: * The string `0 20 14 ? * MON-FRI` indicates that you want to schedule the processor to run at 2:20 PM every Monday through Friday. * The string `0 15 10 ? * 6L 2011-2017` indicates that you want to schedule the processor to run at 10:15 AM, on the last Friday of every month, between 2011 and 2017. -For additional information and examples, see the http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html[Chron Trigger Tutorial] in the Quartz documentation. +For additional information and examples, see the link:http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html[Chron Trigger Tutorial^] in the Quartz documentation. Next, the Scheduling Tab provides a configuration option named `Concurrent tasks`. This controls how many threads the Processor will use. Said a different way, this controls how many FlowFiles should be processed by this Processor at the same time. Increasing @@ -1761,7 +1761,7 @@ will remove the selection. [[versioning_dataflow]] == Versioning a DataFlow -When NiFi is connected to a NiFi Registry, dataflows can be version controlled on the process group level. For more information about NiFi Registry usage and configuration, see the documentation at https://nifi.apache.org/docs/nifi-registry-docs/index.html. +When NiFi is connected to a NiFi Registry, dataflows can be version controlled on the process group level. For more information about NiFi Registry usage and configuration, see the documentation at link:https://nifi.apache.org/docs/nifi-registry-docs/index.html[https://nifi.apache.org/docs/nifi-registry-docs/index.html^]. === Connecting to a NiFi Registry To connect NiFi to a Registry, select Controller Settings from the Global Menu. @@ -1780,7 +1780,7 @@ Click "Add" to complete the registration. image::registry-client-added.png["Registry Client Added"] -NOTE: Versioned flows are stored and organized in registry buckets. Bucket Policies and Special Privileges configured by the registry administrator determine which buckets a user can import versioned flows from and which buckets a user can save versioned flows to. Information on Bucket Policies and Special Privileges can be found in the NiFi Registry User Guide (https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html). +NOTE: Versioned flows are stored and organized in registry buckets. Bucket Policies and Special Privileges configured by the registry administrator determine which buckets a user can import versioned flows from and which buckets a user can save versioned flows to. Information on Bucket Policies and Special Privileges can be found in the NiFi Registry User Guide (link:https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html[https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html^]). [[version_states]] === Version States @@ -2005,7 +2005,7 @@ filter the templates to see only those of interest, export, and delete Templates [[Export_Template]] ==== Exporting a Template -Once a Template has been created, it can be shared with others in the Template Management page (see <<Manage_Templates>>). +Once a Template has been created, it can be shared with others in the Template Management page. To export a Template, locate the Template in the table. The Filter in the top-right corner can be used to help find the appropriate Template if several are available. Then click the Export or Download button ( image:iconExport.png["Export"] @@ -2015,8 +2015,7 @@ into other instances of NiFi (see <<Import_Template>>). ==== Removing a Template -Once it is decided that a Template is no longer needed, it can be easily removed from the Template Management page -(see <<Manage_Templates>>). To delete a Template, locate it in the table (the Filter in the top-right corner +Once it is decided that a Template is no longer needed, it can be easily removed from the Template Management page. To delete a Template, locate it in the table (the Filter in the top-right corner may be used to find the appropriate Template if several are available) and click the Delete button ( image:iconDelete.png["Delete"] ). This will prompt for confirmation. After confirming the deletion, the Template will be removed from this table @@ -2247,10 +2246,10 @@ When switching between implementation "families" (i.e. `VolatileProvenanceReposi ============ * Switching between unencrypted and encrypted repositories -** If a user has an existing repository (`WriteAheadProvenanceRepository` only -- **not** `PersistentProvenanceRepository`) that is not encrypted and switches their configuration to use an encrypted repository, the application writes an error to the log but starts up. However, previous events are not accessible through the provenance query interface and new events will overwrite the existing events. The same behavior occurs if a user switches from an encrypted repository to an unencrypted repository. Automatic roll-over is a future effort (https://issues.apache.org/jira/browse/NIFI-3722[NIFI-3722]) but NiFi is not intended for long-term storage of provenance events so the impact should be minimal. There are two scenarios for roll-over: +** If a user has an existing repository (`WriteAheadProvenanceRepository` only -- **not** `PersistentProvenanceRepository`) that is not encrypted and switches their configuration to use an encrypted repository, the application writes an error to the log but starts up. However, previous events are not accessible through the provenance query interface and new events will overwrite the existing events. The same behavior occurs if a user switches from an encrypted repository to an unencrypted repository. Automatic roll-over is a future effort (link:https://issues.apache.org/jira/browse/NIFI-3722[NIFI-3722^]) but NiFi is not intended for long-term storage of provenance events so the impact should be minimal. There are two scenarios for roll-over: *** Encrypted -> unencrypted -- if the previous repository implementation was encrypted, these events should be handled seamlessly as long as the key provider available still has the keys used to encrypt the events (see **Key Rotation**) *** Unencrypted -> encrypted -- if the previous repository implementation was unencrypted, these events should be handled seamlessly as the previously recorded events simply need to be read with a plaintext schema record reader and then written back with the encrypted record writer -** There is also a future effort to provide a standalone tool in NiFi Toolkit to encrypt/decrypt an existing provenance repository to make the transition easier. The translation process could take a long time depending on the size of the existing repository, and being able to perform this task outside of application startup would be valuable (https://issues.apache.org/jira/browse/NIFI-3723[NIFI-3723]). +** There is also a future effort to provide a standalone tool in NiFi Toolkit to encrypt/decrypt an existing provenance repository to make the transition easier. The translation process could take a long time depending on the size of the existing repository, and being able to perform this task outside of application startup would be valuable (link:https://issues.apache.org/jira/browse/NIFI-3723[NIFI-3723^]). * Multiple repositories -- No additional effort or testing has been applied to multiple repositories at this time. It is possible/likely issues will occur with repositories on different physical devices. There is no option to provide a heterogenous environment (i.e. one encrypted, one plaintext repository). * Corruption -- when a disk is filled or corrupted, there have been reported issues with the repository becoming corrupted and recovery steps are necessary. This is likely to continue to be an issue with the encrypted repository, although still limited in scope to individual records (i.e. an entire repository file won't be irrecoverable due to the encryption).
