Modified: nifi/site/trunk/docs/nifi-docs/html/expression-language-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/expression-language-guide.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/expression-language-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/expression-language-guide.html Thu Jan 23 03:48:17 2020 @@ -506,6 +506,8 @@ body.book #toc,body.book #preamble,body. <li><a href="expression-language-guide.html#replacenull">replaceNull</a></li> <li><a href="expression-language-guide.html#replaceempty">replaceEmpty</a></li> <li><a href="expression-language-guide.html#length">length</a></li> +<li><a href="expression-language-guide.html#evaluateelstring">evaluateELString</a></li> +<li><a href="expression-language-guide.html#repeat">repeat</a></li> </ul> </li> <li><a href="expression-language-guide.html#encode">Encode/Decode Functions</a> @@ -2282,6 +2284,98 @@ Expressions will provide the following r will return 0.</p> </div> </div> +<div class="sect2 function"> +<h3 id="evaluateelstring"><a class="anchor" href="expression-language-guide.html#evaluateelstring"></a>evaluateELString</h3> +<div class="paragraph"> +<p><strong>Description</strong>: <span class="description">This function evaluates the Expression Language inside the variable value.</span></p> +</div> +<div class="paragraph"> +<p><strong>Subject Type</strong>: <span class="subject">String</span></p> +</div> +<div class="paragraph"> +<p><strong>Arguments</strong>: No arguments</p> +</div> +<div class="paragraph"> +<p><strong>Return Type</strong>: <span class="returnType">String</span></p> +</div> +<div class="paragraph"> +<p><strong>Examples</strong>: If one of the registry variable named "query" has value "SELECT * FROM TABLE WHERE ID = ${id}" + and the value of the "id" field, we are getting from the flowfile attributes (i.e. id=20) + then the Expression <code>${query:evaluateELString()}</code> will return SELECT * FROM TABLE WHERE ID = 20</p> +</div> +</div> +<div class="sect2 function"> +<h3 id="repeat"><a class="anchor" href="expression-language-guide.html#repeat"></a>repeat</h3> +<div class="paragraph"> +<p><strong>Description</strong>: +<span class="description">Returns a string that is the Subject repeated a random number of times between <em>min repeats</em> and +<em>max repeats</em>. If <em>max repeats</em> is not supplied, it will return the Subject repeated exactly <em>min repeats</em> times.</span></p> +</div> +<div class="paragraph"> +<p><span class="description">The <em>min repeats</em> and <em>max repeats</em> must be positive numbers, with <em>max repeats</em> greater than or equal +to <em>min repeats</em></span></p> +</div> +<div class="paragraph"> +<p><span class="description">If either <em>min repeats</em> or <em>max repeats</em> is not a number, this function call will result +in an error.</span></p> +</div> +<div class="paragraph"> +<p><strong>Subject Type</strong>: <span class="subject">String</span></p> +</div> +<div class="paragraph"> +<p><strong>Arguments</strong>:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p><span class="argName"><em>min repeats</em></span> : <span class="argDesc">The minimum number (inclusive) of times to repeat the subject, or the exact number +of times to repeat the subject if <em>max repeats</em> is not supplied.</span></p> +</li> +<li> +<p><span class="argName"><em>max repeats</em></span> : <span class="argDesc">The maximum number (inclusive) of times to repeat the subject.</span></p> +</li> +</ul> +</div> +<div class="paragraph"> +<p><strong>Return Type</strong>: <span class="returnType">String</span></p> +</div> +<div class="paragraph"> +<p><strong>Examples</strong>:</p> +</div> +<div class="paragraph"> +<p>If we have an attribute named "str" with the value "abc", +then the following Expressions will result in the following values:</p> +</div> +<table class="tableblock frame-all grid-all spread"> +<caption class="title">Table 14. Repeat Examples</caption> +<colgroup> +<col style="width: 50%;"> +<col style="width: 50%;"> +</colgroup> +<tbody> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">Expression</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Value</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>${str:repeat(1)}</code></p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>abc</code></p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>${str:repeat(2)}</code></p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>abcabc</code></p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>${str:repeat(1,2)}</code></p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>abc</code> or <code>abcabc</code> (at random)</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>${str:repeat( ${str:length()} )}</code></p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock"><code>abc</code> or <code>abcabc</code> or <code>abcabcabc</code> (at random)</p></td> +</tr> +</tbody> +</table> +</div> </div> </div> <div class="sect1"> @@ -2723,7 +2817,7 @@ Expressions will provide the following r Expressions will provide the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 14. find Examples</caption> +<caption class="title">Table 15. find Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -2777,7 +2871,7 @@ Expressions will provide the following r Expressions will provide the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 15. matches Examples</caption> +<caption class="title">Table 16. matches Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -2832,7 +2926,7 @@ Expressions will provide the following r Expressions will provide the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 16. indexOf Examples</caption> +<caption class="title">Table 17. indexOf Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -2891,7 +2985,7 @@ Expressions will provide the following r Expressions will provide the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 17. lastIndexOf Examples</caption> +<caption class="title">Table 18. lastIndexOf Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -2971,7 +3065,7 @@ Expressions will provide the following r </div> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 18. jsonPath Examples</caption> +<caption class="title">Table 19. jsonPath Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3059,7 +3153,7 @@ form of the updated JSON.</span></p> </div> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 19. jsonPathDelete Examples</caption> +<caption class="title">Table 20. jsonPathDelete Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3142,7 +3236,7 @@ path is not in the JSON, the operation r </div> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 20. jsonPathAdd Examples</caption> +<caption class="title">Table 21. jsonPathAdd Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3224,7 +3318,7 @@ form of the updated JSON.</span></p> </div> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 21. jsonPathSet Examples</caption> +<caption class="title">Table 22. jsonPathSet Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3308,7 +3402,7 @@ form of the updated JSON.</span></p> </div> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 22. jsonPathPut Examples</caption> +<caption class="title">Table 23. jsonPathPut Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3499,7 +3593,7 @@ Divide. This is to preserve backwards co the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 23. toRadix Examples</caption> +<caption class="title">Table 24. toRadix Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3566,7 +3660,7 @@ Divide. This is to preserve backwards co the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 24. toRadix Examples</caption> +<caption class="title">Table 25. toRadix Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3695,7 +3789,7 @@ In order to run the correct method, the the following results:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 25. format Examples</caption> +<caption class="title">Table 26. format Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -3797,7 +3891,7 @@ was run then it would output <code>14538 manipulate the value.</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 26. now Examples</caption> +<caption class="title">Table 27. now Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4109,7 +4203,7 @@ provides several functions for evaluatin and "filename" contains "file.txt" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 27. anyAttribute Examples</caption> +<caption class="title">Table 28. anyAttribute Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4159,7 +4253,7 @@ provides several functions for evaluatin and "filename" contains "file.txt" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 28. allAttributes Example</caption> +<caption class="title">Table 29. allAttributes Example</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4210,7 +4304,7 @@ provides several functions for evaluatin and "filename" contains "file.txt" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 29. anyMatchingAttribute Example</caption> +<caption class="title">Table 30. anyMatchingAttribute Example</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4258,7 +4352,7 @@ provides several functions for evaluatin and "filename" contains "file.txt" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 30. anyMatchingAttributes Examples</caption> +<caption class="title">Table 31. anyMatchingAttributes Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4315,7 +4409,7 @@ though it does not have to be.</span></p consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 31. anyDelineatedValue Examples</caption> +<caption class="title">Table 32. anyDelineatedValue Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4368,7 +4462,7 @@ an embedded Expression, though it does n consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 32. allDelineatedValues Examples</caption> +<caption class="title">Table 33. allDelineatedValues Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4425,7 +4519,7 @@ an embedded Expression, though it does n and "filename" contains "file.txt" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 33. join Examples</caption> +<caption class="title">Table 34. join Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4468,7 +4562,7 @@ an embedded Expression, though it does n and "number_list" contains "1,2,3,4,5" consider the following examples:</p> </div> <table class="tableblock frame-all grid-all spread"> -<caption class="title">Table 34. count Examples</caption> +<caption class="title">Table 35. count Examples</caption> <colgroup> <col style="width: 50%;"> <col style="width: 50%;"> @@ -4506,7 +4600,7 @@ an embedded Expression, though it does n </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body>
Modified: nifi/site/trunk/docs/nifi-docs/html/getting-started.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/getting-started.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/getting-started.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/getting-started.html Thu Jan 23 03:48:17 2020 @@ -1623,7 +1623,7 @@ work back to the Apache NiFi community s </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body> Added: nifi/site/trunk/docs/nifi-docs/html/images/encrypted-flowfile-hex.png URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/images/encrypted-flowfile-hex.png?rev=1873052&view=auto ============================================================================== Binary file - no diff available. Propchange: nifi/site/trunk/docs/nifi-docs/html/images/encrypted-flowfile-hex.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Modified: nifi/site/trunk/docs/nifi-docs/html/nifi-in-depth.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/nifi-in-depth.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/nifi-in-depth.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/nifi-in-depth.html Thu Jan 23 03:48:17 2020 @@ -896,7 +896,7 @@ This section relies heavily on informati </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:05 -05:00 </div> </div> </body> Modified: nifi/site/trunk/docs/nifi-docs/html/overview.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/overview.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/overview.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/overview.html Thu Jan 23 03:48:17 2020 @@ -917,7 +917,7 @@ Scalable Internet Services [online]. Re </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body> Modified: nifi/site/trunk/docs/nifi-docs/html/record-path-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/record-path-guide.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/record-path-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/record-path-guide.html Thu Jan 23 03:48:17 2020 @@ -2019,7 +2019,7 @@ only of white space (spaces, tabs, carri </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body> Modified: nifi/site/trunk/docs/nifi-docs/html/toolkit-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/toolkit-guide.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/toolkit-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/toolkit-guide.html Thu Jan 23 03:48:17 2020 @@ -1884,7 +1884,7 @@ JKS keystores and truststores are recomm </div> <div class="literalblock"> <div class="content"> -<pre>./bin/tls-toolkit.sh standlone -h</pre> +<pre>./bin/tls-toolkit.sh standalone -h</pre> </div> </div> <div class="paragraph"> @@ -2940,7 +2940,7 @@ As of NiFi 1.10.x, because of an upgrade </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body> Modified: nifi/site/trunk/docs/nifi-docs/html/user-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/user-guide.html?rev=1873052&r1=1873051&r2=1873052&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/user-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/user-guide.html Thu Jan 23 03:48:17 2020 @@ -627,14 +627,34 @@ body.book #toc,body.book #preamble,body. <li><a href="user-guide.html#potential-issues">Potential Issues</a></li> </ul> </li> +</ul> +</li> <li><a href="user-guide.html#encrypted-content">Encrypted Content Repository</a> -<ul class="sectlevel3"> +<ul class="sectlevel2"> <li><a href="user-guide.html#what-is-it-2">What is it?</a></li> -<li><a href="user-guide.html#how-does-it-work-2">How does it work?</a></li> +<li><a href="user-guide.html#how-does-it-work-2">How does it work?</a> +<ul class="sectlevel3"> +<li><a href="user-guide.html#statickeyprovider-2">StaticKeyProvider</a></li> +<li><a href="user-guide.html#filebasedkeyprovider-2">FileBasedKeyProvider</a></li> +<li><a href="user-guide.html#content-repository-key-rotation">Key Rotation</a></li> +</ul> +</li> <li><a href="user-guide.html#writing-and-reading-content-claims">Writing and Reading Content Claims</a></li> <li><a href="user-guide.html#potential-issues-2">Potential Issues</a></li> </ul> </li> +<li><a href="user-guide.html#encrypted-flowfile">Encrypted FlowFile Repository</a> +<ul class="sectlevel2"> +<li><a href="user-guide.html#what-is-it-3">What is it?</a></li> +<li><a href="user-guide.html#how-does-it-work-3">How does it work?</a> +<ul class="sectlevel3"> +<li><a href="user-guide.html#statickeyprovider-3">StaticKeyProvider</a></li> +<li><a href="user-guide.html#filebasedkeyprovider-3">FileBasedKeyProvider</a></li> +<li><a href="user-guide.html#flowfile-repository-key-rotation">Key Rotation</a></li> +</ul> +</li> +<li><a href="user-guide.html#writing-and-reading-flowfiles">Writing and Reading FlowFiles</a></li> +<li><a href="user-guide.html#potential-issues-3">Potential Issues</a></li> </ul> </li> <li><a href="user-guide.html#experimental_warning">Experimental Warning</a></li> @@ -2880,7 +2900,9 @@ These defaults can be changed by modifyi <p><strong>Partition by attribute</strong>: Determines which node to send a given FlowFile to based on the value of a user-specified FlowFile Attribute. All FlowFiles that have the same value for the Attribute will be sent to the same node in the cluster. If the destination node is disconnected from the cluster or if unable to communicate, the data does not fail over to another node. The data will queue, waiting for the node to be available again. Additionally, if a node joins or leaves the cluster necessitating a rebalance of the data, consistent hashing is applied to avoid having to redistribute all of the data.</p> </li> <li> -<p><strong>Round robin</strong>: FlowFiles will be distributed to nodes in the cluster in a round-robin fashion. If a node is disconnected from the cluster or if unable to communicate with a node, the data that is queued for that node will be automatically redistributed to another node(s).</p> +<p><strong>Round robin</strong>: FlowFiles will be distributed to nodes in the cluster in a round-robin fashion. If a node is disconnected from the cluster or if unable to communicate with a node, the data that is +queued for that node will be automatically redistributed to another node(s). If a node is not able to receive the data as fast other nodes in the cluster, the node may also be skipped for one or +more iterations in order to maximize throughput of data distribution across the cluster.</p> </li> <li> <p><strong>Single node</strong>: All FlowFiles will be sent to a single node in the cluster. Which node they are sent to is not configurable. If the node is disconnected from the cluster or if unable to communicate with the node, the data that is queued for that node will remain queued until the node is available again.</p> @@ -5327,7 +5349,7 @@ key5=c6FzfnKm7UR7xqI2NFpZ+fEKBfSU7+1NvRw </div> </div> <div class="sect4"> -<h5 id="key-rotation"><a class="anchor" href="user-guide.html#key-rotation"></a>Key Rotation</h5> +<h5 id="provenance-repository-key-rotation"><a class="anchor" href="user-guide.html#provenance-repository-key-rotation"></a>Key Rotation</h5> <div class="paragraph"> <p>Simply update <em>nifi.properties</em> to reference a new key ID in <code>nifi.provenance.repository.encryption.key.id</code>. Previously-encrypted events can still be decrypted as long as that key is still available in the key definition file or <code>nifi.provenance.repository.encryption.key.id.<OldKeyID></code> as the key ID is serialized alongside the encrypted record.</p> </div> @@ -5376,7 +5398,7 @@ key5=c6FzfnKm7UR7xqI2NFpZ+fEKBfSU7+1NvRw <div class="ulist"> <ul> <li> -<p>Encrypted → unencrypted — if the previous repository implementation was encrypted, these events should be handled seamlessly as long as the key provider available still has the keys used to encrypt the events (see <strong>Key Rotation</strong>)</p> +<p>Encrypted → unencrypted — if the previous repository implementation was encrypted, these events should be handled seamlessly as long as the key provider available still has the keys used to encrypt the events (see <a href="user-guide.html#provenance-repository-key-rotation">Key Rotation</a>)</p> </li> <li> <p>Unencrypted → encrypted — if the previous repository implementation was unencrypted, these events should be handled seamlessly as the previously recorded events simply need to be read with a plaintext schema record reader and then written back with the encrypted record writer</p> @@ -5400,10 +5422,13 @@ key5=c6FzfnKm7UR7xqI2NFpZ+fEKBfSU7+1NvRw </div> </div> </div> -<div class="sect2"> -<h3 id="encrypted-content"><a class="anchor" href="user-guide.html#encrypted-content"></a>Encrypted Content Repository</h3> +</div> +</div> +<div class="sect1"> +<h2 id="encrypted-content"><a class="anchor" href="user-guide.html#encrypted-content"></a>Encrypted Content Repository</h2> +<div class="sectionbody"> <div class="paragraph"> -<p>While OS-level access control can offer some security over the flowfile content data written to the disk in a repository, there are scenarios where the data may be sensitive, compliance and regulatory requirements exist, or NiFi is running on hardware not under the direct control of the organization (cloud, etc.). In this case, the content repository allows for all data to be encrypted before being persisted to the disk.</p> +<p>While OS-level access control can offer some security over the flowfile content data written to the disk in a repository, there are scenarios where the data may be sensitive, compliance and regulatory requirements exist, or NiFi is running on hardware not under the direct control of the organization (cloud, etc.). In this case, the content repository allows for all data to be encrypted before being persisted to the disk. For more information on the internal workings of the content repository, see <a href="nifi-in-depth.html#content-repository">NiFi In-Depth - Content Repository</a>.</p> </div> <div class="admonitionblock warning"> <table> @@ -5435,22 +5460,22 @@ key5=c6FzfnKm7UR7xqI2NFpZ+fEKBfSU7+1NvRw </tr> </table> </div> -<div class="sect3"> -<h4 id="what-is-it-2"><a class="anchor" href="user-guide.html#what-is-it-2"></a>What is it?</h4> +<div class="sect2"> +<h3 id="what-is-it-2"><a class="anchor" href="user-guide.html#what-is-it-2"></a>What is it?</h3> <div class="paragraph"> <p>The <code>EncryptedFileSystemRepository</code> is a new implementation of the content repository which encrypts all content data before it is written to the repository. This allows for storage on systems where OS-level access controls are not sufficient to protect the data while still allowing querying and access to the data through the NiFi UI/API.</p> </div> </div> -<div class="sect3"> -<h4 id="how-does-it-work-2"><a class="anchor" href="user-guide.html#how-does-it-work-2"></a>How does it work?</h4> +<div class="sect2"> +<h3 id="how-does-it-work-2"><a class="anchor" href="user-guide.html#how-does-it-work-2"></a>How does it work?</h3> <div class="paragraph"> <p>The <code>FileSystemRepository</code> was introduced in NiFi 0.2.1 and provided the only persistent content repository implementation. The encrypted version wraps that implementation with functionality to return to the <code>Session</code> (usually <code>StandardProcessSession</code>) a special <code>OutputStream</code>/<code>InputStream</code> which encrypt and decrypt the serialized bytes respectively. This allows all components to continue interacting with the content repository interface in the same way as before and continue operating on content data in a streaming manner, without requiring any changes to handle the data protection.</p> </div> <div class="paragraph"> <p>The fully qualified class <code>org.apache.nifi.content.EncryptedFileSystemRepository</code> is specified as the content repository implementation in <em>nifi.properties</em> as the value of <code>nifi.content.repository.implementation</code>. In addition, <a href="administration-guide.html#encrypted-file-system-content-repository-properties">new properties</a> must be populated to allow successful initialization.</p> </div> -<div class="sect4"> -<h5 id="statickeyprovider-2"><a class="anchor" href="user-guide.html#statickeyprovider-2"></a>StaticKeyProvider</h5> +<div class="sect3"> +<h4 id="statickeyprovider-2"><a class="anchor" href="user-guide.html#statickeyprovider-2"></a>StaticKeyProvider</h4> <div class="paragraph"> <p>The <code>StaticKeyProvider</code> implementation defines keys directly in <em>nifi.properties</em>. Individual keys are provided in hexadecimal encoding. The keys can also be encrypted like any other sensitive property in <em>nifi.properties</em> using the <a href="administration-guide.html#encrypt-config_tool"><code>./encrypt-config.sh</code></a> tool in the NiFi Toolkit.</p> </div> @@ -5466,8 +5491,8 @@ nifi.content.repository.encryption.key.i </div> </div> </div> -<div class="sect4"> -<h5 id="filebasedkeyprovider-2"><a class="anchor" href="user-guide.html#filebasedkeyprovider-2"></a>FileBasedKeyProvider</h5> +<div class="sect3"> +<h4 id="filebasedkeyprovider-2"><a class="anchor" href="user-guide.html#filebasedkeyprovider-2"></a>FileBasedKeyProvider</h4> <div class="paragraph"> <p>The <code>FileBasedKeyProvider</code> implementation reads from an encrypted definition file of the format:</p> </div> @@ -5494,15 +5519,15 @@ over the key material.</p> </div> </div> </div> -<div class="sect4"> -<h5 id="key-rotation-2"><a class="anchor" href="user-guide.html#key-rotation-2"></a>Key Rotation</h5> +<div class="sect3"> +<h4 id="content-repository-key-rotation"><a class="anchor" href="user-guide.html#content-repository-key-rotation"></a>Key Rotation</h4> <div class="paragraph"> <p>Simply update <em>nifi.properties</em> to reference a new key ID in <code>nifi.content.repository.encryption.key.id</code>. Previously-encrypted content claims can still be decrypted as long as that key is still available in the key definition file or <code>nifi.content.repository.encryption.key.id.<OldKeyID></code> as the key ID is serialized alongside the encrypted content.</p> </div> </div> </div> -<div class="sect3"> -<h4 id="writing-and-reading-content-claims"><a class="anchor" href="user-guide.html#writing-and-reading-content-claims"></a>Writing and Reading Content Claims</h4> +<div class="sect2"> +<h3 id="writing-and-reading-content-claims"><a class="anchor" href="user-guide.html#writing-and-reading-content-claims"></a>Writing and Reading Content Claims</h3> <div class="paragraph"> <p>Once the repository is initialized, all content claim write operations are serialized using <code>RepositoryObjectStreamEncryptor</code> (the only currently existing implementation is <code>RepositoryObjectAESCTREncryptor</code>) to an <code>OutputStream</code>. The actual implementation is <code>EncryptedContentRepositoryOutputStream</code>, which encrypts the data written by the component via <code>StandardProcessSession</code> inline and the encryption metadata (<code>keyId</code>, <code>algorithm</code>, <code>version</code>, <code>IV</code>) is serialized and prepended. The complete <code>OutputStream</code> is then written to the repository on disk as normal.</p> </div> @@ -5516,8 +5541,8 @@ over the key material.</p> <p>Within the NiFi UI/API, there is no detectable difference between an encrypted and unencrypted content repository. The Provenance Query operations to view content work as expected with no change to the process.</p> </div> </div> -<div class="sect3"> -<h4 id="potential-issues-2"><a class="anchor" href="user-guide.html#potential-issues-2"></a>Potential Issues</h4> +<div class="sect2"> +<h3 id="potential-issues-2"><a class="anchor" href="user-guide.html#potential-issues-2"></a>Potential Issues</h3> <div class="admonitionblock warning"> <table> <tr> @@ -5544,7 +5569,7 @@ over the key material.</p> <div class="ulist"> <ul> <li> -<p>Encrypted → unencrypted — if the previous repository implementation was encrypted, these claims should be handled seamlessly as long as the key provider available still has the keys used to encrypt the claims (see <strong>Key Rotation</strong>)</p> +<p>Encrypted → unencrypted — if the previous repository implementation was encrypted, these claims should be handled seamlessly as long as the key provider available still has the keys used to encrypt the claims (see <a href="user-guide.html#content-repository-key-rotation">Key Rotation</a>)</p> </li> <li> <p>Unencrypted → encrypted — if the previous repository implementation was unencrypted, these claims should be handled seamlessly as the previously written claims simply need to be read with a plaintext <code>InputStream</code> and then be written back with the <code>EncryptedContentRepositoryOutputStream</code></p> @@ -5569,6 +5594,168 @@ over the key material.</p> </div> </div> </div> +<div class="sect1"> +<h2 id="encrypted-flowfile"><a class="anchor" href="user-guide.html#encrypted-flowfile"></a>Encrypted FlowFile Repository</h2> +<div class="sectionbody"> +<div class="paragraph"> +<p>While OS-level access control can offer some security over the flowfile attribute and content claim data written to the disk in a repository, there are scenarios where the data may be sensitive, compliance and regulatory requirements exist, or NiFi is running on hardware not under the direct control of the organization (cloud, etc.). In this case, the flowfile repository allows for all data to be encrypted before being persisted to the disk. For more information on the internal workings of the flowfile repository, see <a href="nifi-in-depth.html#flowfile-repository">NiFi In-Depth - FlowFile Repository</a>.</p> +</div> +<div class="admonitionblock warning"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-warning" title="Warning"></i> +</td> +<td class="content"> +<div class="title">Experimental</div> +<div class="paragraph"> +<p>This implementation is marked <a href="user-guide.html#experimental_warning"><strong>experimental</strong></a> as of Apache NiFi 1.11.0 (January 2020). The API, configuration, and internal behavior may change without warning, and such changes may occur during a minor release. Use at your own risk.</p> +</div> +</td> +</tr> +</table> +</div> +<div class="admonitionblock warning"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-warning" title="Warning"></i> +</td> +<td class="content"> +<div class="title">Performance</div> +<div class="paragraph"> +<p>The current implementation of the encrypted flowfile repository intercepts the serialization of flowfile record data via the <code>EncryptedSchemaRepositoryRecordSerde</code> and uses the <code>AES/GCM</code> algorithm, which is fairly performant on commodity hardware. This use of an authenticated encryption algorithm (AEAD) block cipher (because the content length is limited and known a priori) is the same as the <a href="user-guide.html#encrypted-provenance">Encrypted Provenance Repository</a>, but differs from the unauthenticated stream cipher used in the <a href="user-guide.html#encrypted-content">Encrypted Content Repository</a>. In low volume flowfile scenarios, the added cost will be minimal. However, administrators should perform their own risk assessment and performance analysis and decide how to move forward. Switching back and forth between encrypted/unencrypted implementations is not recommended at this time.</p> +</div> +</td> +</tr> +</table> +</div> +<div class="sect2"> +<h3 id="what-is-it-3"><a class="anchor" href="user-guide.html#what-is-it-3"></a>What is it?</h3> +<div class="paragraph"> +<p>The <code>EncryptedSequentialAccessWriteAheadLog</code> is a new implementation of the flowfile write-ahead log which encrypts all flowfile attribute data before it is written to the repository. This allows for storage on systems where OS-level access controls are not sufficient to protect the data while still allowing querying and access to the data through the NiFi UI/API.</p> +</div> +</div> +<div class="sect2"> +<h3 id="how-does-it-work-3"><a class="anchor" href="user-guide.html#how-does-it-work-3"></a>How does it work?</h3> +<div class="paragraph"> +<p>The <code>SequentialAccessWriteAheadLog</code> was introduced in NiFi 1.6.0 and provided a faster flowfile repository implementation. The encrypted version wraps that implementation with functionality to transparently encrypt and decrypt the serialized <code>RepositoryRecord</code> objects during file system interaction. During all writes to disk (swapping, snapshotting, journaling, and checkpointing), the flowfile containers are serialized to bytes based on a schema, and this serialized form is encrypted before writing. This allows the snapshot handler to continue interacting with the flowfile repository interface in the same way as before and continue operating on flowfile data in a random access manner, without requiring any changes to handle the data protection.</p> +</div> +<div class="paragraph"> +<p>The fully qualified class <code>org.apache.nifi.wali.EncryptedSequentialAccessWriteAheadLog</code> is specified as the flowfile repository write-ahead log implementation in <em>nifi.properties</em> as the value of <code>nifi.flowfile.repository.wal.implementation</code>. In addition, <a href="administration-guide.html#encrypted-write-ahead-flowfile-repository-properties">new properties</a> must be populated to allow successful initialization.</p> +</div> +<div class="sect3"> +<h4 id="statickeyprovider-3"><a class="anchor" href="user-guide.html#statickeyprovider-3"></a>StaticKeyProvider</h4> +<div class="paragraph"> +<p>The <code>StaticKeyProvider</code> implementation defines keys directly in <em>nifi.properties</em>. Individual keys are provided in hexadecimal encoding. The keys can also be encrypted like any other sensitive property in <em>nifi.properties</em> using the <a href="administration-guide.html#encrypt-config_tool"><code>./encrypt-config.sh</code></a> tool in the NiFi Toolkit.</p> +</div> +<div class="paragraph"> +<p>The following configuration section would result in a key provider with two available keys, "Key1" (active) and "AnotherKey".</p> +</div> +<div class="literalblock"> +<div class="content"> +<pre>nifi.flowfile.repository.encryption.key.provider.implementation=org.apache.nifi.security.kms.StaticKeyProvider +nifi.flowfile.repository.encryption.key.id=Key1 +nifi.flowfile.repository.encryption.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 +nifi.flowfile.repository.encryption.key.id.AnotherKey=0101010101010101010101010101010101010101010101010101010101010101</pre> +</div> +</div> +</div> +<div class="sect3"> +<h4 id="filebasedkeyprovider-3"><a class="anchor" href="user-guide.html#filebasedkeyprovider-3"></a>FileBasedKeyProvider</h4> +<div class="paragraph"> +<p>The <code>FileBasedKeyProvider</code> implementation reads from an encrypted definition file of the format:</p> +</div> +<div class="literalblock"> +<div class="content"> +<pre>key1=NGCpDpxBZNN0DBodz0p1SDbTjC2FG5kp1pCmdUKJlxxtcMSo6GC4fMlTyy1mPeKOxzLut3DRX+51j6PCO5SznA== +key2=GYxPbMMDbnraXs09eGJudAM5jTvVYp05XtImkAg4JY4rIbmHOiVUUI6OeOf7ZW+hH42jtPgNW9pSkkQ9HWY/vQ== +key3=SFe11xuz7J89Y/IQ7YbJPOL0/YKZRFL/VUxJgEHxxlXpd/8ELA7wwN59K1KTr3BURCcFP5YGmwrSKfr4OE4Vlg== +key4=kZprfcTSTH69UuOU3jMkZfrtiVR/eqWmmbdku3bQcUJ/+UToecNB5lzOVEMBChyEXppyXXC35Wa6GEXFK6PMKw== +key5=c6FzfnKm7UR7xqI2NFpZ+fEKBfSU7+1NvRw+XWQ9U39MONWqk5gvoyOCdFR1kUgeg46jrN5dGXk13sRqE0GETQ==</pre> +</div> +</div> +<div class="paragraph"> +<p>Each line defines a key ID and then the Base64-encoded cipher text of a 16 byte IV and wrapped AES-128, AES-192, or AES-256 key depending on the JCE policies available. The individual keys are wrapped by AES/GCM encryption using the <strong>master key</strong> defined by <code>nifi.bootstrap.sensitive.key</code> in <em>conf/bootstrap.conf</em>.</p> +</div> +</div> +<div class="sect3"> +<h4 id="flowfile-repository-key-rotation"><a class="anchor" href="user-guide.html#flowfile-repository-key-rotation"></a>Key Rotation</h4> +<div class="paragraph"> +<p>Simply update <em>nifi.properties</em> to reference a new key ID in <code>nifi.flowfile.repository.encryption.key.id</code>. Previously-encrypted flowfile records can still be decrypted as long as that key is still available in the key definition file or <code>nifi.flowfile.repository.encryption.key.id.<OldKeyID></code> as the key ID is serialized alongside the encrypted record.</p> +</div> +</div> +</div> +<div class="sect2"> +<h3 id="writing-and-reading-flowfiles"><a class="anchor" href="user-guide.html#writing-and-reading-flowfiles"></a>Writing and Reading FlowFiles</h3> +<div class="paragraph"> +<p>Once the repository is initialized, all flowfile record write operations are serialized using <code>RepositoryObjectBlockEncryptor</code> (the only currently existing implementation is <code>RepositoryObjectAESGCMEncryptor</code>) to the provided <code>DataOutputStream</code>. The original stream is swapped with a temporary wrapped stream, which encrypts the data written by the wrapped serializer/deserializer via <code>EncryptedSchemaRepositoryRecordSerde</code> inline and the encryption metadata (<code>keyId</code>, <code>algorithm</code>, <code>version</code>, <code>IV</code>, <code>cipherByteLength</code>) is serialized and prepended. The complete length and encrypted bytes are then written to the original <code>DataOutputStream</code> on disk as normal.</p> +</div> +<div class="paragraph"> +<p><span class="image"><img src="images/encrypted-flowfile-hex.png" alt="Encrypted flowfile repository journal file on disk"></span></p> +</div> +<div class="paragraph"> +<p>On flowfile record read, the process is reversed. The encryption metadata (<code>RepositoryObjectEncryptionMetadata</code>) is parsed and used to decrypt the serialized bytes, which are then deserialized into a <code>DataInputStream</code> object.</p> +</div> +<div class="paragraph"> +<p>During swaps and recoveries, the flowfile records are deserialized and reserialized, so if the active key has been changed, the flowfile records will be re-encrypted with the new active key.</p> +</div> +<div class="paragraph"> +<p>Within the NiFi UI/API, there is no detectable difference between an encrypted and unencrypted flowfile repository. All framework interactions with flowfiles work as expected with no change to the process.</p> +</div> +</div> +<div class="sect2"> +<h3 id="potential-issues-3"><a class="anchor" href="user-guide.html#potential-issues-3"></a>Potential Issues</h3> +<div class="admonitionblock warning"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-warning" title="Warning"></i> +</td> +<td class="content"> +<div class="title">Switching Implementations</div> +<div class="paragraph"> +<p>It is not recommended to switch between any implementation other than <code>SequentialAccessWriteAheadLog</code> and the <code>EncryptedSequentialAccessWriteAheadLog</code>. To migrate from a different provider, first migrate to the plaintext sequential log, allow NiFi to automatically recover the flowfiles, then stop NiFi and change the configuration to enable encryption. NiFi will automatically recover the plaintext flowfiles from the repository, and begin encrypting them on subsequent writes.</p> +</div> +</td> +</tr> +</table> +</div> +<div class="ulist"> +<ul> +<li> +<p>Switching between unencrypted and encrypted repositories</p> +<div class="ulist"> +<ul> +<li> +<p>If a user has an existing write-ahead repository (<code>WriteAheadFlowFileRepository</code>) that is not encrypted (uses the <code>SequentialAccessWriteAheadLog</code>) and switches their configuration to use an encrypted repository, the application handles this and all flowfile records will be recovered on startup. Future writes (including re-serialization of these same flowfiles) will be encrypted. If a user switches from an encrypted repository to an unencrypted repository, the flowfiles cannot be recovered, and it is recommended to delete the existing flowfile repository before switching in this direction. Automatic roll-over is a future effort (<a href="https://issues.apache.org/jira/browse/NIFI-6994"; target="_blank">NIFI-6994</a>) but NiFi is not intended for long-term storage of flowfile records so the impact should be minimal. There are two scenarios for roll-over:</p> +<div class="ulist"> +<ul> +<li> +<p>Encrypted → unencrypted — if the previous repository implementation was encrypted, these records should be handled seamlessly as long as the key provider available still has the keys used to encrypt the claims (see <a href="user-guide.html#flowfile-repository-key-rotation">Key Rotation</a>)</p> +</li> +<li> +<p>Unencrypted → encrypted — currently handled seamlesssly for <code>SequentialAccessWriteAheadLog</code> but there are other initial implementations which could be handled</p> +</li> +</ul> +</div> +</li> +<li> +<p>There is also a future effort to provide a standalone tool in NiFi Toolkit to encrypt/decrypt an existing flowfile repository to make the transition easier. The translation process could take a long time depending on the size of the existing repository, and being able to perform this task outside of application startup would be valuable (<a href="https://issues.apache.org/jira/browse/NIFI-6994"; target="_blank">NIFI-6994</a>).</p> +</li> +</ul> +</div> +</li> +<li> +<p>Multiple repositories — No additional effort or testing has been applied to multiple repositories at this time. Current implementations of the flowfile repository allow only for one repository, though it can reside across multiple volumes and partitions. It is possible/likely issues will occur with repositories on different physical devices. There is no option to provide a heterogenous environment (i.e. one encrypted, one plaintext partition/directory).</p> +</li> +<li> +<p>Corruption — when a disk is filled or corrupted, there have been reported issues with the repository becoming corrupted and recovery steps are necessary. This is likely to continue to be an issue with the encrypted repository, although still limited in scope to individual records (i.e. an entire repository file won’t be irrecoverable due to the encryption). It is important for the continued operation of NiFi to ensure that the disk storing the flowfile repository does not run out of available space.</p> +</li> +</ul> +</div> +</div> +</div> </div> <div class="sect1"> <h2 id="experimental_warning"><a class="anchor" href="user-guide.html#experimental_warning"></a>Experimental Warning</h2> @@ -5622,7 +5809,7 @@ the <a href="administration-guide.html"> </div> <div id="footer"> <div id="footer-text"> -Last updated 2019-10-29 09:44:15 -07:00 +Last updated 2020-01-19 14:56:06 -05:00 </div> </div> </body>
