Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/additionalDetails.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/additionalDetails.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1,129 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>TailFile</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <h3>Modes</h3> + <p> + This processor is used to tail a file or multiple files according to multiple modes. The + mode to choose depends of the logging pattern followed by the file(s) to tail. In any case, if there + is a rolling pattern, the rolling files must be plain text files (compression is not supported at + the moment). + </p> + <ul> + <li><b>Single file</b>: the processor will tail the file with the path given in 'File(s) to tail' property.</li> + <li><b>Multiple files</b>: the processor will look for files into the 'Base directory'. It will look for file recursively + according to the 'Recursive lookup' property and will tail all the files matching the regular expression + provided in the 'File(s) to tail' property.</li> + </ul> + <h3>Rolling filename pattern</h3> + <p> + In case the 'Rolling filename pattern' property is used, when the processor detects that the file to tail has rolled over, the + processor will look for possible missing messages in the rolled file. To do so, the processor will use the pattern to find the + rolling files in the same directory as the file to tail. + </p> + <p> + In order to keep this property available in the 'Multiple files' mode when multiples files to tail are in the same directory, + it is possible to use the ${filename} tag to reference the name (without extension) of the file to tail. For example, if we have: + </p> + <p> + <code> + /my/path/directory/my-app.log.1<br /> + /my/path/directory/my-app.log<br /> + /my/path/directory/application.log.1<br /> + /my/path/directory/application.log + </code> + </p> + <p> + the 'rolling filename pattern' would be <i>${filename}.log.*</i>. + </p> + <h3>Descriptions for different modes and strategies</h3> + <p> + The '<b>Single file</b>' mode assumes that the file to tail has always the same name even if there is a rolling pattern. + Example: + </p> + <p> + <code> + /my/path/directory/my-app.log.2<br /> + /my/path/directory/my-app.log.1<br /> + /my/path/directory/my-app.log + </code> + </p> + <p> + and new log messages are always appended in my-app.log file. + </p> + <p> + In case recursivity is set to 'true'. The regular expression for the files to tail must embrace the possible intermediate directories + between the base directory and the files to tail. Example: + </p> + <p> + <code> + /my/path/directory1/my-app1.log<br /> + /my/path/directory2/my-app2.log<br /> + /my/path/directory3/my-app3.log + </code> + </p> + <p> + <code> + Base directory: /my/path<br /> + Files to tail: directory[1-3]/my-app[1-3].log<br /> + Recursivity: true + </code> + </p> + <p> + In the '<b>Multiple files</b>' mode, it is possible to specify if the file to tail has always the same name or not. It is done through + the property 'Rolling strategy'. The strategy can be 'Fixed name' in case the files to tail have always the same name (see example above) + or can be 'Changing name' in case the files to tail do not always have the same name. Example: + </p> + <p> + <code> + /my/path/directory/my-app-2016-09-06.log<br /> + /my/path/directory/my-app-2016-09-07.log<br /> + /my/path/directory/my-app-2016-09-08.log + </code> + </p> + <p> + and new log messages are always appended in log file of the current day. + </p> + <p> + If the processor is configured with '<b>Multiple files</b>' mode, two additional properties are relevant: + </p> + <ul> + <li><b>Lookup frequency</b>: specifies the minimum duration the processor will wait before listing again the files to tail.</li> + <li><b>Maximum age</b>: specifies the necessary minimum duration to consider that no new messages will be appended in a file + regarding its last modification date. If the amount of time that has elapsed since the file was modified is larger than this + period of time, the file will not be tailed. For example, if a file was modified 24 hours ago and this property is set to 12 hours, + the file will not be tailed. But if this property is set to 36 hours, then the file will continue to be tailed.</li> + </ul> + <p> + It is necessary to pay attention to 'Lookup frequency' and 'Maximum age' properties, as well as the frequency at which the processor is + triggered, in order to achieve high performance. It is recommended to keep 'Maximum age' > 'Lookup frequency' > processor scheduling + frequency to avoid missing data. It also recommended not to set 'Maximum Age' too low because if messages are appended in a file + after this file has been considered "too old", all the messages in the file may be read again, leading to data duplication. + </p> + <p> + If the processor is configured with '<b>Multiple files</b>' mode, the 'Rolling + filename pattern' property must be specific enough to ensure that only the rolling files will be listed and not other currently tailed + files in the same directory (this can be achieved using ${filename} tag). + </p> + </body> +</html> \ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TailFile/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>TailFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">TailFile</h1><h2>Description: </h2><p>"Tails" a file, or a list of files, ingesting data from the file as it is written to the file. The file is expected to be textual. Data is ingested only when a new line is encountered (carriage return or new-line character or combination). If the file to tail is periodically "rolled over", as is generally the case with log files, an optional Rolling Filename Pattern can be used to retrieve data from files that have rolled over, even if the rollover occurred while NiFi was not running (provided that the data still exists upon restart of NiFi). It is generally advisa ble to set the Run Schedule to a few seconds, rather than running with the default value of 0 secs, as this Processor will consume a lot of resources if scheduled very aggressively. At this time, this Processor does not support ingesting files that have been compressed when 'rolled over'.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>tail, file, log, text, source</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Tailing mode</strong></td><td id="default-value">Single file</td><td id="allowable-values"><ul><li>Single file <img src="../../../../../html/images/iconInfo.png" alt="In this mode, only the one file indicated in the 'Files to tail' property will be watched by the processor. In this mode, the file may not exist when starting the processor." title="In this mode, only the one file indicated in the 'Files to tail' property will be watched by the processor. In this mode, the file may not exist when starting the processor."></img></li><li>Multiple files <img src="../../../../../html/images/iconInfo.png" alt="In this mode, the 'Files to tail' property accepts a regular expression and the processor will look for files in 'Base directory' to list the files to tail by the processor." title="In this mode, the 'Files to tail' property accepts a regular expression and the processor will look for files in 'Base directory' to list the files to tail by the processor."></img></li></ul></td><td id="description">Mode to use: single file will tail only one file, multiple file will look for a list of file. In Multiple mode the Base directory is required.</td></tr><tr><td id="name"><strong>File(s) to Tail</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Path of the file to tail in case of single file mode. If using multifile mode, regular expression to find files to tail in the base directory. In case recursivity is set to true, the regular expression will be used to match the path starting from the base directory (see additional details for examples).<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Rolling Filename Pattern</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">If the file to tail "rolls over" as would be the case with log files, this filename pattern will be used to identify files that have rolled over so that if NiFi is restarted, and the file has rolled over, it will be able to pick up where it left off. This pa ttern supports wildcard characters * and ?, it also supports the notation ${filename} to specify a pattern based on the name of the file (without extension), and will assume that the files that have rolled over live in the same directory as the file being tailed. The same glob pattern will be used for all files.</td></tr><tr><td id="name">Base directory</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Base directory used to look for files to tail. This property is required when using Multifile mode.<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name"><strong>Initial Start Position</strong></td><td id="default-value">Beginning of File</td><td id="allowable-values"><ul><li>Beginning of Time <img src="../../../../../html/images/iconInfo.png" alt="Start with the oldest data that matches the Rolling Filename Pattern and then begin reading from the File to Tail" title="Start with the oldest data that matches the Rolling Filename Pattern and then begin reading from the File to Tail"></img></li><li>Beginning of File <img src="../../../../../html/images/iconInfo.png" alt="Start with the beginning of the File to Tail. Do not ingest any data that has already been rolled over" title="Start with the beginning of the File to Tail. Do not ingest any data that has already been rolled over"></img></li><li>Current Time <img src="../../../../../html/images/iconInfo.png" alt="Start with the data at the end of the File to Tail. Do not ingest any data thas has already been rolled over or any data in the File to Tail that has already been written." title="Start with the data at the end of the File to Tail. Do not ingest any data thas has already been rolled over or any data in the File to Tail that has already been written."></img></li></ul></td><td id="description">When the Processor first begins to tail data, this property specifies where the Processor should begin re ading data. Once data has been ingested from a file, the Processor will continue from the last point from which it has received data.</td></tr><tr><td id="name"><strong>State Location</strong></td><td id="default-value">Local</td><td id="allowable-values"><ul><li>Local <img src="../../../../../html/images/iconInfo.png" alt="State is stored locally. Each node in a cluster will tail a different file." title="State is stored locally. Each node in a cluster will tail a different file."></img></li><li>Remote <img src="../../../../../html/images/iconInfo.png" alt="State is located on a remote resource. This Processor will store state across the cluster so that it can be run on Primary Node Only and a new Primary Node can pick up where the last one left off." title="State is located on a remote resource. This Processor will store state across the cluster so that it can be run on Primary Node Only and a new Primary Node can pick up where the last one left off."></img></li></ul></td><td id=" description">Specifies where the state is located either local or cluster so that state can be stored appropriately in order to ensure that all data is consumed without duplicating data upon restart of NiFi</td></tr><tr><td id="name"><strong>Recursive lookup</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">When using Multiple files mode, this property defines if files must be listed recursively or not in the base directory.</td></tr><tr><td id="name">Lookup frequency</td><td id="default-value">10 minutes</td><td id="allowable-values"></td><td id="description">Only used in Multiple files mode and Changing name rolling strategy. It specifies the minimum duration the processor will wait before listing again the files to tail.</td></tr><tr><td id="name">Maximum age</td><td id="default-value">24 hours</td><td id="allowable-values"></td><td id="description">Only used in Multiple files mode and Changing na me rolling strategy. It specifies the necessary minimum duration to consider that no new messages will be appended in a file regarding its last modification date. This should not be set too low to avoid duplication of data in case new messages are appended at a lower frequency.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All FlowFiles are routed to this Relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>tailfile.original.path</td><td>Path of the original file the flow file comes from.</td></tr></table><h3>State management: </h3><table id="stateful"><tr><th>Scope</th><th>Description</th></tr><tr><td>LOCAL, CLUSTER</td><td>Stores state about where in the Tailed File it left off so that on restart it does not have to duplicate data. State is stored either local or cluste red depend on the <File Location> property.</td></tr></table><h3>Restricted: </h3><table id="restrictions"><tr><th>Required Permission</th><th>Explanation</th></tr><tr><td>read filesystem</td><td>Provides operator the ability to read from any file that NiFi has access to.</td></tr></table><h3>Input requirement: </h3>This component does not allow an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TransformXml/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TransformXml/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TransformXml/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.TransformXml/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>TransformXml</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">TransformXml</h1><h2>Description: </h2><p>Applies the provided XSLT file to the flowfile XML payload. A new FlowFile is created with transformed content and is routed to the 'success' relationship. If the XSL transform fails, the original FlowFile is routed to the 'failure' relationship</p><h3>Tags: </h3><p>xml, xslt, transform</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../. ./html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">XSLT file name</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Provides the name (including full path) of the XSLT file to apply to the flowfile XML content.One of the 'XSLT file name' and 'XSLT Lookup' properties must be defined.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">XSLT Lookup</td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>StringLookupService<br/><strong>Implementations: </strong><a href="../../../nifi-lookup-services-nar/1.9.0/org.apache.nifi.lookup.XMLFileLookupService/index.html">XMLFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.9.0/org.apache.nifi.loo kup.SimpleKeyValueLookupService/index.html">SimpleKeyValueLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.9.0/org.apache.nifi.lookup.DistributedMapCacheLookupService/index.html">DistributedMapCacheLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.9.0/org.apache.nifi.lookup.SimpleCsvFileLookupService/index.html">SimpleCsvFileLookupService</a><br/><a href="../../../nifi-couchbase-nar/1.9.0/org.apache.nifi.couchbase.CouchbaseKeyValueLookupService/index.html">CouchbaseKeyValueLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.9.0/org.apache.nifi.lookup.PropertiesFileLookupService/index.html">PropertiesFileLookupService</a><br/><a href="../../../nifi-elasticsearch-client-service-nar/1.9.0/org.apache.nifi.elasticsearch.ElasticSearchStringLookupService/index.html">ElasticSearchStringLookupService</a></td><td id="description">Controller lookup used to store XSLT definitions. One of the 'XSLT file name' and 'XSLT Lookup' properties must be defined. WARNING: note that the lookup controller service should not be used to store large XSLT files.</td></tr><tr><td id="name">XSLT Lookup key</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Key used to retrieve the XSLT definition from the XSLT lookup controller. This property must be set when using the XSLT controller property.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Indent</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Whether or not to indent the output.</td></tr><tr><td id="name"><strong>Secure processing</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Whether or not to mitigate various XML-related attacks like XXE (XML External Entity) attacks.</td></t r><tr><td id="name"><strong>Cache size</strong></td><td id="default-value">10</td><td id="allowable-values"></td><td id="description">Maximum number of stylesheets to cache. Zero disables the cache.</td></tr><tr><td id="name"><strong>Cache TTL after last access</strong></td><td id="default-value">60 secs</td><td id="allowable-values"></td><td id="description">The cache TTL (time-to-live) or how long to keep stylesheets in the cache after last access.</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">An XSLT transform parameter name</td><td id="value">An XSLT transform parameter value</td><td>These XSLT parameters are passed to the transformer<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr></table></p><h3>Relations hips: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>The FlowFile with transformed content will be routed to this relationship</td></tr><tr><td>failure</td><td>If a FlowFile fails processing for any reason (for example, the FlowFile is not valid XML), it will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UnpackContent/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UnpackContent/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UnpackContent/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UnpackContent/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>UnpackContent</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">UnpackContent</h1><h2>Description: </h2><p>Unpacks the content of FlowFiles that have been packaged with one of several different Packaging Formats, emitting one to many FlowFiles for each input FlowFile</p><h3>Tags: </h3><p>Unpack, un-merge, tar, zip, archive, flowfile-stream, flowfile-stream-v3</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th ><th>Description</th></tr><tr><td id="name"><strong>Packaging >Format</strong></td><td id="default-value">use mime.type attribute</td><td >id="allowable-values"><ul><li>use mime.type >attribute</li><li>tar</li><li>zip</li><li>flowfile-stream-v3</li><li>flowfile-stream-v2</li><li>flowfile-tar-v1</li></ul></td><td > id="description">The Packaging Format used to create the >file</td></tr><tr><td id="name"><strong>File Filter</strong></td><td >id="default-value">.*</td><td id="allowable-values"></td><td >id="description">Only files contained in the archive whose names match the >given regular expression will be extracted (tar/zip >only)</td></tr></table><h3>Relationships: </h3><table >id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Unpacked > FlowFiles are sent to this relationship</td></tr><tr><td>failure</td><td>The >original FlowFile is sent to this relationship when it cannot be unpacked for >some reason</td></tr><tr><td>original</td><td>The original FlowFile i s sent to this relationship after it has been successfully unpacked</td></tr></table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>mime.type</td><td>If the <Packaging Format> property is set to use mime.type attribute, this attribute is used to determine the FlowFile's MIME Type. In this case, if the attribute is set to application/tar, the TAR Packaging Format will be used. If the attribute is set to application/zip, the ZIP Packaging Format will be used. If the attribute is set to application/flowfile-v3 or application/flowfile-v2 or application/flowfile-v1, the appropriate FlowFile Packaging Format will be used. If this attribute is missing, the FlowFile will be routed to 'failure'. Otherwise, if the attribute's value is not one of those mentioned above, the FlowFile will be routed to 'success' without being unpacked. Use the File Filter property only extract files matching a specific regular expression.</td></tr>< /table><h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>mime.type</td><td>If the FlowFile is successfully unpacked, its MIME Type is no longer known, so the mime.type attribute is set to application/octet-stream.</td></tr><tr><td>fragment.identifier</td><td>All unpacked FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute</td></tr><tr><td>fragment.index</td><td>A one-up number that indicates the ordering of the unpacked FlowFiles that were created from a single parent FlowFile</td></tr><tr><td>fragment.count</td><td>The number of unpacked FlowFiles generated from the parent FlowFile</td></tr><tr><td>segment.original.filename </td><td>The filename of the parent FlowFile. Extensions of .tar, .zip or .pkg are removed because the MergeContent processor automatically adds those extensions if it is used to rebuild the original FlowFile</td></tr></table><h3>State manag ement: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.MergeContent/index.html">MergeContent</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateCounter/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateCounter/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateCounter/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateCounter/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>UpdateCounter</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">UpdateCounter</h1><h2>Description: </h2><p>This processor allows users to set specific counters and key points in their flow. It is useful for debugging and basic counting functions.</p><h3>Tags: </h3><p>counter, debug, instrumentation</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties "><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Counter Name</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The name of the counter you want to set the value off - supports expression language like ${counterName}<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Delta</strong></td><td id="default-value">1</td><td id="allowable-values"></td><td id="description">Adjusts the counter by the specified delta for each flow file received. May be a positive or negative integer.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Counter was updated/retrieved</td></tr>< /table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>counterName</td><td>The name of the counter to update/get.</td></tr></table><h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/additionalDetails.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/additionalDetails.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1,390 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>UpdateRecord</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <p> + UpdateRecord makes use of the NiFi <a href="../../../../../html/record-path-guide.html"> + RecordPath Domain-Specific Language (DSL)</a> to allow the user + to indicate which field(s) in the Record should be updated. Users do this by adding a User-defined + Property to the Processor's configuration. The name of the User-defined Property must be the + RecordPath text that should be evaluated against each Record. The value of the Property specifies + what value should go into that selected Record field. + </p> + + <p> + When specifying the replacement value (the value of the User-defined Property), the user is able + to specify a literal value such as the number <code>10</code>; an Expression Language Expression + to reference FlowFile attributes, such as <code>${filename}</code>; or another RecordPath path + from which to retrieve the desired value from the Record itself. Whether the value entered should + be interpreted as a literal or a RecordPath path is determined by the value of the <Replacement + Value Strategy> Property. + </p> + + <p> + If a RecordPath is given and does not match any field in an input Record, that Property will be skipped + and all other Properties will still be evaluated. If the RecordPath matches exactly one field, that field + will be updated with the corresponding value. If multiple fields match the RecordPath, then all fields that + match will be updated. If the replacement value is itself a RecordPath that does not match, then a + <code>null</code> value will be set for the field. For instances where this is not the desired behavior, + RecordPath predicates can be used to filter the fields that match so that no fields will be selected. + See <a href="../../../../../html/record-path-guide.html#predicates">RecordPath Predicates</a> for more information. + </p> + + <p> + Below, we lay out some examples in order to provide clarity about the Processor's behavior. For all of + the examples below, consider the example to operate on the following set of 2 (JSON) records: + </p> + +<code> +<pre> + [{ + "id": 17, + "name": "John", + "child": { + "id": "1" + }, + "siblingIds": [4, 8], + "siblings": [ + { "name": "Jeremy", "id": 4 }, + { "name": "Julia", "id": 8 } + ] + }, + { + "id": 98, + "name": "Jane", + "child": { + "id": 2 + }, + "gender": "F", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + <p> + For brevity, we will omit the corresponding schema and configuration of the RecordReader and + RecordWriter. Otherwise, consider the following set of Properties are configured for the Processor + and their associated outputs. + </p> + + <h3>Example 1 - Replace with Literal</h3> + + <p> + Here, we will replace the name of each Record with the name 'Jeremy' and set the gender to 'M': + </p> + + <table> + <tr> + <th>Property Name</th> + <th>Property Value</th> + </tr> + <tr> + <td>Replacement Value Strategy</td> + <td>Literal Value</td> + </tr> + <tr> + <td>/name</td> + <td>Jeremy</td> + </tr> + <tr> + <td>/gender</td> + <td>M</td> + </tr> + </table> + + <p> + This will yield the following output: + </p> + +<code> +<pre> + [{ + "id": 17, + "name": "Jeremy", + "child": { + "id": "1" + }, + "gender": "M", + "siblingIds": [4, 8], + "siblings": [ + { "name": "Jeremy", "id": 4 }, + { "name": "Julia", "id": 8 } + ] + }, + { + "id": 98, + "name": "Jeremy", + "child": { + "id": 2 + }, + "gender": "M", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + <p> + Note that even though the first record did not have a "gender" field in the input, one + will be added after the "child" field, as that's where the field is located in the schema. + </p> + + + <h3>Example 2 - Replace with RecordPath</h3> + + <p> + This example will replace the value in one field of the Record with the value from another field. + For this example, consider the following set of Properties: + </p> + + <table> + <tr> + <th>Property Name</th> + <th>Property Value</th> + </tr> + <tr> + <td>Replacement Value Strategy</td> + <td>Record Path Value</td> + </tr> + <tr> + <td>/name</td> + <td>/siblings[0]/name</td> + </tr> + </table> + + <p> + This will yield the following output: + </p> + +<code> +<pre> + [{ + "id": 17, + "name": "Jeremy", + "child": { + "id": "1" + }, + "siblingIds": [4, 8], + "siblings": [ + { "name": "Jeremy", "id": 4 }, + { "name": "Julia", "id": 8 } + ] + }, + { + "id": 98, + "name": null, + "child": { + "id": 2 + }, + "gender": "F", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + + + <h3>Example 3 - Replace with Relative RecordPath</h3> + + <p> + In the above example, we replaced the value of field based on another RecordPath. That RecordPath was an "absolute RecordPath," + meaning that it starts with a "slash" character (<code>/</code>) and therefore it specifies the path from the "root" or "outer most" element. + However, sometimes we want to reference a field in such a way that we defined the RecordPath relative to the field being updated. This example + does just that. For each of the siblings given in the "siblings" array, we will replace the sibling's name with their id's. To do so, we will + configure the processor with the following properties: + </p> + + <table> + <tr> + <th>Property Name</th> + <th>Property Value</th> + </tr> + <tr> + <td>Replacement Value Strategy</td> + <td>Record Path Value</td> + </tr> + <tr> + <td>/siblings[*]/name</td> + <td>../id</td> + </tr> + </table> + + <p> + Note that the RecordPath that was given for the value starts with <code>..</code>, which is a reference to the parent. We do this because the field + that we are going to update is the "name" field of the sibling. To get to the associated "id" field, we need to go to the "name" field's parent and then + to its "id" child field. The above example results in the following output: + </p> + +<code> +<pre> + [{ + "id": 17, + "name": "John", + "child": { + "id": "1" + }, + "siblingIds": [4, 8], + "siblings": [ + { "name": "4", "id": 4 }, + { "name": "8", "id": 8 } + ] + }, + { + "id": 98, + "name": "Jane", + "child": { + "id": 2 + }, + "gender": "F", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + + + + <h3>Example 4 - Replace Multiple Values</h3> + + <p> + This example will replace the value of all fields that have the name "id", regardless of + where in the Record hierarchy the field is found. The value that it uses references the Expression Language, + so for this example, let's assume that the incoming FlowFile has an attribute named "replacement.id" that + has a value of "91": + </p> + + <table> + <tr> + <th>Property Name</th> + <th>Property Value</th> + </tr> + <tr> + <td>Replacement Value Strategy</td> + <td>Literal Value</td> + </tr> + <tr> + <td>//id</td> + <td>${replacement.id}</td> + </tr> + </table> + + <p> + This will yield the following output: + </p> + +<code> +<pre> + [{ + "id": 91, + "name": "John", + "child": { + "id": "91" + }, + "siblingIds": [4, 8], + "siblings": [ + { "name": "Jeremy", "id": 91 }, + { "name": "Julia", "id": 91 } + ] + }, + { + "id": 91, + "name": "Jane", + "child": { + "id": 91 + }, + "gender": "F", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + <p> + It is also worth noting that in this example, some of the "id" fields were of type STRING, while + others were of type INT. This is okay because the RecordReaders and RecordWriters should handle + these simple type coercions for us. + </p> + + + + + <h3>Example 5 - Use Expression Language to Modify Value</h3> + + <p> + This example will capitalize the value of all 'name' fields, regardless of + where in the Record hierarchy the field is found. This is done by referencing the 'field.value' variable in the Expression Language. + We can also access the field.name variable and the field.type variable. + </p> + + <table> + <tr> + <th>Property Name</th> + <th>Property Value</th> + </tr> + <tr> + <td>Replacement Value Strategy</td> + <td>Literal Value</td> + </tr> + <tr> + <td>//name</td> + <td>${field.value:toUpper()}</td> + </tr> + </table> + + <p> + This will yield the following output: + </p> + +<code> +<pre> + [{ + "id": 17, + "name": "JOHN", + "child": { + "id": "1" + }, + "siblingIds": [4, 8], + "siblings": [ + { "name": "JEREMY", "id": 4 }, + { "name": "JULIA", "id": 8 } + ] + }, + { + "id": 98, + "name": "JANE", + "child": { + "id": 2 + }, + "gender": "F", + "siblingIds": [], + "siblings": [] + }] +</pre> +</code> + + + </body> +</html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.UpdateRecord/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>UpdateRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">UpdateRecord</h1><h2>Description: </h2><p>Updates the contents of a FlowFile that contains Record-oriented data (i.e., data that can be read via a RecordReader and written by a RecordWriter). This Processor requires that at least one user-defined Property be added. The name of the Property should indicate a RecordPath that determines the field that should be updated. The value of the Property is either a replacement value (optionally making use of the Expression Language) or is itself a RecordPath that extracts a value from the Record. Whether the Property value is determined to be a RecordPath or a literal value depends on the configuration of the <Replacement Value Strategy> Property.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>update, record, generic, schema, json, csv, avro, log, logs, freeform, text</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9. 0/org.apache.nifi.xml.XMLReader/index.html">XMLReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.syslog.Syslog5424Reader/index.html">Syslog5424Reader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-scripting-nar/1.9.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.syslog.SyslogReader/index.html">SyslogReader</a></td><td id="description">Specifies the Controller Service to use for reading incoming data</td></tr><tr><td id="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.xml.XMLRecordSetWriter/index.html">XMLRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.9.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href ="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a></td><td id="description">Specifies the Controller Service to use for writing out the records</td></tr><tr><td id="name"><strong>Replacement Value Strategy</strong></td><td id="default-value">literal-value</td><td id="allowable-values"><ul><li>Literal Value <img src="../../../../../html/images/iconInfo.png" alt="The value entered for a Property (after Expression Language has been evaluated) is the desired value to update the Record Fields with. Expression Language may reference variables 'field.name', 'field.type', and 'field.value' to access information about the field and the value of the field being evaluated." title="The value entered for a Property (after Expression Language has been evaluated) is the desired value to update the Record Fields with. Expression Language may reference variables 'field.name', 'field.type', and 'field.va lue' to access information about the field and the value of the field being evaluated."></img></li><li>Record Path Value <img src="../../../../../html/images/iconInfo.png" alt="The value entered for a Property (after Expression Language has been evaluated) is not the literal value to use but rather is a Record Path that should be evaluated against the Record, and the result of the RecordPath will be used to update the Record. Note that if this option is selected, and the Record Path results in multiple values for a given Record, the input FlowFile will be routed to the 'failure' Relationship." title="The value entered for a Property (after Expression Language has been evaluated) is not the literal value to use but rather is a Record Path that should be evaluated against the Record, and the result of the RecordPath will be used to update the Record. Note that if this option is selected, and the Record Path results in multiple values for a given Record, the input FlowFile will be rout ed to the 'failure' Relationship."></img></li></ul></td><td id="description">Specifies how to interpret the configured replacement values</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles that are successfully transformed will be routed to this relationship</td></tr><tr><td>failure</td><td>If a FlowFile cannot be transformed from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.ConvertRecord/index.html">Conver tRecord</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/additionalDetails.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/additionalDetails.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1,109 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8"/> + <title>ValidateCsv</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/> +</head> + +<body> +<!-- Processor Documentation ================================================== --> +<h2>Usage Information</h2> + +<p> + The Validate CSV processor is based on the super-csv library and the concept of + <a href="http://super-csv.github.io/super-csv/cell_processors.html"; target="_blank">Cell Processors</a>. + The corresponding java documentation can be found + <a href="http://super-csv.github.io/super-csv/apidocs/org/supercsv/cellprocessor/ift/CellProcessor.html"; target="_blank">here</a>. +</p> + +<p> + The cell processors cannot be nested (except with Optional which gives the possibility to define a CellProcessor for values + that could be null) and must be defined in a comma-delimited string as the Schema property. +</p> + +<p> + The supported cell processors are: + <ul> + <li>ParseBigDecimal</li> + <li>ParseBool</li> + <li>ParseChar</li> + <li>ParseDate</li> + <li>ParseDouble</li> + <li>ParseInt</li> + <li>Optional</li> + <li>DMinMax</li> + <li>Equals</li> + <li>ForbidSubStr</li> + <li>LMinMax</li> + <li>NotNull</li> + <li>Null</li> + <li>RequireHashCode</li> + <li>RequireSubStr</li> + <li>Strlen</li> + <li>StrMinMax</li> + <li>StrNotNullOrEmpty</li> + <li>StrRegEx</li> + <li>Unique</li> + <li>UniqueHashCode</li> + <li>IsIncludedIn</li> + </ul> +</p> + +<p> + Here are some examples: + + <ul> + <b>Schema property:</b> Null, ParseDate("dd/MM/yyyy"), Optional(ParseDouble())<br /> + <b>Meaning:</b> the input CSV has three columns, the first one can be null and has no specification, the second one must be a date + formatted as expected, and the third one must a double or null (no value). + </ul> + + <ul> + <b>Schema property:</b> ParseBigDecimal(), ParseBool(), ParseChar(), ParseInt(), ParseLong()<br /> + <b>Meaning:</b> the input CSV has five columns, the first one must be a big decimal, the second one must be a boolean, + the third one must be a char, the fourth one must be an integer and the fifth one must be a long. + </ul> + + <ul> + <b>Schema property:</b> Equals(), NotNull(), StrNotNullOrEmpty()<br /> + <b>Meaning:</b> the input CSV has three columns, all the values of the first column must be equal to each other, all the values + of the second column must be not null, and all the values of the third column are not null/empty string values. + </ul> + + <ul> + <b>Schema property:</b> Strlen(4), StrMinMax(3,5), StrRegex("[a-z0-9\\._]+@[a-z0-9\\.]+")<br /> + <b>Meaning:</b> the input CSV has three columns, all the values of the first column must be 4-characters long, all the values + of the second column must be between 3 and 5 characters (inclusive), and all the values of the last column must match + the provided regular expression (email address). + </ul> + + <ul> + <b>Schema property:</b> Unique(), UniqueHashCode()<br /> + <b>Meaning:</b> the input CSV has two columns. All the values of the first column must be unique (all the values are stored in + memory and this can be consuming depending of the input). All the values of the second column must be unique (only hash + codes of the input values are stored to ensure uniqueness). + </ul> + + <ul> + <b>Schema property:</b> ForbidSubStr("test", "tset"), RequireSubStr("test")<br /> + <b>Meaning:</b> the input CSV has two columns. None of the values in the first column must contain one of the provided strings. + And all the values of the second column must contain the provided string. + </ul> +</p> +</body> +</html> Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateCsv/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ValidateCsv</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ValidateCsv</h1><h2>Description: </h2><p>Validates the contents of FlowFiles against a user-specified CSV schema. Take a look at the additional documentation of this processor for some schema examples.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>csv, schema, validation</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-l anguage-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Schema</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The schema to be used for validation. Is expected a comma-delimited string representing the cell processors to apply. The following cell processors are allowed in the schema definition: [ParseBigDecimal, ParseBool, ParseChar, ParseDate, ParseDouble, ParseInt, ParseLong, Optional, DMinMax, Equals, ForbidSubStr, LMinMax, NotNull, Null, RequireHashCode, RequireSubStr, Strlen, StrMinMax, StrNotNullOrEmpty, StrRegEx, Unique, UniqueHashCode, IsIncludedIn]. Note: cell processors cannot be nested except with Optional.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Header</strong></td><td id="de fault-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">True if the incoming flow file contains a header to ignore, false otherwise.</td></tr><tr><td id="name"><strong>Delimiter character</strong></td><td id="default-value">,</td><td id="allowable-values"></td><td id="description">Character used as 'delimiter' in the incoming data. Example: ,<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Quote character</strong></td><td id="default-value">"</td><td id="allowable-values"></td><td id="description">Character used as 'quote' in the incoming data. Example: "<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>End of line symbols</strong></td><td id="default-value">\n</td><td id="allowable-values"></td><td id="descrip tion">Symbols used as 'end of line' in the incoming data. Example: \n<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Validation strategy</strong></td><td id="default-value">FlowFile validation</td><td id="allowable-values"><ul><li>Line by line validation <img src="../../../../../html/images/iconInfo.png" alt="In case an error is found, the input CSV file will be split into two FlowFiles: one routed to the 'valid' relationship containing all the correct lines and one routed to the 'invalid' relationship containing all the incorrect lines. Take care if choosing this option while using Unique cell processors in schema definition:the first occurrence will be considered valid and the next ones as invalid." title="In case an error is found, the input CSV file will be split into two FlowFiles: one routed to the 'valid' relationship containing all the correct lines and one routed to the 'invalid' relationship containing all the incorrect lines. Take care if choosing this option while using Unique cell processors in schema definition:the first occurrence will be considered valid and the next ones as invalid."></img></li><li>FlowFile validation <img src="../../../../../html/images/iconInfo.png" alt="As soon as an error is found in the CSV file, the validation will stop and the whole flow file will be routed to the 'invalid' relationship. This option offers best performances." title="As soon as an error is found in the CSV file, the validation will stop and the whole flow file will be routed to the 'invalid' relationship. This option offers best performances."></img></li></ul></td><td id="description">Strategy to apply when routing input files to output relationships.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>valid</td><td>FlowFiles that are successfully validated against the schema are routed to this relationship</td></tr><tr><td>invalid</td><td>FlowFiles that are not valid according to the specified schema are routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>count.valid.lines</td><td>If line by line validation, number of valid lines extracted from the source data</td></tr><tr><td>count.invalid.lines</td><td>If line by line validation, number of invalid lines extracted from the source data</td></tr><tr><td>count.total.lines</td><td>If line by line validation, total number of lines in the source data</td></tr><tr><td>validation.error.message</td><td>For flow files routed to invalid, message of the first validation error</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relati onship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateRecord/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateRecord/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateRecord/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateRecord/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ValidateRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ValidateRecord</h1><h2>Description: </h2><p>Validates the Records of an incoming FlowFile against a given schema. All records that adhere to the schema are routed to the "valid" relationship while records that do not adhere to the schema are routed to the "invalid" relationship. It is therefore possible for a single incoming FlowFile to be split into two individual FlowFiles if some records are valid according to the schema and others are not. Any FlowFile that is routed to the "invalid" relationship will emit a ROUTE Provenance Event with the Details field populated to explain why records were i nvalid. In addition, to gain further explanation of why records were invalid, DEBUG-level logging can be enabled for the "org.apache.nifi.processors.standard.ValidateRecord" logger.</p><h3>Tags: </h3><p>record, schema, validate</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonTreeReader/ index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.xml.XMLReader/index.html">XMLReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.syslog.Syslog5424Reader/index.html">Syslog5424Reader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-scripting-nar/1.9.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org. apache.nifi.syslog.SyslogReader/index.html">SyslogReader</a></td><td id="description">Specifies the Controller Service to use for reading incoming data</td></tr><tr><td id="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.xml.XMLRecordSetWriter/index.html">XMLRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.9.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-na r/1.9.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a></td><td id="description">Specifies the Controller Service to use for writing out the records. Regardless of the Controller Service schema access configuration, the schema that is used to validate record is used to write the valid results.</td></tr><tr><td id="name">Record Writer for Invalid Records</td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.xml.XMLRecordSetWriter/index.html">XMLRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.9.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.9.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a></td><td id="description">If specified, this Controller Service will be used to write out any records that are invalid. If not specified, the writer specified by the "Record Writer" property will be used with the schema used to read the input records. This is useful, for example, when the configured Record Writer cannot write data that does not adhere to its schema (as is the case with Avro) or when it is desirable to keep invalid records in their original format while converting valid records to another format.</td></tr><tr><td id="name"><strong>Schema Access Strategy</strong></td><td id="default-value">reader-schema</td><td id="allowable-values"><ul><li>Use Reader's Schema <img src="../../../../../html/images/iconInfo.png" alt="The schema to validate the data against is determined by asking the configured Record Reader for its schema" title="The schema to validate the data against is determined by asking the configured Record Reader for its schema"></img></li><li>Use Schema Name Property <img src="../../../../../html/images/iconInfo.png" alt="The schema to validate the data against is determined by looking at the 'Schema Name' Property and looking up the schema in the configured Schema Registry" title="The schema to validate the data against is determined by looking at the 'Schema Name' Property and looking up the schema in the configured Schema Registry"></img></li><li>Use Schema Text Prop erty <img src="../../../../../html/images/iconInfo.png" alt="The schema to validate the data against is determined by looking at the 'Schema Text' Property and parsing the schema as an Avro schema" title="The schema to validate the data against is determined by looking at the 'Schema Text' Property and parsing the schema as an Avro schema"></img></li></ul></td><td id="description">Specifies how to obtain the schema that should be used to validate records</td></tr><tr><td id="name">Schema Registry</td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>SchemaRegistry<br/><strong>Implementations: </strong><a href="../../../nifi-hwx-schema-registry-nar/1.9.0/org.apache.nifi.schemaregistry.hortonworks.HortonworksSchemaRegistry/index.html">HortonworksSchemaRegistry</a><br/><a href="../../../nifi-confluent-platform-nar/1.9.0/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html">ConfluentSchemaRegistry</a><br/><a href=" ../../../nifi-registry-nar/1.9.0/org.apache.nifi.schemaregistry.services.AvroSchemaRegistry/index.html">AvroSchemaRegistry</a></td><td id="description">Specifies the Controller Service to use for the Schema Registry. This is necessary only if the Schema Access Strategy is set to "Use 'Schema Name' Property".</td></tr><tr><td id="name">Schema Name</td><td id="default-value">${schema.name}</td><td id="allowable-values"></td><td id="description">Specifies the name of the schema to lookup in the Schema Registry property<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Schema Text</td><td id="default-value">${avro.schema}</td><td id="allowable-values"></td><td id="description">The text of an Avro-formatted Schema<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Allow Extra Fiel ds</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If the incoming data has fields that are not present in the schema, this property determines whether or not the Record is valid. If true, the Record is still valid. If false, the Record will be invalid due to the extra fields.</td></tr><tr><td id="name"><strong>Strict Type Checking</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If the incoming data has a Record where a field is not of the correct type, this property determine whether how to handle the Record. If true, the Record will still be considered invalid. If false, the Record will be considered valid and the field will be coerced into the correct type (if possible, according to the type coercion supported by the Record Writer).</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</ th><th>Description</th></tr><tr><td>valid</td><td>Records that are valid according to the schema will be routed to this relationship</td></tr><tr><td>invalid</td><td>Records that are not valid according to the schema will be routed to this relationship</td></tr><tr><td>failure</td><td>If the records cannot be read, validated, or written, for any reason, the original FlowFile will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>mime.type</td><td>Sets the mime.type attribute to the MIME Type specified by the Record Writer</td></tr><tr><td>record.count</td><td>The number of records in the FlowFile routed to a relationship</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>Syste m Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateXml/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateXml/index.html?rev=1854109&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateXml/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ValidateXml/index.html Fri Feb 22 01:03:44 2019 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ValidateXml</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ValidateXml</h1><h2>Description: </h2><p>Validates the contents of FlowFiles against a user-specified XML Schema file</p><h3>Tags: </h3><p>xml, schema, validation, xsd</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th> Description</th></tr><tr><td id="name"><strong>Schema File</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The path to the Schema file that is to be used for validation<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>valid</td><td>FlowFiles that are successfully validated against the schema are routed to this relationship</td></tr><tr><td>invalid</td><td>FlowFiles that are not valid according to the specified schema are routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>validatexml.invalid.error</td><td>If the flow file is routed to the invalid relationship the attribute will contain the error message resulting from the valida tion failure.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file
