This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch phrocker-patch-1 in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 6e217bc727e364c6718d2a00bba05128b5d28fc2 Author: Marc <[email protected]> AuthorDate: Fri Aug 2 11:51:47 2019 -0400 Ran gen-processors-docs To generate new docs --- PROCESSORS.md | 1517 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 781 insertions(+), 736 deletions(-) diff --git a/PROCESSORS.md b/PROCESSORS.md index 079e055..3f0a1af 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -1,41 +1,33 @@ -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - +<!--Licensed to the Apache Software Foundation (ASF) under one or morecontributor license agreements. See the NOTICE file distributed withthis work for additional information regarding copyright ownership.The ASF licenses this file to You under the Apache License, Version 2.0(the "License"); you may not use this file except in compliance withthe License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to i [...] # Processors ## Table of Contents - [AppendHostInfo](#appendhostinfo) - [ApplyTemplate](#applytemplate) +- [BinFiles](#binfiles) - [CapturePacket](#capturepacket) +- [CaptureRTSPFrame](#capturertspframe) - [CompressContent](#compresscontent) - [ConsumeMQTT](#consumemqtt) - [ConvertHeartBeat](#convertheartbeat) - [ConvertJSONAck](#convertjsonack) - [ConvertUpdate](#convertupdate) - [ExecuteProcess](#executeprocess) -- [ExecuteScript](#executescript) +- [ExecutePythonProcessor](#executepythonprocessor) - [ExecuteSQL](#executesql) +- [ExecuteScript](#executescript) - [ExtractText](#extracttext) +- [FetchSFTP](#fetchsftp) - [FocusArchiveEntry](#focusarchiveentry) - [GenerateFlowFile](#generateflowfile) - [GetFile](#getfile) -- [GetUSBCamera](#getusbcamera) +- [GetGPS](#getgps) - [GetTCP](#gettcp) +- [GetUSBCamera](#getusbcamera) - [HashContent](#hashcontent) - [InvokeHTTP](#invokehttp) +- [ListSFTP](#listsftp) - [ListenHTTP](#listenhttp) - [ListenSyslog](#listensyslog) - [LogAttribute](#logattribute) @@ -44,228 +36,333 @@ - [PublishKafka](#publishkafka) - [PublishMQTT](#publishmqtt) - [PutFile](#putfile) +- [PutSFTP](#putsftp) - [PutSQL](#putsql) - [RouteOnAttribute](#routeonattribute) -- [TailFile](#tailfile) - [TFApplyGraph](#tfapplygraph) - [TFConvertImageToTensor](#tfconvertimagetotensor) - [TFExtractTopLabels](#tfextracttoplabels) +- [TailFile](#tailfile) - [UnfocusArchiveEntry](#unfocusarchiveentry) - [UpdateAttribute](#updateattribute) - ## AppendHostInfo -### Description - -Appends host information such as IP address and hostname as an attribute to -incoming flowfiles. +### Description -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Appends host information such as IP address and hostname as an attribute to incoming flowfiles. +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Network Interface Name** | eth0 | | Network interface from which to read an IP v4 address | -| **Hostname Attribute** | source.hostname | | Flowfile attribute to used to record the agent's hostname | -| **IP Attribute** | source.ipv4 | | Flowfile attribute to used to record the agent's IP address | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Hostname Attribute|source.hostname||Flowfile attribute to used to record the agent's hostname| +|IP Attribute|source.ipv4||Flowfile attribute to used to record the agent's IP address| +|Network Interface Name|eth0||Network interface from which to read an IP v4 address| +### Properties | Name | Description | | - | - | -| success | All FlowFiles are routed to this relationship. | +|success|success operational on the flow record| + ## ApplyTemplate -### Description +### Description -Applies the mustache template specified by the "Template" property and writes -the output to the flow file content. FlowFile attributes are used as template -parameters. +Applies the mustache template specified by the "Template" property and writes the output to the flow file content. FlowFile attributes are used as template parameters. +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Template|||Path to the input mustache template file| +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Template** | | | Path to the input mustache template file<br>**Supports Expression Language: true** | +| Name | Description | +| - | - | +|success|success operational on the flow record| -### Relationships + +## BinFiles + +### Description + +Bins flow files into buckets based on the number of entries or size of entries +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Max Bin Age|||The maximum age of a Bin that will trigger a Bin to be complete. Expected format is <duration> <time unit>| +|Maximum Group Size|||The maximum size for the bundle. If not specified, there is no maximum.| +|Maximum Number of Entries|||The maximum number of files to include in a bundle. If not specified, there is no maximum.| +|Maximum number of Bins|100||Specifies the maximum number of bins that can be held in memory at any one time| +|Minimum Group Size|0||The minimum size of for the bundle| +|Minimum Number of Entries|1||The minimum number of files to include in a bundle| +### Properties | Name | Description | | - | - | -| success | All FlowFiles are routed to this relationship. | +|failure|If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure| +|original|The FlowFiles that were used to create the bundle| ## CapturePacket -### Description +### Description -CapturePacket captures and writes one or more packets into a PCAP file that will be used as the content -of a flow file. Configuration options exist to adjust the batching of PCAP files. PCAP batching will -place a single PCAP into a flow file. A regular expression selects network interfaces. Bluetooth -network interfaces can be selected through a separate option. +CapturePacket captures and writes one or more packets into a PCAP file that will be used as the content of a flow file. Configuration options exist to adjust the batching of PCAP files. PCAP batching will place a single PCAP into a flow file. A regular expression selects network interfaces. Bluetooth network interfaces can be selected through a separate option. +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Base Directory|/tmp/||Scratch directory for PCAP files| +|Batch Size|50||The number of packets to combine within a given PCAP| +|Capture Bluetooth|false||True indicates that we support bluetooth interfaces| +|Network Controller|.*||Regular expression of the network controller(s) to which we will attach| +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| Base Directory | /tmp/ | | A base directory where pcap files are stored temporarily | -| Batch Size | 50 | | Number of captured packets written into a PCAP file | -| Capture Bluetooth | false | | Captures bluetooth interfaces if true | -| Network Controller | .\* | | Regular expression of the network controller(s) to which packet capture will be attached| +| Name | Description | +| - | - | +|success|All files are routed to success| -### Relationships + +## CaptureRTSPFrame + +### Description + +Captures a frame from the RTSP stream at specified intervals. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Image Encoding|.jpg||The encoding that should be applied the the frame images captured from the RTSP stream| +|RTSP Hostname|||Hostname of the RTSP stream we are trying to connect to| +|RTSP Password|||Password used to connect to the RTSP stream| +|RTSP Port|||Port that should be connected to to receive RTSP Frames| +|RTSP URI|||URI that should be appended to the RTSP stream hostname| +|RTSP Username|||The username for connecting to the RTSP stream| +### Properties | Name | Description | | - | - | -| success | All FlowFiles are routed to this relationship. | +|failure|Failures to capture RTSP frame| +|success|Successful capture of RTSP frame| -## HashContent +## CompressContent -### Description +### Description -HashContent calculates the checksum of the content of the flowfile and adds it as an attribute. -Configuration options exist to select hashing algorithm and set the name of the attribute. +Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Compression Format|use mime.type attribute||The compression format to use.| +|Compression Level|1||The compression level to use; this is valid only when using GZIP compression.| +|Mode|compress||Indicates whether the processor should compress content or decompress content.| +|Update Filename|false||Determines if filename extension need to be updated| +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| Hash Attribute | Checksum | | Name of the attribute the processor will use to add the checksum | -| Hash Algorithm | MD5 | MD5, SHA1, SHA256 | Name of the algorithm used to calculate the checksum | -| Fail on empty | false | false, true | Route flow files with empty content to failure relationship | +| Name | Description | +| - | - | +|failure|FlowFiles will be transferred to the failure relationship if they fail to compress/decompress| +|success|FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed| -### Relationships + +## ConsumeMQTT + +### Description + +This Processor gets the contents of a FlowFile from a MQTT broker for a specified topic. The the payload of the MQTT message becomes content of a FlowFile +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Broker URI|||The URI to use to connect to the MQTT broker| +|Client ID|||MQTT client ID to use| +|Connection Timeout|30 sec||Maximum time interval the client will wait for the network connection to the MQTT server| +|Keep Alive Interval|60 sec||Defines the maximum time interval between messages sent or received| +|Max Flow Segment Size|||Maximum flow content payload segment size for the MQTT record| +|Password|||Password to use when connecting to the broker| +|Quality of Service|MQTT_QOS_0||The Quality of Service(QoS) to send the message with. Accepts three values '0', '1' and '2'| +|Queue Max Message|||Maximum number of messages allowed on the received MQTT queue| +|Session state|true||Whether to start afresh or resume previous flows. See the allowable value descriptions for more details| +|Topic|||The topic to publish the message to| +|Username|||Username to use when connecting to the broker| +### Properties | Name | Description | | - | - | -| success | By default all flow files are routed to this relationship. | -| failure | In case "Fail on empty" property is set to true, flow files with empty content are routed to this relationship. | +|success|FlowFiles that are sent successfully to the destination are transferred to this relationship| + ## ConvertHeartBeat -This Processor converts MQTT heartbeats into a JSON repreesntation. +### Description -### Properties +ConvertHeartBeat converts heatrbeats into MQTT messages. +### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **MQTT Controller Service** | | | The MQTT Controller service | -| Listening topic | | | The topic on which we will listen to get MQTT C2 messages | +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Listening Topic|||Name of topic to listen to| +|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| +### Properties +| Name | Description | +| - | - | +|success|All files are routed to success| -## ConvertJSONAck -This Processor parses C2 respones (acknowledgements) and forwards them to the MQTT agent. +## ConvertJSONAck -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Converts JSON acks into an MQTT consumable by MQTTC2Protocol +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **MQTT Controller Service** | | | The MQTT Controller service | -| Listening topic | | | The topic on which we will listen to get MQTT C2 messages | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Listening Topic|||Name of topic to listen to| +|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| +### Properties | Name | Description | | - | - | -| success | Any successful http response flow file will be sent to this relationship | +|success|All files are routed to success| + ## ConvertUpdate -This converts MQTT update messages into an HTTP request to retrieve an update. This -processor requires cURL support. If it does not exist this processor will be a NOOP. +### Description -### Properties +onverts update messages into the appropriate RESTFul call +### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Listening Topic|||Name of topic to listen to| +|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| +|SSL Context Service|||The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.| +### Properties + +| Name | Description | +| - | - | +|success|All files are routed to success| -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **MQTT Controller Service** | | | The MQTT Controller service | -| SSL Context Service | | | SSL context service used for HTTP requestor. | -| Listening topic | | | The topic on which we will listen to get MQTT C2 messages | ## ExecuteProcess -### Description +### Description -Runs an operating system command specified by the user and writes the output of -that command to a FlowFile. If the command is expected to be long-running, the -Processor can output the partial data on a specified interval. When this option -is used, the output is expected to be in textual format, as it typically does -not make sense to split binary data on arbitrary time-based intervals. +Runs an operating system command specified by the user and writes the output of that command to a FlowFile. If the command is expected to be long-running,the Processor can output the partial data on a specified interval. When this option is used, the output is expected to be in textual format,as it typically does not make sense to split binary data on arbitrary time-based intervals. +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Batch Duration|0 sec||If the process is expected to be long-running and produce textual output, a batch duration can be specified.| +|Command|||Specifies the command to be executed; if just the name of an executable is provided, it must be in the user's environment PATH.<br/>**Supports Expression Language: true**| +|Command Arguments|||The arguments to supply to the executable delimited by white space. White space can be escaped by enclosing it in double-quotes.<br/>**Supports Expression Language: true**| +|Redirect Error Stream|false||If true will redirect any error stream output of the process to the output stream.| +|Working Directory|||The directory to use as the current working directory when executing the command<br/>**Supports Expression Language: true**| +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Command** | | | Specifies the command to be executed; if just the name of an executable is provided, it must be in the user's environment PATH.<br>**Supports Expression Language: true** | -| Command Arguments | | | The arguments to supply to the executable delimited by white space. White space can be escaped by enclosing it in double-quotes.<br>**Supports Expression Language: true** | -| Working Directory | | | The directory to use as the current working directory when executing the command<br>**Supports Expression Language: true** | -| **Batch Duration** | 0 || If the process is expected to be long-running and produce textual output, a batch duration can be specified. | -| **Redirect Error Stream** | false | | If true will redirect any error stream output of the process to the output stream. | +| Name | Description | +| - | - | +|success|All created FlowFiles are routed to this relationship.| -### Relationships + +## ExecutePythonProcessor + +### Description + +Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each i [...] +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Module Directory|||Comma-separated list of paths to files and/or directories which + contain modules required by the script| +|Script File|||Path to script file to execute. + Only one of Script File or Script Body may be used| +### Properties + +| Name | Description | +| - | - | +|failure|Script failures| +|success|Script successes| + + +## ExecuteSQL + +### Description + +Execute provided SQL query. Query result rows will be outputted as new flow files with attribute keys equal to result column names and values equal to result values. There will be one output FlowFile per result row. This processor can be scheduled to run using the standard timer-based scheduling methods, or it can be triggered by an incoming FlowFile. If it is triggered by an incoming FlowFile, then attributes of that FlowFile will be available when evaluating the query. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Connection URL|||The database URL to connect to| +|SQL Statement|||The SQL statement to execute| +### Properties | Name | Description | | - | - | -| success | All created FlowFiles are routed to this relationship. | +|failure|Failures which will not work if retried| +|original|The original FlowFile is sent here| +|success|After a successful SQL execution, result FlowFiles are sent here| + ## ExecuteScript -### Description +### Description -Executes a script given the flow file and a process session. The script is -responsible for handling the incoming flow file (transfer to SUCCESS or remove, -e.g.) as well as any flow files created by the script. If the handling is -incomplete or incorrect, the session will be rolled back. +Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each i [...] +### Properties -Scripts must define an onTrigger function which accepts NiFi Context and -Property objects. For efficiency, scripts are executed once when the processor -is run, then the onTrigger method is called for each incoming flowfile. This -enables scripts to keep state if they wish, although there will be a script -context per concurrent task of the processor. In order to, e.g., compute an -arithmetic sum based on incoming flow file information, set the concurrent -tasks to 1. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -Multiple variables are provided automatically to script contexts: +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Module Directory|||Comma-separated list of paths to files and/or directories which + contain modules required by the script| +|Script Body|||Body of script to execute. + Only one of Script File or Script Body may be used| +|Script Engine|python||The engine to execute scripts (python, lua)| +|Script File|||Path to script file to execute. + Only one of Script File or Script Body may be used| +### Properties -| Name | Purpose | +| Name | Description | | - | - | -| `log` | the logging object used for logging debug, info, warn, and error | -| `REL_SUCCESS` | the "success" relationship | -| `REL_FAILURE` | the "failure" relationship | +|failure|Script failures| +|success|Script successes| + ### Python Notes @@ -370,449 +467,642 @@ function onTrigger(context, session) end ``` -### Properties +## ExtractText -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +### Description -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Script Engine** | python | python, lua | The engine to execute scripts | -| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used | -| Script Body | | | Body of script to execute. Only one of Script File or Script Body may be used | -| Module Directory | | | Comma-separated list of paths to files and/or directories which contain modules required by the script | +Extracts the content of a FlowFile and places it into an attribute. +### Properties -### Relationships +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Attribute|||Attribute to set from content| +|Enable Case-insensitive Matching|false||Indicates that two characters match even if they are in a different case. | +|Enable repeating capture group|false||f set to true, every string matching the capture groups will be extracted. Otherwise, if the Regular Expression matches more than once, only the first match will be extracted.| +|Include Capture Group 0|true||Indicates that Capture Group 0 should be included as an attribute. Capture Group 0 represents the entirety of the regular expression match, is typically not used, and could have considerable length.| +|Maximum Capture Group Length|1024||Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.| +|Regex Mode|false||Set this to extract parts of flowfile content using regular experssions in dynamic properties| +|Size Limit|2097152||Maximum number of bytes to read into the attribute. 0 for no limit. Default is 2MB.| +### Properties | Name | Description | | - | - | -| success | Script successes | -| failure | Script failures | +|success|success operational on the flow record| + + +## FetchSFTP + +### Description + +Fetches the content of a file from a remote SFTP server and overwrites the contents of an incoming FlowFile with the content of the remote file. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|**Completion Strategy**|None|Delete File<br>Move File<br>None<br>|Specifies what to do with the original file on the server once it has been pulled into NiFi. If the Completion Strategy fails, a warning will be logged but the data will still be transferred.| +|**Connection Timeout**|30 sec||Amount of time to wait before timing out while creating a connection| +|**Create Directory**|false||Specifies whether or not the remote directory should be created if it does not exist.| +|**Data Timeout**|30 sec||When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems| +|Disable Directory Listing|false||Control how 'Move Destination Directory' is created when 'Completion Strategy' is 'Move File' and 'Create Directory' is enabled. If set to 'true', directory listing is not performed prior to create missing directories. By default, this processor executes a directory listing command to see target directory existence before creating missing directories. However, there are situations that you might need to disable the directory listing such as the following [...] +|Host Key File|||If supplied, the given file will be used as the Host Key; otherwise, no use host key file will be used| +|**Hostname**|||The fully qualified hostname or IP address of the remote system<br/>**Supports Expression Language: true**| +|Http Proxy Password|||Http Proxy Password<br/>**Supports Expression Language: true**| +|Http Proxy Username|||Http Proxy Username<br/>**Supports Expression Language: true**| +|Move Destination Directory|||The directory on the remote server to move the original file to once it has been ingested into NiFi. This property is ignored unless the Completion Strategy is set to 'Move File'. The specified directory must already exist on the remote system if 'Create Directory' is disabled, or the rename will fail.<br/>**Supports Expression Language: true**| +|Password|||Password for the user account<br/>**Supports Expression Language: true**| +|**Port**|||The port that the remote system is listening on for file transfers<br/>**Supports Expression Language: true**| +|Private Key Passphrase|||Password for the private key<br/>**Supports Expression Language: true**| +|Private Key Path|||The fully qualified path to the Private Key file<br/>**Supports Expression Language: true**| +|Proxy Host|||The fully qualified hostname or IP address of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Port|||The port of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Type|DIRECT|DIRECT<br>HTTP<br>SOCKS<br>|Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN| +|**Remote File**|||The fully qualified filename on the remote system<br/>**Supports Expression Language: true**| +|**Send Keep Alive On Timeout**|true||Indicates whether or not to send a single Keep Alive message when SSH socket times out| +|**Strict Host Key Checking**|false||Indicates whether or not strict enforcement of hosts keys should be applied| +|**Use Compression**|false||Indicates whether or not ZLIB compression should be used when transferring files| +|**Username**|||Username<br/>**Supports Expression Language: true**| +### Properties -## ExecuteSQL +| Name | Description | +| - | - | +|comms.failure|Any FlowFile that could not be fetched from the remote server due to a communications failure will be transferred to this Relationship.| +|not.found|Any FlowFile for which we receive a 'Not Found' message from the remote server will be transferred to this Relationship.| +|permission.denied|Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship.| +|success|All FlowFiles that are received are routed to success| -### Description -Execute provided SQL query. Query result rows will be outputted as new flow -files with attribute keys equal to result column names and values equal to -result values. There will be one output FlowFile per result row. This processor -can be scheduled to run using the standard timer-based scheduling methods, or -it can be triggered by an incoming FlowFile. If it is triggered by an incoming -FlowFile, then attributes of that FlowFile will be available when evaluating -the query. +## FocusArchiveEntry -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Allows manipulation of entries within an archive (e.g. TAR) by focusing on one entry within the archive at a time. When an archive entry is focused, that entry is treated as the content of the FlowFile and may be manipulated independently of the rest of the archive. To restore the FlowFile to its original state, use UnfocusArchiveEntry. +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Connection URL** | | | The database connection URL (e.g. `sqlite://filename.db?cache=shared`) **Only SQLite is currently supported** | -| SQL Statement | | | The SQL query to execute. The query can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL query, to be issued by the processor to the database. Note that Expression Language is not evaluated for flow file contents.<br>**Supports Expression Lang [...] +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Path|||The path within the archive to focus ("/" to focus the total archive)| +### Properties | Name | Description | | - | - | -| original | Upon successful query execution, the original FlowFile is routed here. | -| success | For each SQL result row, a FlowFile will be written to this relationships. Attributes will be written to result FlowFiles having the same names and values as present in result columns. | -| failure | A FlowFile is routed to this relationship if the SQL statement cannot be executed and retrying the operation will also fail, such as an invalid query or an integrity constraint violation. | +|success|success operational on the flow record| + + +## GenerateFlowFile + +### Description + +This processor creates FlowFiles with random data or custom content. GenerateFlowFile is useful for load testing, configuration, and simulation. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Batch Size|1||The number of FlowFiles to be transferred in each invocation| +|Data Format|Binary|Text<br>Binary<br>|Specifies whether the data should be Text or Binary| +|File Size|1 kB||The size of the file that will be used| +|Unique FlowFiles|true||If true, each FlowFile that is generated will be unique. If false, a random value will be generated and all FlowFiles| +### Properties -### Reads Attributes | Name | Description | | - | - | -| sql.args.N.value | Incoming FlowFiles are expected to be parametrized SQL statements. The value of the Parameters are specified as `sql.args.1.value`, `sql.args.2.value`, `sql.args.3.value`, and so on. The type of the `sql.args.1.value` Parameter is specified by the `sql.args.1.type` attribute. | +|success|success operational on the flow record| + ## GetFile -### Description +### Description -Creates FlowFiles from files in a directory. NiFi will ignore files it doesn't -have at least read permissions for. +Creates FlowFiles from files in a directory. MiNiFi will ignore files for which it doesn't have read permissions. +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Batch Size** | 10 | | The maximum number of files to pull in each iteration | -| **Input Directory** | . | | The input directory from which to pull files<br>**Supports Expression Language: true** | -| **Ignore Hidden Files** | true | | Indicates whether or not hidden files should be ignored | -| **Keep Source File** | false | | If true, the file is not deleted after it has been copied to the Content Repository | -| Maximum File Age | 0 sec | | The minimum age that a file must be in order to be pulled any file younger than this amount of time (according to last modification date) will be ignored | -| **Minimum File Age** | 0 sec | | The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored | -| Maximum File Size | 0 B | | The maximum size that a file can be in order to be pulled | -| **Minimum File Size** | 0 B | | The minimum size that a file must be in order to be pulled | -| **Polling Interval** | 0 sec | | Indicates how long to wait before performing a directory listing | -| **Recurse Subdirectories** | true | | Indicates whether or not to pull files from subdirectories | -| **File Filter** | [^\\\\.].\* | | Only files whose names match the given regular expression will be picked up | - -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Batch Size|10||The maximum number of files to pull in each iteration| +|File Filter|[^\.].*||Only files whose names match the given regular expression will be picked up| +|Ignore Hidden Files|true||Indicates whether or not hidden files should be ignored| +|**Input Directory**|.||The input directory from which to pull files<br/>**Supports Expression Language: true**| +|Keep Source File|false||If true, the file is not deleted after it has been copied to the Content Repository| +|Maximum File Age|0 sec||The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored| +|Minimum File Age|0 sec||The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored| +|Minimum File Size|0 B||The maximum size that a file can be in order to be pulled| +|Polling Interval|0 sec||Indicates how long to wait before performing a directory listing| +|Recurse Subdirectories|true||Indicates whether or not to pull files from subdirectories| +### Properties | Name | Description | | - | - | -| success | All files are routed to success | +|success|All files are routed to success| -## GetUSBCamera -### Description +## GetGPS -Gets images from USB Video Class (UVC)-compatible devices. Outputs one flow -file per frame at the rate specified by the `FPS` property in the format -specified by the `Format` property. +### Description -Camera frames are captured in a separate background thread and are emitted as -flow files upon capture. The onTrigger of this processor is a NOOP and will -report an error if inputs are flowed into the processor. Because of this, the -standard event/timer driven scheduling options have no effect. +Obtains GPS coordinates from the GPSDHost and port. +### Properties -If the Width/Height properties are set, the closest supported image frame -dimensions to the given Width/Height properties are used. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -If no Width/Height properties are set, and the camera supports multiple image -size/quality settings, the highest quality is chosen for the given FPS. For -example: +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|GPSD Host|localhost||The host running the GPSD daemon| +|GPSD Port|2947||The GPSD daemon port| +|GPSD Wait Time|50000000||Timeout value for waiting for data from the GPSD instance| +### Properties -- If the FPS is 10 and the camera supports a maximum of 1920x1080 at this FPS, - output images will be 1920x780 -- If the FPS is 60 and the camera supports a maximum of 320x240 at this FPS, - output images will be 320x240 +| Name | Description | +| - | - | +|success|All files are routed to success| -### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +## GetTCP -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **FPS** | 1 | | Frames per second to capture from USB camera | -| Width | | | Desired frame width (closest supported by camera hardware will be used) | -| Height | | | Desired frame height (closest supported by camera hardware will be used) | -| **Format** | PNG | PNG, RAW | Frame format (currently only PNG and RAW are supported; RAW is a binary pixel buffer of RGB values) | -| USB Vendor ID | | | USB Vendor ID of camera device, in hexadecimal format | -| USB Product ID | | | USB Product ID of camera device, in hexadecimal format| -| USB Serial No. | | | USB Serial No. of camera device | +### Description -### Relationships +Establishes a TCP Server that defines and retrieves one or more byte messages from clients +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|SSL Context Service|||SSL Context Service Name| +|Stay Connected|true||Determines if we keep the same socket despite having no data| +|concurrent-handler-count|1||Number of concurrent handlers for this session| +|connection-attempt-timeout|3||Maximum number of connection attempts before attempting backup hosts, if configured| +|end-of-message-byte|13||Byte value which denotes end of message. Must be specified as integer within the valid byte range (-128 thru 127). For example, '13' = Carriage return and '10' = New line. Default '13'.| +|**endpoint-list**|||A comma delimited list of the endpoints to connect to. The format should be <server_address>:<port>.| +|receive-buffer-size|16 MB||The size of the buffer to receive data in. Default 16384 (16MB).| +### Properties | Name | Description | | - | - | -| success | Sucessfully captured images sent here | -| failure | Failures sent here | +|partial|Indicates an incomplete message as a result of encountering the end of message byte trigger| +|success|All files are routed to success| -## GenerateFlowFile -### Description +## GetUSBCamera -This processor creates FlowFiles with random data or custom content. -GenerateFlowFile is useful for load testing, configuration, and simulation. +### Description -### Properties +Gets images from USB Video Class (UVC)-compatible devices. Outputs one flow file per frame at the rate specified by the FPS property in the format specified by the Format property. +### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **File Size** | 1 kB | | The size of the file that will be used | -| **Batch Size** | 1 | | The number of FlowFiles to be transferred in each invocation | -| **Data Format** | Binary | Binary, Text | Specifies whether the data should be Text or Binary| -| **Unique FlowFiles** | true | true, false | If true, each FlowFile that is generated will be unique. If false, a random value will be generated and all FlowFiles will get the same content but this offers much higher throughput | - -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|FPS|1||Frames per second to capture from USB camera| +|Format|PNG||Frame format (currently only PNG and RAW are supported; RAW is a binary pixel buffer of RGB values)| +|Height|||Target height of image to capture from USB camera| +|USB Product ID|0x0||USB Product ID of camera device, in hexadecimal format| +|USB Serial No.|||USB Serial No. of camera device| +|USB Vendor ID|0x0||USB Vendor ID of camera device, in hexadecimal format| +|Width|||Target width of image to capture from USB camera| +### Properties | Name | Description | | - | - | -| success | Generated FlowFiles are sent here | +|failure|Failures sent here| +|success|Sucessfully captured images sent here| -## InvokeHTTP -### Description +## HashContent -An HTTP client processor which can interact with a configurable HTTP Endpoint. -The destination URL and HTTP Method are configurable. FlowFile attributes are -converted to HTTP headers and the FlowFile contents are included as the body of -the request (if the HTTP Method is PUT, POST or PATCH). +### Description -### Properties +HashContent calculates the checksum of the content of the flowfile and adds it as an attribute. Configuration options exist to select hashing algorithm and set the name of the attribute. +### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **HTTP Method** | GET | | HTTP request method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). Arbitrary methods are also supported. Methods other than POST, PUT and PATCH will be sent without a message body. | -| **Remote URL** | | | Remote URL which will be connected to, including scheme, host, port, path.<br>**Supports Expression Language: true** | -| **Connection Timeout** | 5 secs | | Max wait time for connection to remote service. | -| **Read Timeout** | 15 secs | | Max wait time for response from remote service. | -| **Include Date Header** | True | True, False | Include an RFC-2616 Date header in the request. | -| **Follow Redirects** | True | | Follow HTTP redirects issued by remote server. | -| Attributes to Send | | | Regular expression that defines which attributes to send as HTTP headers in the request. If not defined, no attributes are sent as headers. | -| SSL Context Service | | | The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections. | -| Proxy Host | | | The fully qualified hostname or IP address of the proxy server | -| Proxy Port | | | The port of the proxy server | -| invokehttp-proxy-user | | | Username to set when authenticating against proxy | -| invokehttp-proxy-password | | | Password to set when authenticating against proxy | -| **Content-type** | application/octet-stream | | The Content-Type to specify for when content is being transmitted through a PUT, POST or PATCH. In the case of an empty value after evaluating an expression language expression, Content-Type defaults to application/octet-stream| -| send-message-body | true | true, false | If true, sends the HTTP message body on POST/PUT/PATCH requests (default). If false, suppresses the message body and content-type header for these requests. | -| **Use Chunked Encoding** | false | true, false | When POST'ing, PUT'ing or PATCH'ing content set this property to true in order to not pass the 'Content-length' header and instead send 'Transfer-Encoding' with a value of 'chunked'. This will enable the data transfer mechanism which was introduced in HTTP 1.1 to pass data of unknown lengths in chunks. | -| Put Response Body in Attribute | | | If set, the response body received back will be put into an attribute of the original FlowFile instead of a separate FlowFile. The attribute key to put to is determined by evaluating value of this property. | -| Always Output Response | false | true, false | Will force a response FlowFile to be generated and routed to the 'Response' relationship regardless of what the server status code received is or if the processor is configured to put the server response body in the request attribute. In the later configuration a request FlowFile with the response body in the attribute and a typical response FlowFile will be emitted to their respective relationships. | -| Penalize on "No Retry" | false | true, false | Enabling this property will penalize FlowFiles that are routed to the "No Retry" relationship. | +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Hash Algorithm|SHA256||Name of the algorithm used to generate checksum| +|Hash Attribute|Checksum||Attribute to store checksum to| +### Properties -### Relationships +| Name | Description | +| - | - | +|failure|failure operational on the flow record| +|success|success operational on the flow record| + + +## InvokeHTTP + +### Description + +An HTTP client processor which can interact with a configurable HTTP Endpoint. The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH). +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Always Output Response|false||Will force a response FlowFile to be generated and routed to the 'Response' relationship regardless of what the server status code received is | +|Attributes to Send|||Regular expression that defines which attributes to send as HTTP headers in the request. If not defined, no attributes are sent as headers.| +|Connection Timeout|5 secs||Max wait time for connection to remote service.| +|Content-type|application/octet-stream||The Content-Type to specify for when content is being transmitted through a PUT, POST or PATCH. In the case of an empty value after evaluating an expression language expression, Content-Type defaults to| +|Disable Peer Verification|false||Disables peer verification for the SSL session| +|HTTP Method|GET||HTTP request method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). Arbitrary methods are also supported. Methods other than POST, PUT and PATCH will be sent without a message body.| +|Include Date Header|true||Include an RFC-2616 Date header in the request.| +|Proxy Host|||The fully qualified hostname or IP address of the proxy server| +|Proxy Port|||The port of the proxy server| +|Read Timeout|15 secs||Max wait time for response from remote service.| +|Remote URL|||Remote URL which will be connected to, including scheme, host, port, path.<br/>**Supports Expression Language: true**| +|SSL Context Service|||The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.| +|Use Chunked Encoding|false||When POST'ing, PUT'ing or PATCH'ing content set this property to true in order to not pass the 'Content-length' header and instead send 'Transfer-Encoding' with a value of 'chunked'. This will enable the data transfer mechanism which was introduced in HTTP 1.1 to pass data of unknown lengths in chunks.| +|invokehttp-proxy-password|||Password to set when authenticating against proxy| +|invokehttp-proxy-username|||Username to set when authenticating against proxy| +|send-message-body|true||If true, sends the HTTP message body on POST/PUT/PATCH requests (default). If false, suppresses the message body and content-type header for these requests.| +### Properties | Name | Description | | - | - | -| success | All files are routed to success | -| response | A Response FlowFile will be routed upon success (2xx status codes). If the 'Output Response Regardless' property is true then the response will be sent to this relationship regardless of the status code received. | -| retry | The original FlowFile will be routed on any status code that can be retried (5xx status codes). It will have new attributes detailing the request. | -| no retry | The original FlowFile will be routed on any status code that should NOT be retried (1xx, 3xx, 4xx status codes). It will have new attributes detailing the request. | -| failure | The original FlowFile will be routed on any type of connection failure, timeout or general exception. It will have new attributes detailing the request. | +|success|All files are routed to success| + + +## ListSFTP + +### Description + +Performs a listing of the files residing on an SFTP server. For each file that is found on the remote server, a new FlowFile will be created with the filename attribute set to the name of the file on the remote server. This can then be used in conjunction with FetchSFTP in order to fetch those files. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|**Connection Timeout**|30 sec||Amount of time to wait before timing out while creating a connection| +|**Data Timeout**|30 sec||When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems| +|Entity Tracking Initial Listing Target|All Available|All Available<br>Tracking Time Window<br>|Specify how initial listing should be handled. Used by 'Tracking Entities' strategy.| +|Entity Tracking Time Window|||Specify how long this processor should track already-listed entities. 'Tracking Entities' strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed ent [...] +|File Filter Regex|||Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched| +|**Follow symlink**|false||If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories| +|Host Key File|||If supplied, the given file will be used as the Host Key; otherwise, no use host key file will be used| +|**Hostname**|||The fully qualified hostname or IP address of the remote system<br/>**Supports Expression Language: true**| +|Http Proxy Password|||Http Proxy Password<br/>**Supports Expression Language: true**| +|Http Proxy Username|||Http Proxy Username<br/>**Supports Expression Language: true**| +|**Ignore Dotted Files**|true||If true, files whose names begin with a dot (".") will be ignored| +|**Listing Strategy**|Tracking Timestamps|Tracking Entities<br>Tracking Timestamps<br>|Specify how to determine new/updated entities. See each strategy descriptions for detail.| +|Maximum File Age|||The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored| +|Maximum File Size|||The maximum size that a file must be in order to be pulled| +|**Minimum File Age**|0 sec||The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored| +|**Minimum File Size**|0 B||The minimum size that a file must be in order to be pulled| +|Password|||Password for the user account<br/>**Supports Expression Language: true**| +|Path Filter Regex|||When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned| +|**Port**|||The port that the remote system is listening on for file transfers<br/>**Supports Expression Language: true**| +|Private Key Passphrase|||Password for the private key<br/>**Supports Expression Language: true**| +|Private Key Path|||The fully qualified path to the Private Key file<br/>**Supports Expression Language: true**| +|Proxy Host|||The fully qualified hostname or IP address of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Port|||The port of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Type|DIRECT|DIRECT<br>HTTP<br>SOCKS<br>|Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN| +|Remote Path|||The fully qualified filename on the remote system<br/>**Supports Expression Language: true**| +|**Search Recursively**|false||If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories| +|**Send Keep Alive On Timeout**|true||Indicates whether or not to send a single Keep Alive message when SSH socket times out| +|**State File**|ListSFTP||Specifies the file that should be used for storing state about what data has been ingested so that upon restart MiNiFi can resume from where it left off| +|**Strict Host Key Checking**|false||Indicates whether or not strict enforcement of hosts keys should be applied| +|**Target System Timestamp Precision**|Auto Detect|Auto Detect<br>Milliseconds<br>Minutes<br>Seconds<br>|Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision.| +|**Username**|||Username<br/>**Supports Expression Language: true**| +### Properties -## LogAttribute +| Name | Description | +| - | - | +|success|All FlowFiles that are received are routed to success| -### Description -Logs attributes of flow files in the MiNiFi application log. +## ListenHTTP -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Starts an HTTP Server and listens on a given base path to transform incoming requests into FlowFiles. The default URI of the Service will be http://{hostname}:{port}/contentListener. Only HEAD, POST, and GET requests are supported. PUT, and DELETE will result in an error and the HTTP response status code 405. The response body text for all requests, by default, is empty (length of 0). A static response body can be set for a given URI by sending input files to ListenHTTP with the http.typ [...] +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Log Level** | info | trace, debug, info, warn, error | The Log Level to use when logging the Attributes | -| Attributes to Log | | | A comma-separated list of Attributes to Log. If not specified, all attributes will be logged. | -| Attributes to Ignore | | | A comma-separated list of Attributes to ignore. If not specified, no attributes will be ignored. | -| **Log Payload** | false | true, false | If true, the FlowFile's payload will be logged, in addition to its attributes; otherwise, just the Attributes will be logged. | -| Log prefix | | | Log prefix appended to the log lines. It helps to distinguish the output of multiple LogAttribute processors. | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Authorized DN Pattern|.*||A Regular Expression to apply against the Distinguished Name of incoming connections. If the Pattern does not match the DN, the connection will be refused.| +|Base Path|contentListener||Base path for incoming connections| +|HTTP Headers to receive as Attributes (Regex)|||Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes| +|**Listening Port**|80||The Port to listen on for incoming connections. 0 means port is going to be selected randomly.| +|SSL Certificate|||File containing PEM-formatted file including TLS/SSL certificate and key| +|SSL Certificate Authority|||File containing trusted PEM-formatted certificates| +|SSL Minimum Version|SSL2|SSL2<br>SSL3<br>TLS1.0<br>TLS1.1<br>TLS1.2<br>|Minimum TLS/SSL version allowed (SSL2, SSL3, TLS1.0, TLS1.1, TLS1.2)| +|SSL Verify Peer|no|no<br>yes<br>|Whether or not to verify the client's certificate (yes/no)| +### Properties | Name | Description | | - | - | -| success | All FlowFiles are routed to this relationship | +|success|All files are routed to success| -## ListenHTTP -### Description +## ListenSyslog -Starts an HTTP Server and listens on a given base path to transform incoming -requests into FlowFiles. The default URI of the Service will be -`http://{hostname}:{port}/contentListener`. Only HEAD, POST, and GET requests are -supported. PUT, and DELETE will result in an error and the HTTP response -status code 405. - -The response body text for all requests, by default, is empty (length of 0). A -static response body can be set for a given URI by sending input files to -ListenHTTP with the `http.type` attribute set to `response_body`. The response -body FlowFile `filename` attribute is appended to the `Base Path` property -(separated by a `/`) when mapped to incoming requests. The `mime.type` -attribute of the response body FlowFile is used for the `Content-type` header -in responses. Response body content can be cleared by sending an empty (size 0) -FlowFile for a given URI mapping. +### Description -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Listens for Syslog messages being sent to a given port over TCP or UDP. Incoming messages are checked against regular expressions for RFC5424 and RFC3164 formatted messages. The format of each message is: (<PRIORITY>)(VERSION )(TIMESTAMP) (HOSTNAME) (BODY) where version is optional. The timestamp can be an RFC5424 timestamp with a format of "yyyy-MM-dd'T'HH:mm:ss.SZ" or "yyyy-MM-dd'T'HH:mm:ss.S+hh:mm", or it can be an RFC3164 timestamp with a format of "MMM d HH:mm:ss". If an incoming me [...] +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| Base Path | contentListener | | Base path for incoming connections | -| **Listening Port** | | | The Port to listen on for incoming connections | -| SSL Certificate | | | File containing PEM-formatted file including TLS/SSL certificate and key | -| SSL Certificate Authority | | | File containing trusted PEM-formatted certificates | -| SSL Verify Peer | no | yes, no | Whether or not to verify the client's certificate | -| SSL Minimum Version | SSL2 | SSL2, SSL3, TLS1.0, TLS1.1, TLS1.2 | Minimum TLS/SSL version allowed | -| HTTP Headers to receive as Attributes (Regex) | | | Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes | -| Authorized DN Pattern | .\* | | Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Max Batch Size|1||The maximum number of Syslog events to add to a single FlowFile.| +|Max Number of TCP Connections|2||The maximum number of concurrent connections to accept Syslog messages in TCP mode.| +|Max Size of Socket Buffer|1 MB||The maximum size of the socket buffer that should be used.| +|Message Delimiter|\n||Specifies the delimiter to place between Syslog messages when multiple messages are bundled together (see <Max Batch Size> core::Property).| +|Parse Messages|false||Indicates if the processor should parse the Syslog messages. If set to false, each outgoing FlowFile will only.| +|Port|514||The port for Syslog communication| +|Protocol|UDP|UDP<br>TCP<br>|The protocol for Syslog communication.| +|Receive Buffer Size|65507 B||The size of each buffer used to receive Syslog messages.| +### Properties | Name | Description | | - | - | -| success | Relationship for successfully received FlowFiles | +|invalid|SysLog message format invalid| +|success|All files are routed to success| -## ListenSyslog - -### Description -Listens for Syslog messages being sent to a given port over TCP or UDP. -Incoming messages are checked against regular expressions for RFC5424 and -RFC3164 formatted messages. The format of each message is: -(\<PRIORITY\>)(VERSION )(TIMESTAMP) (HOSTNAME) (BODY) where version is -optional. The timestamp can be an RFC5424 timestamp with a format of -"yyyy-MM-dd'T'HH:mm:ss.SZ" or "yyyy-MM-dd'T'HH:mm:ss.S+hh:mm", or it can be an -RFC3164 timestamp with a format of "MMM d HH:mm:ss". If an incoming messages -matches one of these patterns, the message will be parsed and the individual -pieces will be placed in FlowFile attributes, with the original message in the -content of the FlowFile. If an incoming message does not match one of these -patterns it will not be parsed and the syslog.valid attribute will be set to -false with the original message in the content of the FlowFile. Valid messages -will be transferred on the success relationship, and invalid messages will be -transferred on the invalid relationship. +## LogAttribute -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Logs attributes of flow files in the MiNiFi application log. +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Receive Buffer Size** | 65507 B | | The size of each buffer used to receive Syslog messages. Adjust this value appropriately based on the expected size of the incoming Syslog messages. When UDP is selected each buffer will hold one Syslog message. When TCP is selected messages are read from an incoming connection until the buffer is full, or the connection is closed. | -| **Max Size of Socket Buffer** | 1 MB | | The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. | -| **Max Number of TCP Connections** | 2 | | The maximum number of concurrent connections to accept Syslog messages in TCP mode. | -| **Max Batch Size** | 1 | | The maximum number of Syslog events to add to a single FlowFile. If multiple events are available, they will be concatenated along with the \<Message Delimiter\> up to this configured maximum number of messages| -| **Message Delimiter"** | \n | | Specifies the delimiter to place between Syslog messages when multiple messages are bundled together (see \<Max Batch Size\> property). | -| **Parse Messages** | true | true, false | Indicates if the processor should parse the Syslog messages. If set to false, each outgoing FlowFile will only contain the sender, protocol, and port, and no additional attributes. | -| **Port** | 514 | | The port for Syslog communication.| +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Attributes to Ignore|||A comma-separated list of Attributes to ignore. If not specified, no attributes will be ignored.| +|Attributes to Log|||A comma-separated list of Attributes to Log. If not specified, all attributes will be logged.| +|FlowFiles To Log|1||Number of flow files to log. If set to zero all flow files will be logged. Please note that this may block other threads from running if not used judiciously.| +|Log Level||debug<br>error<br>info<br>trace<br>warn<br>|The Log Level to use when logging the Attributes| +|Log Payload|false||If true, the FlowFile's payload will be logged, in addition to its attributes.otherwise, just the Attributes will be logged| +|Log Prefix|||Log prefix appended to the log lines. It helps to distinguish the output of multiple LogAttribute processors.| +### Properties | Name | Description | | - | - | -| success | Syslog messages that match one of the expected formats will be sent out this relationship as a FlowFile per message. | -| invalid | Syslog messages that do not match one of the expected formats will be sent out this relationship as a FlowFile per message. | +|success|success operational on the flow record| -## PutFile -### Description +## ManipulateArchive -Writes the contents of a FlowFile to the local file system +### Description -### Properties +Performs an operation which manipulates an archive without needing to split the archive into multiple FlowFiles. +### Properties -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| Directory | . | | The directory to which files should be written. You may use expression language such as `/aa/bb/${path}`<br>**Supports Expression Language: true** | -| Conflict Resolution Strategy | fail | replace, ignore, fail | Indicates what should happen when a file with the same name already exists in the output directory | -| **Create Missing Directories** | true | true, false | If true, then missing destination directories will be created. If false, flowfiles are penalized and sent to failure. | -| Maximum File Count | | | Specifies the maximum number of files that can exist in the output directory | +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|After|||For operations which result in new entries, places the new entry after the entry specified by this property.| +|Before|||For operations which result in new entries, places the new entry before the entry specified by this property.| +|Destination|||Destination for operations (touch, move or copy) which result in new entries.| +|Operation|||Operation to perform on the archive (touch, remove, copy, move).| +|Target|||An existing entry within the archive to perform the operation on.| +### Properties -### Relationships +| Name | Description | +| - | - | +|failure|FlowFiles will be transferred to the failure relationship if the operation fails.| +|success|FlowFiles will be transferred to the success relationship if the operation succeeds.| + + +## MergeContent + +### Description + +Merges a Group of FlowFiles together based on a user-defined strategy and packages them into a single FlowFile. MergeContent should be configured with only one incoming connection as it won't create grouped Flow Files.This processor updates the mime.type attribute as appropriate. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Correlation Attribute Name|||Correlation Attribute Name| +|Delimiter Strategy|Filename||Determines if Header, Footer, and Demarcator should point to files| +|Demarcator File|||Filename specifying the demarcator to use| +|Footer File|||Filename specifying the footer to use| +|Header File|||Filename specifying the header to use| +|Keep Path|false||If using the Zip or Tar Merge Format, specifies whether or not the FlowFiles' paths should be included in their entry| +|Max Bin Age|||The maximum age of a Bin that will trigger a Bin to be complete. Expected format is <duration> <time unit>| +|Maximum Group Size|||The maximum size for the bundle. If not specified, there is no maximum.| +|Maximum Number of Entries|||The maximum number of files to include in a bundle. If not specified, there is no maximum.| +|Maximum number of Bins|100||Specifies the maximum number of bins that can be held in memory at any one time| +|Merge Format|Binary Concatenation||Merge Format| +|Merge Strategy|Defragment||Defragment or Bin-Packing Algorithm| +|Minimum Group Size|0||The minimum size of for the bundle| +|Minimum Number of Entries|1||The minimum number of files to include in a bundle| +### Properties | Name | Description | | - | - | -| success | Files that have been successfully written to the output directory are transferred to this relationship | -| failure | Files that could not be written to the output directory for some reason are transferred to this relationship | +|failure|If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure| +|merged|The FlowFile containing the merged content| +|original|The FlowFiles that were used to create the bundle| + + +## PublishKafka + +### Description + +This Processor puts the contents of a FlowFile to a Topic in Apache Kafka. The content of a FlowFile becomes the contents of a Kafka message. This message is optionally assigned a key by using the <Kafka Key> Property. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Attributes to Send as Headers|||Any attribute whose name matches the regex will be added to the Kafka messages as a Header| +|Batch Size|||Maximum number of messages batched in one MessageSet| +|**Client Name**|||Client Name to use when communicating with Kafka<br/>**Supports Expression Language: true**| +|Compress Codec|none||compression codec to use for compressing message sets| +|Delivery Guarantee|DELIVERY_ONE_NODE||TSpecifies the requirement for guaranteeing that a message is sent to Kafka<br/>**Supports Expression Language: true**| +|Kerberos Keytab Path|||The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required.| +|Kerberos Principal|||Keberos Principal| +|Kerberos Service Name|||Kerberos Service Name| +|**Known Brokers**|||A comma-separated list of known Kafka Brokers in the format <host>:<port><br/>**Supports Expression Language: true**| +|Max Flow Segment Size|||Maximum flow content payload segment size for the kafka record| +|Max Request Size|||Maximum Kafka protocol request message size| +|Message Key Field|||The name of a field in the Input Records that should be used as the Key for the Kafka message. +Supports Expression Language: true (will be evaluated using flow file attributes)| +|Queue Buffering Max Time|||Delay to wait for messages in the producer queue to accumulate before constructing message batches| +|Queue Max Buffer Size|||Maximum total message size sum allowed on the producer queue| +|Queue Max Message|||Maximum number of messages allowed on the producer queue| +|Request Timeout|||The ack timeout of the producer request in milliseconds<br/>**Supports Expression Language: true**| +|Security CA|||File or directory path to CA certificate(s) for verifying the broker's key| +|Security Cert|||Path to client's public key (PEM) used for authentication| +|Security Pass Phrase|||Private key passphrase| +|Security Private Key|||Path to client's private key (PEM) used for authentication| +|Security Protocol|||Protocol used to communicate with brokers| +|**Topic Name**|||The Kafka Topic of interest<br/>**Supports Expression Language: true**| +### Properties -## PutSQL +| Name | Description | +| - | - | +|failure|Any FlowFile that cannot be sent to Kafka will be routed to this Relationship| +|success|Any FlowFile that is successfully sent to Kafka will be routed to this Relationship| -### Description -Executes a SQL UPDATE or INSERT command. The content of an incoming FlowFile is -expected to be the SQL command to execute. The SQL command may use the `?` -character to bind parameters. In this case, the parameters to use must exist as -FlowFile attributes with the naming convention `sql.args.N.type` and -`sql.args.N.value`, where `N` is a positive integer. The content of the -FlowFile is expected to be in UTF-8 format. +## PublishMQTT -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +PublishMQTT serializes FlowFile content as an MQTT payload, sending the message to the configured topic and broker. +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Connection URL** | | | The database connection URL (e.g. `sqlite://filename.db?cache=shared`) **Only SQLite is currently supported** | -| SQL Statement | | | The SQL statement to execute. The statement can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL statement, to be issued by the processor to the database.<br>**Supports Expression Language: true** | -| **Batch Size** | 1 | | The maximum number of FlowFiles to put to the database in a single transaction | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Broker URI|||The URI to use to connect to the MQTT broker| +|Client ID|||MQTT client ID to use| +|Connection Timeout|30 sec||Maximum time interval the client will wait for the network connection to the MQTT server| +|Keep Alive Interval|60 sec||Defines the maximum time interval between messages sent or received| +|Max Flow Segment Size|||Maximum flow content payload segment size for the MQTT record| +|Password|||Password to use when connecting to the broker| +|Quality of Service|MQTT_QOS_0||The Quality of Service(QoS) to send the message with. Accepts three values '0', '1' and '2'| +|Retain|false||Retain MQTT published record in broker| +|Session state|true||Whether to start afresh or resume previous flows. See the allowable value descriptions for more details| +|Topic|||The topic to publish the message to| +|Username|||Username to use when connecting to the broker| +### Properties | Name | Description | | - | - | -| retry | A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed | -| success | A FlowFile is routed to this relationship after the database is successfully updated | -| failure | A FlowFile is routed to this relationship if the database cannot be updated and retrying the operation will also fail, such as an invalid query or an integrity constraint violation | +|failure|FlowFiles that failed to send to the destination are transferred to this relationship| +|success|FlowFiles that are sent successfully to the destination are transferred to this relationship| -### Reads Attributes -| Name | Description | -| - | - | -| sql.args.N.value | Incoming FlowFiles are expected to be parametrized SQL statements. The value of the Parameters are specified as `sql.args.1.value`, `sql.args.2.value`, `sql.args.3.value`, and so on. The type of the `sql.args.1.value` Parameter is specified by the `sql.args.1.type` attribute. | -## RouteOnAttribute - -### Description +## PutFile -Routes FlowFiles based on their Attributes using the Attribute Expression Language. +### Description -### Dynamic Properties +Writes the contents of a FlowFile to the local file system +### Properties -Dynamic Properties allow the user to specify both the name and value of a property. +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Value | Description | -| - | - | - | -| Relationship Name | Attribute Expression Language | Routes FlowFiles whose attributes match the Attribute Expression Language specified in the Dynamic Property Value to the Relationship specified in the Dynamic Property Key<br>**Supports Expression Language: true** | +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Conflict Resolution Strategy|fail|fail<br>ignore<br>replace<br>|Indicates what should happen when a file with the same name already exists in the output directory| +|**Create Missing Directories**|true||If true, then missing destination directories will be created. If false, flowfiles are penalized and sent to failure.| +|Directory|.||The output directory to which to put files<br/>**Supports Expression Language: true**| +|Maximum File Count|-1||Specifies the maximum number of files that can exist in the output directory| +### Properties -### Relationships +| Name | Description | +| - | - | +|failure|Failed files (conflict, write failure, etc.) are transferred to failure| +|success|All files are routed to success| + + +## PutSFTP + +### Description + +Sends FlowFiles to an SFTP Server +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|**Batch Size**|500||The maximum number of FlowFiles to send in a single connection| +|**Conflict Resolution**|NONE|FAIL<br>IGNORE<br>NONE<br>REJECT<br>RENAME<br>REPLACE<br>|Determines how to handle the problem of filename collisions| +|**Connection Timeout**|30 sec||Amount of time to wait before timing out while creating a connection| +|**Create Directory**|false||Specifies whether or not the remote directory should be created if it does not exist.| +|**Data Timeout**|30 sec||When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems| +|Disable Directory Listing|false||If set to 'true', directory listing is not performed prior to create missing directories. By default, this processor executes a directory listing command to see target directory existence before creating missing directories. However, there are situations that you might need to disable the directory listing such as the following. Directory listing might fail with some permission setups (e.g. chmod 100) on a directory. Also, if any other SFTP client create [...] +|Dot Rename|true||If true, then the filename of the sent file is prepended with a "." and then renamed back to the original once the file is completely sent. Otherwise, there is no rename. This property is ignored if the Temporary Filename property is set.| +|Host Key File|||If supplied, the given file will be used as the Host Key; otherwise, no use host key file will be used| +|**Hostname**|||The fully qualified hostname or IP address of the remote system<br/>**Supports Expression Language: true**| +|Http Proxy Password|||Http Proxy Password<br/>**Supports Expression Language: true**| +|Http Proxy Username|||Http Proxy Username<br/>**Supports Expression Language: true**| +|Last Modified Time|||The lastModifiedTime to assign to the file after transferring it. If not set, the lastModifiedTime will not be changed. Format must be yyyy-MM-dd'T'HH:mm:ssZ. You may also use expression language such as ${file.lastModifiedTime}. If the value is invalid, the processor will not be invalid but will fail to change lastModifiedTime of the file.<br/>**Supports Expression Language: true**| +|Password|||Password for the user account<br/>**Supports Expression Language: true**| +|Permissions|||The permissions to assign to the file after transferring it. Format must be either UNIX rwxrwxrwx with a - in place of denied permissions (e.g. rw-r--r--) or an octal number (e.g. 644). If not set, the permissions will not be changed. You may also use expression language such as ${file.permissions}. If the value is invalid, the processor will not be invalid but will fail to change permissions of the file.<br/>**Supports Expression Language: true**| +|**Port**|||The port that the remote system is listening on for file transfers<br/>**Supports Expression Language: true**| +|Private Key Passphrase|||Password for the private key<br/>**Supports Expression Language: true**| +|Private Key Path|||The fully qualified path to the Private Key file<br/>**Supports Expression Language: true**| +|Proxy Host|||The fully qualified hostname or IP address of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Port|||The port of the proxy server<br/>**Supports Expression Language: true**| +|Proxy Type|DIRECT|DIRECT<br>HTTP<br>SOCKS<br>|Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN| +|Reject Zero-Byte Files|true||Determines whether or not Zero-byte files should be rejected without attempting to transfer| +|Remote Group|||Integer value representing the Group ID to set on the file after transferring it. If not set, the group will not be set. You may also use expression language such as ${file.group}. If the value is invalid, the processor will not be invalid but will fail to change the group of the file.<br/>**Supports Expression Language: true**| +|Remote Owner|||Integer value representing the User ID to set on the file after transferring it. If not set, the owner will not be set. You may also use expression language such as ${file.owner}. If the value is invalid, the processor will not be invalid but will fail to change the owner of the file.<br/>**Supports Expression Language: true**| +|Remote Path|||The path on the remote system from which to pull or push files<br/>**Supports Expression Language: true**| +|**Send Keep Alive On Timeout**|true||Indicates whether or not to send a single Keep Alive message when SSH socket times out| +|**Strict Host Key Checking**|false||Indicates whether or not strict enforcement of hosts keys should be applied| +|Temporary Filename|||If set, the filename of the sent file will be equal to the value specified during the transfer and after successful completion will be renamed to the original filename. If this value is set, the Dot Rename property is ignored.<br/>**Supports Expression Language: true**| +|**Use Compression**|false||Indicates whether or not ZLIB compression should be used when transferring files| +|**Username**|||Username<br/>**Supports Expression Language: true**| +### Properties | Name | Description | | - | - | -| unmatched | FlowFiles that do not match any user-define expression will be routed here | +|failure|FlowFiles that failed to send to the remote system; failure is usually looped back to this processor| +|reject|FlowFiles that were rejected by the destination system| +|success|FlowFiles that are successfully sent will be routed to success| + + +## PutSQL + +### Description + +Executes a SQL UPDATE or INSERT command. The content of an incoming FlowFile is expected to be the SQL command to execute. The SQL command may use the ? character to bind parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args.N.type and sql.args.N.value, where N is a positive integer. The content of the FlowFile is expected to be in UTF-8 format. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Dynamic Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Batch Size|1||The maximum number of flow files to process in one batch| +|Connection URL|||The database URL to connect to| +|SQL Statement|||The SQL statement to execute| +### Properties | Name | Description | | - | - | -| Name from Dynamic Property | FlowFiles that match the Dynamic Property's Attribute Expression Language | +|failure|Failures which will not work if retried| +|retry|Failures which might work if retried| +|success|After a successful put SQL operation, FlowFiles are sent here| -## TailFile -### Description +## RouteOnAttribute -"Tails" a file, or a list of files, ingesting data from the file as it is -written to the file. The file is expected to be textual. Data is ingested only -when a new line is encountered (carriage return or new-line character or -combination). If the file to tail is periodically "rolled over", as is -generally the case with log files, an optional Rolling Filename Pattern can be -used to retrieve data from files that have rolled over, even if the rollover -occurred while NiFi was not running (provided that the data still exists upon -restart of NiFi). It is generally advisable to set the Run Schedule to a few -seconds, rather than running with the default value of 0 secs, as this -Processor will consume a lot of resources if scheduled very aggressively. At -this time, this Processor does not support ingesting files that have been -compressed when 'rolled over'. +### Description -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +Routes FlowFiles based on their Attributes using the Attribute Expression Language. +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **File to Tail** | | | Fully-qualified filename of the file that should be tailed | -| **State File** | TailFileState | | Specifies the file that should be used for storing state about what data has been ingested so that upon restart NiFi can resume from where it left off | -| Input Delimiter | | | Specifies the character that should be used for delimiting the data being tailed from the incoming file. | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +### Properties | Name | Description | | - | - | -| success | All FlowFiles are routed to this Relationship. | +|failure|Failed files are transferred to failure| +|unmatched|Files which do not match any expression are routed here| + -## TFApplyGraph +## TFApplyGraph ### Description @@ -842,7 +1132,6 @@ default values, and whether a property supports the NiFi Expression Language. | success | Successful graph application outputs as tensor protobufs | | retry | Inputs which fail graph application but may work if sent again | | failure | Failures which will not work if retried | - ## TFConvertImageToTensor ### Description @@ -875,7 +1164,6 @@ default values, and whether a property supports the NiFi Expression Language. | - | - | | success | Successfully read tensor protobufs | | failure | Inputs which could not be converted to tensor protobufs | - ## TFExtractTopLabels ### Description @@ -910,306 +1198,63 @@ default values, and whether a property supports the NiFi Expression Language. | success | Successful FlowFiles are sent here with labels as attributes | | retry | Failures which might work if retried | | failure | Failures which will not work if retried | +## TailFile -## MergeContent - -Merges a Group of FlowFiles together based on a user-defined strategy and -packages them into a single FlowFile. It is recommended that the Processor be -configured with only a single incoming connection, as Group of FlowFiles will -not be created from FlowFiles in different connections. This processor updates -the mime.type attribute as appropriate. - -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Merge Strategy** | Defragment | Defragment, Bin-Packing Algorithm | | -| **Merge Format** | Binary Concatenation | TAR, ZIP, "FlowFile Stream, v3", "FlowFile Stream, v2", "FlowFile Tar, v1", Binary Concatenation, Avro | Determines the format that will be used to merge the content. | -| Correlation Attribute Name | | | Correlation Attribute Name | -| **Delimiter Strategy** | Filename | Filename, Text | Determines if Header, Footer, and Demarcator should point to files containing the respective content, or if the values of the properties should be used as the content. | -| Header File | | | Filename specifying the header to use | -| Footer File | | | Filename specifying the footer to use | -| Demarcator File | | | Filename specifying the demarcator to use | -| **Keep Path** | false | false, true | If using the Zip or Tar Merge Format, specifies whether or not the FlowFiles' paths should be included in their entry | - -### Relationships - -| Name | Description | -| - | - | -| original | The FlowFiles that were used to create the bundle | -| merged | The FlowFile containing the merged content | -| failure | If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure | - -## ExtractText - -Extracts the content of a FlowFile and places it into an attribute. - -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Attribute** | | | Attribute to set from content | -| **Size Limit** | 2 MB | | Maximum number of bytes to read into the attribute. 0 for no limit. | - -### Relationships - -| Name | Description | -| - | - | -| success | All FlowFiles are routed to this Relationship. | - -## CompressContent - -Compresses or decompresses the contents of FlowFiles using a user-specified -compression algorithm and updates the mime.type attribute as appropriate - -### Properties +### Description -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. +"Tails" a file, or a list of files, ingesting data from the file as it is written to the file. The file is expected to be textual. Data is ingested only when a new line is encountered (carriage return or new-line character or combination). If the file to tail is periodically "rolled over", as is generally the case with log files, an optional Rolling Filename Pattern can be used to retrieve data from files that have rolled over, even if the rollover occurred while NiFi was not running (pr [...] +### Properties -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Compression Level** | 1 | 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 | The compression level to use; this is valid only when using GZIP compression. | -| **Mode** | compress | compress, decompress | Indicates whether the processor should compress content or decompress content. | -| **Compression Format** | use mime.type attribute | gzip, bzip2, xz-lzma2, lzma, snappy, snappy framed | The compression format to use. | -| **Update Filename** | false | true, false | If true, will remove the filename extension when decompressing data (only if the extension indicates the appropriate compression format) and add the appropriate extension when compressing data | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|File to Tail|||Fully-qualified filename of the file that should be tailed when using single file mode, or a file regex when using multifile mode| +|Input Delimiter|||Specifies the character that should be used for delimiting the data being tailedfrom the incoming file.If none is specified, data will be ingested as it becomes available.| +|State File|TailFileState||Specifies the file that should be used for storing state about what data has been ingested so that upon restart NiFi can resume from where it left off| +|tail-base-directory|||| +|**tail-mode**|Single file|Single file<br>Multiple file<br>|Specifies the tail file mode. In 'Single file' mode only a single file will be watched. In 'Multiple file' mode a regex may be used. Note that in multiple file mode we will still continue to watch for rollover on the initial set of watched files. The Regex used to locate multiple files will be run during the schedule phrase. Note that if rotated files are matched by the regex, those files will be tailed.| +### Properties | Name | Description | | - | - | -| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed | -| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress | +|success|All files are routed to success| -## FocusArchiveEntry - -Allows manipulation of entries within an archive (e.g. TAR) by focusing on one -entry within the archive at a time. When an archive entry is focused, that -entry is treated as the content of the FlowFile and may be manipulated -independently of the rest of the archive. To restore the FlowFile to its -original state, use UnfocusArchiveEntry. - -Archives may be compressed. - -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Path** | | | The path of the archive entry to focus | - -### Relationships - -| Name | Description | -| - | - | -| success | The FlowFile, with the given entry focused, is sent to this relationship | ## UnfocusArchiveEntry -Restores a FlowFile which has had an archive entry focused via -FocusArchiveEntry to its original state. +### Description -### Relationships +Restores a FlowFile which has had an archive entry focused via FocusArchiveEntry to its original state. +### Properties -| Name | Description | -| - | - | -| success | All FlowFiles are routed to this Relationship | +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -## ManipulateArchive - -Performs an operation which manipulates an archive without needing to split the -archive into multiple FlowFiles. - -### Operations +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +### Properties | Name | Description | | - | - | -| `remove` | Removes the target | -| `copy` | Copies the target to a new archive entry | -| `move` | Moves the target to a new archive entry | -| `touch` | Creates a new archive entry | - -### Properties +|success|success operational on the flow record| -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Operation** | | touch, remove, copy, move | Operation to perform on the archive | -| Target | | | An existing entry within the archive to perform the operation on | -| Destination | | | Destination for operations (touch, move or copy) which result in new entries. | -| Before | | | For operations which result in new entries, places the new entry before the entry specified by this property | -| After | | | For operations which result in new entries, places the new entry after the entry specified by this property | - -### Relationships - -| Name | Description | -| - | - | -| success | FlowFiles will be transferred to the success relationship if the operation succeeds. | -| failure | FlowFiles will be transferred to the failure relationship if the operation fails. | - -## PublishKafka - -This Processor puts the contents of a FlowFile to a Topic in Apache Kafka. The -content of a FlowFile becomes the contents of a Kafka message. This message is -optionally assigned a key by using the \<Kafka Key\> Property. - -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Known Brokers** | | | A comma-separated list of known Kafka Brokers in the format \<host\>:\<port\> | -| **Topic Name** | | | The Kafka Topic of interest | -| **Delivery Guarantee** | 1 | all, 1, 0 | Specifies the requirement for guaranteeing that a message is sent to Kafka | -| Request Timeout | | | The ack timeout of the producer request in milliseconds | -| Client Name | | | Client Name to use when communicating with Kafka | -| Batch Size | | | Maximum number of messages batched in one MessageSet | -| Attributes to Send as Headers | | | Any attribute whose name matches the regex will be added to the Kafka messages as a Header | -| Queue Buffering Max Time | | | Delay to wait for messages in the producer queue to accumulate before constructing message batches | -| Queue Max Buffer Size | | | Maximum total message size sum allowed on the producer queue | -| Queue Max Message | | | Maximum number of messages allowed on the producer queue | -| Compress Codec | none | none, gzip, snappy | compression codec to use for compressing message sets | -| Max Flow Segment Size | | | Maximum flow content payload segment size for the kafka record | -| Security Protocol | | plaintext, ssl, sasl\_plaintext, sasl\_ssl | Protocol used to communicate with brokers | -| Security Cert | | | Path to client's public key (PEM) used for authentication | -| Security Private Key | | | Path to client's private key (PEM) used for authentication | -| Security Pass Phrase | | | Private key passphrase | - -### Relationships - -| Name | Description | -| - | - | -| success | Any FlowFile that is successfully sent to Kafka will be routed to this Relationship | -| failure | Any FlowFile that cannot be sent to Kafka will be routed to this Relationship | - -## PublishMQTT - -This Processor puts the contents of a FlowFile to a MQTT broker for a sepcified topic. The -content of a FlowFile becomes the payload of the MQTT message. - -### Properties - -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Broker URI** | | | The URI to use to connect to the MQTT broker | -| **Topic** | | | The topic to publish the message to | -| Session state | | | Whether to start afresh or resume previous flows | -| Client ID | | | MQTT client ID to use | -| Username | | | Username to use when connecting to the broker | -| Password | | | Password to use when connecting to the broker | -| Keep Alive Interval | | | Defines the maximum time interval between messages sent or received | -| Connection Timeout | | | Maximum time interval the client will wait for the network connection to the MQTT server | -| Quality of Service | | | The Quality of Service(QoS) to send the message with. Accepts three values '0', '1' and '2' | -| Retain | | | Retain MQTT published record in broker | -| Max Flow Segment Size | | Maximum flow content payload segment size for the MQTT record | - -### Relationships - -| Name | Description | -| - | - | -| success | Any FlowFile that is successfully sent to broker will be routed to this Relationship | -| failure | Any FlowFile that cannot be sent to broker will be routed to this Relationship | +## UpdateAttribute -## ConsumeMQTT +### Description -This Processor gets the contents of a FlowFile from a MQTT broker for a sepcified topic. The -the payload of the MQTT message becomes content of a FlowFile +This processor updates the attributes of a FlowFile using properties that are added by the user. This allows you to set default attribute changes that affect every FlowFile going through the processor, equivalent to the "basic" usage in Apache NiFi. +### Properties -### Properties +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -In the list below, the names of required properties appear in bold. Any other -properties (not in bold) are considered optional. The table also indicates any -default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -| **Broker URI** | | | The URI to use to connect to the MQTT broker | -| **Topic** | | | The topic to publish the message to | -| Session state | | | Whether to start afresh or resume previous flows | -| Client ID | | | MQTT client ID to use | -| Username | | | Username to use when connecting to the broker | -| Password | | | Password to use when connecting to the broker | -| Keep Alive Interval | | | Defines the maximum time interval between messages sent or received | -| Connection Timeout | | | Maximum time interval the client will wait for the network connection to the MQTT server | -| Quality of Service | | | The Quality of Service(QoS) to send the message with. Accepts three values '0', '1' and '2' | -| Max Flow Segment Size | | Maximum flow content payload segment size for the MQTT record | - -### Relationships +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +### Properties | Name | Description | | - | - | -| success | Any FlowFile that is successfully sent to broker will be routed to this Relationship | +|failure|Failed files are transferred to failure| +|success|All files are routed to success| -## UpdateAttribute - -This processor updates the attributes of a FlowFile using properties that are -added by the user. This allows you to set default attribute changes that affect -every FlowFile going through the processor, equivalent to the "basic" usage in -Apache NiFi. -### Properties - -The properties in this processor are added by the user. The expression language -is supported in user-added properties for this processor. See the [NiFi -Expression Language Guide](EXPRESSIONS.md) to learn how to formulate proper -expression language statements to perform the desired functions. - -### Relationships - -| Name | Description | -| - | - | -| success | If the processor successfully updates the specified attribute(s), then the FlowFile follows this relationship. | - -### Basic Usage - -For basic usage, changes are made by adding a new processor property and -referencing as its name the attribute you want to change. Then enter the -desired attribute value as the Value. The Value can be as simple as any text -string or it can be a NiFi Expression Language statement that specifies how to -formulate the value. (See the [NiFi Expression Language Usage -Guide](EXPRESSIONS.md) for details on crafting NiFi Expression Language -statements.) - -As an example, to alter the standard `filename` attribute so that it has `.txt` -appended to the end of it, add a new property and make the property name -`filename` (to reference the desired attribute), and as the value, use the NiFi -Expression Language statement shown below: - -- **Property**: `filename` -- **Value**: `${filename}.txt` - -The preceding example illustrates how to modify an existing attribute. If an -attribute does not already exist, this processor can also be used to add a new -attribute. For example, the following property could be added to create a new -attribute called `myAttribute` that has the value `myValue`: - -- **Property**: `myAttribute` -- **Value**: `myValue` - -In this example, all FlowFiles passing through this processor will receive an -additional FlowFile attribute called `myAttribute` with the value `myValue`. This -type of configuration might be used in a flow where you want to tag every -FlowFile with an attribute so that it can be used later in the flow, such as -for routing in a `RouteOnAttribute` processor.
