Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.ISPEnrichIP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.ISPEnrichIP/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.ISPEnrichIP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.ISPEnrichIP/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ISPEnrichIP</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ISPEnrichIP</h1><h2>Description: </h2><p>Looks up ISP information for an IP address and adds the information to FlowFile attributes. The ISP data is provided as a MaxMind ISP database (Note that this is NOT the same as the GeoLite database utilizedby some geo enrichment tools). The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the the attributes added by enrichment will take the form X.isp.<fieldName></p><h3>Tags: </h3><p>ISP, enrich, ip, maxmind</p><h3>Properties: </h3><p>In the list belo w, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>MaxMind Database File</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Path to Maxmind IP Enrichment Database File</td></tr><tr><td id="name"><strong>IP Address Attribute</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The name of an attribute whose value is a dotted decimal IP address for which enrichment should occur<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>D escription</th></tr><tr><td>not found</td><td>Where to route flow files after unsuccessfully enriching attributes because no data was found</td></tr><tr><td>found</td><td>Where to route flow files after successfully enriching attributes with data provided by database</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>X.isp.lookup.micros</td><td>The number of microseconds that the geo lookup took</td></tr><tr><td>X.isp.asn</td><td>The Autonomous System Number (ASN) identified for the IP address</td></tr><tr><td>X.isp.asn.organization</td><td>The Organization Associated with the ASN identified</td></tr><tr><td>X.isp.name</td><td>The name of the ISP associated with the IP address provided</td></tr><tr><td>X.isp.organization</td><td>The Organization associated with the IP address provided</td></tr></table><h3>State management: </h3>This component does not store state.<h3>R estricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html> \ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryDNS/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryDNS/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryDNS/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryDNS/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,2 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>QueryDNS</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">QueryDNS</h1><h2>Description: </h2><p>A powerful DNS query processor primary designed to enrich DataFlows with DNS based APIs (e.g. RBLs, ShadowServer's ASN lookup) but that can be also used to perform regular DNS lookups.</p><h3>Tags: </h3><p>dns, enrich, ip</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><t able id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Lookup value</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The value that should be used to populate the query<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Results Parser</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>Split <img src="../../../../../html/images/iconInfo.png" alt="Use a delimiter character or RegEx to split the results into attributes" title="Use a delimiter character or RegEx to split the results into attributes"></img></li><li>RegEx <img src="../../../../../html/images/iconInfo.png" alt="Use a regular expression to split the results into attributes " title="Use a regular expression to split the results into attributes "></img></li><li>None <img src="../../../../../html/images/iconInfo.png" alt="Do not split results" title="Do not split results"></img></li></ul></td><td id="description">The method used to slice the results into attribute groups</td></tr><tr><td id="name">Parser RegEx</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Choice between a splitter and regex matcher used to parse the results of the query into attribute groups. +NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters.</td></tr><tr><td id="name"><strong>DNS Query Retries</strong></td><td id="default-value">1</td><td id="allowable-values"></td><td id="description">The number of attempts before giving up and moving on</td></tr><tr><td id="name"><strong>DNS Query Timeout</strong></td><td id="default-value">1500 ms</td><td id="allowable-values"></td><td id="description">The amount of time to wait until considering a query as failed</td></tr><tr><td id="name">DNS Servers</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma separated list of DNS servers to be used. (Defaults to system wide if none is used)</td></tr><tr><td id="name"><strong>DNS Query Type</strong></td><td id="default-value">TXT</td><td id="allowable-values"></td><td id="description">The DNS query type to be used by the processor (e.g. TXT, A)</td></tr></table><h3>Relationships : </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>not found</td><td>Where to route flow files if data enrichment query rendered no results</td></tr><tr><td>found</td><td>Where to route flow files after successfully enriching attributes with data</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>enrich.dns.record*.group*</td><td>The captured fields of the DNS query response for each of the records received</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryWhois/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryWhois/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryWhois/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.3.0/org.apache.nifi.processors.enrich.QueryWhois/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,2 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>QueryWhois</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">QueryWhois</h1><h2>Description: </h2><p>A powerful whois query processor primary designed to enrich DataFlows with whois based APIs (e.g. ShadowServer's ASN lookup) but that can be also used to perform regular whois lookups.</p><h3>Tags: </h3><p>whois, enrich, ip</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>. </p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Lookup value</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The value that should be used to populate the query<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Whois Query Type</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The Whois query type to be used by the processor (if used)</td></tr><tr><td id="name"><strong>Whois Server</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The Whois server to be used</td></tr><tr><td id="name"><strong>Whois Server Port</strong></td><td id="default-value">43</td><td id="allowable-values"></td><td id="description">The TCP port of the remote Whois server</td></tr><tr><td id="name"><strong>Whois Query Timeout</strong></td><td id="default-value">1500 m s</td><td id="allowable-values"></td><td id="description">The amount of time to wait until considering a query as failed</td></tr><tr><td id="name"><strong>Batch Size</strong></td><td id="default-value">25</td><td id="allowable-values"></td><td id="description">The number of incoming FlowFiles to process in a single execution of this processor. </td></tr><tr><td id="name"><strong>Bulk Protocol</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>Begin/End <img src="../../../../../html/images/iconInfo.png" alt="The evaluated input of each flowfile is enclosed within begin and end tags. Each row contains a delimited set of fields" title="The evaluated input of each flowfile is enclosed within begin and end tags. Each row contains a delimited set of fields"></img></li><li>None <img src="../../../../../html/images/iconInfo.png" alt="Queries are made without any particular dialect" title="Queries are made without any particular dialect"></img></li></ul></td><td id="description">The protocol used to perform the bulk query. </td></tr><tr><td id="name"><strong>Results Parser</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>Split <img src="../../../../../html/images/iconInfo.png" alt="Use a delimiter character or RegEx to split the results into attributes" title="Use a delimiter character or RegEx to split the results into attributes"></img></li><li>RegEx <img src="../../../../../html/images/iconInfo.png" alt="Use a regular expression to split the results into attributes " title="Use a regular expression to split the results into attributes "></img></li><li>None <img src="../../../../../html/images/iconInfo.png" alt="Do not split results" title="Do not split results"></img></li></ul></td><td id="description">The method used to slice the results into attribute groups</td></tr><tr><td id="name">Parser RegEx</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Choice between a splitte r and regex matcher used to parse the results of the query into attribute groups. +NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters.</td></tr><tr><td id="name">Key lookup group (multiline / batch)</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">When performing a batched lookup, the following RegEx numbered capture group or Column number will be used to match the whois server response with the lookup field</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>not found</td><td>Where to route flow files if data enrichment query rendered no results</td></tr><tr><td>found</td><td>Where to route flow files after successfully enriching attributes with data</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>enrich.dns.record*.group*</td><td>The captured fields of the Whois query response for each of the records received</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,91 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>ParseEvtx</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <!-- Processor Documentation ================================================== --> + <h2>Description:</h2> + <p>This processor is used to parse Windows event logs in the binary evtx format. The input flow files' content should be evtx files. The processor has 4 outputs: + <ul> + <li>The original unmodified FlowFile</li> + <li>The XML resulting from parsing at the configured granularity</li> + <li>Failed parsing with partial output</li> + <li>Malformed chunk in binary form</li> + </ul> + </p> + <h2>Output XML Example:</h2> + <p> + <pre> +<?xml version="1.0"?> +<Events> + <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> + <System> + <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" Ev +entSourceName="Service Control Manager"/> + <EventID Qualifiers="16384">7036</EventID> + <Version>0</Version> + <Level>4</Level> + <Task>0</Task> + <Opcode>0</Opcode> + <Keywords>0x8080000000000000</Keywords> + <TimeCreated SystemTime="2016-01-08 16:49:47.518"/> + <EventRecordID>780</EventRecordID> + <Correlation ActivityID="" RelatedActivityID=""/> + <Execution ProcessID="480" ThreadID="596"/> + <Channel>System</Channel> + <Computer>win7-pro-vm</Computer> + <Security UserID=""/> + </System> + <EventData> + <Data Name="param1">Workstation</Data> + <Data Name="param2">running</Data> + <Binary>TABhAG4AbQBhAG4AVwBvAHIAawBzAHQAYQB0AGkAbwBuAC8ANAAAAA==</Binary> + </EventData> + </Event> + <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> + <System> + <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager"/> + <EventID Qualifiers="16384">7036</EventID> + <Version>0</Version> + <Level>4</Level> + <Task>0</Task> + <Opcode>0</Opcode> + <Keywords>0x8080000000000000</Keywords> + <TimeCreated SystemTime="2016-01-08 16:49:47.535"/> + <EventRecordID>781</EventRecordID> + <Correlation ActivityID="" RelatedActivityID=""/> + <Execution ProcessID="480" ThreadID="576"/> + <Channel>System</Channel> + <Computer>win7-pro-vm</Computer> + <Security UserID=""/> + </System> + <EventData> + <Data Name="param1">Cryptographic Services</Data> + <Data Name="param2">running</Data> + <Binary>QwByAHkAcAB0AFMAdgBjAC8ANAAAAA==</Binary> + </EventData> + </Event> +</Events> + </pre> + </p> +</body> +</html> Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.3.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ParseEvtx</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ParseEvtx</h1><h2>Description: </h2><p>Parses the contents of a Windows Event Log file (evtx) and writes the resulting XML to the FlowFile</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>logs, windows, event, evtx, message, file</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td i d="name"><strong>Granularity</strong></td><td id="default-value">Chunk</td><td id="allowable-values"><ul><li>Record</li><li>Chunk</li><li>File</li></ul></td><td id="description">Output flow file for each Record, Chunk, or File encountered in the event log</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Any FlowFile that was successfully converted from evtx to XML</td></tr><tr><td>failure</td><td>Any FlowFile that encountered an exception during conversion will be transferred to this relationship with as much parsing as possible done</td></tr><tr><td>original</td><td>The unmodified input FlowFile will be transferred to this relationship</td></tr><tr><td>bad chunk</td><td>Any bad chunks of records will be transferred to this relationship in their original binary form</td></tr></table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td> <td>The filename of the evtx file</td></tr></table><h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The output filename</td></tr><tr><td>mime.type</td><td>The output filetype (application/xml for success and failure relationships, original value for bad chunk and original relationships)</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,155 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8" /> + <title>ExecuteFlumeSink</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> +</head> + +<body> +<h2>Data Model</h2> +<p> + This processor executes an Apache Flume sink. FlowFiles are wrapped in Flume's + Event interface. The content of the FlowFile becomes the body of the Event and + the attributes of the FlowFile become Event headers. The following special + headers are also set: +</p> +<table id="headers"> + <tr> + <th>Flume Event Header</th> + <th>FlowFile Attribute</th> + </tr> + <tr> + <td>nifi.entry.date</td> + <td>FlowFile#getEntryDate()</td> + </tr> + <tr> + <td>nifi.id</td> + <td>FlowFile#getId()</td> + </tr> + <tr> + <td>nifi.last.queue.date</td> + <td>FlowFile#getLastQueueDate()</td> + </tr> + <tr> + <td>nifi.lineage.start.date</td> + <td>FlowFile#getLineageStartDate()</td> + </tr> + <tr> + <td>nifi.size</td> + <td>FlowFile#getSize()</td> + </tr> +</table> +<h2>Warning</h2> +<p> + In NiFi, the contents of a FlowFile are accessed via a stream, but in Flume it is + stored in a byte array. This means the full content will be loaded into memory when + a FlowFile is processed by the ExecuteFlumeSink processor. You should consider the + typical size of the FlowFiles you'll process and the batch size, if any, your sink + is configured with when setting NiFi's heap size. +</p> +<h2>Configuration Details</h2> +<p> + This processor is designed to execute arbitrary Flume sinks. Most of the details + of configuring the sink is deferred to Flume's built-in configuration system. + For details on the available settings for each sink type, refer to the Flume + <a href="http://flume.apache.org/FlumeUserGuide.html#flume-sinks";>User Guide</a>. + Configuring the Flume sink is a four step process: +</p> +<ol> + <li>Set the Sink Type property to a valid Flume sink type.</li> + <li> + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: <code>tier1</code> + </li> + <li> + Set the Sink Name property to the name of the sink in your Flume + configuration. If Agent Name is <code>tier1</code>, then the Sink Name + is the value of the <code>tier1.sinks</code> property. Example: <code>sink-1</code> + </li> + <li> + Copy and paste the configuration for the sink from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Sink Name as in the examples above, this will be all + of the properties that start with <code>tier1.sinks.sink-1</code>. + Do not copy the <code>tier1.sinks.sink-1.type</code> or + <code>tier1.sinks.sink-1.channel</code> properties. + </li> +</ol> +<h2>Usage Example</h2> +<p> + Assuming you had the following existing Flume configuration file: +</p> + <pre> +a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = netcat +a1.sources.r1.bind = localhost +a1.sources.r1.port = 44444 + +a1.sinks.k1.type = hdfs +a1.sinks.k1.channel = c1 +a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S +a1.sinks.k1.hdfs.filePrefix = events- +a1.sinks.k1.hdfs.round = true +a1.sinks.k1.hdfs.roundValue = 10 +a1.sinks.k1.hdfs.roundUnit = minute + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1</pre> +<p> + Then you'd configure the ExecuteFlumeSink as follows: +</p> +<table id="example"> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td>Sink Type</td> + <td>hdfs</td> + </tr> + <tr> + <td>Agent Name</td> + <td>a1</td> + </tr> + <tr> + <td>Sink Name</td> + <td>k1</td> + </tr> + <tr> + <td>Flume Configuration</td> + <td> + <code> + a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S<br> + a1.sinks.k1.hdfs.filePrefix = events-<br> + a1.sinks.k1.hdfs.round = true<br> + a1.sinks.k1.hdfs.roundValue = 10<br> + a1.sinks.k1.hdfs.roundUnit = minute + </code> + </td> + </tr> +</table> +</body> +</html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ExecuteFlumeSink</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ExecuteFlumeSink</h1><h2>Description: </h2><p>Execute a Flume sink. Each input FlowFile is converted into a Flume Event for processing by the sink.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>flume, hadoop, put, sink, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></t r><tr><td id="name"><strong>Sink Type</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The component type name for the sink. For some sinks, this is a short, symbolic name (e.g. hdfs). For others, it's the fully-qualified name of the Sink class. See the Flume User Guide for details.</td></tr><tr><td id="name"><strong>Agent Name</strong></td><td id="default-value">tier1</td><td id="allowable-values"></td><td id="description">The name of the agent used in the Flume sink configuration</td></tr><tr><td id="name"><strong>Sink Name</strong></td><td id="default-value">sink-1</td><td id="allowable-values"></td><td id="description">The name of the sink used in the Flume sink configuration</td></tr><tr><td id="name"><strong>Flume Configuration</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The Flume configuration for the sink copied from the flume.properties file</td></tr></table><h3>Relationships: </h3> <table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td></td></tr><tr><td>failure</td><td></td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,114 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8" /> + <title>ExecuteFlumeSource</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> +</head> + +<body> +<h2>Data Model</h2> +<p> + This processor executes an Apache Flume source. Each Flume Event is turned into + a FlowFile. The content of the FlowFile is set to the body of the Event and + the Event headers become FlowFile attributes. +</p> +<h2>Configuration Details</h2> +<p> + This processor is designed to execute arbitrary Flume sources. Most of the details + of configuring the source is deferred to Flume's built-in configuration system. + For details on the available settings for each source type, refer to the Flume + <a href="http://flume.apache.org/FlumeUserGuide.html#flume-sources";>User Guide</a>. + Configuring the Flume source is a four step process: +</p> +<ol> + <li>Set the Source Type property to a valid Flume source type.</li> + <li> + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: <code>tier1</code> + </li> + <li> + Set the Source Name property to the name of the source in your Flume + configuration. If Agent Name is <code>tier1</code>, then the Source Name + is the value of the <code>tier1.sources</code> property. Example: <code>src-1</code> + </li> + <li> + Copy and paste the configuration for the source from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Source Name as in the examples above, this will be all + of the properties that start with <code>tier1.sources.src-1</code>. + Do not copy the <code>tier1.sources.src-1.type</code> or + <code>tier1.sources.src-1.channel</code> properties. + </li> +</ol> +<h2>Usage Example</h2> +<p> + Assuming you had the following existing Flume configuration file: +</p> + <pre> +a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = multiport_syslogtcp +a1.sources.r1.channels = c1 +a1.sources.r1.host = 0.0.0.0 +a1.sources.r1.ports = 10001 10002 10003 +a1.sources.r1.portHeader = port + +a1.sinks.k1.type = logger + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1</pre> +<p> + Then you'd configure the ExecuteFlumeSource as follows: +</p> +<table id="example"> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td>Source Type</td> + <td>multiport_syslogtcp</td> + </tr> + <tr> + <td>Agent Name</td> + <td>a1</td> + </tr> + <tr> + <td>Source Name</td> + <td>r1</td> + </tr> + <tr> + <td>Flume Configuration</td> + <td> + <code> + a1.sources.r1.host = 0.0.0.0<br> + a1.sources.r1.ports = 10001 10002 10003<br> + a1.sources.r1.portHeader = port + </code> + </td> + </tr> +</table> +</body> +</html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.3.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ExecuteFlumeSource</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ExecuteFlumeSource</h1><h2>Description: </h2><p>Execute a Flume source. Each Flume Event is sent to the success relationship as a FlowFile</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>flume, hadoop, get, source, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr ><td id="name"><strong>Source Type</strong></td><td >id="default-value"></td><td id="allowable-values"></td><td >id="description">The component type name for the source. For some sources, >this is a short, symbolic name (e.g. spooldir). For others, it's the >fully-qualified name of the Source class. See the Flume User Guide for >details.</td></tr><tr><td id="name"><strong>Agent Name</strong></td><td >id="default-value">tier1</td><td id="allowable-values"></td><td >id="description">The name of the agent used in the Flume source >configuration</td></tr><tr><td id="name"><strong>Source Name</strong></td><td >id="default-value">src-1</td><td id="allowable-values"></td><td >id="description">The name of the source used in the Flume source >configuration</td></tr><tr><td id="name"><strong>Flume >Configuration</strong></td><td id="default-value"></td><td >id="allowable-values"></td><td id="description">The Flume configuration for >the source copied from the flume.properties file</td></tr></table><h3>Rela tionships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td></td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.<h3>Input requirement: </h3>This component does not allow an incoming relationship.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GCPCredentialsControllerService</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GCPCredentialsControllerService</h1><h2>Description: </h2><p>Defines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by `gcloud auth application-default login`, AppEngine/Compute Engine service accounts, etc.</p><h3>Tags: </h3><p>gcp, credentials, provider</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any oth er properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Use Application Default Credentials</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the A pp Engine service account, and the Compute Engine service account.</td></tr><tr><td id="name">Use Compute Engine Credentials</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on.</td></tr><tr><td id="name">Service Account JSON File</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Path to a file containing a Service Account key file in JSON format.</td></tr><tr><td id="name">Service Account JSON</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The raw JSON containing a Service Account keyfile.<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>DeleteGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">DeleteGCSObject</h1><h2>Description: </h2><p>Deletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.</p><h3>Tags: </h3><p>google cloud, gcs, google, storage, delete</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properti es"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucke t</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Generation</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The generation of the object to be deleted. If null, will use latest version of the object.<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cloud S torage operation fails.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html">ListGCSBucket</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchGCSObject</h1><h2>Description: </h2><p>Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.</p><h3>Tags: </h3><p>google cloud, google, storage, gcs, fetch</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number o f retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Object Generation</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The generation of the Object to download. If null, will download latest generation.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Server Side Encryption Key</td><td id="default-value"></td><td id="al lowable-values"></td><td id="description">An AES256 Key (encoded in base64) which the object has been encrypted in.<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cloud Storage operation fails.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file, parsed if possible from the Content-Disposition response header</td></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the object .</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mime.type</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</td></tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the object</td ></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the >object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for >the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the >object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data >encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download >link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The >metageneration of the object.</td></tr><tr><td>gcs.owner</td><td>The owner >(uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL >entity type of the uploader of the >object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a >string.</td></tr></table><h3>State management: </h3>This component does not >store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input >requirement: </h3>This component requires an incoming relationship.<h3>See >Also:</h3><p><a >href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/in dex.html">ListGCSBucket</a>, <a href="../org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ListGCSBucket</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ListGCSBucket</h1><h2>Description: </h2><p>Retrieves a listing of objects from an GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.</p><h3>Tags: </h3><p>google cloud, google, storage, gcs, list</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</str ong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id="des cription">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Bucket of the object.</td></tr><tr><td id="name">Prefix</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The prefix used to filter the object list. In most cases, it should end with a forward slash ('/').</td></tr><tr><td id="name"><strong>Use Generations</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Specifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage oper ation.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file</td></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the object.</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mime.type</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</td></ tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the object</td></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The metageneration of the object.</td></tr><tr><td>gcs.owner</td><td>The owner (uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL entity type of the uploader of the object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a string.</td></tr></table><h3>State management: </h3><table id="stateful"><tr><th>Scope</th><th>Description</th></tr><tr><td>CLUSTER</td><td>After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.</td></tr></table><h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component does not allow an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processor s.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.3.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>PutGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">PutGCSObject</h1><h2>Description: </h2><p>Puts flow files to a Google Cloud Bucket.</p><h3>Tags: </h3><p>google, google cloud, gcs, archive, put</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td><td i d="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Content Type</td><td id="default-value">${mime.type}</td><td id="allowable-values"></td><td id="description">Content Type for the file, i.e. text/plain<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">MD5 Hash</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">MD5 Hash (encoded in Base64) of the file for server-side validation .<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">CRC32C Checksum</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">CRC32C Checksum (encoded in Base64, big-Endian order) of the file for server-side validation.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Object ACL</td><td id="default-value"></td><td id="allowable-values"><ul><li>All Authenticated Users <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER and WRITER permissions. All other permissions are removed." title="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER and WRITER permissions. All other permissions are removed."></img></li><li>Authenticated Read <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER perm ission, and gives all authenticated Google account holders READER permission. All other permissions are removed." title="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER permission. All other permissions are removed."></img></li><li>Bucket Owner Full Control <img src="../../../../../html/images/iconInfo.png" alt="Gives the object and bucket owners OWNER permission. All other permissions are removed." title="Gives the object and bucket owners OWNER permission. All other permissions are removed."></img></li><li>Bucket Owner Read Only <img src="../../../../../html/images/iconInfo.png" alt="Gives the object owner OWNER permission, and gives the bucket owner READER permission. All other permissions are removed." title="Gives the object owner OWNER permission, and gives the bucket owner READER permission. All other permissions are removed."></img></li><li>Private <img src="../../../../../html/images/iconInfo.png" alt="Gives the b ucket or object owner OWNER permission for a bucket or object, and removes all other access permissions." title="Gives the bucket or object owner OWNER permission for a bucket or object, and removes all other access permissions."></img></li><li>Project Private <img src="../../../../../html/images/iconInfo.png" alt="Gives permission to the project team based on their roles. Anyone who is part of the team has READER permission. Project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed." title="Gives permission to the project team based on their roles. Anyone who is part of the team has READER permission. Project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed."></img></li><li>Public Read Only <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission, and gives all users, both authenticated and anonymous, READER permission. When you apply this to an object, anyone on the Internet can read the object without authenticating." title="Gives the bucket or object owner OWNER permission, and gives all users, both authenticated and anonymous, READER permission. When you apply this to an object, anyone on the Internet can read the object without authenticating."></img></li></ul></td><td id="description">Access Control to be attached to the object uploaded. Not providing this will revert to bucket defaults.</td></tr><tr><td id="name">Server Side Encryption Key</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br/><strong>Sensitive Property: true</strong><br/><strong>Support s Expression Language: true</strong></td></tr><tr><td id="name"><strong>Overwrite Object</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If false, the upload to GCS will succeed only if the object does not exist.</td></tr><tr><td id="name">Content Disposition Type</td><td id="default-value"></td><td id="allowable-values"><ul><li>Inline <img src="../../../../../html/images/iconInfo.png" alt="Indicates that the object should be loaded and rendered within the browser." title="Indicates that the object should be loaded and rendered within the browser."></img></li><li>Attachment <img src="../../../../../html/images/iconInfo.png" alt="Indicates that the object should be saved (using a Save As... dialog) rather than opened directly within the browser" title="Indicates that the object should be saved (using a Save As... dialog) rather than opened directly within the browser"></img></li></ul></td><td id="des cription">Type of RFC-6266 Content Disposition to be attached to the object</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">The name of a User-Defined Metadata field to add to the GCS Object</td><td id="value">The value of a User-Defined Metadata field to add to the GCS Object</td><td>Allows user-defined metadata to be added to the GCS object as key/value pairs<br/><strong>Supports Expression Language: true</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cloud Storage operation fails.</td></tr></table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>Uses the FlowFile's filename as the filename for the GCS object</td></tr><tr><td>mime.type</td><td>Uses the FlowFile's MIME type as the content-type for the GCS object</td></tr></table><h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the object.</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mime.ty pe</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</td></tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the object</td></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The metageneration of the object.</td></tr><tr><td>gcs.owner</td><td>The owner (uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL entity type of the uploader of the object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a string.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html">ListGCSBucket</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1,46 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>CreateHadoopSequenceFile</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <!-- Processor Documentation ================================================== --> + <h2>Description:</h2> + <p>This processor is used to create a Hadoop Sequence File, which essentially is a file of key/value pairs. The key + will be a file name and the value will be the flow file content. The processor will take either a merged (a.k.a. packaged) flow + file or a singular flow file. Historically, this processor handled the merging by type and size or time prior to creating a + SequenceFile output; it no longer does this. If creating a SequenceFile that contains multiple files of the same type is desired, + precede this processor with a <code>RouteOnAttribute</code> processor to segregate files of the same type and follow that with a + <code>MergeContent</code> processor to bundle up files. If the type of files is not important, just use the + <code>MergeContent</code> processor. When using the <code>MergeContent</code> processor, the following Merge Formats are + supported by this processor: + <ul> + <li>TAR</li> + <li>ZIP</li> + <li>FlowFileStream v3</li> + </ul> + The created SequenceFile is named the same as the incoming FlowFile with the suffix '.sf'. For incoming FlowFiles that are + bundled, the keys in the SequenceFile are the individual file names, the values are the contents of each file. + </p> + NOTE: The value portion of a key/value pair is loaded into memory. While there is a max size limit of 2GB, this could cause memory + issues if there are too many concurrent tasks and the flow file sizes are large. +</body> +</html> Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html?rev=1798216&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.3.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html Fri Jun 9 13:20:11 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>CreateHadoopSequenceFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">CreateHadoopSequenceFile</h1><h2>Description: </h2><p>Creates Hadoop Sequence Files from incoming flow files</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>hadoop, sequence file, create, sequencefile</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hado op Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Relogin Period</td><td id="default-value">4 hours</td><td id="allowable-values"></td><td id="description">Period of time which should pass befo re attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories that will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name">Compression type</td><td id="default-value"></td><td id="allowable-values"><ul><li>NONE</li><li>RECORD</li><li>BLOCK</li></ul></td><td id="description">Type of compression to use when creating Sequence File</td></tr><tr><td id="name"><strong>Compression codec</strong></td><td id="default-value">NONE</td><td id="allowable-values"><ul><li>NONE <img src="../../../../../html/images/iconInfo.png" alt="No compression" title="No compression"></img></li><li>DEFAULT <img src="../../../../../html/images/iconInfo.png" alt="Default ZLIB compression" title="Defa ult ZLIB compression"></img></li><li>BZIP <img src="../../../../../html/images/iconInfo.png" alt="BZIP compression" title="BZIP compression"></img></li><li>GZIP <img src="../../../../../html/images/iconInfo.png" alt="GZIP compression" title="GZIP compression"></img></li><li>LZ4 <img src="../../../../../html/images/iconInfo.png" alt="LZ4 compression" title="LZ4 compression"></img></li><li>LZO <img src="../../../../../html/images/iconInfo.png" alt="LZO compression - it assumes LD_LIBRARY_PATH has been set and jar is available" title="LZO compression - it assumes LD_LIBRARY_PATH has been set and jar is available"></img></li><li>SNAPPY <img src="../../../../../html/images/iconInfo.png" alt="Snappy compression" title="Snappy compression"></img></li><li>AUTOMATIC <img src="../../../../../html/images/iconInfo.png" alt="Will attempt to automatically detect the compression codec." title="Will attempt to automatically detect the compression codec."></img></li></ul></td><td id="description">No Description Provided.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Generated Sequence Files are sent to this relationship</td></tr><tr><td>failure</td><td>Incoming files that failed to generate a Sequence File are sent to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.PutHDFS/index.html">PutHDFS</a></p></body></html> \ No newline at end of file
