Author: aldrin
Date: Wed Dec 28 21:43:39 2016
New Revision: 1776334
URL: http://svn.apache.org/viewvc?rev=1776334&view=rev
Log:
Updating site with artifacts from MiNiFi 0.1.0 release.
Modified:
nifi/site/trunk/minifi/minifi-toolkit.html
nifi/site/trunk/minifi/system-admin-guide.html
Modified: nifi/site/trunk/minifi/minifi-toolkit.html
URL:
http://svn.apache.org/viewvc/nifi/site/trunk/minifi/minifi-toolkit.html?rev=1776334&r1=1776333&r2=1776334&view=diff
==============================================================================
--- nifi/site/trunk/minifi/minifi-toolkit.html (original)
+++ nifi/site/trunk/minifi/minifi-toolkit.html Wed Dec 28 21:43:39 2016
@@ -140,8 +140,8 @@
<li><p>Linux / OS X</p>
<pre><code>$ ./config.sh</code></pre>
</li>
-<li><p>Windows</p>
-<pre><code>execute bin/config.bat</code></pre>
+<li><p>Windows
+ execute bin/config.bat</p>
</li>
</ul>
</li>
@@ -155,6 +155,7 @@ transform: Transform template xml into M
validate: Validate config YAML</code></pre>
</li>
</ul>
+<p>Note: Currently MiNiFi does not support external Input Ports or Output
Ports. Any templates that contain these will fail transformation.</p>
<h2 id="getting-help">Getting Help</h2>
<p>If you have questions, you can reach out to our mailing list:
[email protected]
(<a href="https://mail-archives.apache.org/mod_mbox/nifi-dev";>archive</a>).
Modified: nifi/site/trunk/minifi/system-admin-guide.html
URL:
http://svn.apache.org/viewvc/nifi/site/trunk/minifi/system-admin-guide.html?rev=1776334&r1=1776333&r2=1776334&view=diff
==============================================================================
--- nifi/site/trunk/minifi/system-admin-guide.html (original)
+++ nifi/site/trunk/minifi/system-admin-guide.html Wed Dec 28 21:43:39 2016
@@ -104,12 +104,245 @@
<div class="large-space"></div>
<h1 id="minifi-system-administrator-s-guide">MiNiFi System
Administrator's Guide</h1>
<p><a href="[email protected]">Apache NiFi Team</a></p>
-<p><a href="https://nifi.apache.org";>NiFi Homepage</a></p>
-<h1 id="flowstatus-query-options">FlowStatus Query Options</h1>
-<p>From the minifi.sh script there is the ability to query to get the current
status of the flow. This section will give an overview of the different
options.</p>
-<p>Note: Currently the script only accepts one high level option at a time.
Also any names of connections, remote process groups or processors that contain
":", ";" or "," will cause parsing errors when
querying.</p>
-<h2 id="processors">Processors</h2>
-<p>To query the processors use the "processor" flag followed by the
id of the processor to get (or "all") followed by one of the
processor options. The processor options are below.</p>
+<p><a href="https://nifi.apache.org/minifi/index.html";>MiNiFi Homepage</a></p>
+<h1 id="automatic-warm-redeploy">Automatic Warm-Redeploy</h1>
+<p>When many MiNiFi agents running on the edge, it may not be possible to
manually stop, edit the config.yml and then restart every one every time their
configuration needs to change. The Config Change Coordinator and its Ingestors
were designed to automatically redeploy in response to a configuration
update.</p>
+<p>The Config Change Ingestors are the means by which the agent is notified of
a potential new configuration. Currently there are three:</p>
+<ul>
+<li>FileChangeIngestor</li>
+<li>RestChangeIngestor</li>
+<li>PullHttpChangeIngestor</li>
+</ul>
+<p>After a new configuration has been pulled/received the Ingestors use a
Differentiator in order to determine if the currently running config is
different than the new config. Which Differentiator is used, is configurable
for each Ingestor. Currently there is only one Differentiator:</p>
+<ul>
+<li>WholeConfigDifferentiator: Compares the entire new config with the
currently running one, byte for byte.</li>
+</ul>
+<p>After a new config is determined to be new, the MiNiFi agent will attempt
to restart. The bootstrap first saves the old config into a swap file. The
bootstrap monitors the agent as it restarts and if it fails it will roll back
to the old config. If it succeeds then the swap file will be deleted and the
agent will start processing using the new config.</p>
+<p><strong>Note:</strong> Data left in connections when the agent attempts to
restart will either be mapped to a connection with the same ID in the new
config, or orphaned and deleted.</p>
+<p>The configuration for Warm-Redeploy is done in the bootstrap.conf and
primarily revolve around the Config Change Ingestors. The configuration in the
bootstrap.conf is done using the "nifi.minifi.notifier.ingestors" key
followed by the full path name of the desired Ingestor implementation to run.
Use a comma separated list to define more than one Ingestor implementation.
For example:</p>
+<pre><code>nifi.minifi.notifier.ingestors=org.apache.nifi.minifi.bootstrap.configuration.ingestors.PullHttpChangeIngestor</code></pre>
+<p>Ingestor specific configuration is also necessary and done in the
bootstrap.conf as well. Specifics for each are detailed below.</p>
+<h2 id="filechangeingestor">FileChangeIngestor</h2>
+<p>class name:
org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestor</p>
+<p>This Config Change Ingestor watches a file and when the file is updated,
the file is ingested as a new config.</p>
+<p>Note: The config file path configured here and in
"nifi.minifi.config" cannot be the same. This is due to the swapping
mechanism and other implementation limitations.</p>
+<p>Below are the configuration options. The file config path is the only
required property.</p>
+<table>
+<thead>
+<tr>
+<th>Option</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>nifi.minifi.notifier.ingestors.file.config.path</td>
+<td>Path of the file to monitor for changes. When these occur, the
FileChangeNotifier, if configured, will begin the configuration reloading
process</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.file.polling.period.seconds</td>
+<td>How frequently the file specified by
'nifi.minifi.notifier.file.config.path' should be evaluated for
changes. If not set then a default polling period of 15 seconds will be
used.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.file.differentiator</td>
+<td>Which differentiator to use. If not set then it uses the
WholeConfigDifferentiator as a default.</td>
+</tr>
+</tbody>
+</table>
+<h2 id="restchangeingestor">RestChangeIngestor</h2>
+<p>class name:
org.apache.nifi.minifi.bootstrap.configuration.ingestors.RestChangeIngestor</p>
+<p>This Config Change Ingestor sets up a light-weight Jetty HTTP(S) REST
service in order to listen to HTTP(S) requests. A potential new configuration
is sent via a POST request with the BODY being the potential new config.</p>
+<p><strong>Note:</strong> The encoding is expected to be Unicode and the exact
version specified by the BOM mark ('UTF-8','UTF-16BE' or
'UTF-16LE'). If there is no BOM mark, then UTF-8 is used.</p>
+<p>Here is an example post request using 'curl' hitting the local
machine on pot 8338 and it is executed with the config file
"config.yml" in the directory the command is run from:</p>
+<pre><code>curl --request POST --data-binary "@config.yml"
http://localhost:8338/</code></pre>
+<p>Below are the configuration options. There are no required options. If no
properties are set then the server will bind to hostname "localhost"
on a random open port, will only connect via HTTP and will use the
WholeConfigDifferentiator.</p>
+<table>
+<thead>
+<tr>
+<th>Option</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.host</td>
+<td>Hostname on which the Jetty server will bind to. If not specified then it
will bind to localhost.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.port</td>
+<td>Port on which the Jetty server will bind to. If not specified then it will
bind to a random open port.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.truststore.location</td>
+<td>If using HTTPS, this specifies the location of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.truststore.password</td>
+<td>If using HTTPS, this specifies the password of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.truststore.type</td>
+<td>If using HTTPS, this specifies the type of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.keystore.location</td>
+<td>If using HTTPS, this specifies the location of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.keystore.password</td>
+<td>If using HTTPS, this specifies the password of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.keystore.type</td>
+<td>If using HTTPS, this specifies the type of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.need.client.auth</td>
+<td>If using HTTPS, this specifies whether or not to require client
authentication.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.receive.http.differentiator</td>
+<td>Which differentiator to use. If not set then it uses the
WholeConfigDifferentiator as a default.</td>
+</tr>
+</tbody>
+</table>
+<h2 id="pullhttpchangeingestor">PullHttpChangeIngestor</h2>
+<p>class name:
org.apache.nifi.minifi.bootstrap.configuration.ingestors.PullHttpChangeIngestor</p>
+<p>This Config Change Ingestor periodically sends a GET request to a REST
endpoint using HTTP(S) to order to pull the potential new config.</p>
+<p>Below are the configuration options. The hostname and port are the only
required properties.</p>
+<table>
+<thead>
+<tr>
+<th>Option</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.hostname</td>
+<td>Hostname on which to pull configurations from</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.port</td>
+<td>Port on which to pull configurations from</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.path</td>
+<td>Path on which to pull configurations from</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.period.ms</td>
+<td>Period on which to pull configurations from, defaults to 5 minutes if not
set.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.use.etag</td>
+<td>If the destination server is set up with cache control ability and
utilizes an "ETag" header, then this should be set to true to utilize
it. Very simply, the Ingestor remembers the "ETag" of the last
successful pull (returned 200) then uses that "ETag" in a
"If-None-Match" header on the next request.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.connect.timeout.ms</td>
+<td>Sets the connect timeout for new connections. A value of 0 means no
timeout, otherwise values must be a positive whole number in milliseconds.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.read.timeout.ms</td>
+<td>Sets the read timeout for new connections. A value of 0 means no timeout,
otherwise values must be a positive whole number in milliseconds.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.truststore.location</td>
+<td>If using HTTPS, this specifies the location of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.truststore.password</td>
+<td>If using HTTPS, this specifies the password of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.truststore.type</td>
+<td>If using HTTPS, this specifies the type of the truststore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.keystore.location</td>
+<td>If using HTTPS, this specifies the location of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.keystore.password</td>
+<td>If using HTTPS, this specifies the password of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.keystore.type</td>
+<td>If using HTTPS, this specifies the type of the keystore.</td>
+</tr>
+<tr>
+<td>nifi.minifi.notifier.ingestors.pull.http.differentiator</td>
+<td>Which differentiator to use. If not set then it uses the
WholeConfigDifferentiator as a default.</td>
+</tr>
+</tbody>
+</table>
+<h1 id="status-reporting-and-querying">Status Reporting and Querying</h1>
+<p>In NiFi there is a lot of information, such as stats and bulletins, that is
only available to view through the UI. MiNiFi provides access to this
information through a query mechanism. You can query FlowStatus either using
the MiNiFi.sh script or by configuring one of the Periodic Status Reporters.
The API for the query is the same for the reporters and the
"flowStatus" script option. The API is outlined in the
"FlowStatus Query Options" section below.</p>
+<h2 id="flowstatus-script-query">FlowStatus Script Query</h2>
+<p>From the minifi.sh script, you can manually query to get the current status
of your dataflow. The following is an example of a minifi.sh query you might
run to view health, stats, and bulletins for the TailFile processor. This query
returns information to your command-line.</p>
+<pre><code>minifi.sh flowStatus
processor:TailFile:health,stats,bulletins</code></pre>
+<p>Currently the script only accepts one high level option at a time. Also any
names of connections, remote process groups, or processors that contain "
" (a space), ":", ";" or "," cause parsing
errors when querying.</p>
+<h2 id="periodic-status-reporters">Periodic Status Reporters</h2>
+<p>You can set up Periodic Status Reporters to periodically report the status
of your dataflow. The query executes at configurable intervals and the results
are reported using the configured implementation. Configure the Reporters in
the bootstrap.conf file, using the
"nifi.minifi.status.reporter.components" key followed by the full
path name of the desired Reporter implementation to run. Use a comma separated
list to define more than one Reporter implementation. For example:</p>
+<pre><code>nifi.minifi.status.reporter.components=org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger</code></pre>
+<h3 id="statuslogger">StatusLogger</h3>
+<p>class name:
org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger</p>
+<p>The Periodic Status Reporter logs the results of the query to the logs. By
default it is logged to the minifi-bootstrap.log but you can modify logback.xml
to log to an alternate file and location.</p>
+<table>
+<thead>
+<tr>
+<th>Option</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>nifi.minifi.status.reporter.log.query</td>
+<td>The FlowStatus query to run.</td>
+</tr>
+<tr>
+<td>nifi.minifi.status.reporter.log.level</td>
+<td>The log level at which to log the status. Available options are
"TRACE", "DEBUG", "INFO", "WARN" and
"ERROR".</td>
+</tr>
+<tr>
+<td>nifi.minifi.status.reporter.log.period</td>
+<td>The delay (in milliseconds) between each query.</td>
+</tr>
+</tbody>
+</table>
+<p>Example bootstrap.conf configuration:</p>
+<pre><code># The FlowStatus query to submit to the MiNiFi instance
+nifi.minifi.status.reporter.log.query=instance:health,bulletins
+# The log level at which the status will be logged
+nifi.minifi.status.reporter.log.level=INFO
+# The period (in milliseconds) at which to log the status
+nifi.minifi.status.reporter.log.period=60000</code></pre>
+<p>Example logback.xml configuration to output the status to its own rolling
log file:</p>
+<pre><code><appender name="STATUS_LOG_FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
+ <file>logs/minifi-status.log</file>
+ <rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+ <!--
+ For daily rollover, use 'user_%d.log'.
+ For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
+ To GZIP rolled files, replace '.log' with '.log.gz'.
+ To ZIP rolled files, replace '.log' with '.log.zip'.
+ -->
+
<fileNamePattern>./logs/minifi-status_%d.log</fileNamePattern>
+ <!-- keep 5 log files worth of history -->
+ <maxHistory>5</maxHistory>
+ </rollingPolicy>
+ <encoder
class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
+ <pattern>%date %level [%thread] %logger{40}
%msg%n</pattern>
+ </encoder>
+</appender>
+
+<logger
name="org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger"
level="INFO" additivity="false">
+ <appender-ref ref="STATUS_LOG_FILE" />
+</logger></code></pre>
+<h2 id="flowstatus-query-options">FlowStatus Query Options</h2>
+<p>This section outlines each option to query the MiNiFi instance for the
FlowStatus.</p>
+<h3 id="processors">Processors</h3>
+<p>To query the processors use the "processor" flag followed by the
processor ID to get (or "all") followed by one of the processor
options. The processor options are below.</p>
<table>
<thead>
<tr>
@@ -134,8 +367,8 @@
</table>
<p>An example query to get the health, bulletins and stats of the
"TailFile" processor is below.</p>
<pre><code>minifi.sh flowStatus
processor:TailFile:health,stats,bulletins</code></pre>
-<h2 id="connections">Connections</h2>
-<p>To query the connections use the "connection" flag followed by
the id of the connection to get (or "all") followed by one of the
connection options. The connection options are below.</p>
+<h3 id="connections">Connections</h3>
+<p>To query the connections use the "connection" flag followed by
the connection ID to get (or "all") followed by one of the connection
options. The connection options are below.</p>
<table>
<thead>
<tr>
@@ -156,8 +389,8 @@
</table>
<p>An example query to get the health and stats of the "TailToS2S"
connection is below.</p>
<pre><code>minifi.sh flowStatus connection:TailToS2S:health,stats</code></pre>
-<h2 id="remote-process-groups">Remote Process Groups</h2>
-<p>To query the remote process groups (RPG) use the
"remoteProcessGroup" flag followed by the id of the remote process
group to get (or "all") followed by one of the remote process group
options. The remote process group options are below.</p>
+<h3 id="remote-process-groups">Remote Process Groups</h3>
+<p>To query the remote process groups (RPG) use the
"remoteProcessGroup" flag followed by the RPG ID to get (or
"all") followed by one of the remote process group options. The
remote process group options are below.</p>
<table>
<thead>
<tr>
@@ -175,10 +408,6 @@
<td>A list of all the current bulletins (if there are any).</td>
</tr>
<tr>
-<td>authorizationIssues</td>
-<td>A list of all the current authorization issues (if there are any).</td>
-</tr>
-<tr>
<td>inputPorts</td>
<td>A list of every input port for this RPG and their status. Their status
includes it's name, whether the target exit and whether it's currently
running.</td>
</tr>
@@ -188,9 +417,9 @@
</tr>
</tbody>
</table>
-<p>An example query to get the health, bulletins, authorization issues, input
ports and stats of all the RPGS is below.</p>
-<pre><code>minifi.sh flowStatus
remoteprocessinggroup:all:health,bulletins,authorizationIssues,inputports,stats</code></pre>
-<h2 id="controller-services">Controller Services</h2>
+<p>An example query to get the health, bulletins, input ports and stats of all
the RPGS is below.</p>
+<pre><code>minifi.sh flowStatus
remoteprocessgroup:all:health,bulletins,inputports,stats</code></pre>
+<h3 id="controller-services">Controller Services</h3>
<p>To query the controller services use the "controllerServices"
flag followed by one of the controller service options. The controller service
options are below.</p>
<table>
<thead>
@@ -212,7 +441,7 @@
</table>
<p>An example query to get the health and bulletins of all the controller
services is below.</p>
<pre><code>minifi.sh flowStatus
controllerservices:health,bulletins</code></pre>
-<h2 id="provenance-reporting">Provenance Reporting</h2>
+<h3 id="provenance-reporting">Provenance Reporting</h3>
<p>To query the status of the provenance reporting use the
"provenancereporting" flag followed by one of the provenance
reporting options. The provenance reporting options are below.</p>
<table>
<thead>
@@ -234,8 +463,8 @@
</table>
<p>An example query to get the health and bulletins of the provenance
reporting is below.</p>
<pre><code>minifi.sh flowStatus
provenancereporting:health,bulletins</code></pre>
-<h2 id="instance">Instance</h2>
-<p>To query the status of the MiNiFi instance in general use the
"instance" flag followed by one of the instance options. The instance
options are below.</p>
+<h3 id="instance">Instance</h3>
+<p>To query the status of the MiNiFi instance use the "instance"
flag followed by one of the instance options. The instance options are
below.</p>
<table>
<thead>
<tr>
@@ -260,7 +489,7 @@
</table>
<p>An example query to get the health, stats and bulletins of the instance is
below.</p>
<pre><code>minifi.sh flowStatus instance:health,stats,bulletins</code></pre>
-<h2 id="system-diagnostics">System Diagnostics</h2>
+<h3 id="system-diagnostics">System Diagnostics</h3>
<p>To query the system diagnostics use the "systemdiagnostics" flag
followed by one of the system diagnostics options. The system diagnostics
options are below.</p>
<table>
<thead>
@@ -294,7 +523,7 @@
</table>
<p>An example query to get the heap, processor stats, content repository
usage, FlowFile repository usage and garbage collection from the system
diagnostics is below.</p>
<pre><code>minifi.sh flowStatus
systemdiagnostics:heap,processorstats,contentrepositoryusage,flowfilerepositoryusage,garbagecollection</code></pre>
-<h2 id="example">Example</h2>
+<h3 id="example">Example</h3>
<p>This is an example of a simple query to get the health of all the
processors and its results from a simple flow:</p>
<pre><code>User:minifi-0.0.1-SNAPSHOT user ./bin/minifi.sh flowStatus
processor:all:health
@@ -303,16 +532,32 @@ MiNiFi home: /Users/user/projects/nifi-m
Bootstrap Config File:
/Users/user/projects/nifi-minifi/minifi-assembly/target/minifi-0.0.1-SNAPSHOT-bin/minifi-0.0.1-SNAPSHOT/conf/bootstrap.conf
-Args
-flowStatus
-processor:all:health
-FlowStatusReport{controllerServiceStatusList=null,
processorStatusList=[{name='TailFile',
processorHealth={runStatus='Running', hasBulletins=false,
validationErrorList=[]}, processorStats=null,
-bulletinList=null}], connectionStatusList=null,
remoteProcessingGroupStatusList=null, instanceStatus=null,
systemDiagnosticsStatus=null, reportingTaskStatusList=null,
errorsGeneratingReport=[]}</code></pre>
+{"controllerServiceStatusList":null,"processorStatusList":[{"name":"Connection
Diagnostics","processorHealth":{"runStatus":"Running","hasBulletins":false,"validationErrorList":[]},"processorStats":null,"bulletinList":null},{"name":"UpdateAttribute","processorHealth":{"runStatus":"Running","hasBulletins":false,"validationErrorList":[]},"processorStats":null,"bulletinList":null},{"name":"Processor
Diagnostics","processorHealth":{"runStatus":"Running","hasBulletins":false,"validationErrorList":[]},"processorStats":null,"bulletinList":null},{"name":"System
Diagnostics","processorHealth":{"runStatus":"Running","hasBulletins":false,"validat
ionErrorList":[]},"processorStats":null,"bulletinList":null},{"name":"GenerateFlowFile","processorHealth":{"runStatus":"Running","hasBulletins":false,"validationErrorList":[]},"processorStats":null,"bulletinList":null}],"connectionStatusList":null,"remoteProcessGroupStatusList":null,"instanceStatus":null,"systemDiagnosticsStatus":null,"reportingTaskStatusList":null,"errorsGeneratingReport":[]}</code></pre>
+<h1 id="periodic-status-reporters">Periodic Status Reporters</h1>
<h1 id="config-file">Config File</h1>
<p>The config.yml in the <em>conf</em> directory is the main configuration
file for controlling how MiNiFi runs. This section provides an overview of the
properties in this file. The file is a YAML
and follows the YAML format laid out <a
href="http://www.yaml.org/";>here</a>.</p>
-<p>NOTE: Note that values for periods of time and data sizes must include the
unit of measure,
+<p>Alternatively, the MiNiFi Toolkit Converter can aid in creating a
config.yml from a generated template exported from a NiFi instance. This
+tool can be downloaded from <a
href="http://nifi.apache.org/minifi/download.html";>http://nifi.apache.org/minifi/download.html</a>
under the <code>MiNiFi Toolkit Binaries</code> section. Information on the
toolkit's usage is
+available at <a
href="https://nifi.apache.org/minifi/minifi-toolkit.html";>https://nifi.apache.org/minifi/minifi-toolkit.html</a>.</p>
+<p><strong>Note:</strong> Values for periods of time and data sizes must
include the unit of measure,
for example "10 sec" or "10 MB", not simply
"10".</p>
+<h2 id="versioning">Versioning</h2>
+<p>The "MiNiFi Config Version" property is used to indicate to the
configuration parser which version of the config file it is looking at. If the
property is empty or missing, version 1 is assumed.</p>
+<p>The MiNiFi Toolkit Converter is capable of parsing previous versions
(possibly subject to a future deprecation policy) and writing out the current
version. It can also validate that a given config file
+parses and upconverts to the current version without issue.</p>
+<h3 id="version-1-version-2-changes">Version 1 -> Version 2 changes</h3>
+<ol>
+<li>Use ids instead of names for processors, connections.</li>
+<li>Allow multiple source relationships for connections.</li>
+<li>Added support for process groups, and internal input ports an output
ports.</li>
+<li>Change Id Key for RPGs from "Remote Processing Groups" to the
proper "Remote Process Groups" (not "ing").</li>
+</ol>
+<h3 id="version-2-version-3-changes">Version 2 -> Version 3 changes</h3>
+<ol>
+<li>Added support for Controller Services.</li>
+<li>Added support for Site-To-Site over proxy.</li>
+</ol>
<h2 id="flow-controller">Flow Controller</h2>
<p>The first section of config.yml is for naming and commenting on the
file.</p>
<table>
@@ -324,6 +569,10 @@ for example "10 sec" or "
</thead>
<tbody>
<tr>
+<td>MiNiFi Config Version</td>
+<td>The version of the configuration file. The default value if this property
is missing or empty is 1, the current value is 2.</td>
+</tr>
+<tr>
<td>name</td>
<td>The name of the file.</td>
</tr>
@@ -470,7 +719,7 @@ FlowFile Repository, if also on that dis
</tr>
</tbody>
</table>
-<h2 id="-component-status-repository-"><em>Component Status
Repository</em></h2>
+<h2 id="component-status-repository">Component Status Repository</h2>
<p>The Component Status Repository contains the information for the Component
Status History tool in the User Interface. These
properties govern how that tool works.</p>
<p>The buffer.size and snapshot.frequency work together to determine the
amount of historical data to retain. As an example to
@@ -496,7 +745,7 @@ of 576.</p>
</tr>
</tbody>
</table>
-<h2 id="-security-properties-"><em>Security Properties</em></h2>
+<h2 id="security-properties">Security Properties</h2>
<p>These properties pertain to various security features in NiFi. Many of
these properties are covered in more detail in the
Security Configuration section of this Administrator's Guide.</p>
<table>
@@ -541,6 +790,7 @@ Security Configuration section of this A
</tr>
</tbody>
</table>
+<p><strong>Note:</strong> A StandardSSLContextService will be made
automatically with the ID "SSL-Context-Service" if "ssl
protocol" is configured.</p>
<h4 id="sensitive-properties-subsection">Sensitive Properties Subsection</h4>
<p>Some properties for processors are marked as <em>sensitive</em> and should
be encrypted. These following properties will be used to encrypt the properties
while in use by MiNiFi. This will currently <em>not</em> be used to encrypt
properties in the config file.</p>
<table>
@@ -566,7 +816,7 @@ Security Configuration section of this A
</tbody>
</table>
<h2 id="processors">Processors</h2>
-<p>The current implementation of MiNiFi supports multiple processors. the
"Processors" subsection is a list of these processors. Each processor
must specify these properties. They are the basic configuration general to all
processor implementations. Make sure that all relationships for a processor are
accounted for in the auto-terminated relationship list or are used in a
connection.</p>
+<p>The current implementation of MiNiFi supports multiple processors. The
"Processors" subsection is a list of these processors. Each processor
must specify these properties. They are the basic configuration general to all
processor implementations. Make sure that all relationships for a processor are
accounted for in the auto-terminated relationship list or are used in a
connection.</p>
<table>
<thead>
<tr>
@@ -580,6 +830,10 @@ Security Configuration section of this A
<td>The name of what this processor will do. This is not used for any
underlying implementation but solely for the users of this configuration and
MiNiFi agent.</td>
</tr>
<tr>
+<td>id</td>
+<td>The id of this processor. This can be omitted but in processors without
this field, there should not be any duplicate names and connections will need
to specify source and destination name instead of id. If set it should be a
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+<tr>
<td>class</td>
<td>The fully qualified java class name of the processor to run. For example
for the standard TailFile processor it would be:
org.apache.nifi.processors.standard.TailFile</td>
</tr>
@@ -611,6 +865,10 @@ Security Configuration section of this A
<td>auto-terminated relationships list</td>
<td>A YAML list of the relationships to auto-terminate for the processor.</td>
</tr>
+<tr>
+<td>annotation data</td>
+<td>Some processors make use of "Annotation Data" in order to do
more complex configuration, such as the Advanced portion of UpdateAttribute.
This data will be unique to each implementing processor and more than likely
will not be written out manually.</td>
+</tr>
</tbody>
</table>
<h4 id="processor-properties">Processor Properties</h4>
@@ -620,6 +878,146 @@ Security Configuration section of this A
Rolling Filename Pattern: nifi-app*
State File: ./conf/state/tail-file
Initial Start Position: Beginning of File</code></pre>
+<h3 id="controller-services">Controller Services</h3>
+<p>The current implementation of MiNiFi supports Controller Services. The
"Controller Services" subsection is a list of these services. Each
Controller Service must specify the following properties. They are the basic
configuration general to all Controller Service implementations.</p>
+<table>
+<thead>
+<tr>
+<th><em>Property</em></th>
+<th><em>Description</em></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>name</td>
+<td>The name of what this Controller Service will do. This is not used for any
underlying implementation but solely for the users of this configuration and
MiNiFi agent.</td>
+</tr>
+<tr>
+<td>id</td>
+<td>The id of this Controller Service. This must be a valid UUID. To reference
this Controller Service in the properties of another component, this ID is
used.</td>
+</tr>
+<tr>
+<td>type</td>
+<td>The fully qualified java class name of the processor to run. For example
for the standard StandardSSLContextService processor would be:
org.apache.nifi.ssl.StandardSSLContextService</td>
+</tr>
+</tbody>
+</table>
+<p><strong>Note:</strong> If the "Security Properties" is configured
with an "ssl protocol" then a StandardSSLContextService will be made
automatically with the ID "SSL-Context-Service".</p>
+<h4 id="controller-service-properties">Controller Service Properties</h4>
+<p>Within the Controller Service Configuration section, there is the
<code>Properties</code> subsection. The keys and values in this section are the
property names and values for the service. For example the
StandardSSLContextService would have a section like this:</p>
+<pre><code>Properties:
+ Keystore Filename: /tmp/localhost/keystore.jks
+ Keystore Password: keystorePassword
+ Keystore Type: JKS
+ SSL Protocol: TLS
+ Truststore Filename: /tmp/truststore.jks
+ Truststore Password: truststorePassword
+ Truststore Type: JKS
+ key-password: keyPassword</code></pre>
+<h2 id="process-groups">Process Groups</h2>
+<p>Process groups can be nested from the top level. They can contain other
process groups as well and can be used to logically group related
operations.</p>
+<table>
+<thead>
+<tr>
+<th><em>Property</em></th>
+<th><em>Description</em></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>name</td>
+<td>The name of what this process group will do.</td>
+</tr>
+<tr>
+<td>id</td>
+<td>The id of this process group. This needs to be set to a unique
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+<tr>
+<td>Processors</td>
+<td>The processors contained in this Process Group. (Defined above)</td>
+</tr>
+<tr>
+<td>Remote Process Groups</td>
+<td>The remote process groups contained in this Process Group. (Defined
below)</td>
+</tr>
+<tr>
+<td>Connections</td>
+<td>The connections contained in this Process Group. (Defined below)</td>
+</tr>
+<tr>
+<td>Input Ports</td>
+<td>The input ports contained in this Process Group. (Defined below)</td>
+</tr>
+<tr>
+<td>Output Ports</td>
+<td>The output ports contained in this Process Group. (Defined below)</td>
+</tr>
+<tr>
+<td>Funnels</td>
+<td>The funnels contained in this Process Group. (Defined below)</td>
+</tr>
+<tr>
+<td>Process Groups</td>
+<td>The child Process Groups contained in this Process Group.</td>
+</tr>
+</tbody>
+</table>
+<h2 id="input-ports">Input Ports</h2>
+<p>These ports provide input to the Process Group they reside on. (Currently
only for internal Input ports.)</p>
+<table>
+<thead>
+<tr>
+<th><em>Property</em></th>
+<th><em>Description</em></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>name</td>
+<td>The name of what this input port will do.</td>
+</tr>
+<tr>
+<td>id</td>
+<td>The id of this input port. This needs to be set to a unique
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+</tbody>
+</table>
+<h2 id="output-ports">Output Ports</h2>
+<p>These ports provide output from the Process Group they reside on.
(Currently only for internal Output ports.)</p>
+<table>
+<thead>
+<tr>
+<th><em>Property</em></th>
+<th><em>Description</em></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>name</td>
+<td>The name of what this output port will do.</td>
+</tr>
+<tr>
+<td>id</td>
+<td>The id of this output port. This needs to be set to a unique
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+</tbody>
+</table>
+<h2 id="funnels">Funnels</h2>
+<p>Funnels can be used to combine outputs from multiple processors into a
single connection for ease of design.</p>
+<table>
+<thead>
+<tr>
+<th><em>Property</em></th>
+<th><em>Description</em></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>id</td>
+<td>The id of this funnel. This needs to be set to a unique
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+</tbody>
+</table>
<h2 id="connections">Connections</h2>
<p>There can be multiple connections in this version of MiNiFi. The
"Connections" subsection is a list of connections. Each connection
must specify these properties.</p>
<table>
@@ -635,6 +1033,10 @@ Security Configuration section of this A
<td>The name of what this connection will do. This is used for the id of the
connection so it must be unique.</td>
</tr>
<tr>
+<td>id</td>
+<td>The id of this connection. This needs to be left empty or set to a
filesystem-friendly value (regex: [A-Za-z0-9_-]+)</td>
+</tr>
+<tr>
<td>source name</td>
<td>The name of what of the processor that is the source for this
connection.</td>
</tr>
@@ -664,8 +1066,8 @@ Security Configuration section of this A
</tr>
</tbody>
</table>
-<h2 id="remote-processing-groups">Remote Processing Groups</h2>
-<p>MiNiFi can be used to send data using the Site to Site protocol (via a
Remote Processing Group) or a Processor. These properties configure the Remote
Processing Groups that use Site-To-Site to send data to a core instance.</p>
+<h2 id="remote-process-groups">Remote Process Groups</h2>
+<p>MiNiFi can be used to send data using the Site to Site protocol (via a
Remote Process Group) or a Processor. These properties configure the Remote
Process Groups that use Site-To-Site to send data to a core instance. The
proxy settings are for HTTP Site-To-Site.</p>
<table>
<thead>
<tr>
@@ -676,11 +1078,11 @@ Security Configuration section of this A
<tbody>
<tr>
<td>name</td>
-<td>The name of what this Remote Processing Group points to. This is not used
for any underlying implementation but solely for the users of this
configuration and MiNiFi agent.</td>
+<td>The name of what this Remote Process Group points to. This is not used for
any underlying implementation but solely for the users of this configuration
and MiNiFi agent.</td>
</tr>
<tr>
<td>comment</td>
-<td>A comment about the Remote Processing Group. This is not used for any
underlying implementation but solely for the users of this configuration and
MiNiFi agent.</td>
+<td>A comment about the Remote Process Group. This is not used for any
underlying implementation but solely for the users of this configuration and
MiNiFi agent.</td>
</tr>
<tr>
<td>url</td>
@@ -692,7 +1094,27 @@ Security Configuration section of this A
</tr>
<tr>
<td>yield period</td>
-<td>When communication with this Remote Processing Group fails, it will not be
scheduled again for this amount of time.</td>
+<td>When communication with this Remote Process Group fails, it will not be
scheduled again for this amount of time.</td>
+</tr>
+<tr>
+<td>transport protocol</td>
+<td>The transport protocol to use for this Remote Process Group. Can be either
"RAW" or "HTTP"</td>
+</tr>
+<tr>
+<td>proxy host</td>
+<td>The hostname of the proxy server</td>
+</tr>
+<tr>
+<td>proxy port</td>
+<td>The port to connect to on the proxy server</td>
+</tr>
+<tr>
+<td>proxy user</td>
+<td>The user name on the proxy server</td>
+</tr>
+<tr>
+<td>proxy password</td>
+<td>The password for the proxy server</td>
</tr>
</tbody>
</table>
@@ -776,9 +1198,11 @@ Security Configuration section of this A
</tr>
</tbody>
</table>
+<p><strong>Note:</strong> In order to send via HTTPS, the "Security
Properties" must be fully configured. A StandardSSLContextService will be
made automatically with the ID "SSL-Context-Service" and used by the
Provenance Reporting.</p>
<h1 id="example-config-file">Example Config File</h1>
<p>Below are two example config YAML files. The first tails the
minifi-app.log, send the tailed log and provenance data back to a secure
instance of NiFi. The second uses a series of processors to tail the app log,
routes off only lines that contain "WriteAheadFlowFileRepository" and
puts it as a file in the "./" directory.</p>
-<pre><code class="language-yaml">Flow Controller:
+<pre><code class="language-yaml">MiNiFi Config Version: 1
+Flow Controller:
name: MiNiFi Flow
comment:
