Author: aldrin
Date: Mon Jun 4 18:56:10 2018
New Revision: 1832881
URL: http://svn.apache.org/viewvc?rev=1832881&view=rev
Log:
MINIFI-440 Updating MiNiFi Sys Admin guide
Modified:
nifi/site/trunk/minifi/system-admin-guide.html
Modified: nifi/site/trunk/minifi/system-admin-guide.html
URL:
http://svn.apache.org/viewvc/nifi/site/trunk/minifi/system-admin-guide.html?rev=1832881&r1=1832880&r2=1832881&view=diff
==============================================================================
--- nifi/site/trunk/minifi/system-admin-guide.html (original)
+++ nifi/site/trunk/minifi/system-admin-guide.html Mon Jun 4 18:56:10 2018
@@ -106,6 +106,7 @@
<h1>MiNiFi System Administrator's Guide</h1>
<p><a href="mailto:[email protected]";>Apache NiFi Team</a></p>
<p><a href="https://nifi.apache.org/minifi/index.html";>MiNiFi Homepage</a></p>
+<p>This documentation is for MiNiFi 0.4.0.</p>
<h1>Automatic Warm-Redeploy</h1>
<p>When many MiNiFi agents running on the edge, it may not be possible to
manually stop, edit the config.yml and then restart every one every time their
configuration needs to change. The Config Change Coordinator and its Ingestors
were designed to automatically redeploy in response to a configuration
update.</p>
<p>The Config Change Ingestors are the means by which the agent is notified of
a potential new configuration. Currently there are three:</p>
@@ -175,8 +176,13 @@
<tbody>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.hostname</td><td>Hostname on
which to pull configurations from</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.port</td><td>Port on which to
pull configurations from</td></tr>
+<tr><td>nifi.minifi.notifier.ingestors.pull.http.proxy.hostname</td><td>Proxy
server hostname</td></tr>
+<tr><td>nifi.minifi.notifier.ingestors.pull.http.proxy.port</td><td>Proxy
server port</td></tr>
+<tr><td>nifi.minifi.notifier.ingestors.pull.http.proxy.username</td><td>Proxy
username</td></tr>
+<tr><td>nifi.minifi.notifier.ingestors.pull.http.proxy.password</td><td>Proxy
password</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.path</td><td>Path on which to
pull configurations from</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.period.ms</td><td>Period on
which to pull configurations from, defaults to 5 minutes if not set.</td></tr>
+<tr><td>nifi.minifi.notifier.ingestors.pull.http.query</td><td>Querystring
value for the URL</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.use.etag</td><td>If the
destination server is set up with cache control ability and utilizes an
"ETag" header, then this should be set to true to utilize it. Very
simply, the Ingestor remembers the "ETag" of the last successful pull
(returned 200) then uses that "ETag" in a "If-None-Match"
header on the next request.</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.connect.timeout.ms</td><td>Sets
the connect timeout for new connections. A value of 0 means no timeout,
otherwise values must be a positive whole number in milliseconds.</td></tr>
<tr><td>nifi.minifi.notifier.ingestors.pull.http.read.timeout.ms</td><td>Sets
the read timeout for new connections. A value of 0 means no timeout, otherwise
values must be a positive whole number in milliseconds.</td></tr>
@@ -192,10 +198,16 @@
<h1>Status Reporting and Querying</h1>
<p>In NiFi there is a lot of information, such as stats and bulletins, that is
only available to view through the UI. MiNiFi provides access to this
information through a query mechanism. You can query FlowStatus either using
the MiNiFi.sh script or by configuring one of the Periodic Status Reporters.
The API for the query is the same for the reporters and the
"flowStatus" script option. The API is outlined in the
"FlowStatus Query Options" section below.</p>
<h2>FlowStatus Script Query</h2>
-<p>From the minifi.sh script, you can manually query to get the current status
of your dataflow. The following is an example of a minifi.sh query you might
run to view health, stats, and bulletins for the TailFile processor. This query
returns information to your command-line.</p>
+<p>From the minifi.sh script, you can manually query to get the current status
of your dataflow. The following is an example of a minifi.sh query you might
run to view health, stats, and bulletins for the TailFile processor. This query
returns information to your command-line.</p>
<pre><code>minifi.sh flowStatus processor:TailFile:health,stats,bulletins
</code></pre>
<p>Currently the script only accepts one high level option at a time. Also any
names of connections, remote process groups, or processors that contain "
" (a space), ":", ";" or "," cause parsing
errors when querying.</p>
+<p><strong>Note:</strong> The examples in this documentation are provided for
*nix based environments. Windows support is also provided by by the
<code>flowstatus-minifi.bat</code> file. To perform one of the listed
interactions, <code>minifi.sh flowStatus</code> would simply be replaced by
<code>flowstatus-minifi.bat</code>. For example, the sample query above in
*nix environments,</p>
+<pre><code>minifi.sh flowStatus processor:TailFile:health,stats,bulletins`
+</code></pre>
+<p>would translate to Windows environments as,</p>
+<pre><code>flowstatus-minifi.bat processor:TailFile:health,stats,bulletins
+</code></pre>
<h2>Periodic Status Reporters</h2>
<p>You can set up Periodic Status Reporters to periodically report the status
of your dataflow. The query executes at configurable intervals and the results
are reported using the configured implementation. Configure the Reporters in
the bootstrap.conf file, using the
"nifi.minifi.status.reporter.components" key followed by the full
path name of the desired Reporter implementation to run. Use a comma separated
list to define more than one Reporter implementation. For example:</p>
<pre><code>nifi.minifi.status.reporter.components=org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger
@@ -247,7 +259,7 @@ nifi.minifi.status.reporter.log.period=6
<h2>FlowStatus Query Options</h2>
<p>This section outlines each option to query the MiNiFi instance for the
FlowStatus.</p>
<h3>Processors</h3>
-<p>To query the processors use the "processor" flag followed by the
processor ID to get (or "all") followed by one of the processor
options. The processor options are below.</p>
+<p>To query the processors use the "processor" flag followed by the
processor ID or name, to get (or "all") followed by one of the
processor options. The processor options are below. <strong>Note:</strong> In
Windows environments, all <code>minifi.sh flowStatus</code> invocations should
be replaced with <code>flowstatus-minifi.bat</code>. See <a
href="#flowstatus-script-query">FlowStatus Script Query</a> for an
illustration.</p>
<table>
<thead>
<tr><th>Option</th><th>Description</th></tr>
@@ -262,7 +274,7 @@ nifi.minifi.status.reporter.log.period=6
<pre><code>minifi.sh flowStatus processor:TailFile:health,stats,bulletins
</code></pre>
<h3>Connections</h3>
-<p>To query the connections use the "connection" flag followed by
the connection ID to get (or "all") followed by one of the connection
options. The connection options are below.</p>
+<p>To query the connections use the "connection" flag followed by
the connection ID or name, to get (or "all") followed by one of the
connection options. The connection options are below.</p>
<table>
<thead>
<tr><th>Option</th><th>Description</th></tr>
@@ -276,7 +288,7 @@ nifi.minifi.status.reporter.log.period=6
<pre><code>minifi.sh flowStatus connection:TailToS2S:health,stats
</code></pre>
<h3>Remote Process Groups</h3>
-<p>To query the remote process groups (RPG) use the
"remoteProcessGroup" flag followed by the RPG ID to get (or
"all") followed by one of the remote process group options. The
remote process group options are below.</p>
+<p>To query the remote process groups (RPG) use the
"remoteProcessGroup" flag followed by the RPG ID or name to get (or
"all") followed by one of the remote process group options. The
remote process group options are below.</p>
<table>
<thead>
<tr><th>Option</th><th>Description</th></tr>
@@ -284,11 +296,12 @@ nifi.minifi.status.reporter.log.period=6
<tbody>
<tr><td>health</td><td>The connections's queued bytes and queued FlowFile
count.</td></tr>
<tr><td>bulletins</td><td>A list of all the current bulletins (if there are
any).</td></tr>
-<tr><td>inputPorts</td><td>A list of every input port for this RPG and their
status. Their status includes it's name, whether the target exit and whether
it's currently running.</td></tr>
+<tr><td>inputPorts</td><td>A list of every input port for this RPG and their
status. Their status includes its name, whether the target exists and whether
it's currently running.</td></tr>
+<tr><td>outputPorts</td><td>A list of every output port for this RPG and their
status. Their status includes its name, whether the target exists and whether
it's currently running.</td></tr>
<tr><td>stats</td><td>The current stats of the RPG. This includes the active
threads, sent content size and count.</td></tr>
</tbody>
</table>
-<p>An example query to get the health, bulletins, input ports and stats of all
the RPGs is below.</p>
+<p>An example query to get the health, bulletins, input ports and stats of all
the RPGS is below.</p>
<pre><code>minifi.sh flowStatus
remoteprocessgroup:all:health,bulletins,inputports,stats
</code></pre>
<h3>Controller Services</h3>
@@ -326,7 +339,7 @@ nifi.minifi.status.reporter.log.period=6
<tr><th>Option</th><th>Description</th></tr>
</thead>
<tbody>
-<tr><td>health</td><td>The provenance reporting state, active threads, whether
or not it has bulletins and any validation errors.</td></tr>
+<tr><td>health</td><td>The instance reporting state, active threads, whether
or not it has bulletins and any validation errors.</td></tr>
<tr><td>bulletins</td><td>A list of all the current bulletins (if there are
any).</td></tr>
<tr><td>stats</td><td>The current stats of the instance. This including but
not limited to bytes read/written and FlowFiles sent/transferred.</td></tr>
</tbody>
@@ -367,7 +380,7 @@ Bootstrap Config File: /Users/user/proje
<p>The config.yml in the <em>conf</em> directory is the main configuration
file for controlling how MiNiFi runs. This section provides an overview of the
properties in this file. The file is a YAML<br>
and follows the YAML format laid out <a
href="http://www.yaml.org/";>here</a>.</p>
<p>Alternatively, the MiNiFi Toolkit Converter can aid in creating a
config.yml from a generated template exported from a NiFi instance. This<br>
-tool can be downloaded from http://nifi.apache.org/minifi/download.html under
the <code>MiNiFi Toolkit Binaries</code> section. Information on the toolkit's
usage is<br>
+tool can be downloaded from https://nifi.apache.org/minifi/download.html under
the <code>MiNiFi Toolkit Binaries</code> section. Information on the toolkit's
usage is<br>
available at https://nifi.apache.org/minifi/minifi-toolkit.html.</p>
<p><strong>Note:</strong> Values for periods of time and data sizes must
include the unit of measure,<br>
for example "10 sec" or "10 MB", not simply
"10".</p>
@@ -386,6 +399,8 @@ parses and upconverts to the current ver
<ol>
<li>Added support for Controller Services.</li>
<li>Added support for Site-To-Site over proxy.</li>
+<li>Added support for overriding nifi.properties values</li>
+<li>Added support for Output Ports to Remote Process Groups</li>
</ol>
<h2>Flow Controller</h2>
<p>The first section of config.yml is for naming and commenting on the
file.</p>
@@ -468,6 +483,7 @@ FlowFile Repository, if also on that dis
</thead>
<tbody>
<tr><td>provenance rollover time</td><td>The amount of time to wait before
rolling over the latest data provenance information so that it is available to
be accessed by components. The default value is 1 min.</td></tr>
+<tr><td>implementation</td><td>The implementation of
<code>ProvenanceRepository</code> to use. The default value is
<code>org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository</code>.</td></tr>
</tbody>
</table>
<h2>Component Status Repository</h2>
@@ -656,18 +672,20 @@ Security Configuration section of this A
<tr><td>proxy port</td><td>The port to connect to on the proxy server</td></tr>
<tr><td>proxy user</td><td>The user name on the proxy server</td></tr>
<tr><td>proxy password</td><td>The password for the proxy server</td></tr>
+<tr><td>Input Ports</td><td>The Input Ports for this Remote Process Group (See
below)</td></tr>
+<tr><td>Output Ports</td><td>The Output Ports for this Remote Process Group
(See below)</td></tr>
</tbody>
</table>
-<h4>Input Ports Subsection</h4>
-<p>When connecting via Site to Site, MiNiFi needs to know which input port to
communicate to of the core NiFi instance. These properties designate and
configure communication with that port.</p>
+<h4>Input/Output Ports Subsection</h4>
+<p>When connecting via Site to Site, MiNiFi needs to know which input or
output port to communicate to of the core NiFi instance. These properties
designate and configure communication with that port.</p>
<table>
<thead>
<tr><th><em>Property</em></th><th><em>Description</em></th></tr>
</thead>
<tbody>
-<tr><td>id</td><td>The id of the input port as it exists on the core NiFi
instance. To get this information access the UI of the core instance, right the
input port that is desired to be connect to and select "configure".
The id of the port should under the "Id" section.</td></tr>
-<tr><td>name</td><td>The name of the input port as it exists on the core NiFi
instance. To get this information access the UI of the core instance, right the
input port that is desired to be connect to and select "configure".
The id of the port should under the "Port name" section.</td></tr>
-<tr><td>comments:</td><td>A comment about the Input Port. This is not used for
any underlying implementation but solely for the users of this configuration
and MiNiFi agent.</td></tr>
+<tr><td>id</td><td>The id of the port as it exists on the core NiFi instance.
To get this information access the UI of the core instance, right click the
port that is desired to be connect to and select "configure". The id
of the port should under the "Id" section.</td></tr>
+<tr><td>name</td><td>The name of the port as it exists on the core NiFi
instance. To get this information access the UI of the core instance, right
click the port that is desired to be connect to and select
"configure". The name of the port should under the "Port
name" section.</td></tr>
+<tr><td>comments:</td><td>A comment about the Port. This is not used for any
underlying implementation but solely for the users of this configuration and
MiNiFi agent.</td></tr>
<tr><td>max concurrent tasks</td><td>The number of tasks that this port should
be scheduled for at maximum.</td></tr>
<tr><td>use compression</td><td>Whether or not compression should be used when
communicating with the port. This is a boolean value of either "true"
or "false"</td></tr>
</tbody>
@@ -691,6 +709,23 @@ Security Configuration section of this A
</tbody>
</table>
<p><strong>Note:</strong> In order to send via HTTPS, the "Security
Properties" must be fully configured. A StandardSSLContextService will be
made automatically with the ID "SSL-Context-Service" and used by the
Provenance Reporting.</p>
+<h2>NiFi Properties Overrides</h2>
+<p>This is a yaml map that contains values to be put into nifi.properties.
This will supercede any hardcoded or other schema values that are substituted
into nifi.properties file.</p>
+<h3>Example NiFi Properties Overrides</h3>
+<pre><code class="language-yaml">NiFi Properties Overrides:
+ nifi.flowfile.repository.directory: ./flowfile_repository_override
+ nifi.content.repository.directory.default: ./content_repository_override
+ nifi.database.directory: ./database_repository_override
+</code></pre>
+<h1>Running as a Windows Service</h1>
+<p>MiNiFi can run as a Windows service. To do so, you must modify the
<code>conf/bootstrap.conf</code> to set absolute paths for some properties. The
properties are:</p>
+<ul>
+<li><code>lib.dir</code> (e.g. <code>C:\\minifi-x.x.x\\lib</code>)</li>
+<li><code>conf.dir</code> (e.g. <code>C:\\minifi-x.x.x\\conf</code>)</li>
+<li><code>nifi.minifi.config</code> (e.g.
<code>C:\\minifi-x.x.x\\conf\\config.yml</code>)</li>
+</ul>
+<p>You can now install the MiNiFi service by running the
<code>install-service.bat</code> script. To remove the service run the
<code>delete-service.bat</code> file.</p>
+<p>The <code>minifi.exe</code> in MiNiFi <code>bin</code> directory is used to
run MiNiFi Windows service. The bundled one is for 64 bit architecture and
requires 64 bit JRE. If you have to use 32 bit JRE for some reason, you need to
replace the <code>minifi.exe</code> file with the one for 32 bit to make MiNiFi
service runs successfully. To do so, go to <a
href="https://commons.apache.org/proper/commons-daemon/download_daemon.cgi";>Commons
Daemon project download page</a>, download the binary (e.g.
commons-daemon-1.1.0-bin.zip), extract it and replace
<code>bin/minifi.exe</code> by copying
<code>commons-daemon-x.x.x-bin/prunsrv.exe</code> into MiNiFi <code>bin</code>
directory as <code>minifi.exe</code> to overwrite the 64 bit exe with the 32
bit one.</p>
<h1>Example Config File</h1>
<p>Below are two example config YAML files. The first tails the
minifi-app.log, send the tailed log and provenance data back to a secure
instance of NiFi. The second uses a series of processors to tail the app log,
routes off only lines that contain "WriteAheadFlowFileRepository" and
puts it as a file in the "./" directory.</p>
<pre><code class="language-yaml">MiNiFi Config Version: 1
