NIFI-286 Fixes typos in Overview; adds Clustering section to Administration Guide; adds to Installation instructions in Administration Guide
Signed-off-by: Mark Payne <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-nifi/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-nifi/commit/ff6e5e68 Tree: http://git-wip-us.apache.org/repos/asf/incubator-nifi/tree/ff6e5e68 Diff: http://git-wip-us.apache.org/repos/asf/incubator-nifi/diff/ff6e5e68 Branch: refs/heads/NIFI-250 Commit: ff6e5e681c51f51509ba4bbddff4c54d40a8a0e3 Parents: eabf2d5 Author: Jenn Barnabee <[email protected]> Authored: Mon Feb 2 15:18:25 2015 -0500 Committer: Mark Payne <[email protected]> Committed: Fri Feb 13 07:17:20 2015 -0500 ---------------------------------------------------------------------- .../src/main/asciidoc/administration-guide.adoc | 139 ++++++++++++++++--- nifi/nifi-docs/src/main/asciidoc/images/ncm.png | Bin 0 -> 339522 bytes nifi/nifi-docs/src/main/asciidoc/overview.adoc | 20 +-- 3 files changed, 132 insertions(+), 27 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-nifi/blob/ff6e5e68/nifi/nifi-docs/src/main/asciidoc/administration-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi/nifi-docs/src/main/asciidoc/administration-guide.adoc index 30cca83..b2fdc5a 100644 --- a/nifi/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -21,21 +21,45 @@ Apache NiFi Team <[email protected]> How to install and start NiFi ----------------------------- -NOTE: This section is incomplete. -* Linux -** Decompress and untar into desired installation directory. +* Linux/Unix/OSX +** Decompress and untar into desired installation directory ** Make any desired edits in files found under <installdir>/conf -** Execute the following commands +*** At a minimum, we recommend editing the _nifi.properties_ file and entering a password for the nifi.sensitive.props.key (see <<system_properties>> below) +** From the <installdir>/bin directory, execute the following commands by typing ./nifi.sh <command>: *** start: starts NiFi in the background *** stop: stops NiFi that is running in the background *** status: provides the current status of NiFi -*** run: runs NiFi in the foreground and waits for a Ctl-C to initiate shutdown of NiFi -*** intall: installs NiFi as a service that can then be controlled via +*** run: runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi +*** install: installs NiFi as a service that can then be controlled via **** service nifi start **** service nifi stop **** service nifi status +* Windows +** Decompress into the desired installation directory +** Make any desired edits in the files found under <installdir>/conf +*** At a minimum, we recommend editing the _nifi.properties_ file and entering a password for the nifi.sensitive.props.key (see <<system_properties>> below) +** Navigate to the <installdir>/bin directory +** Double-click run-nifi.bat. This runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi +** Alternatively, to start NiFi in the background, double-click start-nifi.bat +** To stop NiFi running in the background, double-click stop-nifi.bat +** To see the current status of NiFi, double-click status-nifi.bat + + +When NiFi first starts up, the following files and directories are created: + +* content_repository +* database_repository +* flowfile_repository +* provenance_repository +* work directory +* logs directory +* Within the conf directory, the _flow.xml.gz_ file and the templates directory are created + +See the <<system_properties>> section of this guide for more information about configuring NiFi repositories and configuration files. + + Best Practice Configuration --------------------------- NOTE: Typical Linux defaults are not necessarily well tuned for the needs of an IO intensive application like @@ -76,14 +100,14 @@ it but to adjust do something like sudo sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait="1" ---- -Tell linux you never want NiFi to swap:: +Tell Linux you never want NiFi to swap:: Swapping is fantastic for some applications. It isn't good for something like -NiFi that always wants to be running. To tell linux you'd like swapping off you +NiFi that always wants to be running. To tell Linux you'd like swapping off you can edit '/etc/sysctl.conf' to add the following line ---- vm.swappiness = 0 ---- -For the partions handling the various NiFi repos turn off things like 'atime'. +For the partitions handling the various NiFi repos turn off things like 'atime'. Doing so can cause a surprising bump in throughput. Edit the '/etc/fstab' file and for the partition(s) of interest add the 'noatime' option. @@ -94,15 +118,14 @@ NOTE: This section is incomplete. Controlling Levels of Access ---------------------------- -NOTE: This section is incomplete. Once NiFi is configured to run securely as discussed in the previous section, it is necessary -to manually designate an ADMIN user in the authorized-users.xml file, which is located in the +to manually designate an ADMIN user in the _authorized-users.xml_ file, which is located in the root installation's conf directory. After this ADMIN user has been added, s/he may grant access -to other users, systems, and other instances of NiFi, through the User Interface (UI) without having to manually edit the authorized-users.xml +to other users, systems, and other instances of NiFi, through the User Interface (UI) without having to manually edit the _authorized-users.xml_ file. If you are the administrator, you would add yourself as the ADMIN user in this file. -Open the authorized-users.xml file in a text editor. You will notice that it includes a template +Open the _authorized-users.xml_ file in a text editor. You will notice that it includes a template to guide you, with example entries that are commented out. It is only necessary to manually add one user, the ADMIN user, @@ -128,9 +151,9 @@ Here is an example entry using the name John Smith: </users> ---- -After the authorized-users.xml file has been edited and saved, restart NiFi. +After the _authorized-users.xml_ file has been edited and saved, restart NiFi. Once the application starts, the ADMIN user is -able to access the UI at the https URL that is configured in the nifi.properties file. +able to access the UI at the https URL that is configured in the _nifi.properties_ file. From the UI, click on the Users icon ( image:iconUsers.png["Users", width=32] ) in the Management Toolbar (upper-right corner of the UI), and the User Management Page opens. @@ -154,6 +177,88 @@ in the remote cluster can be included in the same group. When the ADMIN wants to cluster, s/he can grant it to the group and avoid having to grant it individually to each node in the cluster. +[[clustering]] +Clustering Configuration +------------------------ + +This section provides a quick overview of NiFi Clustering and instructions on how to set up a basic cluster. In the future, we hope to provide supplemental documentation that covers the NiFi Cluster Architecture in depth. + +The design of NiFi clustering is a simple master/slave model where there is a master and one or more slaves. While the model is that of master and slave, if the master dies, the slaves are all instructed to continue operating as they were to ensure the dataflow remains live. The absence of the master simply means new slaves cannot come on-line and flow changes cannot occur until the master is restored. In NiFi clustering, we call the master the NiFi Cluster Manager (NCM), and the slaves are called Nodes. See a full description of each in the Terminology section below. + +*Why Cluster?* + + +NiFi Administrators or Dataflow Managers (DFMs) may find that using one instance of NiFi on a single server is not enough to process the amount of data they have. So, one solution is to run the same dataflow on multiple NiFi servers. However, this creates a management problem, because each time DFMs want to change or update the dataflow, they must make those changes on each server and then monitor each server individually. By clustering the NiFi servers, it's possible to have that increased processing capability along with a single interface through which to make dataflow changes and monitor the dataflow. Clustering allows the DFM to make each change only once, and that change is then replicated to all the nodes of the cluster. Through the single interface, the DFM may also monitor the health and status of all the nodes. + +NiFi Clustering is unique and has its own terminology. It's important to understand the following terms before setting up a cluster. + +[template="glossary", id="terminology"] +*Terminology* + + +*NiFi Cluster Manager*: A NiFi Cluster Manager (NCM) is an instance of NiFi that provides the sole management point for the cluster. It communicates dataflow changes to the nodes and receives health and status information from the nodes. It also ensures that a uniform dataflow is maintained across the cluster. When DFMs manage a dataflow in a cluster, they do so through the User Interface of the NCM (i.e., via the URL of the NCM's User Interface). Fundamentally, the NCM keeps the state of the cluster consistent. + +*Nodes*: Each cluster is made up of the NCM and one or more nodes. The nodes do the actual data processing. (The NCM does not process any data; all data runs through the nodes.) While nodes are connected to a cluster, the DFM may not access the User Interface for any of the individual nodes. The User Interface of a node may only be accessed if the node is manually removed from the cluster. + +*Primary Node*: Every cluster has one Primary Node. On this node, it is possible to run "Isolated Processors" (see below). By default, the NCM will elect the first node that connects to the cluster as the Primary Node; however, the DFM may select a new node as the Primary Node in the Cluster Management page of the User Interface if desired. If the cluster restarts, the NCM will "remember" which node was he Primary Node and wait for that node to re-connect before allowing the DFM to make any changes to the dataflow. The ADMIN may adjust how long the NCM waits for the Primary Node to reconnect by adjusting the property _nifi.cluster.manager.safemode.duration_ in the _nifi.properties_ file, which is discussed in the <<system_properties>> section of this document. + +*Isolated Processors*: In a NiFi cluster, the same dataflow runs on all the nodes. As a result, every component in the flow runs on every node. However, there may be cases when the DFM would not want every processor to run on every node. The most common case is when using a processor like the GetSFTP processor, which is pulling from a remote directory. If the GetSFTP on every node tries simultaneously to pull from the same remote directory, there could be race conditions. Therefore, the DFM could configure the GetSFTP on the Primary Node to run in isolation, meaning that it only runs on that node. It could pull in data and -with the proper dataflow configuration- load-balance it across the rest of the nodes in the cluster. Note that while this feature exists, it is also very common to simply use a standalone NiFi instance to pull data and feed it to the cluster. It just depends on the resources available and how the Administrator decides to configure the cluster. + +*Heartbeats*: The nodes communicate their health and status to the NCM via "heartbeats", which let the NCM know they are still connected to the cluster and working properly. By default, the nodes emit heartbeats to the NCM every 5 seconds, and if the NCM does not receive a heartbeat from a node within 45 seconds, it disconnects the node due to "lack of heartbeat". (The 5-second and 45-second settings are configurable in the _nifi.properties_ file. See the <<system_properties>> section of this document for more information.) The reason that the NCM disconnects the node is because the NCM needs to ensure that every node in the cluster is in sync, and if a node is not heard from regularly, the NCM cannot be sure it is still in sync with the rest of the cluster. If, after 45 seconds, the node does send a new heartbeat, the NCM will automatically reconnect the node to the cluster. Both the disconnection due to lack of heartbeat and the reconnection once a heartbeat is received are report ed to the DFM in the NCM's User Interface. + +*Communication within the Cluster* + + +As noted, the nodes communicate with the NCM via heartbeats. The NCM-to-node communication may be set up as multicast or unicast, depending on the properties that are configured in the _nifi.properties_ file (See <<system_properties>> ). By default, unicast is used. It is important to note that the nodes in a NiFi cluster are not aware of each other. They only communicate with the NCM. Therefore, if one of the nodes goes down, the other nodes in the cluster will not automatically pick up the load of the missing node. It is possible for the DFM to configure the dataflow for failover contingencies; however, this is dependent on the dataflow design and does not happen automatically. + +When the DFM makes changes to the dataflow, the NCM communicates those changes to the nodes and waits for each node to respond, indicating that it has made the change on its local flow. If the DFM wants to make another change, the NCM will only allow this to happen once all the nodes have acknowledged that they've implemented the last change. As such, the speed with which dataflow changes may be made is as fast as the slowest node. When all nodes are located in close proximity and the network is stable, this response time is not an issue. However, if your cluster is comprised of nodes that are geographically dispersed and/or operating over a latent network, there may be times when DFMs cannot make changes as quickly as they would like. Keep this in mind when setting up a cluster. + +*Dealing with Disconnected Nodes* + + +A DFM may manually disconnect a node from the cluster. But if a node becomes disconnected for any other reason (such as due to lack of heartbeat), the NCM will show a bulletin on the User Interface, and the DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes may be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is definitely down; it just means that the NCM cannot communicate with the node. + + +*Basic Cluster Setup* + + +This section describes the setup for a simple two-node, non-secure, unicast cluster comprised of three instances of NiFi: + +* The NCM +* Node 1 +* Node 2 + +Administrators may install each instance on a separate server; however, it is also perfectly fine to install the NCM and one of the nodes on the same server, as the NCM is very lightweight. Just keep in mind that the ports assigned to each instance must not collide if the NCM and one of the nodes share the same server. + +For each instance, the clustering properties in the _nifi.properties_ file will need to be updated. All the clustering properties are described in the <<system_properties>> section of this guide; however, in this section, we will focus on the minimum properties that must be set for simple cluster. + +For all three instances, the Cluster Common Properties can be left with the default settings. Note, however, that if you change these settings, they must be set the same on every instance in the cluster (NCM and nodes). + +For the NCM, the minimum properties to configure are as follows: + +* Under the Web Properties, set either the http or https port that you want the NCM to run on. If the NCM and one of the nodes are on the same server, make sure this port is different from the web port used by the node. +* Under the Cluster Manager Properties, set the following: +** nifi.cluster.is.manager - Set this to _true_. +** nifi.cluster.protocol.manager.port - Set this to an open port that is higher than 1024 (anything lower requires root). Take note of this setting, as you will need to reference it when you set up the nodes. + +For Node 1, the minimum properties to configure are as follows: + +* Under the Web Properties, set either the http or https port that you want Node 1 to run on. If the NCM is running on the same server, choose a different web port for Node 1. +* Under Cluster Node Properties, set the following: +** nifi.cluster.is.node - Set this to _true_. +** nifi.cluster.node.protocol.port - Set this to an open port that is higher than 1024 (anything lower requires root). If Node 1 and the NCM are on the same server, make sure this port is different from the nifi.cluster.protocol.manager.port. +** nifi.cluster.node.unicast.manager.protocol.port - Set this to exactly the same port that was set on the NCM for the property nifi.cluster.manager.protocol.port. + +For Node 2, the minimum properties to configure are as follows: + +* Under the Web Properties, set either the http or https port that you want Node 2 to run on. +* Under the Cluster Node Properties, set the following: +** nifi.cluster.is.node - Set this to _true_. +** nifi.cluster.node.protocol.port - Set this to an open port that is higher than 1024 (anything lower requires root). +** nifi.cluster.node.unicast.manager.protocol.port - Set this to exactly the same port that was set on the NCM for the property nifi.cluster.manager.protocol.port. + +Now, it is possible to start up the cluster. Technically, it does not matter which instance starts up first. However, you could start the NCM first, then Node 1 and then Node 2. Since the first node that connects is automatically elected as the Primary Node, this sequence should create a cluster where Node 1 is the Primary Node. Navigate to the URL for the NCM in your web browser, and the User Interface should look similar to the following: + +image:ncm.png["NCM User Interface", width=940] + + + +[[system_properties]] System Properties ----------------- The _nifi.properties_ file in the conf directory is the main configuration file for controlling how NiFi runs. This section provides an overview of the properties in this file and includes some notes on how to configure it in a way that will make upgrading easier. *After making changes to this file, restart NiFi in order @@ -248,7 +353,7 @@ FlowFile Repository, if also on that disk, could become corrupt. To avoid this s |nifi.content.repository.directory.default*|The location of the Content Repository. The default value is ./content_repository. |nifi.content.repository.archive.max.retention.period|If archiving is enabled (see nifi.content.repository.archive.enabled below), then this property specifies the maximum amount of time to keep the archived data. It is blank by default. -|nifi.content.repository.archive.max.usage.percentage|If archiving is enabled (see nifi.content.repository.archive.enabled below), then this property also must have a value to indiciate the maximum percentage of disk space that may be used for archiving. It is blank by default. +|nifi.content.repository.archive.max.usage.percentage|If archiving is enabled (see nifi.content.repository.archive.enabled below), then this property also must have a value to indicate the maximum percentage of disk space that may be used for archiving. It is blank by default. |nifi.content.repository.archive.enabled|To enable archiving, set this to _true_ and specify a value for the nifi.content.repository.archive.max.usage.percentage property above. By default, archiving is not enabled. |nifi.content.repository.always.sync|If set to _true_, any change to the repository will be synchronized to the disk, meaning that NiFi will ask the operating system not to cache the information. This is very expensive and can significantly reduce NiFi performance. However, if it is _false_, there could be the potential for data loss if either there is a sudden power loss or the operating system crashes. The default value is _false_. |nifi.content.viewer.url|The URL for a web-based content viewer if one is available. It is blank by default. @@ -332,7 +437,7 @@ Security Configuration section of this Administrator's Guide. |==== |*Property*|*Description* -|nifi.sensitive.props.key|This is the password used to encrypt any sensitive property values that are configured in processors. By default, it is blank, but the system administrator should provide a value for it. It can be a string of any length. Be aware that once this password is set and one or more sensitive processor properties has been configured, this password should not be changed. +|nifi.sensitive.props.key|This is the password used to encrypt any sensitive property values that are configured in processors. By default, it is blank, but the system administrator should provide a value for it. It can be a string of any length. Be aware that once this password is set and one or more sensitive processor properties have been configured, this password should not be changed. |nifi.sensitive.props.algorithm|The algorithm used to encrypt sensitive properties. The default value is PBEWITHMD5AND256BITAES-CBC-OPENSSL. |nifi.sensitive.props.provider|The sensitive property provider. The default value is BC. |nifi.security.keystore*|The full path and name of the keystore. It is blank by default. http://git-wip-us.apache.org/repos/asf/incubator-nifi/blob/ff6e5e68/nifi/nifi-docs/src/main/asciidoc/images/ncm.png ---------------------------------------------------------------------- diff --git a/nifi/nifi-docs/src/main/asciidoc/images/ncm.png b/nifi/nifi-docs/src/main/asciidoc/images/ncm.png new file mode 100644 index 0000000..a63e313 Binary files /dev/null and b/nifi/nifi-docs/src/main/asciidoc/images/ncm.png differ http://git-wip-us.apache.org/repos/asf/incubator-nifi/blob/ff6e5e68/nifi/nifi-docs/src/main/asciidoc/overview.adoc ---------------------------------------------------------------------- diff --git a/nifi/nifi-docs/src/main/asciidoc/overview.adoc b/nifi/nifi-docs/src/main/asciidoc/overview.adoc index fa1a5f1..a5183d7 100644 --- a/nifi/nifi-docs/src/main/asciidoc/overview.adoc +++ b/nifi/nifi-docs/src/main/asciidoc/overview.adoc @@ -39,7 +39,7 @@ Data access exceeds capacity to consume:: Sometimes a given data source can outpace some part of the processing or delivery chain - it only takes one weak-link to have an issue. Boundary conditions are mere suggestions:: -You will get data that is too big, too small, too fast, too slow, corrupt, wrong, wrong format +You will invariably get data that is too big, too small, too fast, too slow, corrupt, wrong, or in the wrong format. What is noise one day becomes signal the next:: Priorities of an organization change - rapidly. Enabling new flows and changing existing ones must be fast. @@ -92,7 +92,7 @@ and either commit that work or rollback. | Connection | Bounded Buffer | Connections provide the actual linkage between processors. These act as queues and allow various processes to interact at differing rates. These queues then -can be prioritized dynamically and can have upper bounds on load which enable +can be prioritized dynamically and can have upper bounds on load, which enable back pressure. | Flow Controller | Scheduler | @@ -102,14 +102,14 @@ Flow Controller acts as the broker facilitating the exchange of FlowFiles between processors. | Process Group | subnet | -A Process Group is a specific set of processes and their connections which can +A Process Group is a specific set of processes and their connections, which can receive data via input ports and send data out via output ports. In this manner process groups allow creation of entirely new components simply by composition of other components. |=========================== -This design model, also similar to <<seda>>, provides many beneficial consequences which help NiFi +This design model, also similar to <<seda>>, provides many beneficial consequences that help NiFi to be a very effective platform for building powerful and scalable dataflows. A few of these benefits include: @@ -141,7 +141,7 @@ FlowFile Repository:: The FlowFile Repository is where NiFi keeps track of the state of what it knows about a given FlowFile that is presently active in the flow. The implementation of the repository is pluggable. The default approach is a persistent Write-Ahead Log that lives on a specified disk partition. Content Repository:: -The Content Repository is where the actual content bytes of a given FlowFile live. The implementation of the repository is pluggable. The default approach is a fairly simple mechanism which stores blocks of data in the file system. More than one file system storage location can be specified so as to get different physical partitions engaged to reduce contention on any single volume. +The Content Repository is where the actual content bytes of a given FlowFile live. The implementation of the repository is pluggable. The default approach is a fairly simple mechanism, which stores blocks of data in the file system. More than one file system storage location can be specified so as to get different physical partitions engaged to reduce contention on any single volume. Provenance Repository:: The Provenance Repository is where all provenance event data is stored. The repository construct is pluggable with the default implementation being to use one or more physical disk volumes. Within each location event data is indexed and searchable. @@ -161,8 +161,8 @@ instructed to continue operating as they were to ensure the data flow remains li The absence of the NCM simply means new nodes cannot come on-line and flow changes cannot occur until the NCM is restored. -Performance Expections and Characteristics of NiFi --------------------------------------------------- +Performance Expectations and Characteristics of NiFi +---------------------------------------------------- NiFi is designed to fully leverage the capabilities of the underlying host system it is operating on. This maximization of resources is particularly strong with regard to CPU and disk. Many more details will @@ -181,7 +181,7 @@ efficiently reach 100 or more MB/s of throughput. That is because linear growth is expected for each physical partition and content repository added to NiFi. This will bottleneck at some point on the FlowFile repository and provenance repository. We plan to provide a benchmarking/performance test template to -include in the build which will allow users to easily test their system and +include in the build, which will allow users to easily test their system and to identify where bottlenecks are and at which point they might become a factor. It should also make it easy for system administrators to make changes and to verify the impact. @@ -272,8 +272,8 @@ Designed for Extension:: Points of extension;; Processors, Controller Services, Reporting Tasks, Prioritizers, Customer User Interfaces Classloader Isolation;; - For any component based system one problem that can quickly occur is dependency nightmares. NiFi addresses this by providing a custom class loader model - ensuring that each extension bundle is exposed to a very limited set of dependencies. As a result extensions can be built with little concern for whether + For any component-based system, dependency nightmares can quickly occur. NiFi addresses this by providing a custom class loader model, + ensuring that each extension bundle is exposed to a very limited set of dependencies. As a result, extensions can be built with little concern for whether they might conflict with another extension. The concept of these extension bundles is called 'NiFi Archives' and will be discussed in greater detail in the developer's guide. Clustering (scale-out)::
