http://git-wip-us.apache.org/repos/asf/nifi/blob/623d56c0/nifi-docs/src/main/asciidoc/user-guide.adoc ---------------------------------------------------------------------- diff --git a/nifi-docs/src/main/asciidoc/user-guide.adoc b/nifi-docs/src/main/asciidoc/user-guide.adoc index 8af0321..f84440e 100644 --- a/nifi-docs/src/main/asciidoc/user-guide.adoc +++ b/nifi-docs/src/main/asciidoc/user-guide.adoc @@ -29,8 +29,8 @@ along several dimensions of quality of service, such as loss-tolerant versus gua high throughput, and priority-based queuing. NiFi provides fine-grained data provenance for all data received, forked, joined cloned, modified, sent, and ultimately dropped upon reaching its configured end-state. -See the Admin Guide for information about system requirements, installation, and configuration. Once NiFi is installed, -use a supported web browser to view the User Interface (UI). +See the link:administration-guide.html[System Administratorâs Guide] for information about system requirements, installation, and configuration. Once NiFi is installed, +use a supported web browser to view the UI. Browser Support @@ -57,6 +57,12 @@ against the supported browsers. Any problem using a supported browser should be While the UI may run successfully in unsupported browsers, it is not actively tested against them. Additionally, the UI is designed as a desktop experience and is not currently supported in mobile browsers. +=== Viewing the UI in Variably Sized Browsers +In most environments, all of the UI is visible in your browser. However, the UI has a responsive design that allows you +to scroll through screens as needed, in smaller sized browsers or tablet environments. + +In environments where your browser width is less than 800 pixels and the height less than 600 pixels, portions of the +UI may become unavailable. [template="glossary", id="terminology"] Terminology @@ -110,7 +116,7 @@ Terminology Whenever a component reports a Bulletin, a bulletin icon is displayed on that component. System-level bulletins are displayed on the Status bar near the top of the page. Using the mouse to hover over that icon will provide a tool-tip that shows the time and severity (Debug, Info, Warning, Error) of the Bulletin, as well as the message of the Bulletin. - Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Management Toolbar. + Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Global Menu. *Template*: Often times, a dataflow is comprised of many sub-flows that could be reused. NiFi allows DFMs to select a part of the dataflow (or the entire dataflow) and create a Template. This Template is given a name and can then be dragged onto the canvas just like the other components. @@ -123,7 +129,7 @@ Terminology You can use these archived files to rollback flow configuration. To do so, stop NiFi, replace flow.xml.gz with a desired backup copy, then restart NiFi. In a clustered environment, stop the entire NiFi cluster, replace the flow.xml.gz of one of nodes, and restart the node. Remove flow.xml.gz from other nodes. Once you confirmed the node starts up as a one-node cluster, start the other nodes. The replaced flow configuration will be synchronized across the cluster. - The name and location of flow.xml.gz, and auto archive behavior are configurable. See the link:administration-guide.html#core-properties-br[Admin Guide] for further details. + The name and location of flow.xml.gz, and auto archive behavior are configurable. See the link:administration-guide.html#core-properties-br[System Administratorâs Guide] for further details. @@ -131,105 +137,129 @@ Terminology NiFi User Interface ------------------- -The NiFi User Interface (UI) provides mechanisms for creating automated dataflows, as well as visualizing, +The NiFi UI provides mechanisms for creating automated dataflows, as well as visualizing, editing, monitoring, and administering those dataflows. The UI can be broken down into several segments, each responsible for different functionality of the application. This section provides screenshots of the application and highlights the different segments of the UI. Each segment is discussed in further detail later in the document. -When the application is started, the user is able to navigate to the User Interface by going to the default address of +When the application is started, the user is able to navigate to the UI by going to the default address of `http://<hostname>:8080/nifi` in a web browser. There are no permissions configured by default, so anyone is -able to view and modify the dataflow. For information on securing the system, see the Systems Administrator guide. +able to view and modify the dataflow. For information on securing the system, see the link:administration-guide.html[System Administratorâs Guide]. When a DFM navigates to the UI for the first time, a blank canvas is provided on which a dataflow can be built: -image::new-flow.png["Empty Flow"] +image::images/nifi-toolbar-components.png["NiFi Components Toolbar"] -Along the top of the of the screen is a toolbar that contains several of these segments. -To the left is the Components Toolbar. This toolbar consists of the different components that can be dragged onto the canvas. +The Components Toolbar runs across the top left portion of your screen. It consists of the components you can drag onto the +canvas to build your dataflow. Each component is described in more detail in link:building-dataflow.html[Building a Dataflow]. -Next to the Components Toolbar is the Actions Toolbar. This toolbar consists of buttons to manipulate the existing -components on the canvas. To the right of the Actions Toolbar is the Search Toolbar. This toolbar consists of a single -Search field that allows users to easily find components on the canvas. Users are able to search by component name, -type, identifier, configuration properties, and their values. +The Status Bar is under the Components Toolbar. The Status bar provides information about how many Processors exist on the canvas in +each state (Stopped, Running, Invalid, Disabled), how many Remote Process Groups exist on the canvas in each state +(Transmitting, Not Transmitting), the number of threads that are currently active in the flow, the amount of data that currently +exists in the flow, and the timestamp at which all of this information was last refreshed. Additionally, if the instance of NiFi is clustered, the Status bar shows how many nodes +are in the cluster and how many are currently connected. -The Management Toolbar sits to the right-hand side of the screen. This toolbar consists of buttons that are -used by DFMs to manage the flow as well as by administrators who manage user access +The Operate Palette sits to the left-hand side of the screen. It consists of buttons that are +used by DFMs to manage the flow, as well as by administrators who manage user access and configure system properties, such as how many system resources should be provided to the application. -image::nifi-toolbar-components.png["NiFi Components Toolbar"] +On the right side of the canvas is Search, and the Global Menu. You can use Search to easily find components on the +canvas and can to search by component name, type, identifier, configuration properties, and their values. The Global Menu +contain options that allow you to manipulate existing components on the canvas: -Next, we have segments that provide capabilities to easily navigate around the canvas. On the left-hand side is a toolbar that -provides the ability to pan around the canvas and zoom in and out. On the right-hand side is a âBirds-Eye Viewâ of the dataflow. -This provides a high-level view of the dataflow and allows the user to quickly and easily pan across large portions of the dataflow. -Along the top of the screen is a trail of breadcrumbs. As users navigate into and out of Process Groups, the breadcrumbs show -the depth in the flow and each Process Group that was entered to reach this depth. Each of the Process Groups listed in the breadcrumbs -is a link that will take you back up to that level in the flow. +image::images/global-menu.png[NiFi Global Menu] -image::nifi-navigation.png["NiFi Navigation"] +Additionally, the UI has allows has some features that allow you to easily navigate around the canvas. You can use the +Navigate Palette to pan around the canvas, and to zoom in and out. The âBirds Eye Viewâ of the dataflow provides a high-level +view of the dataflow and allows you to pan across large portions of the dataflow. You can also find breadcrumbs along the +bottom of the screen. As you navigate into and out of Process Groups, the breadcrumbs show +the depth in the flow, and each Process Group that you entered to reach this depth. Each of the Process Groups listed in the +breadcrumbs is a link that will take you back up to that level in the flow. -[[status_bar]] -Below the breadcrumbs lives the Status bar. The Status bar provides information about how many Processors exist on the canvas in -each state (Stopped, Running, Invalid, Disabled), how many Remote Process Groups exist on the canvas in each state -(Transmitting, Not Transmitting), the number of threads that are currently active in the flow, the amount of data that currently -exists in the flow, and the timestamp at which all of this information was last refreshed. If there are any System-Level bulletins, -these are shown in the Status bar as well. Additionally, if the instance of NiFi is clustered, the Status bar shows how many nodes -are in the cluster and how many are currently connected. +image::images/nifi-navigation.png["NiFi Navigation"] -image::status-bar.png["NiFi Status Bar"] +[[UI-with-multi-tenant-authorization]] +Accessing the UI with Multi-Tenant Authorization +------------------------------------------------ +Multi-tenant authorization enables multiple groups of users (tenants) to command, control, and observe different parts of the dataflow, +with varying levels of authorization. When an authenticated user attempts to view or modify a NiFi resource, the system checks whether the +user has privileges to perform that action. These privileges are defined by policies that you can apply system wide or to individual +components. What this means from a Dataflow Manager perspective is that once you have access to the NiFi canvas, a range of functionality +is visible and available to you, depending on the privileges assigned to you. +The available global access policies are: +[options="header"] +|====================== +|Policy |Privilege +|view the UI |Allows users to view the UI +|access the controller |Allows users to view and modify the controller including reporting tasks, Controller Services, and nodes in the cluster +|query provenance |Allows users to submit a provenance search and request even lineage +|access all policies |Allows users to view and modify the policies for all components +|access users/groups |Allows users view and modify the users and user groups +|retrieve site-to-site details | Allows other NiFi instances to retrieve Site-To-Site details +|view system diagnostics |Allows users to view System Diagnostics +|proxy user requests |Allows proxy machines to send requests on the behalf of others +|access counters |Allows users to view and modify counters +|====================== + +The available component-level access policies are: + +[options="header"] +|====================== +|Policy |Privilege +|view the component |Allows users to view component configuration details +|modify the component |Allows users to modify component configuration details +|view the data |Allows users to view metadata and content for this component through provenance data and flowfile queues in outbound connection +|modify the data |Allows users to empty flowfile queues in outbound connections and to submit replays +|view the policies |Allows users to view the list of users who can view and modify a component +|modify the policies |Allows users to modify the list of users who can view and modify a component +|retrieve data via site-to-site |Allows a port to receive data from NiFi instances +|send data via site-to-site |Allows a port to send data from NiFi instances +|====================== + +If you are unable to view or modify a NiFi resource, contact your System Administrator or see Configuring Users and Access Policies in the +link:administration-guide.html[System Administratorâs Guide] for more information. [[logging-in]] Logging In --------- -If NiFi is configured to run securely, users will be able to request access to the DataFlow. For information on configuring NiFi to run -securely, see the link:administration-guide.html[Admin Guide]. If NiFi supports anonymous access, users will be given access +If NiFi is configured to run securely, users will be able to request access to the DataFlow. For information on configuring NiFi to run +securely, see the link:administration-guide.html[System Administratorâs Guide]. If NiFi supports anonymous access, users will be given access accordingly and given an option to log in. -image::anonymous-access.png["Anonymous Access"] - Clicking the 'login' link will open the log in page. If the user is logging in with their username/password they will be presented with -a form to do so. If NiFi is not configured to support anonymous access and the user is logging in with their username/password, they will +a form to do so. If NiFi is not configured to support anonymous access and the user is logging in with their username/password, they will be immediately sent to the login form bypassing the canvas. -image::login.png["Log In"] - -Once the user has logged in or if they are accessing NiFi using a client certificate loaded in their browser, they will be prompted -to request access by submitting a justification if this is the first time they have accessed this NiFi. Fill in an optional justification -that the administrator will review while granting the account access. If NiFi is not configured to support anonymous access and the -user is using a client certificate, they will be immediately sent to the form to request access bypassing the canvas and login form. - -image::request-access.png["Request Access"] - -Press Submit to send the account request. If NiFi supports anonymous access, the user can continue accessing the DataFlow by closing the -login page. Returning to the login page will check the status of the account request. If access has been granted, press the home link or -reload the page to assume the new roles. +image::images/login.png["Log In"] [[building-dataflow]] Building a DataFlow ------------------- -A DFM is able to build an automated dataflow using the NiFi User Interface (UI). Simply drag components from the toolbar to the canvas, configure the components to meet specific needs, and connect +A DFM is able to build an automated dataflow using the NiFi UI. Simply drag components from the toolbar to the canvas, +configure the components to meet specific needs, and connect the components together. === Adding Components to the Canvas -In the User Interface section above outlined the different segments of the UI and pointed out a Components Toolbar. +The User Interface section above outlined the different segments of the UI and pointed out a Components Toolbar. This section looks at each of the Components in that toolbar: -image::components.png["Components"] +image::images/components.png["Components"] [[processor]] -image:iconProcessor.png["Processor", width=32] +image:images/iconProcessor.png["Processor", width=32] *Processor*: The Processor is the most commonly used component, as it is responsible for data ingress, egress, routing, and manipulating. There are many different types of Processors. In fact, this is a very common Extension Point in NiFi, meaning that many vendors may implement their own Processors to perform whatever functions are necessary for their use case. When a Processor is dragged onto the canvas, the user is presented with a dialog to choose which type of Processor to use: -image::add-processor.png["Add Processor Dialog"] +image::images/add-processor.png["Add Processor Dialog"] In the top-right corner, the user is able to filter the list based on the Processor Type or the Tags associated with a Processor. Processor developers have the ability to add Tags to their Processors. These tags are used in this dialog for filtering and are @@ -238,24 +268,26 @@ in the Tag Cloud. Clicking a Tag in the Cloud will filter the available Processo Tags are selected, only those Processors that contain all of those Tags are shown. For example, if we want to show only those Processors that allow us to ingest data via HTTP, we can select both the `http` Tag and the `ingest` Tag: -image::add-processor-with-tag-cloud.png["Add Processor with Tag Cloud"] +image::images/add-processor-with-tag-cloud.png["Add Processor with Tag Cloud"] Clicking the `Add` button or double-clicking on a Processor Type will add the selected Processor to the canvas at the location that it was dropped. -*Note*: For any component added to the canvas, it is possible to select it with the mouse and move it anywhere on the canvas. Also, it is possible to select multiple items at once by either holding down the Shift key and selecting each item or by holding down the Shift key and dragging a selection box around the desired components. +*Note*: For any component added to the canvas, it is possible to select it with the mouse and move it anywhere on the canvas. +Also, it is possible to select multiple items at once by either holding down the Shift key and selecting each item or by holding +down the Shift key and dragging a selection box around the desired components. -Once a Processor has been dragged onto the canvas, the user may interact with it by right-clicking on the Processor and selecting an option from the context menu. +Once you have dragged a Processor onto the canvas, you can interact with it by right-clicking on the Processor and +selecting an option from the context menu. The options available to you from the context menu vary, depending on the privileges assigned to you. -image::nifi-processor-menu.png["Processor Menu", width=300] +image::images/nifi-processor-menu.png["Processor Menu"] -The following options are available: +While the options available from the context menu vary, the following options are typically available when you have full privileges to work with a Processor: - *Configure*: This option allows the user to establish or change the configuration of the Processor. (See <<Configuring_a_Processor>>.) - *Start* or *Stop*: This option allows the user to start or stop a Processor; the option will be either Start or Stop, depending on the current state of the Processor. - *Stats*: This option opens a graphical representation of the Processor's statistical information over time. -- *Upstream connections*: This option allows the user to see and "jump to" upstream connections that are coming into the Processor. This is particularly useful when processors connect into and out of other Process Groups. -- *Downstream connections*: This option allows the user to see and "jump to" downstream connections that are going out of the Processor. This is particularly useful when processors connect into and out of other Process Groups. +- *Data provenance*: This option displays the NiFi Data Provenance table, with information about data provenance events for the FlowFiles routed through that Processor - *Usage*: This option takes the user to the Processor's usage documentation. - *Change color*: This option allows the user to change the color of the Processor, which can make the visual management of large flows easier. - *Center in view*: This option centers the view of the canvas on the given Processor. @@ -265,53 +297,51 @@ The following options are available: [[input_port]] -image:iconInputPort.png["Input Port", width=32] +image:images/iconInputPort.png["Input Port", width=32] *Input Port*: Input Ports provide a mechanism for transferring data into a Process Group. When an Input Port is dragged onto the canvas, the DFM is prompted to name the Port. All Ports within a Process Group must have unique names. -All components exist only within a Process Group. When a user initially navigates to the NiFi page, the user is placed +All components exist only within a Process Group. When a user initially navigates to the NiFi page, the user is placed in the Root Process Group. If the Input Port is dragged onto the Root Process Group, the Input Port provides a mechanism -to receive data from remote instances of NiFi via <<site-to-site,Site-to-Site>>. In this case, the Input Port can be configured -to restrict access to appropriate users, if NiFi is configured to run securely. For information on configuring NiFi to run -securely, see the -link:administration-guide.html[Admin Guide]. +to receive data from remote instances of NiFi via <<site-to-site,Site-to-Site>>. In this case, the Input Port can be configured +to restrict access to appropriate users, if NiFi is configured to run securely. For information on configuring NiFi to run +securely, see the +link:administration-guide.html[System Administratorâs Guide]. [[output_port]] -image:iconOutputPort.png["Output Port", width=32] +image:images/iconOutputPort.png["Output Port", width=32] *Output Port*: Output Ports provide a mechanism for transferring data from a Process Group to destinations outside of the Process Group. When an Output Port is dragged onto the canvas, the DFM is prompted to name the Port. All Ports within a Process Group must have unique names. If the Output Port is dragged onto the Root Process Group, the Output Port provides a mechanism for sending data to -remote instances of NiFi via <<site-to-site,Site-to-Site>>. In this case, the Port acts as a queue. As remote instances -of NiFi pull data from the port, that data is removed from the queues of the incoming Connections. If NiFi is configured -to run securely, the Output Port can be configured to restrict access to appropriate users. For information on configuring -NiFi to run securely, see the -link:administration-guide.html[Admin Guide]. +remote instances of NiFi via <<site-to-site,Site-to-Site>>. In this case, the Port acts as a queue. As remote instances +of NiFi pull data from the port, that data is removed from the queues of the incoming Connections. If NiFi is configured +to run securely, the Output Port can be configured to restrict access to appropriate users. For information on configuring +NiFi to run securely, see the +link:administration-guide.html[System Administratorâs Guide]. [[process_group]] -image:iconProcessGroup.png["Process Group", width=32] +image:images/iconProcessGroup.png["Process Group", width=32] *Process Group*: Process Groups can be used to logically group a set of components so that the dataflow is easier to understand and maintain. When a Process Group is dragged onto the canvas, the DFM is prompted to name the Process Group. All Process Groups within the same parent group must have unique names. The Process Group will then be nested within that parent group. -Once a Process Group has been dragged onto the canvas, the user may interact with it by right-clicking on the Process Group and selecting an option from -context menu. +Once you have dragged a Process Group onto the canvas, you can interact with it by right-clicking on the Process Group and selecting an option from +context menu.The options available to you from the context menu vary, depending on the privileges assigned to you. -image::nifi-process-group-menu.png["Process Group Menu", width=300] +image::images/nifi-process-group-menu.png["Process Group Menu"] -The following options are available: +While the options available from the context menu vary, the following options are typically available when you have full privileges to work with the Process Group: - *Configure*: This option allows the user to establish or change the configuration of the Process Group. - *Enter group*: This option allows the user to enter the Process Group. It is also possible to double-click on the Process Group to enter it. - *Start*: This option allows the user to start a Process Group. - *Stop*: This option allows the user to stop a Process Group. - *Stats*: This option opens a graphical representation of the Process Group's statistical information over time. -- *Upstream connections*: This option allows the user to see and "jump to" upstream connections that are coming into the Process Group. -- *Downstream connections*: This option allows the user to see and "jump to" downstream connections that are going out of the Process Group. - *Center in view*: This option centers the view of the canvas on the given Process Group. - *Copy*: This option places a copy of the selected Process Group on the clipboard, so that it may be pasted elsewhere on the canvas by right-clicking on the canvas and selecting Paste. The Copy/Paste actions also may be done using the keystrokes Ctrl-C (Command-C) and Ctrl-V (Command-V). - *Delete*: This option allows the DFM to delete a Process Group. @@ -319,22 +349,22 @@ The following options are available: [[remote_process_group]] -image:iconRemoteProcessGroup.png["Remote Process Group", width=32] +image:images/iconRemoteProcessGroup.png["Remote Process Group", width=32] *Remote Process Group*: Remote Process Groups appear and behave similar to Process Groups. However, the Remote Process Group (RPG) references a remote instance of NiFi. When an RPG is dragged onto the canvas, rather than being prompted for a name, the DFM is prompted for the URL of the remote NiFi instance. If the remote NiFi is a clustered instance, the URL that should be used is the URL of the remote instance's NiFi Cluster Manager (NCM). When data is transferred to a clustered instance of NiFi via an RPG, the RPG will first connect to the remote instance's NCM to determine which nodes are in the cluster and how busy each node is. This information is then used to load balance the data that is pushed to each node. The remote NCM is -then interrogated periodically to determine information about any nodes that are dropped from or added to the cluster and to +then interrogated periodically to determine information about any nodes that are dropped from or added to the cluster and to recalculate the load balancing based on each node's load. For more information, see the section on <<site-to-site,Site-to-Site>>. Once a Remote Process Group has been dragged onto the canvas, the user may interact with it by right-clicking on the Remote Process Group and selecting an option from -context menu. +context menu. The options available to you from the context menu vary, depending on the privileges assigned to you. -image::nifi-rpg-menu.png["Remote Process Group Menu", width=300] +image::images/nifi-rpg-menu.png["Remote Process Group Menu"] -The following options are available: +While the options available from the context menu vary, the following options are typically available when you have full privileges to work with the Remote Process Group: - *Configure*: This option allows the user to establish or change the configuration of the Remote Process Group. - *Remote Ports*: This option allows the user to see input ports and/or output ports that exist on the remote instance of NiFi that the Remote Process Group is connected to. Note that if the Site-to-Site configuration is secure, only the ports that the connecting NiFi has been given accessed to will be visible. @@ -352,7 +382,7 @@ The following options are available: [[funnel]] -image:iconFunnel.png["Funnel", width=32] +image:images/iconFunnel.png["Funnel"] *Funnel*: Funnels are used to combine the data from many Connections into a single Connection. This has two advantages. First, if many Connections are created with the same destination, the canvas can become cluttered if those Connections have to span a large space. By funneling these Connections into a single Connection, that single Connection can then be @@ -362,22 +392,22 @@ one Connection, rather than prioritizing the data on each Connection independent [[template]] -image:iconTemplate.png["Template", width=32] +image:images/iconTemplate.png["Template"] *Template*: Templates can be created by DFMs from sections of the flow, or they can be imported from other dataflows. These Templates provide larger building blocks for creating a complex flow quickly. When the Template is dragged onto the canvas, the DFM is provided a dialog to choose which Template to add to the canvas: -image::instantiate-template.png["Instantiate Template Dialog"] +image::images/instantiate-template.png["Instantiate Template Dialog"] Clicking the drop-down box shows all available Templates. Any Template that was created with a description will show a question mark icon, indicating that there is more information. Hovering over the icon with the mouse will show this description: -image::instantiate-template-description.png["Instantiate Template Dialog"] +image::images/instantiate-template-description.png["Instantiate Template Dialog"] [[label]] -image:iconLabel.png["Label"] +images/iconLabel.png["Label"] *Label*: Labels are used to provide documentation to parts of a dataflow. When a Label is dropped onto the canvas, it is created with a default size. The Label can then be resized by dragging the handle in the bottom-right corner. The Label has no text when initially created. The text of the Label can be added by right-clicking on the Label and @@ -402,7 +432,7 @@ the Processor again. The first tab in the Processor Configuration dialog is the Settings tab: -image::settings-tab.png["Settings Tab"] +image::images/settings-tab.png["Settings Tab"] This tab contains several different configuration items. First, it allows the DFM to change the name of the Processor. The name of a Processor by default is the same as the Processor type. Next to the Processor Name is a checkbox, indicating @@ -418,7 +448,7 @@ piece of data (a FlowFile), an event may occur that indicates that the data cann data may be processable at a later time. When this occurs, the Processor may choose to Penalize the FlowFile. This will prevent the FlowFile from being Processed for some period of time. For example, if the Processor is to push the data to a remote service, but the remote service already has a file with the same name as the filename that the Processor -is specifying, the Processor may penalize the FlowFile. The `Penalty duration' allows the DFM to specify how long the +is specifying, the Processor may penalize the FlowFile. The `Penalty duration' allows the DFM to specify how long the FlowFile should be penalized. The default value is 30 seconds. Similarly, the Processor may determine that some situation exists such that the Processor can no longer make any progress, @@ -447,31 +477,30 @@ auto-terminated, the auto-termination status will be cleared (turned off) if the The second tab in the Processor Configuration dialog is the Scheduling Tab: -image::scheduling-tab.png["Scheduling Tab"] +image::images/scheduling-tab.png["Scheduling Tab"] The first configuration option is the Scheduling Strategy. There are three possible options for scheduling components: -- *Timer driven*: This is the default mode. The Processor will be scheduled to run on a regular interval. The interval +* *Timer driven*: This is the default mode. The Processor will be scheduled to run on a regular interval. The interval at which the Processor is run is defined by the `Run schedule' option (see below). -- *Event driven*: When this mode is selected, the Processor will be triggered to run by an event, and that event occurs when FlowFiles enter Connections +* *Event driven*: When this mode is selected, the Processor will be triggered to run by an event, and that event occurs when FlowFiles enter Connections feeding this Processor. This mode is currently considered experimental and is not supported by all Processors. When this mode is selected, the `Run schedule' option is not configurable, as the Processor is not triggered to run periodically but as the result of an event. Additionally, this is the only mode for which the `Concurrent tasks' option can be set to 0. In this case, the number of threads is limited only by the size of the Event-Driven Thread Pool that the administrator has configured. -- *CRON driven*: When using the CRON driven scheduling mode, the Processor is scheduled to run periodically, similar to the +* *CRON driven*: When using the CRON driven scheduling mode, the Processor is scheduled to run periodically, similar to the Timer driven scheduling mode. However, the CRON driven mode provides significantly more flexibility at the expense of increasing the complexity of the configuration. This value is made up of six fields, each separated by a space. These fields include: -+ - . Seconds - . Minutes - . Hours - . Day of Month - . Month - . Day of Week - . Year -+ +** Seconds +** Minutes +** Hours +** Day of Month +** Month +** Day of Week +** Year + The value for each of these fields should be a number, range, or increment. Range here refers to a syntax of <number>-<number>. For example,the Seconds field could be set to 0-30, meaning that the Processor should only be scheduled if the time is 0 to 30 seconds @@ -479,9 +508,9 @@ after the minute. Additionally, a value of `*` indicates that all values are val be entered using a `,` as a separator: `0,5,10,15,30`. An increment is written as <start value>/<increment>. For example, settings a value of `0/10` for the seconds fields means that valid values are 0, 10, 20, 30, 40, and 50. However, if we change this to `5/10`, valid values become 5, 15, 25, 35, 45, and 55. -+ + For the Month field, valid values are 1 (January) through 12 (December). -+ + For the Day of Week field, valid values are 1 (Sunday) through 7 (Saturday). Additionally, a value of `L` may be appended to one of these values to indicate the last occurrence of this day in the month. For example, `1L` can be used to indicate the last Monday of the month. @@ -515,11 +544,11 @@ Lower Latency or Higher Throughput. The Properties Tab provides a mechanism to configure Processor-specific behavior. There are no default properties. Each type of Processor must define which Properties make sense for its use case. Below, we see the Properties Tab for a RouteOnAttribute Processor: -image::properties-tab.png["Properties Tab"] +image::images/properties-tab.png["Properties Tab"] This Processor, by default, has only a single property: `Routing Strategy.' The default value is `Route to Property name.' Next to the name of this property is a small question-mark symbol ( -image:iconInfo.png["Question Mark"] +image:images/iconInfo.png["Question Mark"] ). This help symbol is seen in other places throughout the User Interface, and it indicates that more information is available. Hovering over this symbol with the mouse will provide additional details about the property and the default value, as well as historical values that have been set for the Property. @@ -527,98 +556,123 @@ historical values that have been set for the Property. Clicking on the value for the property will allow a DFM to change the value. Depending on the values that are allowed for the property, the user is either provided a drop-down from which to choose a value or is given a text area to type a value: -image::edit-property-dropdown.png["Edit Property with Dropdown"] +image::images/edit-property-dropdown.png["Edit Property with Dropdown"] In the top-right corner of the tab is a button for adding a New Property. Clicking this button will provide the DFM with a dialog to -enter the name and value of a new property. Not all Processors allow User-Defined properties. In processors that do not allow them, -the Processor becomes invalid when User-Defined properties are applied. RouteOnAttribute, however, does allow User-Defined properties. +enter the name and value of a new property. Not all Processors allow User-Defined properties. In processors that do not allow them, +the Processor becomes invalid when User-Defined properties are applied. RouteOnAttribute, however, does allow User-Defined properties. In fact, this Processor will not be valid until the user has added a property. -image:edit-property-textarea.png["Edit Property with Text Area"] +image:images/edit-property-textarea.png["Edit Property with Text Area"] Note that after a User-Defined property has been added, an icon will appear on the right-hand side of that row ( -image:iconDelete.png["Delete Icon"] +image:images/iconDelete.png["Delete Icon"] ). Clicking it will remove the User-Defined property from the Processor. -Some processors also have an Advanced User Interface (UI) built into them. For example, the UpdateAttribute processor has an Advanced UI. To access the Advanced UI, click the `Advanced` button that appears at the bottom of the Configure Processor window. Only processors that have an Advanced UI will have this button. +Some processors also have an Advanced User Interface (UI) built into them. For example, the UpdateAttribute processor has an Advanced UI. To access the Advanced UI, click the `Advanced` button that appears at the bottom of the Configure Processor window. Only processors that have an Advanced UI will have this button. Some processors have properties that refer to other components, such as Controller Services, which also need to be configured. For example, the GetHTTP processor has an SSLContextService property, which refers to the StandardSSLContextService controller service. When DFMs want to configure this property but have not yet created and configured the controller service, they have the option to create the service on the spot, as depicted in the image below. For more information about configuring Controller Services, see the <<Controller_Services_and_Reporting_Tasks>> section. -image:create-service-ssl-context.png["Create Service", width=700] +image:images/create-service-ssl-context.png["Create Service", width=700] ==== Comments Tab The last tab in the Processor configuration dialog is the Comments tab. This tab simply provides an area for users to include whatever comments are appropriate for this component. Use of the Comments tab is optional: -image::comments-tab.png["Comments Tab"] +image::images/comments-tab.png["Comments Tab"] === Additional Help -The user may access additional documentation about each Processor's usage by right-clicking -on the Processor and then selecting `Usage' from the context menu. Alternatively, clicking the `Help' link in the top-right -corner of the User Interface will provide a Help page with all of the documentation, including usage documentation -for all the Processors that are available. Clicking on the desired Processor in the list will display its usage documentation. +You can access additional documentation about each Processor's usage by right-clicking +on the Processor and selecting `Usage' from the context menu. Alternatively, select Help from the Global Menu in the top-right +corner of the UI to display a Help page with all of the documentation, including usage documentation +for all the Processors that are available. Click on the desired Processor to view usage documentation. [[Controller_Services_and_Reporting_Tasks]] === Controller Services and Reporting Tasks -While DFMs have the ability to create Controller Services from the Configure Processor window, there is also a central place within the User Interface for adding and configuring both Controller Services and Reporting Tasks. To get there, click on the Controller Settings button in the Management section of the toolbar. +While DFMs have the ability to create Controller Services from the Configure Processor window, there is also a central place within the +UI for adding and configuring both Controller Services and Reporting Tasks. To get there, select Controller Settings from the Global Menu. [[Controller_Settings]] ==== Controller Settings -image:controller-settings-button.png["Controller Settings Button", width=200] +image:images/controller-settings-button.png["Controller Settings Button"] -The Controller Settings window has three tabs across the top: General, Controller Services, and Reporting Tasks. The General tab is for settings that pertain to general information about the NiFi instance. For example, here, the DFM can provide a unique name for the overall dataflow, as well as comments that describe the flow. Be aware that this information is visible to any other NiFi instance that connects remotely to this instance (using Remote Process Groups, a.k.a., Site-to-Site). +The Controller Settings window has three tabs across the top: General, Controller Services, and Reporting Tasks. The General tab is +for settings that pertain to general information about the NiFi instance. For example, here, the DFM can provide a unique name for +the overall dataflow, as well as comments that describe the flow. Be aware that this information is visible to any other NiFi instance +that connects remotely to this instance (using Remote Process Groups, a.k.a., Site-to-Site). The General tab also provides settings for the overall maximum thread counts of the instance. -image:settings-general-tab.png["Controller Settings General Tab", width=700] +image:images/settings-general-tab.png["Controller Settings General Tab"] -To the right of the General tab is the Controller Services tab. From this tab, the DFM may click the "+" button in the upper-right corner to create a new Controller Service. +To the right of the General tab is the Controller Services tab. From this tab, the DFM may click the "+" button in the upper-right +corner to create a new Controller Service. -image:controller-services-tab.png["Controller Services Tab", width=900] +image:images/controller-services-tab.png["Controller Services Tab"] -The Add Controller Service window opens. This window is similar to the Add Processor window. It provides a list of the available Controller Services on the right and a tag cloud, showing the most common category tags used for Controller Services, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Controller Services to those that fit the categories desired. The DFM may also use the Filter field at the top of the window to search for the desired Controller Service. Upon selecting a Controller Service from the list, the DFM can see a description of the the service below. Select the desired controller service and click Add, or simply double-click the name of the service to add it. +The Add Controller Service window opens. This window is similar to the Add Processor window. It provides a list of the +available Controller Services on the right and a tag cloud, showing the most common category tags used for Controller +Services, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Controller Services +to those that fit the categories desired. The DFM may also use the Filter field at the top of the window to search +for the desired Controller Service. Upon selecting a Controller Service from the list, the DFM can see a description of +the the service below. Select the desired controller service and click Add, or simply double-click the name of the service +to add it. -image:add-controller-service-window.png["Add Controller Service Window", width=700] +image:images/add-controller-service-window.png["Add Controller Service Window"] -Once a Controller Service has been added, the DFM may configure it by clicking the Edit button (pencil icon) in the far-right column. Other buttons in this column include the Enable button (to enable a configured service), the Remove button, and the Usage button, which links to the documentation for the particular Controller Service. -image:controller-services-edit-buttons.png["Controller Services Buttons"] +Once you have added a Controller Service, you can configure it by clicking the Edit button in the +far-right column. Other buttons in this column include Remove and Access Policies. -When the DFM clicks the Edit button, a Configure Controller Service window opens. It has three tabs: Settings, Properties, and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Controller Service a unique name (if desired). It also lists the UUID for the service and provides a list of other components (processors or other controller services) that reference the service. +image:images/controller-services-edit-buttons.png["Controller Services Buttons"] -image:configure-controller-service-settings.png["Configure Controller Service Settings", width=700] +You can obtain information about Controller Services by clicking the Details, Usage, and Alerts buttons in the left-hand column. -The Properties tab lists the various properties that apply to the particular controller service. As with configuring processors, the DFM may hover the mouse over the question mark icons to see more information about each property. +image:images/controller-services-information-buttons.png["Controller Services Information Buttons"] -image:configure-controller-service-properties.png["Configure Controller Service Properties", width=700] +When the DFM clicks the Edit button, a Configure Controller Service window opens. It has three tabs: Settings, Properties, +and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM +to give the Controller Service a unique name (if desired). It also lists the UUID for the service and provides a list +of other components (processors or other controller services) that reference the service. -The Comments tab is just an open-text field, where the DFM may include comments about the service. After configuring a Controller Service, click the Apply button to apply the configuration and close the window, or click the Cancel button to cancel the changes and close the window. +image:images/configure-controller-service-settings.png["Configure Controller Service Settings"] -Note that after a Controller Service has been configured, it must be enabled in order to run. Do this using the Enable button in the far-right column of the Controller Services tab of the Controller Settings window. Then, in order to modify an existing/running controller service, the DFM needs to stop/disable it (as well as all referencing processors, reporting tasks, and controller services). Rather than having to hunt down each component that is referenced by that controller service, the DFM has the ability to stop/disable them when disabling the controller service in question. Likewise, when enabling a controller service, the DFM has the option to start/enable all referencing processors, reporting tasks, and controller services. +The Properties tab lists the various properties that apply to the particular controller service. As with configuring +processors, the DFM may hover the over the question mark icons to see more information about each property. + +image:images/configure-controller-service-properties.png["Configure Controller Service Properties"] + +The Comments tab is just an open-text field, where the DFM may include comments about the service. After configuring +a Controller Service, click the Apply button to apply the configuration and close the window, or click the Cancel +button to cancel the changes and close the window. + +Note that after a Controller Service has been configured, it must be enabled in order to run. Do this using the +Enable button in the far-right column of the Controller Services tab of the Controller Settings window. Then, +in order to modify an existing/running controller service, the DFM needs to stop/disable it (as well as all referencing processors, reporting tasks, and controller services). Rather than having to hunt down each component that is referenced by that controller service, the DFM has the ability to stop/disable them when disabling the controller service in question. Likewise, when enabling a controller service, the DFM has the option to start/enable all referencing processors, reporting tasks, and controller services. The Reporting Tasks tab behaves similarly to the Controller Services tab. The DFM has the option to add Reporting Tasks and configure them in the same way as Controller Services. -image:reporting-tasks-tab.png["Reporting Tasks Tab", width=900] +image:images/reporting-tasks-tab.png["Reporting Tasks Tab"] Once a Reporting Task has been added, the DFM may configure it by clicking the Edit (pencil icon) in the far-right column. Other buttons in this column include the Start button, Remove button, and Usage button, which links to the documentation for the particular Reporting Task. -image:reporting-tasks-edit-buttons2.png["Reporting Tasks Buttons"] +image:images/reporting-tasks-edit-buttons2.png["Reporting Tasks Buttons"] When the DFM clicks the Edit button, a Configure Reporting Task window opens. It has three tabs: Settings, Properties, and Comments. This window is also similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Reporting Task a unique name (if desired). It also lists a UUID for the Reporting Task and provides settings for the task's Scheduling Strategy and Run Schedule (similar to the same settings in a processor). The DFM may hover the mouse over the question mark icons to see more information about each setting. -image:configure-reporting-task-settings.png["Configure Reporting Task Settings", width=700] +image:images/configure-reporting-task-settings.png["Configure Reporting Task Settings"] The Properties tab for a Reporting Task lists the properties that may be configured for the task. The DFM may hover the mouse over the question mark icons to see more information about each property. -image:configure-reporting-task-properties.png["Configure Reporting Task Properties", width=700] +image:images/configure-reporting-task-properties.png["Configure Reporting Task Properties"] The Comments tab is just an open-text field, where the DFM may include comments about the task. After configuring the Reporting Task, click the Apply button to apply the configuration and close the window, or click Cancel to cancel the changes and close the window. @@ -631,23 +685,24 @@ When you want to run the Reporting Task, click the Start button in the far-right Once processors and other components have been added to the canvas and configured, the next step is to connect them to one another so that NiFi knows what to do with each FlowFile after it has been processed. This is accomplished by creating a Connection between each component. When the user hovers the mouse over the center of a component, a new Connection icon ( -image:addConnect.png["Connection Bubble"] +image:images/addConnect.png["Connection Bubble"] ) appears: -image:processor-connection-bubble.png["Processor with Connection Bubble"] +image:images/processor-connection-bubble.png["Processor with Connection Bubble"] The user drags the Connection bubble from one component to another until the second component is highlighted. When the user releases the mouse, a `Create Connection' dialog appears. This dialog consists of two tabs: `Details' and `Settings'. They are -discussed in detail below. Note that it is possible to draw a connection so that it loops back on the same processor. This can be +discussed in detail below. Note that it is possible to draw a connection so that it loops back on the same processor. This can be useful if the DFM wants the processor to try to re-process FlowFiles if they go down a failure Relationship. To create this type of looping -connection, simply drag the connection bubble away and then back to the same processor until it is highlighted. Then release the mouse and the same 'Create Connection' dialog appears. +connection, simply drag the connection bubble away and then back to the same processor until it is highlighted. Then release the mouse +and the same 'Create Connection' dialog appears. ==== Details Tab The Details Tab of the 'Create Connection' dialog provides information about the source and destination components, including the component name, the component type, and the Process Group in which the component lives: -image::create-connection.png["Create Connection"] +image::images/create-connection.png["Create Connection"] Additionally, this tab provides the ability to choose which Relationships should be included in this Connection. At least one Relationship must be selected. If only one Relationship is available, it is automatically selected. @@ -660,7 +715,7 @@ automatically be `cloned', and a copy will be sent to each of those Connections. The Settings Tab provides the ability to configure the Connection's name, FlowFile expiration, Back Pressure thresholds, and Prioritization: -image:connection-settings.png["Connection Settings"] +image:images/connection-settings.png["Connection Settings"] The Connection name is optional. If not specified, the name shown for the Connection will be names of the Relationships that are active for the Connection. @@ -676,7 +731,7 @@ NiFi provides two configuration elements for Back Pressure. These thresholds ind allowed to exist in the queue before the component that is the source of the Connection is no longer scheduled to run. This allows the system to avoid being overrun with data. The first option provided is the ``Back pressure object threshold.'' This is the number of FlowFiles that can be in the queue before back pressure is applied. The second configuration option -is the ``Back pressure data size threshold.'' +is the ``Back pressure data size threshold.'' This specifies the maximum amount of data (in size) that should be queued up before applying back pressure. This value is configured by entering a number followed by a data size (`B` for bytes, `KB` for kilobytes, `MB` for megabytes, `GB` for gigabytes, or `TB` for terabytes). @@ -697,12 +752,12 @@ The following prioritizers are available: *Note*: After a connection has been drawn between two components, the connection's configuration may be changed, and the connection may be moved to a new destination; however, the processors on either side of the connection must be stopped before a configuration or destination change may be made. -image:nifi-connection.png["Connection", width=300] +image:images/nifi-connection.png["Connection"] To change a connection's configuration or interact with the connection in other ways, right-click on the connection to open the connection context menu. -image:nifi-connection-menu.png["Connection Menu", width=200] +image:images/nifi-connection-menu.png["Connection Menu"] The following options are available: @@ -720,22 +775,22 @@ To add a bend point (or elbow) to an existing connection, simply double-click on the bend point and drag it so that the connection is bent in the desired way. You can add as many bend points as you want. You can also use the mouse to drag and move the label on the connection to any existing bend point. To remove a bend point, simply double-click it again. -image:nifi-connection-bend-points.png["Connection Bend Points", width=500] +image:images/nifi-connection-bend-points.png["Connection Bend Points"] === Processor Validation Before trying to start a Processor, it's important to make sure that the Processor's configuration is valid. A status indicator is shown in the top-left of the Processor. If the Processor is invalid, the indicator -will show a yellow Warning indicator with an exclamation mark indicating that there is a problem: +will show a red Warning indicator with an exclamation mark indicating that there is a problem: -image::invalid-processor.png["Invalid Processor"] +image::images/invalid-processor.png["Invalid Processor"] In this case, hovering over the indicator icon with the mouse will provide a tooltip showing all of the validation errors for the Processor. Once all of the validation errors have been addressed, the status indicator will change to a Stop icon, indicating that the Processor is valid and ready to be started but currently is not running: -image::valid-processor.png["Valid Processor"] +image::images/valid-processor.png["Valid Processor"] @@ -756,7 +811,7 @@ Using Site-to-Site provides the following benefits: to allow only specific users, and only those users will be able to see that the port even exists. For information on configuring the Certificates, see the link:administration-guide.html#security-configuration[Security Configuration] section of the -link:administration-guide.html[Admin Guide]. +link:administration-guide.html[System Administratorâs Guide]. * Scalable ** As nodes in the remote cluster change, those changes are automatically detected and data is scaled out across all nodes in the cluster. @@ -799,8 +854,8 @@ It is important to understand which NiFi instance will be the client or server i [[Site-to-Site_Remote_Process_Group]] *Remote Process Group*: In order to communicate with a remote NiFi instance via Site-to-Site, simply drag a <<remote_process_group,Remote Process Group>> onto the canvas -and enter the URL of the remote NiFi instance (for more information on the components of a Remote Process Group, see -<<Remote_Group_Transmission,Remote Process Group Transmission>> section of this guide.) The URL is the same +and enter the URL of the remote NiFi instance (for more information on the components of a Remote Process Group, see +<<Remote_Group_Transmission,Remote Process Group Transmission>> section of this guide.) The URL is the same URL you would use to go to that instance's User Interface. At that point, you can drag a connection to or from the Remote Process Group in the same way you would drag a connection to or from a Processor or a local Process Group. When you drag the connection, you will have a chance to choose which Port to connect to. Note that it may take up to one minute for the Remote Process Group to determine @@ -812,12 +867,12 @@ the ports shown will be the Input Ports of the remote group, as this implies tha *Note*: if the remote instance is configured to use secure data transmission, you will see only ports that you are authorized to communicate with. For information on configuring NiFi to run securely, see the -link:administration-guide.html[Admin Guide]. +link:administration-guide.html[System Administratorâs Guide]. [[Site-to-Site_Transport_Protocol]] *Transport Protocol*: On a Remote Process Group creation or configuration dialog, you can choose Transport Protocol to use for Site-to-Site communication as shown in the following image: -image:configure-remote-process-group.png["Configure Remote Process Group", width=395] +image:images/configure-remote-process-group.png["Configure Remote Process Group"] By default, it is set to _RAW_ which uses raw socket communication using a dedicated port. _HTTP_ transport protocol is especially useful if the remote NiFi instance is in a restricted network that only allow access through HTTP(S) protocol or only accessible from a specific HTTP Proxy server. For accessing through a HTTP Proxy Server, BASIC and DIGEST authentication are supported. @@ -836,7 +891,7 @@ Process Group and choose to "Refresh" the flow. [[Site-to-Site_Output_Port]] *Output Port*: Similar to an Input Port, a DataFlow Manager may choose to add an <<output_port,Output Port>> to the Root Process Group. The Output Port allows an authorized NiFi instance to remotely connect to your instance and pull data from the Output Port. Configuring the Output Port will again allow the -DFM to control how many concurrent tasks are allowed, as well as which NiFi instances are authorized to pull data from the instance being configured. +DFM to control how many concurrent tasks are allowed, as well as which NiFi instances are authorized to pull data from the instance being configured. In addition to other instances of NiFi, some other applications may use a Site-to-Site client in order to push data to or receive data from a NiFi instance. For example, NiFi provides an Apache Storm spout and an Apache Spark Receiver that are able to pull data from NiFi's Root Group Output Ports. @@ -850,12 +905,12 @@ section of the link:administration-guide.html[Admin Guide]. For information on how to enable and configure Site-to-Site on a NiFi instance, see the link:administration-guide.html#site_to_site_properties[Site-to-Site Properties] section of the -link:administration-guide.html[Admin Guide]. +link:administration-guide.html[System Administratorâs Guide]. === Example Dataflow -This section has described the steps required to build a dataflow. Now, to put it all together. The following example dataflow +This section has described the steps required to build a dataflow. Now, to put it all together. The following example dataflow consists of just two processors: GenerateFlowFile and LogAttribute. These processors are normally used for testing, but they can also be used to build a quick flow for demonstration purposes and see NiFi in action. @@ -871,7 +926,7 @@ After you drag the GenerateFlowFile and LogAttribute processors to the canvas an The dataflow should look like the following: -image::simple-flow.png["Simple Flow", width=900] +image::images/simple-flow.png["Simple Flow"] Now see the following section on how to start and stop the dataflow. When the dataflow is running, be sure to note the statistical information that is displayed on the face of each processor (see <<processor_anatomy>>). @@ -896,11 +951,11 @@ In order to start a component, the following conditions must be met: - The component must be enabled. -- The component must have no active tasks. For more information about active tasks, see the ``Anatomy of ...'' +- The component must have no active tasks. For more information about active tasks, see the ``Anatomy of ...'' sections under <<monitoring>> (<<processor_anatomy>>, <<process_group_anatomy>>, <<remote_group_anatomy>>). Components can be started by selecting all of the components to start and then clicking the Start icon ( -image:iconRun.png["Start"] +image:images/iconRun.png["Start"] ) in the Actions Toolbar or by right-clicking a single component and choosing Start from the context menu. @@ -908,7 +963,7 @@ If starting a Process Group, all components within that Process Group (including be started, with the exception of those components that are invalid or disabled. Once started, the status indicator of a Processor will change to a Play symbol ( -image:iconRun.png["Run"] +image:images/iconRun.png["Run"] ). @@ -916,14 +971,14 @@ image:iconRun.png["Run"] A component can be stopped any time that it is running. A component is stopped by right-clicking on the component and clicking Stop from the context menu, or by selecting the component and clicking the Stop icon ( -image:iconStop.png["Stop"] +image:images/iconStop.png["Stop"] ) in the Actions Toolbar. If a Process Group is stopped, all of the components within the Process Group (including child Process Groups) will be stopped. Once stopped, the status indicator of a component will change to the Stop symbol ( -image:iconStop.png["Stop"] +image:images/iconStop.png["Stop"] ). Stopping a component does not interrupt its currently running tasks. Rather, it stops scheduling new tasks to @@ -933,26 +988,26 @@ for more information). === Enabling/Disabling a Component When a component is enabled, it is able to be started. Users may choose to disable components when they are part of a -dataflow that is still being assembled, for example. Typically, if a component is not intended to be run, the component -is disabled, rather than being left in the Stopped state. This helps to distinguish between components that are -intentionally not running and those that may have been stopped temporarily (for instance, to change the component's -configuration) and inadvertently were never restarted. +dataflow that is still being assembled, for example. Typically, if a component is not intended to be run, the component +is disabled, rather than being left in the Stopped state. This helps to distinguish between components that are +intentionally not running and those that may have been stopped temporarily (for instance, to change the component's +configuration) and inadvertently were never restarted. -When it is desirable to re-enable a component, it can be enabled by selecting the component and +When it is desirable to re-enable a component, it can be enabled by selecting the component and clicking the Enable icon ( -image:iconEnable.png["Enable"] +image:images/iconEnable.png["Enable"] ) in the Actions Toolbar. This is available only when the selected component or components are disabled. -Alternatively, a component can be enabled by checking the checkbox next to the ``Enabled'' option in +Alternatively, a component can be enabled by checking the checkbox next to the ``Enabled'' option in the Settings tab of the Processor configuration dialog or the configuration dialog for a Port. Once enabled, the component's status indicator will change to either Invalid ( -image:iconAlert.png["Invalid"] +image:images/iconAlert.png["Invalid"] ) or Stopped ( -image:iconStop.png["Stopped"] +image:images/iconStop.png["Stopped"] ), depending on whether or not the component is valid. A component is then disabled by selecting the component and clicking the Disable icon ( -image:iconDisable.png["Disable"] +image:images/iconDisable.png["Disable"] ) in the Actions Toolbar, or by clearing the checkbox next to the ``Enabled'' option in the Settings tab of the Processor configuration dialog or the configuration dialog for a Port. @@ -965,31 +1020,31 @@ Only Ports and Processors can be enabled and disabled. Remote Process Groups provide a mechanism for sending data to or retrieving data from a remote instance of NiFi. When a Remote Process Group (RPG) is added to the canvas, it is added with the Transmission Disabled, as indicated by the icon ( -image:iconTransmissionInactive.png["Transmission Disabled"] +image:images/iconTransmissionInactive.png["Transmission Disabled"] ) in the top-left corner. When Transmission is Disabled, it can be enabled by right-clicking on the RPG and clicking the ``Enable Transmission'' menu item. This will cause all ports for which there is a Connection to begin transmitting data. This will cause the status indicator to then change to the Transmission Enabled icon ( -image:iconTransmissionActive.png["Transmission Enabled"] -). +image:images/iconTransmissionActive.png["Transmission Enabled"] +). If there are problems communicating with the Remote Process Group, a Warning indicator ( -image:iconAlert.png["Warning"] +image:images/iconAlert.png["Warning"] ) may instead be present in the top-left corner. Hovering over this Warning indicator with the mouse will provide more information about the problem. [[Remote_Port_Configuration]] ==== Individual Port Transmission -There are times when the DFM may want to either enable or disable transmission for only a specific +There are times when the DFM may want to either enable or disable transmission for only a specific Port within the Remote Process Group. This can be accomplished by right-clicking on the Remote Process Group and choosing the ``Remote ports'' menu item. This provides a configuration dialog from which each Port can be configured: -image::remote-group-ports-dialog.png["Remote Process Groups"] +image::images/remote-group-ports-dialog.png["Remote Process Groups"] The left-hand side lists all of the Input Ports that the remote instance of NiFi allows data to be sent to. The right-hand side lists all of the Output Ports from which this instance is able to pull data. -If the remote instance is using secure communications (the URL of the NiFi instance begins with `https://`, +If the remote instance is using secure communications (the URL of the NiFi instance begins with `https://`, rather than `http://`), any Ports that the remote instance has not made available to this instance will not be shown. @@ -1003,12 +1058,12 @@ Each Port is shown with the Port name, followed by its description, currently co tasks, and whether or not data sent to this port will be compressed. To the left of this information is a switch to turn the Port on or off. Those Ports that have no Connections attached to them are grayed out: -image::remote-port-connection-status.png["Remote Port Statuses"] +image::images/remote-port-connection-status.png["Remote Port Statuses"] The on/off switch provides a mechanism to enable and disable transmission for each Port in the Remote -Process Group independently. Those Ports that are connected but are not currently transmitting can be +Process Group independently. Those Ports that are connected but are not currently transmitting can be configured by clicking the pencil icon ( -image:iconEdit.png["Edit"] +image:images/iconEdit.png["Edit"] ) below the on/off switch. Clicking this icon will allow the DFM to change the number of Concurrent tasks and whether or not compression should be used when transmitting data to or from this Port. @@ -1026,15 +1081,15 @@ the NiFi canvas; however, once a flow exists on the canvas, there are additional == Monitoring of DataFlow NiFi provides a great deal of information about the status of the DataFlow in order to monitor the -health and status. The Status bar provides information about the overall system health +health and status. The Status bar provides information about the overall system health (See <<status_bar>> above for more information). Processors, Process Groups, and Remote Process Groups -provide fine-grained details about their operations. Connections and Process Groups provide information +provide fine-grained details about their operations. Connections and Process Groups provide information about the amount of data in their queues. The Summary Page provides information about all of the components on the canvas in a tabular format and also provides System Diagnostics information that includes disk usage, CPU utilization, and Java Heap and Garbage Collection information. In a clustered environment, this -information is available per-node or as aggregates across the entire cluster. We will explore each of these +information is available per-node or as aggregates across the entire cluster. We will explore each of these monitoring artifacts below. - + [[processor_anatomy]] === Anatomy of a Processor @@ -1042,7 +1097,7 @@ monitoring artifacts below. NiFi provides a significant amount of information about each Processor on the canvas. The following diagram shows the anatomy of a Processor: -image:processor-anatomy.png["Anatomy of a Processor"] +image:images/processor-anatomy.png["Anatomy of a Processor"] The image outlines the following elements: @@ -1050,7 +1105,7 @@ The image outlines the following elements: of tasks to be performed. Each type of Processor is designed to perform one specific task. The Processor type (PutFile, in this example) describes the task that this Processor performs. In this case, the Processor writes a FlowFile to disk - or ``Puts'' a FlowFile to a File. - + - *Bulletin Indicator*: When a Processor logs that some event has occurred, it generates a Bulletin to notify those who are monitoring NiFi via the User Interface. The DFM is able to configure which bulletins should be displayed in the User Interface by updating the ``Bulletin level'' field in the @@ -1059,32 +1114,32 @@ The image outlines the following elements: Processor. When it is present, hovering over the icon with the mouse will provide a tooltip explaining the message provided by the Processor as well as the Bulletin level. If the instance of NiFi is clustered, it will also show the Node that emitted the Bulletin. Bulletins automatically expire after five minutes. - + - *Status Indicator*: Shows the current Status of the Processor. The following indicators are possible: - ** image:iconRun.png["Running"] + ** image:images/iconRun.png["Running"] *Running*: The Processor is currently running. - ** image:iconStop.png["Stopped"] + ** image:images/iconStop.png["Stopped"] *Stopped*: The Processor is valid and enabled but is not running. - ** image:iconAlert.png["Invalid"] - *Invalid*: The Processor is enabled but is not currently valid and cannot be started. + ** image:images/iconAlert.png["Invalid"] + *Invalid*: The Processor is enabled but is not currently valid and cannot be started. Hovering over this icon will provide a tooltip indicating why the Processor is not valid. - ** image:iconDisable.png["Disabled"] + ** image:images/iconDisable.png["Disabled"] *Disabled*: The Processor is not running and cannot be started until it has been enabled. This status does not indicate whether or not the Processor is valid. - + - *Processor Name*: This is the user-defined name of the Processor. By default, the name of the Processor is the same as the Processor Type. In the example, this value is "Copy to /review". - + - *Active Tasks*: The number of tasks that this Processor is currently executing. This number is constrained by the ``Concurrent tasks'' setting in the ``Scheduling'' tab of the Processor configuration dialog. Here, we can see that the Processor is currently performing two tasks. If the NiFi instance is clustered, this value represents the number of tasks that are currently executing across all nodes in the cluster. - + - *5-Minute Statistics*: The Processor shows several different statistics in tabular form. Each of these statistics represents the amount of work that has been performed in the past five minutes. If the NiFi instance is clustered, these values indicate how much work has been done by all of the Nodes combined in the past five minutes. These metrics are: - + ** *In*: The amount of data that the Processor has pulled from the queues of its incoming Connections. This value is represented as <count> / <size> where <count> is the number of FlowFiles that have been pulled from the queues and <size> is the total size of those FlowFiles' content. In this example, @@ -1102,7 +1157,7 @@ The image outlines the following elements: which already existed in the output directory, data was neither read nor written to disk. ** *Out*: The amount of data that the Processor has transferred to its outbound Connections. This does not include FlowFiles that the Processor removes itself, or FlowFiles that are routed to connections - that are auto-terminated. Like the ``In'' metric above, this value is represented as <count> / <size> + that are auto-terminated. Like the ``In'' metric above, this value is represented as <count> / <size> where <count> is the number of FlowFiles that have been transferred to outbound Connections and <size> is the total size of those FlowFiles' content. In this example, all of the Relationships are configured to be auto-terminated, so no FlowFiles are reported as having been transferred Out. @@ -1114,8 +1169,8 @@ The image outlines the following elements: case we will see the Time metric showing that it took 60 seconds, instead of 1 second. This time can be thought of as ``System Time,'' or said another way, this value is 60 seconds because that's the amount of time it would have taken to perform the action if only a single concurrent task were used. - - + + @@ -1123,26 +1178,26 @@ The image outlines the following elements: === Anatomy of a Process Group The Process Group provides a mechanism for grouping components together into a logical construct in order -to organize the DataFlow in a way that makes it more understandable from a higher level. +to organize the DataFlow in a way that makes it more understandable from a higher level. The following image highlights the different elements that make up the anatomy of a Process Group: -image::process-group-anatomy.png["Anatomy of a Process Group"] +image::images/process-group-anatomy.png["Anatomy of a Process Group"] The Process Group consists of the following elements: - *Name*: This is the user-defined name of the Process Group. This name is set when the Process Group is added to the canvas. The name can later by changed by right-clicking on the Process Group and clicking the ``Configure'' menu option. In this example, the name of the Process Group is ``Process Group ABC.'' - + - *Bulletin Indicator*: When a child component of a Process Group emits a bulletin, that bulletin is propagated to the component's parent Process Group, as well. When any component has an active Bulletin, this indicator will appear, allowing the user to hover over the icon with the mouse to see the Bulletin. - + - *Active Tasks*: The number of tasks that are currently executing by the components within this - Process Group. Here, we can see that the Process Group is currently performing one task. If the - NiFi instance is clustered, this value represents the number of tasks that are currently executing + Process Group. Here, we can see that the Process Group is currently performing one task. If the + NiFi instance is clustered, this value represents the number of tasks that are currently executing across all nodes in the cluster. - + - *Comments*: When the Process Group is added to the canvas, the user is given the option of specifying Comments in order to provide information about the Process Group. The comments can later be changed by right-clicking on the Process Group and clicking the ``Configure'' menu option. In this example, the Comments are set to ``Example Process Group.'' @@ -1154,57 +1209,49 @@ The Process Group consists of the following elements: This field is represented as <count> / <size> where <count> is the number of FlowFiles that are currently enqueued in the Process Group and <size> is the total size of those FlowFiles' content. In this example, the Process Group currently has 1,738 FlowFiles enqueued; those FlowFiles have a total size of 350.03 megabytes (MB). - + ** *In*: The number of FlowFiles that have been transferred into the Process Group through all of its Input Ports over the past 5 minutes. This field is represented as <count> / <size> where <count> is the number of FlowFiles that - have entered the Process Group in the past 5 minutes and <size> is the total size of those FlowFiles' content. + have entered the Process Group in the past 5 minutes and <size> is the total size of those FlowFiles' content. In this example, 686 FlowFiles have entered the Process Group and their total size is 214.01 MB. - - ** *Read/Write*: The total size of the FlowFile content that the components within the Process Group have - read from disk and written to disk. This provides valuable information about the I/O performance that this - Process Group requires. In this example, we see that in the past five minutes, components within this + + ** *Read/Write*: The total size of the FlowFile content that the components within the Process Group have + read from disk and written to disk. This provides valuable information about the I/O performance that this + Process Group requires. In this example, we see that in the past five minutes, components within this Process Group have read 72.9 MB of the FlowFile content and have written 686.65 MB. - + ** *Out*: The number of FlowFiles that have been transferred out of the Process Group through its Output Ports over the past 5 minutes. This field is represented as <count> / <size> where <count> is the number of FlowFiles that have exited the Process Group in the past 5 minutes and <size> is the total size of those FlowFiles' content. In this example, 657 FlowFiles have exited the Process Group and their total size is 477.74 MB. - + - *Component Counts*: The Component Counts element provides information about how many components of each type exist within the Process Group. The following provides information about each of these icons and their meanings: - - ** image:iconInputPortSmall.png["Input Port", width=16] - *Input Ports*: The number of Input Ports that exist directly within this Process Group. This does not include any - Input Ports that exist within child Process Groups, as child groups' ports cannot be accessed directly. - - ** image:iconOutputPortSmall.png["Output Port"] - *Output Ports*: The number of Output Ports that exist directly within this Process Group. This does not include any - Output Ports that exist within child Process Groups as child groups' ports cannot be accessed directly. - - ** image:iconTransmissionActive.png["Transmission Active"] + + ** image:images/iconTransmissionActive.png["Transmission Active"] *Transmitting Ports*: The number of Remote Process Group Ports that currently are configured to transmit data to remote instances of NiFi or pull data from remote instances of NiFi. - - ** image:iconTransmissionInactive.png["Transmission Inactive"] + + ** image:images/iconTransmissionInactive.png["Transmission Inactive"] *Non-Transmitting Ports*: The number of Remote Process Group Ports that are currently connected to components within this Process Group but currently have their transmission disabled. - - ** image:iconRun.png["Running"] + + ** image:images/iconRun.png["Running"] *Running Components*: The number of Processors, Input Ports, and Output Ports that are currently running within this Process Group. - - ** image:iconStop.png["Stopped Components"] + + ** image:images/iconStop.png["Stopped Components"] *Stopped Components*: The number of Processors, Input Ports, and Output Ports that are currently not running but are valid and enabled. These components are ready to be started. - - ** image:iconAlert.png["Invalid Components"] - *Invalid Components*: The number of Processors, Input Ports, and Output Ports that are enabled but are currently + + ** image:images/iconAlert.png["Invalid Components"] + *Invalid Components*: The number of Processors, Input Ports, and Output Ports that are enabled but are currently not in a valid state. This may be due to misconfigured properties or missing Relationships. - - ** image:iconDisable.png["Disabled Components"] + + ** image:images/iconDisable.png["Disabled Components"] *Disabled Components*: The number of Processors, Input Ports, and Output Ports that are currently disabled. These components may or may not be valid. If the Process Group is started, these components will not cause any errors - but will not be started. + but will not be started. @@ -1221,7 +1268,7 @@ and state of a Remote Process Group, such as queue sizes, the information render Process Group is related to the interaction that occurs between this instance of NiFi and the remote instance. -image::remote-group-anatomy.png["Anatomy of a Remote Process Group"] +image::images/remote-group-anatomy.png["Anatomy of a Remote Process Group"] The image above shows the different elements that make up a Remote Process Group. Here, we provide an explanation of the icons and details about the information provided. @@ -1229,10 +1276,10 @@ explanation of the icons and details about the information provided. - *Transmission Status*: The Transmission Status indicates whether or not data Transmission between this instance of NiFi and the remote instance is currently enabled. The icon shown will be the Transmission Enabled icon ( -image:iconTransmissionActive.png["Transmission Active"] - ) if any of the Input Ports or Output Ports is currently configured to transmit or the Transmission +image:images/iconTransmissionActive.png["Transmission Active"] + ) if any of the Input Ports or Output Ports is currently configured to transmit or the Transmission Disabled icon ( -image:iconTransmissionInactive.png["Transmission Inactive"] +image:images/iconTransmissionInactive.png["Transmission Inactive"] ) if all of the Input Ports and Output Ports that are currently connected are stopped. - *Remote Instance Name*: This is the name of the NiFi instance that was reported by the remote instance. @@ -1241,56 +1288,24 @@ image:iconTransmissionInactive.png["Transmission Inactive"] - *Remote Instance URL*: This is the URL of the remote instance that the Remote Process Group points to. This URL is entered when the Remote Process Group is added to the canvas and it cannot be changed. - + - *Secure Indicator*: This icon indicates whether or not communications with the remote NiFi instance are secure. If communications with the remote instance are secure, this will be indicated by the ``locked'' icon ( -image:iconSecure.png["Secure"] +image:images/iconSecure.png["Secure"] ). If the communications are not secure, this will be indicated by the ``unlocked'' icon ( -image:iconNotSecure.png["Not Secure"] +image:images/iconNotSecure.png["Not Secure"] ). If the communications are secure, this instance of NiFi will not be able to communicate with the remote instance until an administrator for the remote instance grants access. Whenever the Remote Process Group is added to the canvas, this will automatically initiate a request to have a user for this instance of NiFi created on the - remote instance. This instance will be unable to communicate with the remote instance until an administrator + remote instance. This instance will be unable to communicate with the remote instance until an administrator on the remote instance adds the user to the system and adds the ``NiFi'' role to the user. In the event that communications are not secure, the Remote Process Group is able to receive data from anyone, and the data is not encrypted while it is transferred between instances of NiFi. - -- *Input Ports*: This section shows three pieces of information: - ** image:iconInputPortSmall.png["Input Ports"] - *Input Ports*: The number of Input Ports that are available to send data to on the remote instance of NiFi. - If the remote instance is secure, only the ports to which this instance of NiFi has been granted access - will be counted. - - ** image:iconTransmissionActive.png["Transmitting"] - *Transmitting Ports*: The number of Input Ports to which this NiFi is connected and currently configured to - send data to. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <<Remote_Group_Transmission>>) or via the <<Remote_Port_Configuration>> dialog. - - ** image:iconTransmissionInactive.png["Not Transmitting"] - *Non-Transmitting Ports*: The number of Input Ports to which this NiFi is connected but is not currently configured - to send data to. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <<Remote_Group_Transmission>>) or via the <<Remote_Port_Configuration>> dialog. - -- *Output Ports*: Similar to the ``Input Ports'' section above, this element shows three pieces of information: - ** image:iconOutputPortSmall.png["Output Ports"] - *Output Ports*: The number of Output Ports that are available to pull data from the remote instance of NiFi. - If the remote instance is secure, only the ports to which this instance of NiFi has been granted access - will be counted. - - ** image:iconTransmissionActive.png["Transmitting"] - *Transmitting Ports*: The number of Output Ports from which this NiFi is connected and currently configured - to pull data from. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <<Remote_Group_Transmission>>) or via the <<Remote_Port_Configuration>> dialog. - - ** image:iconTransmissionInactive.png["Not Transmitting"] - *Non-Transmitting Ports*: The number of Output Ports to which this NiFi is connected but is not currently configured - to pull data from. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <<Remote_Group_Transmission>>) or via the <<Remote_Port_Configuration>> dialog. - + - *5-Minute Statistics*: Two statistics are shown for Remote Process Groups: *Sent* and *Received*. Both of these are in the format <count> / <size> where <count> is the number of FlowFiles that have been sent or received in the previous - five minutes and <size> is the total size of those FlowFiles' content. + f
<TRUNCATED>
