Repository: nifi-minifi-cpp Updated Branches: refs/heads/master 628687280 -> ee4bb1353
MINIFI-174 Documentation for bidirectional communication configuration This closes #33. Signed-off-by: Aldrin Piri <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/commit/ee4bb135 Tree: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/tree/ee4bb135 Diff: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/diff/ee4bb135 Branch: refs/heads/master Commit: ee4bb13535783abda26677bc13652a1a54ff12a0 Parents: 6286872 Author: Jeremy Dyer <[email protected]> Authored: Tue Dec 27 16:57:10 2016 -0500 Committer: Aldrin Piri <[email protected]> Committed: Tue Dec 27 20:37:04 2016 -0500 ---------------------------------------------------------------------- examples/BidirectionalSiteToSite/README.md | 108 ++++++++++++ .../assets/Apache_NiFi_Incoming_Data.png | Bin 0 -> 96992 bytes .../assets/Apache_NiFi_Outgoing_Data.png | Bin 0 -> 97630 bytes .../assets/Apache_NiFi_Workflow.png | Bin 0 -> 97556 bytes .../assets/BidirectionalSite2SiteExample.xml | 169 +++++++++++++++++++ .../assets/BidirectionalSiteToSite.png | Bin 0 -> 81229 bytes examples/README.md | 17 ++ 7 files changed, 294 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/README.md ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/README.md b/examples/BidirectionalSiteToSite/README.md new file mode 100644 index 0000000..78c0c58 --- /dev/null +++ b/examples/BidirectionalSiteToSite/README.md @@ -0,0 +1,108 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# Bidirectional Site-To-Site Configuration + +## Overview + +Site-To-Site is a cornerstone feature to the Apache NiFi platform. Currently configuring Site-To-Site with MiNiFi-CPP can be a little tricky however. There are plans in the roadmap to improve this configuration flow but given what we have today this example will demonstrate how you can configure Apache MiNiFi-C++ for bidirectional communication with Apache NiFi. + +Below is a very high level diagram illustrating our end goal in this example. + + + +## Apache NiFi Setup + +Before we start setting up our MiNiFi-CPP instance we need to first setup our Apache NiFi instance that the MiNiFi instance will be communicating with. If we recall from the deployment diagram above our example will have a simple environment consisting of a single NiFi instance. While we are using a small environment here the same principals would apply for larger deployments. + +First we must [download and install NiFi](https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#downloading-and-installing-nifi) if you haven't done so already. + +As you can see from the screenshot below our contrived NiFi workflow example simply accepts data from MiNiFi, logs that data, and also writes that data back to MiNiFi. + + + +You can download this workflow from the screenshot [here](assets/BidirectionalSite2SiteExample.xml) + +After installing NiFi, uploading the workflow, and ensuring site-to-site is enabled there are a few environment specific values that you should take note of as you will need them when configuring MiNiFi. + +- ```{NIFI_HOST_OR_IP}``` - The hostname or IP of your NiFi and MiNiFi instances. + - ```nifi.dev``` and ```minifi.dev``` are the hostnames for this example instance but your instance will vary. +- ```{NIFI_OUTPUT_PORT_ID}``` and ```{NIFI_INPUT_PORT_ID}``` - The ID of the Input and Output ports from your NiFi workflow. + - Even if you download and use the same workflow from [here](assets/BidirectionalSite2SiteExample.xml) these ID values are autogenerated and will be different in your environment. + - You can find these values by highlighting the Input or Output port in the NiFi workflow and copying the ID value from the ```Operate``` tile in the NiFi UI. +- ```{NIFI_SITE_TO_SITE_PORT}``` - Ensure Apache NiFi site-to-site is enabled by following [this guide](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_properties). Note the port you specify for ```nifi.remote.input.socket.port``` + +## Apache MiNiFi-CPP Setup + +Setup for the MiNiFi instance really only involves properly configuring your flow yml file which by default is located at ```$MINIFI_HOME/conf/config.yml```. The below configuration can be used after replacing the ```{}``` values with those called out above in the NiFi configuration. After updating this config.yml it can be placed in your ```$MINIFI_HOME/conf/config.yml``` and then MiNiFi can be restarted. After restarting MiNiFi you should start seeing flowfiles accumulate in the output connection from the ```MiNiFiData``` input port in your NiFi UI when you place or touch a file in the directory ```/tmp/getfile/``` on the MiNiFi host. + +``` +Flow Controller: + name: MiNiFi Bidirectional SiteToSite Example + +Processors: + - name: GetFile + class: org.apache.nifi.processors.standard.GetFile + max concurrent tasks: 1 + scheduling strategy: TIMER_DRIVEN + scheduling period: 1 sec + penalization period: 30 sec + yield period: 1 sec + run duration nanos: 0 + auto-terminated relationships list: + Properties: + Input Directory: /tmp/getfile + Keep Source File: true + +Connections: + - name: TransferFilesToRPG + source name: GetFile + source relationship name: success + destination name: {NIFI_INPUT_PORT_ID} + max work queue size: 0 + max work queue data size: 1 MB + flowfile expiration: 60 sec + +Remote Processing Groups: + - name: NiFi Flow + url: http://{NIFI_HOST_OR_IP}:{NIFI_WEB_UI_PORT}/nifi + timeout: 30 secs + yield period: 10 sec + Input Ports: + - id: {NIFI_INPUT_PORT_ID} + name: MiNiFiData + max concurrent tasks: 1 + Properties: + Port: {NIFI_SITE_TO_SITE_PORT} + Host Name: {NIFI_HOST_OR_IP} + Output Ports: + - id: {NIFI_OUTPUT_PORT_ID} + name: ToMiNiFi + max concurrent tasks: 1 + Properties: + Port: {NIFI_SITE_TO_SITE_PORT} + Host Name: {NIFI_HOST_OR_IP} +``` + +## Conclusion + +After starting MiNiFi we can see that the connections going to both ```LogAttribute``` and ```ToMiNiFi``` both contain 24 flowfiles. Since both of the processors are stopped this makes perfect sense and validates that the data is flowing from MiNiFi to NiFi as expected. + + + +Now to validate that the flowfiles will flow back to MiNiFi we can start the ```ToMiNiFi``` output port and the connection queue should drain its flowfiles. + + + +As you can see from the screenshot after starting ```ToMiNiFi``` the connection successfully drained and the flowfiles were successfully transfered back to MiNiFi completing the bidirectional lifecycle. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png new file mode 100644 index 0000000..63cf3be Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png differ http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png new file mode 100644 index 0000000..c206e80 Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png differ http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png new file mode 100644 index 0000000..b467c35 Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png differ http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml b/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml new file mode 100644 index 0000000..186231f --- /dev/null +++ b/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml @@ -0,0 +1,169 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<?xml version="1.0" ?> +<template encoding-version="1.0"> + <description></description> + <groupId>cfed8382-0158-1000-1395-192d30f8b560</groupId> + <name>BidirectionSite2SiteExample</name> + <snippet> + <connections> + <id>08757b0a-0159-1000-0000-000000000000</id> + <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId> + <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> + <backPressureObjectThreshold>10000</backPressureObjectThreshold> + <destination> + <groupId>cfed8382-0158-1000-0000-000000000000</groupId> + <id>08756e38-0159-1000-0000-000000000000</id> + <type>PROCESSOR</type> + </destination> + <flowFileExpiration>0 sec</flowFileExpiration> + <labelIndex>1</labelIndex> + <name></name> + <source> + <groupId>cfed8382-0158-1000-0000-000000000000</groupId> + <id>08755c4f-0159-1000-0000-000000000000</id> + <type>INPUT_PORT</type> + </source> + <zIndex>0</zIndex> + </connections> + <connections> + <id>4152969d-0159-1000-0000-000000000000</id> + <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId> + <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> + <backPressureObjectThreshold>10000</backPressureObjectThreshold> + <destination> + <groupId>cfed8382-0158-1000-0000-000000000000</groupId> + <id>37052b39-0159-1000-0000-000000000000</id> + <type>OUTPUT_PORT</type> + </destination> + <flowFileExpiration>0 sec</flowFileExpiration> + <labelIndex>1</labelIndex> + <name></name> + <source> + <groupId>cfed8382-0158-1000-0000-000000000000</groupId> + <id>08755c4f-0159-1000-0000-000000000000</id> + <type>INPUT_PORT</type> + </source> + <zIndex>0</zIndex> + </connections> + <inputPorts> + <id>08755c4f-0159-1000-0000-000000000000</id> + <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId> + <position> + <x>0.0</x> + <y>6.696388244628906</y> + </position> + <comments></comments> + <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> + <name>MiNiFiData</name> + <state>RUNNING</state> + <transmitting>false</transmitting> + <type>INPUT_PORT</type> + </inputPorts> + <outputPorts> + <id>37052b39-0159-1000-0000-000000000000</id> + <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId> + <position> + <x>478.088623046875</x> + <y>0.0</y> + </position> + <comments></comments> + <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> + <name>ToMiNiFi</name> + <state>RUNNING</state> + <transmitting>false</transmitting> + <type>OUTPUT_PORT</type> + </outputPorts> + <processors> + <id>08756e38-0159-1000-0000-000000000000</id> + <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId> + <position> + <x>179.394287109375</x> + <y>151.79802703857422</y> + </position> + <config> + <bulletinLevel>WARN</bulletinLevel> + <comments></comments> + <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> + <descriptors> + <entry> + <key>Log Level</key> + <value> + <name>Log Level</name> + </value> + </entry> + <entry> + <key>Log Payload</key> + <value> + <name>Log Payload</name> + </value> + </entry> + <entry> + <key>Attributes to Log</key> + <value> + <name>Attributes to Log</name> + </value> + </entry> + <entry> + <key>Attributes to Ignore</key> + <value> + <name>Attributes to Ignore</name> + </value> + </entry> + <entry> + <key>Log prefix</key> + <value> + <name>Log prefix</name> + </value> + </entry> + </descriptors> + <executionNode>ALL</executionNode> + <lossTolerant>false</lossTolerant> + <penaltyDuration>30 sec</penaltyDuration> + <properties> + <entry> + <key>Log Level</key> + <value>info</value> + </entry> + <entry> + <key>Log Payload</key> + <value>true</value> + </entry> + <entry> + <key>Attributes to Log</key> + </entry> + <entry> + <key>Attributes to Ignore</key> + </entry> + <entry> + <key>Log prefix</key> + </entry> + </properties> + <runDurationMillis>0</runDurationMillis> + <schedulingPeriod>0 sec</schedulingPeriod> + <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> + <yieldDuration>1 sec</yieldDuration> + </config> + <name>Log Incoming Data</name> + <relationships> + <autoTerminate>true</autoTerminate> + <name>success</name> + </relationships> + <style></style> + <type>org.apache.nifi.processors.standard.LogAttribute</type> + </processors> + </snippet> + <timestamp>12/27/2016 17:29:28 UTC</timestamp> +</template> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png ---------------------------------------------------------------------- diff --git a/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png b/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png new file mode 100644 index 0000000..51bf1f4 Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png differ http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/README.md ---------------------------------------------------------------------- diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..f12935a --- /dev/null +++ b/examples/README.md @@ -0,0 +1,17 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# Apache NiFi - MiNiFi - C++ Examples + +- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md) \ No newline at end of file
