This is an automated email from the ASF dual-hosted git repository. szaszm pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit a178c9f10c676f212bf0110789e89757e90d233c Author: Gabor Gyimesi <[email protected]> AuthorDate: Mon Jul 19 15:50:39 2021 +0200 MINIFICPP-1601 Add example flow configurations New files under examples/: - getfile_putfile_config.yml: Getting File Data and Putting It in an Output Directory - tailfile_config.yml: Tailing a Single File - cwel_config.yml: Consuming Windows Event Logs (Win-only) - pdh_config.yml: Reading System Performance Data (Win-only) - consumejournald_config.yml: Consume Systemd-Journald System Journal Messages (Linux-only) - http_post_config.yml: HTTP POST Invocation - site_to_site_config.yml: Transfer Data to Remote Nifi Instance - publishkafka_config.yml: Publish Message to Kafka Broker - publishkafka_ssl_config.yml: Publish Message to Kafka Broker Through SSL - consumekafka_config.yml: Consume Messages from Kafka - azure_storage_config.yml: Upload Blob to Azure Storage - puts3_config.yml: Put Object in AWS S3 Bucket - lists3_fetchs3_config.yml: List and Fetch Content from AWS S3 Bucket - querydbtable_config.yml: Query Database Table Closes #1127 Signed-off-by: Marton Szasz <[email protected]> --- PROCESSORS.md | 10 +-- README.md | 4 ++ conf/config.yml | 2 +- examples/README.md | 128 ++++++++++++++++++++++++++++++++++- examples/azure_storage_config.yml | 46 +++++++++++++ examples/consumejournald_config.yml | 43 ++++++++++++ examples/consumekafka_config.yml | 53 +++++++++++++++ examples/cwel_config.yml | 50 ++++++++++++++ examples/getfile_putfile_config.yml | 50 ++++++++++++++ examples/http_post_config.yml | 85 +++++++++++++++++++++++ examples/lists3_fetchs3_config.yml | 66 ++++++++++++++++++ examples/pdh_config.yml | 49 ++++++++++++++ examples/publishkafka_config.yml | 49 ++++++++++++++ examples/publishkafka_ssl_config.yml | 55 +++++++++++++++ examples/puts3_config.yml | 45 ++++++++++++ examples/querydbtable_config.yml | 52 ++++++++++++++ examples/site_to_site_config.yml | 43 ++++++++++++ examples/tailfile_config.yml | 47 +++++++++++++ 18 files changed, 869 insertions(+), 8 deletions(-) diff --git a/PROCESSORS.md b/PROCESSORS.md index e2f7b30..c110348 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -1195,11 +1195,11 @@ In the list below, the names of required properties appear in bold. Any other pr | Name | Default Value | Allowable Values | Description | | - | - | - | - | |**Container Name**|||Name of the Azure storage container. In case of PutAzureBlobStorage processor, container can be created if it does not exist.<br/>**Supports Expression Language: true**| -|Storage Account Name||||The storage account name.<br/>**Supports Expression Language: true**| -|Storage Account Key||||The storage account key. This is an admin-like password providing access to every container in this account. It is recommended one uses Shared Access Signature (SAS) token instead for fine-grained control with policies.<br/>**Supports Expression Language: true**| -|SAS Token||||Shared Access Signature token. Specify either SAS Token (recommended) or Account Key.<br/>**Supports Expression Language: true**| -|Common Storage Account Endpoint Suffix||||Storage accounts in public Azure always use a common FQDN suffix. Override this endpoint suffix with a different suffix in certain circumstances (like Azure Stack or non-public Azure regions).<br/>**Supports Expression Language: true**| -|Connection String||||Connection string used to connect to Azure Storage service. This overrides all other set credential properties.<br/>**Supports Expression Language: true**| +|Storage Account Name|||The storage account name.<br/>**Supports Expression Language: true**| +|Storage Account Key|||The storage account key. This is an admin-like password providing access to every container in this account. It is recommended one uses Shared Access Signature (SAS) token instead for fine-grained control with policies.<br/>**Supports Expression Language: true**| +|SAS Token|||Shared Access Signature token. Specify either SAS Token (recommended) or Account Key.<br/>**Supports Expression Language: true**| +|Common Storage Account Endpoint Suffix|||Storage accounts in public Azure always use a common FQDN suffix. Override this endpoint suffix with a different suffix in certain circumstances (like Azure Stack or non-public Azure regions).<br/>**Supports Expression Language: true**| +|Connection String|||Connection string used to connect to Azure Storage service. This overrides all other set credential properties.<br/>**Supports Expression Language: true**| |Azure Storage Credentials Service|||Name of the Azure Storage Credentials Service used to retrieve the connection string from.| |**Blob**|||The filename of the blob.<br/>**Supports Expression Language: true**| |**Create Container**|false||Specifies whether to check if the container exists and to automatically create it if it does not. Permission to list containers is required. If false, this check is not made, but the Put operation will fail if the container does not exist.| diff --git a/README.md b/README.md index bf71160..3928a94 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ MiNiFi is a child project effort of Apache NiFi. This repository is for a nativ - [Operations](#operations) - [Issue Tracking](#issue-tracking) - [Documentation](#documentation) +- [Examples](#examples) - [License](#license) ## Features @@ -588,6 +589,9 @@ See https://issues.apache.org/jira/projects/MINIFICPP/issues for the issue track ## Documentation See https://nifi.apache.org/minifi for the latest documentation. +## Examples +See our [examples page](examples/README.md) for flow examples. + ## Contributing We welcome all contributions to Apache MiNiFi. To make development easier, we've included diff --git a/conf/config.yml b/conf/config.yml index 0f7f2a5..a3e1837 100644 --- a/conf/config.yml +++ b/conf/config.yml @@ -18,4 +18,4 @@ Flow Controller: Processors: [] Connections: [] Remote Processing Groups: [] -Provenance Reporting: \ No newline at end of file +Provenance Reporting: diff --git a/examples/README.md b/examples/README.md index f12935a..5c154d7 100644 --- a/examples/README.md +++ b/examples/README.md @@ -12,6 +12,130 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Apache NiFi - MiNiFi - C++ Examples +# Apache NiFi - MiNiFi - C++ Examples -- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md) \ No newline at end of file +The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out. + +- [Filesystem Operations](#filesystem-operations) + - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory) + - [Tailing a Single File](#tailing-a-single-file) +- [Windows Specific Processors](#windows-specific-processors) + - [Consuming Windows Event Logs](#consuming-windows-event-logs) + - [Reading System Performance Data](#reading-system-performance-data) +- [Linux Specific Processors](#linux-specific-processors) + - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages) +- [HTTP Operations](#http-operations) + - [HTTP POST Invocation](#http-post-invocation) +- [Site to Site Operations](#site-to-site-operations) + - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance) + - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration) +- [Kafka Operations](#kafka-operations) + - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker) + - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl) + - [Consume Messages from Kafka](#consume-messages-from-kafka) +- [Public Cloud Operations](#public-cloud-operations) + - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage) + - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket) + - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket) +- [SQL Operations](#sql-operations) + - [Query Database Table](#query-database-table) + +## Filesystem Operations + +### Getting File Data and Putting It in an Output Directory + +Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory. + +The flow: GetFile ➔ success ➔ PutFile + +### Tailing a Single File + +Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes. + +The flow: TailFile ➔ success ➔ LogAttribute + +## Windows Specific Processors + +### Consuming Windows Event Logs + +Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format. + +The flow: ConsumeWindowsEventLog ➔ PutFile + +### Reading System Performance Data + +Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format. + +The flow: PerformanceDataMonitor ➔ PutFile + +## Linux Specific Processors + +### Consume Systemd-Journald System Journal Messages + +Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level. + +The flow: ConsumeJournald ➔ LogAttribute + +## HTTP Operations + +### HTTP POST Invocation + +Using the [http_post_config.yml](http_post_config.yml) flow configuration MiNiFi transfers flowfile data received from the GetFile processor by invoking an HTTP endpoint with POST method. + +The flow: GetFile ➔ success ➔ InvokeHTTP + +## Site to Site Operations + +### Transfer Data to Remote Nifi Instance + +Using the [site_to_site_config.yml](site_to_site_config.yml) flow configuration MiNiFi transfers data received from the GetFile processor to a remote NiFi instance located at `http://nifi:8080/nifi`. + +### [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md) + +## Kafka Operations + +### Publish Message to Kafka Broker + +Using the [publishkafka_config.yml](publishkafka_config.yml) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic. + +The flow: GetFile ➔ success ➔ PublishKafka + +### Publish Message to Kafka Broker Through SSL + +Using the [publishkafka_ssl_config.yml](publishkafka_ssl_config.yml) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic through SSL connection. + +The flow: GetFile ➔ success ➔ PublishKafka + +### Consume Messages from Kafka + +Using the [consumekafka_config.yml](consumekafka_config.yml) flow configuration MiNiFi consumes messages from the configured Kafka broker's `ConsumeKafkaTest` topic from the earliest available message. The messages are forwarded to the `PutFile` processor and put in the `/tmp/output` directory. + +The flow: ConsumeKafka ➔ success ➔ PutFile + +## Public Cloud Operations + +### Upload Blob to Azure Storage + +Using the [azure_storage_config.yml](azure_storage_config.yml) flow configuration MiNiFi uploads data received from the GetFile processor to Azure's blob storage container `test-container`. + +The flow: GetFile ➔ success ➔ PutAzureBlobStorage + +### Put Object in AWS S3 Bucket + +Using the [puts3_config.yml](puts3_config.yml) flow configuration MiNiFi uploads data received from the GetFile processor to AWS S3 bucket `test_bucket`. + +The flow: GetFile ➔ success ➔ PutS3Object + +### List and Fetch Content from AWS S3 Bucket + +Using the [lists3_fetchs3_config.yml](lists3_fetchs3_config.yml) flow configuration MiNiFi lists S3 bucket `test_bucket` and fetches its contents in flowfiles then logs the attributes. The flow uses `AWSCredentialsService` controller service to provide credentials for all S3 processors. It has `Use Default Credentials` property set which retrieves credentials from AWS default credentials provider chain (environment variables, configuration file, instance profile). + +The flow: ListS3 ➔ FetchS3Object ➔ LogAttribute + +## SQL Operations + +### Query Database Table + +Using the [querydbtable_config.yml](querydbtable_config.yml) flow configuration MiNiFi queries the `id` and `name` columns of the `users` table with a `where` clause and the results are put in the `/tmp/output` directory. The database connection data is set in the `ODBCService` controller service. + +The flow: QueryDatabaseTable ➔ PutFile diff --git a/examples/azure_storage_config.yml b/examples/azure_storage_config.yml new file mode 100644 index 0000000..777be17 --- /dev/null +++ b/examples/azure_storage_config.yml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: d5fef466-fef0-4d90-814e-9c19a67e454a + class: org.apache.nifi.processors.standard.GetFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 2 sec + Properties: + Input Directory: /tmp/input +- name: Upload Blob to test-container in Azure Storage + id: e7b3f9c6-d1ee-4d9b-b676-48c872cc7ae3 + class: org.apache.nifi.processors.standard.PutAzureBlobStorage + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success + - failure + Properties: + Blob: test-blob + Storage Account Name: test-account + Storage Account Key: test-key + Container Name: test-container + Create Container: 'true' +Connections: +- name: GetFile/success/PutAzureBlobStorage + source id: d5fef466-fef0-4d90-814e-9c19a67e454a + source relationship name: success + destination id: e7b3f9c6-d1ee-4d9b-b676-48c872cc7ae3 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/consumejournald_config.yml b/examples/consumejournald_config.yml new file mode 100644 index 0000000..89ea553 --- /dev/null +++ b/examples/consumejournald_config.yml @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the \"License\"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an \"AS IS\" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Consume journal messages + id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35 + class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald + scheduling strategy: TIMER_DRIVEN + scheduling period: 5 sec +- name: Log messages to info level + id: f7281113-9576-4680-9a87-a8e961101b5c + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: [success] + Properties: + FlowFiles To Log: 0 + Log Level: info + Log Payload: true + Log Prefix: '=====' + Maxumim Payload Line Length: 0 +Connections: +- name: ConsumeJournald/success/LogAttribute + id: f57713d9-fa0b-4416-948d-4b316b1a1c0d + source id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35 + source relationship name: success + destination id: f7281113-9576-4680-9a87-a8e961101b5c +Remote Processing Groups: [] +Provenance Reporting: diff --git a/examples/consumekafka_config.yml b/examples/consumekafka_config.yml new file mode 100644 index 0000000..3878bd3 --- /dev/null +++ b/examples/consumekafka_config.yml @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Consume Kafka messages from ConsumeKafkaTest topic + id: 37d6e8e5-6284-423e-8620-c5168a71add7 + class: org.apache.nifi.processors.standard.ConsumeKafka + scheduling strategy: TIMER_DRIVEN + scheduling period: 1 sec + auto-terminated relationships list: + - failure + Properties: + Group ID: test_group + Honor Transactions: 'true' + Kafka Brokers: kafka-broker:9092 + Key Attribute Encoding: UTF-8 + Max Poll Time: 4 sec + Message Header Encoding: UTF-8 + Offset Reset: earliest + Session Timeout: 60 sec + Topic Name Format: Names + Topic Names: ConsumeKafkaTest +- name: Put files to /tmp/output + class: org.apache.nifi.processors.standard.PutFile + id: 7f9c6f0b-e936-499e-b659-acfaf6872772 + auto-terminated relationships list: + - success + - failure + scheduling strategy: EVENT_DRIVEN + Properties: + Directory: /tmp/output +Connections: +- name: ConsumeKafka/success/PutFile + source id: 37d6e8e5-6284-423e-8620-c5168a71add7 + source relationship name: success + destination id: 7f9c6f0b-e936-499e-b659-acfaf6872772 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/cwel_config.yml b/examples/cwel_config.yml new file mode 100644 index 0000000..252e16b --- /dev/null +++ b/examples/cwel_config.yml @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Consume Windows Events in JSON format + id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + class: org.apache.nifi.minifi.processors.ConsumeWindowsEventLog + scheduling strategy: TIMER_DRIVEN + scheduling period: 1000 ms + Properties: + Channel: "System" + Query: "*" + Output Format: "JSON" + JSON Format: "Flattened" + Batch Commit Size: "10" +- name: Put files to C:\temp\ + id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + - success + Properties: + Conflict Resolution Strategy: fail + Create Missing Directories: 'true' + Directory: C:\temp\ +Connections: +- name: ConsumeWindowsEventLog/success/PutFile + id: 098a56ba-f4bf-4323-a3f3-6f8a5e3586bf + source id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + source relationship names: + - success + destination id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 +Controller Services: [] +Remote Process Groups: [] diff --git a/examples/getfile_putfile_config.yml b/examples/getfile_putfile_config.yml new file mode 100644 index 0000000..e1fe675 --- /dev/null +++ b/examples/getfile_putfile_config.yml @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/getfile_dir + id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + class: org.apache.nifi.minifi.processors.GetFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 1000 ms + Properties: + Input Directory: /tmp/getfile_dir + Keep Source File: 'true' + Recurse Subdirectories: 'true' + Ignore Hidden Files: 'false' + Minimum File Size: '1 MB' +- name: Put files to /tmp/out_dir + id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + - success + Properties: + Conflict Resolution Strategy: fail + Create Missing Directories: 'true' + Directory: /tmp/out_dir +Connections: +- name: GetFile/success/PutFile + id: 098a56ba-f4bf-4323-a3f3-6f8a5e3586bf + source id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + source relationship names: + - success + destination id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 +Controller Services: [] +Remote Process Groups: [] diff --git a/examples/http_post_config.yml b/examples/http_post_config.yml new file mode 100644 index 0000000..4ce683b --- /dev/null +++ b/examples/http_post_config.yml @@ -0,0 +1,85 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: 962790e7-ea35-4096-9362-96f527288669 + class: org.apache.nifi.processors.standard.GetFile + scheduling period: 2 sec + scheduling strategy: TIMER_DRIVEN + Properties: + Input Directory: /tmp/input + Keep Source File: 'true' +- name: Invoke POST request on remote URL + id: 33b373b6-6f19-4194-b45a-1ef73c357a8e + class: org.apache.nifi.processors.standard.InvokeHTTP + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success + - response + Properties: + HTTP Method: POST + Remote URL: http://1.2.3.4:8080/contentListener +- name: Retry on HTTP 5xx status codes + id: 3b1c6971-541b-4530-9f72-175757500699 + class: org.apache.nifi.processors.standard.RetryFlowFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + Properties: + Maximum Retries: 5 +- name: Log failed flow files + id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9 + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: [success] + Properties: + FlowFiles To Log: 0 + Log Level: warn + Log Prefix: '=== Failed HTTP request ===' + Maxumim Payload Line Length: 0 +Connections: +- name: GetFile/success/InvokeHTTP + source id: 962790e7-ea35-4096-9362-96f527288669 + source relationship name: success + destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e +- name: InvokeHTTP/failure/InvokeHTTP + source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e + source relationship name: failure + destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e +- name: InvokeHTTP/retry/RetryFlowFile + id: c8a96f1d-48cd-4ee3-8a2f-081c3e3c7bcd + source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e + source relationship names: retry + destination id: 3b1c6971-541b-4530-9f72-175757500699 +- name: RetryFlowFile/retry/InvokeHTTP + id: cadf51f0-3468-423a-9d12-5c40270aa7ca + source id: 3b1c6971-541b-4530-9f72-175757500699 + source relationship names: retry + destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e +- name: RetryFlowFile/retries_exceeded/LogAttribute + id: 160fc0f4-6efe-4274-8bc7-2a416570bdeb + source id: 3b1c6971-541b-4530-9f72-175757500699 + source relationship names: retries_exceeded + destination id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9 +- name: InvokeHTTP/no retry/LogAttribute + source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e + source relationship name: no retry + destination id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/lists3_fetchs3_config.yml b/examples/lists3_fetchs3_config.yml new file mode 100644 index 0000000..e9e7d88 --- /dev/null +++ b/examples/lists3_fetchs3_config.yml @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Fetch S3 object from test_bucket + id: 6ccd8be9-4aa3-4b86-ae10-0a68668f1c25 + class: org.apache.nifi.minifi.aws.processors.FetchS3Object + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + Properties: + AWS Credentials Provider service: AWSCredentialsService + Bucket: test_bucket + Communications Timeout: 30 sec +- name: List test_bucket S3 bucket + id: 6f2c20f5-e990-4d9f-bfd6-f7471e3df469 + class: org.apache.nifi.minifi.aws.processors.ListS3 + scheduling strategy: TIMER_DRIVEN + scheduling period: 10000 ms + Properties: + AWS Credentials Provider service: AWSCredentialsService + Bucket: test_bucket + Communications Timeout: 30 sec +- name: Log S3 object attributes + id: 73fd7f54-7639-4f27-b20b-889708f493cb + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success + Properties: + FlowFiles To Log: '0' +Controller Services: +- name: AWSCredentialsService + id: 2094d776-2006-4d02-9bb9-28eac9d0fc95 + class: org.apache.nifi.minifi.aws.controllers.AWSCredentialsService + Properties: + Use Default Credentials: 'true' +Connections: +- name: FetchS3Object/success/LogAttribute + id: 2c13ddd6-543e-4e37-a295-e4cb8fe85fa5 + source id: 6ccd8be9-4aa3-4b86-ae10-0a68668f1c25 + source relationship names: + - success + destination id: 73fd7f54-7639-4f27-b20b-889708f493cb +- name: ListS3/success/FetchS3Object + id: 7895c12e-9657-4baa-a8ec-6b8cb11f32cd + source id: 6f2c20f5-e990-4d9f-bfd6-f7471e3df469 + source relationship names: + - success + destination id: 6ccd8be9-4aa3-4b86-ae10-0a68668f1c25 +Remote Process Groups: [] diff --git a/examples/pdh_config.yml b/examples/pdh_config.yml new file mode 100644 index 0000000..143cb98 --- /dev/null +++ b/examples/pdh_config.yml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Read disk and CPU performance data + id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + class: org.apache.nifi.minifi.processors.PerformanceDataMonitor + scheduling strategy: TIMER_DRIVEN + scheduling period: 10000 ms + Properties: + Predefined Groups: "Disk,CPU" + Custom PDH Counters: "\\System\\Processes,\\Process(*)\\ID Process,\\Process(*)\\Private Bytes" + Output Format: "JSON" + Output Compactness: "Compact" +- name: Put files to C:\temp\ + id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + - success + Properties: + Conflict Resolution Strategy: fail + Create Missing Directories: 'true' + Directory: C:\temp\ +Connections: +- name: PerformanceDataMonitor/success/PutFile + id: 098a56ba-f4bf-4323-a3f3-6f8a5e3586bf + source id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + source relationship names: + - success + destination id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 +Controller Services: [] +Remote Process Groups: [] diff --git a/examples/publishkafka_config.yml b/examples/publishkafka_config.yml new file mode 100644 index 0000000..104c931 --- /dev/null +++ b/examples/publishkafka_config.yml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: 7fd166aa-0662-4c42-affa-88f6fb39807f + class: org.apache.nifi.processors.standard.GetFile + scheduling period: 2 sec + scheduling strategy: TIMER_DRIVEN + Properties: + Input Directory: /tmp/input +- name: Publish messages to Kafka topic test + id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848 + class: org.apache.nifi.processors.standard.PublishKafka + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success + - failure + Properties: + Batch Size: '10' + Client Name: test-client + Compress Codec: none + Delivery Guarantee: '1' + Known Brokers: kafka-broker:9092 + Message Timeout: 12 sec + Request Timeout: 10 sec + Topic Name: test +Connections: +- name: GetFile/success/PublishKafka + source id: 7fd166aa-0662-4c42-affa-88f6fb39807f + source relationship name: success + destination id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/publishkafka_ssl_config.yml b/examples/publishkafka_ssl_config.yml new file mode 100644 index 0000000..16f90b8 --- /dev/null +++ b/examples/publishkafka_ssl_config.yml @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d + class: org.apache.nifi.processors.standard.GetFile + scheduling period: 2 sec + scheduling strategy: TIMER_DRIVEN + Properties: + Input Directory: /tmp/input +- name: Publish messages to Kafka topic test + id: 8c0791c7-920c-4541-bcde-f5d559f42ed2 + class: org.apache.nifi.processors.standard.PublishKafka + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success + - failure + Properties: + Batch Size: 10 + Client Name: test_client + Compress Codec: none + Delivery Guarantee: 1 + Known Brokers: kafka-broker:9093 + Message Timeout: 12 sec + Message Timeout Phrase: 12 sec + Request Timeout: 10 sec + Security CA Key: /tmp/resources/certs/ca-cert + Security Cert: /tmp/resources/certs/client_test_client_client.pem + Security Pass Phrase: abcdefgh + Security Private Key: /tmp/resources/certs/client_test_client_client.key + Security Protocol: ssl + Topic Name: test +Connections: +- name: GetFile/success/PublishKafka + source id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d + source relationship name: success + destination id: 8c0791c7-920c-4541-bcde-f5d559f42ed2 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/puts3_config.yml b/examples/puts3_config.yml new file mode 100644 index 0000000..5786d6b --- /dev/null +++ b/examples/puts3_config.yml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: bfd376a8-c845-453b-9f74-4cf6f3a95e74 + class: org.apache.nifi.processors.standard.GetFile + scheduling period: 2 sec + scheduling strategy: TIMER_DRIVEN + Properties: + Input Directory: /tmp/input +- name: Upload to S3 test_bucket + id: 8c9e2512-e814-4b73-ac5c-f33753332128 + class: org.apache.nifi.processors.standard.PutS3Object + scheduling period: 1 sec + scheduling strategy: TIMER_DRIVEN + auto-terminated relationships list: + - success + - failure + Properties: + Access Key: test_access_key + Bucket: test_bucket + Secret Key: test_secret +Connections: +- name: GetFile/success/PutS3Object + source id: bfd376a8-c845-453b-9f74-4cf6f3a95e74 + source relationship name: success + destination id: 8c9e2512-e814-4b73-ac5c-f33753332128 +Controller Services: [] +Remote Processing Groups: [] diff --git a/examples/querydbtable_config.yml b/examples/querydbtable_config.yml new file mode 100644 index 0000000..903bd5f --- /dev/null +++ b/examples/querydbtable_config.yml @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Query users table + id: 0bd6ebcd-7fef-47e3-8e36-4413d39d01d5 + class: org.apache.nifi.processors.standard.QueryDatabaseTable + scheduling strategy: TIMER_DRIVEN + scheduling period: 1 sec + Properties: + Columns to Return: id,name + DB Controller Service: ODBCService + Output Format: JSON + Table Name: users + Where Clause: id = 1 +- name: Put files to /tmp/output + id: 32700228-e078-42a6-89d7-03e4e478c9da + class: org.apache.nifi.processors.standard.PutFile + scheduling strategy: EVENT_DRIVEN + scheduling period: 1 sec + auto-terminated relationships list: + - success + - failure + Properties: + Directory: /tmp/output +Controller Services: +- name: ODBCService + id: 64fafacd-74ad-4c3b-b1cc-a816ef128e09 + class: ODBCService + Properties: + Connection String: Driver={PostgreSQL ANSI};Server=postgresql-server;Port=5432;Database=postgres;Uid=postgres;Pwd=password; +Connections: +- name: QueryDatabaseTable/success/PutFile + destination id: 32700228-e078-42a6-89d7-03e4e478c9da + source id: 0bd6ebcd-7fef-47e3-8e36-4413d39d01d5 + source relationship name: success +Remote Processing Groups: [] diff --git a/examples/site_to_site_config.yml b/examples/site_to_site_config.yml new file mode 100644 index 0000000..2030b16 --- /dev/null +++ b/examples/site_to_site_config.yml @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: 45c2a56e-7c29-42f3-8c1d-89c0c84a6ee7 + class: org.apache.nifi.processors.standard.GetFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 2 sec + Properties: + Input Directory: /tmp/input +Remote Processing Groups: +- Input Ports: + - Properties: {} + id: 4222cd3a-746e-3497-abd0-b7695faa750c + max concurrent tasks: 1 + name: to_nifi + id: 38878f84-23cd-4aaa-9c45-45151bb2d681 + name: RemoteProcessGroup + timeout: 30 sec + url: http://nifi:8080/nifi + yield period: 3 sec +Connections: +- name: GetFile/success/RPG + source id: 45c2a56e-7c29-42f3-8c1d-89c0c84a6ee7 + source relationship name: success + destination id: 4222cd3a-746e-3497-abd0-b7695faa750c +Controller Services: [] diff --git a/examples/tailfile_config.yml b/examples/tailfile_config.yml new file mode 100644 index 0000000..ada27ff --- /dev/null +++ b/examples/tailfile_config.yml @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Log tailfile attributes + id: 41141af1-3c43-4fa4-8236-2b207b0fe9d8 + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: TIMER_DRIVEN + scheduling period: 1000 ms + auto-terminated relationships list: + - success + Properties: + FlowFiles To Log: '0' +- name: Tail /tmp/test_file.log + id: 0ef22940-ac0a-4813-9ad9-fe41c2b51fff + class: org.apache.nifi.minifi.processors.TailFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 1000 ms + Properties: + File to Tail: /tmp/test_file.log + Initial Start Position: Beginning of File + Input Delimiter: \n + tail-mode: Single file +Connections: +- name: TailFile/success/LogAttribute + id: eda7f758-c548-46d6-aba0-faa6e9937d8c + source id: 0ef22940-ac0a-4813-9ad9-fe41c2b51fff + source relationship names: + - success + destination id: 41141af1-3c43-4fa4-8236-2b207b0fe9d8 +Controller Services: [] +Remote Process Groups: []
