szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666937904
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use
cases. Use the path of these yaml files as the flow configuration set in the
`nifi.flow.configuration.file` property of `conf/minifi.properties` file or
replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output
Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi
Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional
Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through
SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3
Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow
configuration MiNiFi gets all files of minimum 1MB size from the
`/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output
directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi
tails a single file `/tmp/test_file.log` and creates flowfiles from every
single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries
all Windows system events and puts them to the `C:\temp\` directory in
flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow
configuration MiNiFi reads incoming journal messages and logs the attributes on
`info` level.
Review comment:
They are not really incoming messages, as they are produced on the
system. The contents are also logged, not only the attributes.
```suggestion
Using the [consumejournald_config.yml](consumejournald_config.yml) flow
configuration MiNiFi reads systemd-journald journal messages and logs them on
`info` level.
```
##########
File path: examples/consumekafka_config.yml
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume Kafka messages from ConsumeKafkaTest topic
+ id: 37d6e8e5-6284-423e-8620-c5168a71add7
+ class: org.apache.nifi.processors.standard.ConsumeKafka
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ Properties:
+ Group ID: test_group
+ Honor Transactions: 'true'
+ Kafka Brokers: kafka-broker:9092
+ Key Attribute Encoding: UTF-8
+ Max Poll Time: 4 sec
+ Message Header Encoding: UTF-8
+ Offset Reset: earliest
+ Session Timeout: 60 sec
+ Topic Name Format: Names
+ Topic Names: ConsumeKafkaTest
Review comment:
Missing auto-terminated relationships list, but ConsumeKafka also has a
failure relationship with no connections from it.
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - retry
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
Review comment:
It would be nice to implement retry here. Flow files at the retry
relationship should be retried. Maybe we should also limit the number of
retries.
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
+ id: f7281113-9576-4680-9a87-a8e961101b5c
+ class: org.apache.nifi.minifi.processors.LogAttribute
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list: [success]
+ Properties:
+ FlowFiles To Log: 0
+ Log Level: info
+ Log Payload: true
+ Log Prefix: '====='
+ Maxumim Payload Line Length: 0
+Connections:
+- name: ConsumeJournald/success/LogAttribute
+ id: f57713d9-fa0b-4416-948d-4b316b1a1c0d
+ source name: ConsumeJournald
Review comment:
```suggestion
source name: Consume journal messages
```
I wonder if this could be omitted.
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use
cases. Use the path of these yaml files as the flow configuration set in the
`nifi.flow.configuration.file` property of `conf/minifi.properties` file or
replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output
Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi
Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional
Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through
SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3
Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
Review comment:
I would call this "Consume systemd-journald system journal messages" for
clarity, since there are many things called journals, and this is only about
the systemd one on linux.
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
Review comment:
With `Log Payload: true`, it would be more accurate to name this "Log
flow files" or "Log messages"
```suggestion
- name: Log messages to info level
```
##########
File path: examples/publishkafka_config.yml
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 7fd166aa-0662-4c42-affa-88f6fb39807f
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling strategy: EVENT_DRIVEN
+ Properties:
+ Batch Size: '10'
+ Client Name: test-client
+ Compress Codec: none
+ Delivery Guarantee: '1'
+ Known Brokers: kafka-broker:9092
+ Message Timeout: 12 sec
+ Request Timeout: 10 sec
+ Topic Name: test
Review comment:
This one is missing auto-terminated relationships list
##########
File path: examples/publishkafka_ssl_config.yml
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8c0791c7-920c-4541-bcde-f5d559f42ed2
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ Properties:
Review comment:
What about `failure`? No connections starting from there and not
auto-terminated either. I think you meant to have it either auto-terminated or
retried.
auto-terminated (errors ignored):
```suggestion
auto-terminated relationships list:
- success
- failure
Properties:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
