[nifi-minifi-cpp] branch master updated: Added py processor to make predictions with h2o3 model
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 7f38a65 Added py processor to make predictions with h2o3 model
7f38a65 is described below
commit 7f38a656bee4316184cb3240a6f89275d250490f
Author: james94
AuthorDate: Tue May 12 19:23:46 2020 -0700
Added py processor to make predictions with h2o3 model
Replaced previous py processors that use Driverless AI models to make
predictions
with a new py processor that uses open source ALv2 h2o3 library to make
predictions.
This new py processor uses the h2o3 MOJO model to do batch scoring or
real-time
scoring. All the user needs to run this processor is the filepath of h2o3
MOJO model
and pass that filepath into the 'MOJO Model Filetpath' minifi processor
property.
The other required property 'Is First Line Header' is set to True by
default. There
are 3 optional properties: 'Input Schema', 'Use Output Header', 'Output
Schema'.
This processor makes use of the h2o's open source datatable library
licensed under
Mozilla Public License 2.0. Datatable is used to load in the flow file
content.
Then that datatable frame is converted to a numpy array. Then the numpy
array and
input column names are passed to H2OFrame(np, col_names) to create
h2oframe. Then
the h2o3 MOJO model is used to make predictions on the h2oframe and the
result is
stored into a new h2oframe. That frame is then converted to pandas frame,
which is
stored into the outgoing flow file content. Thus, the predictions are
stored in
the outgoing flow file content.
Added to comments install Java to use H2O3 algos
In the comments section of the code, I added new recommendations.
The first is to make all packages available on your machine by
running an update. The next is to install Java since it is
required to use H2O-3 algorithms.
This closes #781.
Signed-off-by: Marc Parisi
---
.../h2o/dai/msp/H2oMojoPwScoring.py| 121 ---
.../h2o/dai/psp/H2oPspScoreBatches.py | 98
.../h2o/dai/psp/H2oPspScoreRealTime.py | 108 --
.../h2o/h2o3/mojo/ExecuteH2oMojoScoring.py | 165 +
4 files changed, 165 insertions(+), 327 deletions(-)
diff --git a/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
b/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
deleted file mode 100644
index 0679018..000
--- a/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
+++ /dev/null
@@ -1,121 +0,0 @@
-#!/usr/bin/env python
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""
-Install the following with pip
-
--- after downloading the mojo scoring pipeline from Driverless AI,
- the following packages were needed for helping to make predictions
-
-pip install datatable pandas scipy
-
--- after downloading the mojo2 py runtime from Driverless AI depending on
- your OS, you need to install the appropriate package:
-
-# Install the MOJO2 Py runtime on Mac OS X
-pip install path/to/daimojo-2.2.0-cp36-cp36m-macosx_10_7_x86_64.whl
-
-# Install the MOJO2 Py runtime on Linux x86
-pip install path/to/daimojo-2.2.0-cp36-cp36m-linux_x86_64.whl
-
-# Install the MOJO2 Py runtime on Linux PPC
-pip install path/to/daimojo-2.2.0-cp36-cp36m-linux_ppc64le.whl
-"""
-import codecs
-import pandas as pd
-import datatable as dt
-from collections import Counter
-from scipy.special._ufuncs import expit
-import daimojo.model
-
-def describe(processor):
-""" describe what this processor does
-"""
-processor.setDescription("Executes H2O's MOJO Scoring Pipeline in C++
Runtime Python Wrapper \
-to do batch scoring or real time scoring for one or more predicted
label(s) on the tabular \
-test data in the incoming flow file content. If tabular data is one
row, then MOJO does \
-real ti
[nifi-minifi-cpp] branch master updated: MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 7206c62 MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766)
7206c62 is described below
commit 7206c62240647520cf35649868d5d87903a256c2
Author: James Medel
AuthorDate: Wed Apr 29 12:38:04 2020 -0700
MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766)
MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766)
---
.../h2o/dai/msp/H2oMojoPwScoring.py| 121 +
1 file changed, 121 insertions(+)
diff --git a/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
b/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
new file mode 100644
index 000..0679018
--- /dev/null
+++ b/extensions/pythonprocessors/h2o/dai/msp/H2oMojoPwScoring.py
@@ -0,0 +1,121 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Install the following with pip
+
+-- after downloading the mojo scoring pipeline from Driverless AI,
+ the following packages were needed for helping to make predictions
+
+pip install datatable pandas scipy
+
+-- after downloading the mojo2 py runtime from Driverless AI depending on
+ your OS, you need to install the appropriate package:
+
+# Install the MOJO2 Py runtime on Mac OS X
+pip install path/to/daimojo-2.2.0-cp36-cp36m-macosx_10_7_x86_64.whl
+
+# Install the MOJO2 Py runtime on Linux x86
+pip install path/to/daimojo-2.2.0-cp36-cp36m-linux_x86_64.whl
+
+# Install the MOJO2 Py runtime on Linux PPC
+pip install path/to/daimojo-2.2.0-cp36-cp36m-linux_ppc64le.whl
+"""
+import codecs
+import pandas as pd
+import datatable as dt
+from collections import Counter
+from scipy.special._ufuncs import expit
+import daimojo.model
+
+def describe(processor):
+""" describe what this processor does
+"""
+processor.setDescription("Executes H2O's MOJO Scoring Pipeline in C++
Runtime Python Wrapper \
+to do batch scoring or real time scoring for one or more predicted
label(s) on the tabular \
+test data in the incoming flow file content. If tabular data is one
row, then MOJO does \
+real time scoring. If tabular data is multiple rows, then MOJO does
batch scoring.")
+
+def onInitialize(processor):
+""" onInitialize is where you can set properties
+"""
+processor.addProperty("MOJO Pipeline Filepath", "Add the filepath to the
MOJO pipeline file. For example, \
+'path/to/mojo-pipeline/pipeline.mojo'.", "", True, False)
+
+class ContentExtract(object):
+""" ContentExtract callback class is defined for reading streams of data
through the session
+and has a process function that accepts the input stream
+"""
+def __init__(self):
+self.content = None
+
+def process(self, input_stream):
+""" Use codecs getReader to read that data
+"""
+self.content = codecs.getreader('utf-8')(input_stream).read()
+return len(self.content)
+
+class ContentWrite(object):
+""" ContentWrite callback class is defined for writing streams of data
through the session
+"""
+def __init__(self, data):
+self.content = data
+
+def process(self, output_stream):
+""" Use codecs getWriter to write data encoded to the stream
+"""
+codecs.getwriter('utf-8')(output_stream).write(self.content)
+return len(self.content)
+
+def onTrigger(context, session):
+""" onTrigger is executed and passed processor context and session
+"""
+flow_file = session.get()
+# lambda compares two lists: does header equal expected header
+compare = lambda header, exp_header: Counter(header) == Counter(exp_header)
+if flow
[nifi-minifi-cpp] branch master updated (68e2482 -> 6e5f965)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 68e2482 MINIFICPP-1208 - Enable rocksdb tools build add 6e5f965 MINIFI-1199: Integrate H2O Driverless AI PSP in MiNiFi (#763) No new revisions were added by this update. Summary of changes: extensions/pythonprocessors/h2o/ConvertDsToCsv.py | 86 .../h2o/dai/psp/H2oPspScoreBatches.py | 98 +++ .../h2o/dai/psp/H2oPspScoreRealTime.py | 108 + 3 files changed, 292 insertions(+) create mode 100644 extensions/pythonprocessors/h2o/ConvertDsToCsv.py create mode 100644 extensions/pythonprocessors/h2o/dai/psp/H2oPspScoreBatches.py create mode 100644 extensions/pythonprocessors/h2o/dai/psp/H2oPspScoreRealTime.py
[nifi] branch master updated: NIFI-6742 Use JUnit TemporaryFolder when creating test databases
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new decb5d0 NIFI-6742 Use JUnit TemporaryFolder when creating test
databases
decb5d0 is described below
commit decb5d062ea20714b666906e1414e0c9a8e4064c
Author: Adam Taft
AuthorDate: Wed Oct 23 21:03:38 2019 -0600
NIFI-6742 Use JUnit TemporaryFolder when creating test databases
- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines
Signed-off-by: Marc Parisi
This closes #4137.
---
.../java/org/apache/nifi/dbcp/DBCPServiceTest.java | 88 ++
1 file changed, 25 insertions(+), 63 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
index 7522125..fdad692 100644
---
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
+++
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
@@ -23,13 +23,16 @@ import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
import org.h2.tools.Server;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
import java.io.File;
+import java.io.IOException;
import java.io.PrintWriter;
import java.net.InetAddress;
import java.net.MalformedURLException;
@@ -51,13 +54,21 @@ import static org.junit.Assert.assertTrue;
public class DBCPServiceTest {
-final static String DB_LOCATION = "target/db";
+@Rule
+public TemporaryFolder tempFolder = new TemporaryFolder(new
File("target"));
+
+private String dbLocation;
@BeforeClass
public static void setup() {
System.setProperty("derby.stream.error.file", "target/derby.log");
}
+@Before
+public void before() throws IOException {
+this.dbLocation = new File(tempFolder.getRoot(), "db").getPath();
+}
+
/**
* Missing property values.
*/
@@ -79,12 +90,8 @@ public class DBCPServiceTest {
final DBCPConnectionPool service = new DBCPConnectionPool();
runner.addControllerService("test-good1", service);
-// remove previous test database, if any
-final File dbLocation = new File(DB_LOCATION);
-dbLocation.delete();
-
// set embedded Derby database connection url
-runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + dbLocation + ";create=true");
runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD, "testerp");
runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
@@ -103,12 +110,8 @@ public class DBCPServiceTest {
final DBCPConnectionPool service = new DBCPConnectionPool();
runner.addControllerService("test-good1", service);
-// remove previous test database, if any
-final File dbLocation = new File(DB_LOCATION);
-dbLocation.delete();
-
// set embedded Derby database connection url
-runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + dbLocation + ";create=true");
runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD, "testerp");
runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
@@ -129,12 +132,8 @@ public class DBCPServiceTest {
final DBCPConnectionPool service = new DBCPConnectionPool();
runner.addControllerService("test-good1", service);
-// remove previous test database, if any
-final File dbLocation = new File(DB_LOCATION);
-dbLocation.
[nifi-minifi-cpp] branch master updated: change CapturePacket header guard to be unique
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new b051d7e change CapturePacket header guard to be unique b051d7e is described below commit b051d7ec8b92518f7f0887ff380cfbe6b42cd76c Author: Otto Fowler AuthorDate: Wed Dec 4 18:17:00 2019 -0500 change CapturePacket header guard to be unique per review, comment end of header guard This closes #689. Signed-off-by: Marc Parisi --- extensions/pcap/CapturePacket.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/extensions/pcap/CapturePacket.h b/extensions/pcap/CapturePacket.h index 6dfd276..26f499a 100644 --- a/extensions/pcap/CapturePacket.h +++ b/extensions/pcap/CapturePacket.h @@ -16,8 +16,8 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -#ifndef __INVOKE_HTTP_H__ -#define __INVOKE_HTTP_H__ +#ifndef EXTENSIONS_CAPTUREPACKET_H +#define EXTENSIONS_CAPTUREPACKET_H #include #include @@ -175,4 +175,4 @@ REGISTER_RESOURCE(CapturePacket, "CapturePacket captures and writes one or more } /* namespace apache */ } /* namespace org */ -#endif +#endif /* EXTENSIONS_CAPTUREPACKET_H */
[nifi-minifi-cpp] branch master updated: WIP: Minificpp 1025 -- add metadata (#648)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new a4cd78b WIP: Minificpp 1025 -- add metadata (#648)
a4cd78b is described below
commit a4cd78ba35e4c8ea10aa3c9046bc57646043f0c0
Author: Marc
AuthorDate: Thu Sep 26 14:49:37 2019 -0400
WIP: Minificpp 1025 -- add metadata (#648)
* MINIFICPP-1025: Add message string functionality
* minor update
* updates
* MINIFICPP-1025: Update CWEL
* updates
* update message type
* update message type
* MINIFICPP-1025: add multiple SID resolution test
---
.../windows-event-log/ConsumeWindowsEventLog.cpp | 122 ++---
.../windows-event-log/ConsumeWindowsEventLog.h | 18 +-
.../tests/MetadataWalkerTests.cpp | 30
.../tests/resources/multiplesids.xml | 24 +++
.../windows-event-log/wel/MetadataWalker.cpp | 103 ++-
extensions/windows-event-log/wel/MetadataWalker.h | 36 ++--
.../windows-event-log/wel/WindowsEventLog.cpp | 200 +
extensions/windows-event-log/wel/WindowsEventLog.h | 193
libminifi/include/utils/Deleters.h | 40 +
9 files changed, 728 insertions(+), 38 deletions(-)
diff --git a/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
index b2d6114..6e7b2e0 100644
--- a/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
+++ b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
@@ -36,7 +36,9 @@
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
+
#pragma comment(lib, "wevtapi.lib")
+#pragma comment(lib, "ole32.lib")
namespace org {
namespace apache {
@@ -44,6 +46,8 @@ namespace nifi {
namespace minifi {
namespace processors {
+
+
static std::string to_string(const wchar_t* pChar) {
return std::wstring_convert>().to_bytes(pChar);
}
@@ -108,6 +112,21 @@ core::Property ConsumeWindowsEventLog::ResolveAsAttributes(
withDescription("If true, any metadata that is resolved ( such as IDs
or keyword metadata ) will be placed into attributes, otherwise it will be
replaced in the XML or text output")->
build());
+
+core::Property ConsumeWindowsEventLog::EventHeaderDelimiter(
+ core::PropertyBuilder::createProperty("Event Header Delimiter")->
+ isRequired(false)->
+ withDescription("If set, the chosen delimiter will be used in the Event
output header. Otherwise, a colon followed by spaces will be used.")->
+ build());
+
+
+core::Property ConsumeWindowsEventLog::EventHeader(
+ core::PropertyBuilder::createProperty("Event Header")->
+ isRequired(false)->
+ withDefaultValue("LOG_NAME=Log Name, SOURCE = Source, TIME_CREATED =
Date,EVENT_RECORDID=Record ID,EVENTID = Event ID,TASK_CATEGORY = Task
Category,LEVEL = Level,KEYWORDS = Keywords,USER = User,COMPUTER = Computer,
EVENT_TYPE = EventType")->
+ withDescription("Comma seperated list of key/value pairs with the
following keys LOG_NAME, SOURCE,
TIME_CREATED,EVENT_RECORDID,EVENTID,TASK_CATEGORY,LEVEL,KEYWORDS,USER,COMPUTER,
and EVENT_TYPE. Eliminating fields will remove them from the header.")->
+ build());
+
core::Relationship ConsumeWindowsEventLog::Success("success", "Relationship
for successfully consumed events.");
ConsumeWindowsEventLog::ConsumeWindowsEventLog(const std::string& name,
utils::Identifier uuid)
@@ -127,16 +146,53 @@ ConsumeWindowsEventLog::~ConsumeWindowsEventLog() {
void ConsumeWindowsEventLog::initialize() {
//! Set the supported properties
- setSupportedProperties({Channel, Query, MaxBufferSize,
InactiveDurationToReconnect, IdentifierMatcher, IdentifierFunction,
ResolveAsAttributes });
+ setSupportedProperties({Channel, Query, MaxBufferSize,
InactiveDurationToReconnect, IdentifierMatcher, IdentifierFunction,
ResolveAsAttributes, EventHeaderDelimiter, EventHeader });
//! Set the supported relationships
setSupportedRelationships({Success});
}
+bool ConsumeWindowsEventLog::insertHeaderName(wel::METADATA_NAMES ,
const std::string , const std::string & value) {
+
+ wel::METADATA name =
wel::WindowsEventLogMetadata::getMetadataFromString(key);
+
+ if (name != wel::METADATA::UNKNOWN) {
+ header.emplace_back(std::make_pair(name, value));
+ return true;
+ }
+ else {
+ return false;
+ }
+}
+
void ConsumeWindowsEventLog::onSchedule(const
std::shared_ptr , const
std::shared_ptr ) {
context->getProperty(IdentifierMatcher.get
[nifi-minifi-cpp] branch master updated: MINIFICPP-1028 - http url parsing without port fails
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new d7d2a58 MINIFICPP-1028 - http url parsing without port fails
d7d2a58 is described below
commit d7d2a582ce627b9f2b2816210c1a7af46c28ebe7
Author: Arpad Boda
AuthorDate: Tue Sep 10 12:22:24 2019 +0200
MINIFICPP-1028 - http url parsing without port fails
This closes #645.
Signed-off-by: Marc Parisi
---
libminifi/include/utils/HTTPClient.h | 4 +-
libminifi/src/core/FlowConfiguration.cpp | 2 +-
libminifi/src/utils/HTTPClient.cpp | 68 +---
libminifi/test/unit/HTTPUtilTests.cpp| 67 +++
4 files changed, 97 insertions(+), 44 deletions(-)
diff --git a/libminifi/include/utils/HTTPClient.h
b/libminifi/include/utils/HTTPClient.h
index d477f73..8cdc3ef 100644
--- a/libminifi/include/utils/HTTPClient.h
+++ b/libminifi/include/utils/HTTPClient.h
@@ -356,8 +356,8 @@ public:
extern std::string get_token(utils::BaseHTTPClient *client, std::string
username, std::string password);
-extern void parse_url(std::string *url, std::string *host, int *port,
std::string *protocol);
-extern void parse_url(std::string *url, std::string *host, int *port,
std::string *protocol, std::string *path, std::string *query);
+extern void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol);
+extern void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol, std::string *path, std::string *query);
} /* namespace utils */
} /* namespace minifi */
} /* namespace nifi */
diff --git a/libminifi/src/core/FlowConfiguration.cpp
b/libminifi/src/core/FlowConfiguration.cpp
index 1842c21..dafa6c5 100644
--- a/libminifi/src/core/FlowConfiguration.cpp
+++ b/libminifi/src/core/FlowConfiguration.cpp
@@ -71,7 +71,7 @@ std::unique_ptr
FlowConfiguration::updateFromPayload(const s
auto payload = getRootFromPayload(yamlConfigPayload);
if (!source.empty() && payload != nullptr) {
std::string host, protocol, path, query, url = source;
-int port;
+int port = -1;
utils::parse_url(, , , , , );
std::string flow_id, bucket_id;
diff --git a/libminifi/src/utils/HTTPClient.cpp
b/libminifi/src/utils/HTTPClient.cpp
index 35d8f6b..3328aa5 100644
--- a/libminifi/src/utils/HTTPClient.cpp
+++ b/libminifi/src/utils/HTTPClient.cpp
@@ -48,9 +48,9 @@ std::string get_token(utils::BaseHTTPClient *client,
std::string username, std::
return token;
}
-void parse_url(std::string *url, std::string *host, int *port, std::string
*protocol) {
- std::string http("http://;);
- std::string https("https://;);
+void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol) {
+ static std::string http("http://;);
+ static std::string https("https://;);
if (url->compare(0, http.size(), http) == 0)
*protocol = http;
@@ -76,53 +76,39 @@ void parse_url(std::string *url, std::string *host, int
*port, std::string *prot
if (portStr.size() > 0) {
*port = std::stoi(portStr);
}
+} else {
+ // In case the host contains no port, the first part is needed only
+ // For eg.: nifi.io/nifi
+ size_t ppos = host->find_first_of("/");
+ if (ppos != std::string::npos) {
+*host = host->substr(0, ppos);
+ }
}
}
}
-void parse_url(std::string *url, std::string *host, int *port, std::string
*protocol, std::string *path, std::string *query) {
- std::string http("http://;);
- std::string https("https://;);
+void parse_url(const std::string *url, std::string *host, int *port,
std::string *protocol, std::string *path, std::string *query) {
+ int temp_port = -1;
- if (url->compare(0, http.size(), http) == 0)
-*protocol = http;
-
- if (url->compare(0, https.size(), https) == 0)
-*protocol = https;
+ parse_url(url, host, _port, protocol);
- if (!protocol->empty()) {
-size_t pos = url->find_first_of(":", protocol->size());
+ if (host->empty() || protocol->empty()) {
+return;
+ }
-if (pos == std::string::npos) {
- pos = url->size();
-}
-size_t ppos = url->find_first_of("/", protocol->size());
-if (pos == url->size() && ppos < url->size()) {
- *host = url->substr(protocol->size(), ppos - protocol->size());
-} else {
- if (ppos < url->size())
-*host = url->substr(protocol->size(), pos - protocol->size());
- else
-return;
-}
-if (pos < url->size() && (*url)[pos] == ':') {
- if (ppos == std::string::npos) {
-ppos = url->size();
- }
- std::string portStr(url->
[nifi-minifi-cpp] branch master updated: MINIFICPP-14: Prune class names. Add option to config (#643)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new ad917bb MINIFICPP-14: Prune class names. Add option to config (#643)
ad917bb is described below
commit ad917bb79660e6e029fe52581180a2ad1b93b5de
Author: Marc
AuthorDate: Mon Sep 9 10:28:25 2019 -0400
MINIFICPP-14: Prune class names. Add option to config (#643)
* MINIFICPP-14: Prune class names. Add option to config
* MINIFICPP-14: update API
Signed-off-by: Marc Parisi
---
conf/minifi-log.properties | 2 +
.../include/core/logging/LoggerConfiguration.h | 33 +---
libminifi/include/utils/ClassUtils.h | 44 +++
libminifi/include/utils/StringUtils.h | 3 +-
libminifi/src/core/logging/LoggerConfiguration.cpp | 15
libminifi/src/utils/ClassUtils.cpp | 58 ++
libminifi/test/TestBase.h | 92 ++
libminifi/test/unit/ClassUtilsTests.cpp| 61 ++
libminifi/test/unit/LoggerTests.cpp| 36 -
9 files changed, 316 insertions(+), 28 deletions(-)
diff --git a/conf/minifi-log.properties b/conf/minifi-log.properties
index dc25406..55d07c3 100644
--- a/conf/minifi-log.properties
+++ b/conf/minifi-log.properties
@@ -16,6 +16,8 @@
#More verbose pattern by default
#Format details at https://github.com/gabime/spdlog/wiki/3.-Custom-formatting
spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] %v
+# uncomment to prune package names
+#spdlog.shorten_names=true
#Old format
#spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [minifi log] [%l] %v
diff --git a/libminifi/include/core/logging/LoggerConfiguration.h
b/libminifi/include/core/logging/LoggerConfiguration.h
index 7f9fdd2..ea714e9 100644
--- a/libminifi/include/core/logging/LoggerConfiguration.h
+++ b/libminifi/include/core/logging/LoggerConfiguration.h
@@ -57,7 +57,9 @@ struct LoggerNamespace {
class LoggerProperties : public Properties {
public:
- LoggerProperties() : Properties("Logger properties") {}
+ LoggerProperties()
+ : Properties("Logger properties") {
+ }
/**
* Gets all keys that start with the given prefix and do not have a "."
after the prefix and "." separator.
*
@@ -92,13 +94,21 @@ class LoggerConfiguration {
return logger_configuration;
}
- void disableLogging(){
+ static std::unique_ptr newInstance() {
+return std::unique_ptr(new LoggerConfiguration());
+ }
+
+ void disableLogging() {
controller_->setEnabled(false);
}
- void enableLogging(){
- controller_->setEnabled(true);
-}
+ void enableLogging() {
+controller_->setEnabled(true);
+ }
+
+ bool shortenClassNames() const {
+return shorten_names_;
+ }
/**
* (Re)initializes the logging configuation with the given logger properties.
*/
@@ -118,10 +128,11 @@ class LoggerConfiguration {
class LoggerImpl : public Logger {
public:
-LoggerImpl(std::string name, std::shared_ptr controller,
std::shared_ptr delegate)
-: Logger(delegate,controller),
+explicit LoggerImpl(const std::string , const
std::shared_ptr , const
std::shared_ptr )
+: Logger(delegate, controller),
name(name) {
}
+
void set_delegate(std::shared_ptr delegate) {
std::lock_guard lock(mutex_);
delegate_ = delegate;
@@ -137,6 +148,8 @@ class LoggerConfiguration {
std::mutex mutex;
std::shared_ptr logger_ = nullptr;
std::shared_ptr controller_;
+ bool shorten_names_;
+
};
template
@@ -151,9 +164,9 @@ class LoggerFactory {
}
static std::shared_ptr getAliasedLogger(const std::string ) {
- std::shared_ptr logger =
LoggerConfiguration::getConfiguration().getLogger(alias);
- return logger;
-}
+std::shared_ptr logger =
LoggerConfiguration::getConfiguration().getLogger(alias);
+return logger;
+ }
};
} /* namespace logging */
diff --git a/libminifi/include/utils/ClassUtils.h
b/libminifi/include/utils/ClassUtils.h
new file mode 100644
index 000..3d82c43
--- /dev/null
+++ b/libminifi/include/utils/ClassUtils.h
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distribute
[nifi-minifi-cpp] branch master updated (1f62304 -> dfbc8ed)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 1f62304 MINFICPP-1005 - Disable
[nifi-minifi-cpp] branch master updated (dbc1beb -> 1f62304)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from dbc1beb MINIFICPP-963 - Zero byte (no content) flow files cannot be received … (#615) add 1f62304 MINFICPP-1005 - Disable
[nifi-minifi-cpp] branch master updated (5b5e0c6 -> dbc1beb)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from 5b5e0c6 MINIFICPP-1024 - Fix CentOS and Fedora docker builds (#641)
add dbc1beb MINIFICPP-963 - Zero byte (no content) flow files cannot be
received … (#615)
No new revisions were added by this update.
Summary of changes:
docker/test/integration/minifi/__init__.py | 18 +-
docker/test/integration/minifi/test/__init__.py| 53 +++---
...{test_zero_file.py => test_filter_zero_file.py} | 7 +--
docker/test/integration/test_zero_file.py | 5 +-
extensions/http-curl/HTTPCurlLoader.h | 2 +-
extensions/http-curl/sitetosite/HTTPProtocol.cpp | 58 +---
libminifi/include/Connection.h | 10
libminifi/src/Connection.cpp | 8 +++
libminifi/src/core/yaml/YamlConfiguration.cpp | 8 +++
libminifi/src/sitetosite/RawSocketProtocol.cpp | 6 +-
libminifi/src/sitetosite/SiteToSiteClient.cpp | 64 +-
11 files changed, 158 insertions(+), 81 deletions(-)
copy docker/test/integration/{test_zero_file.py => test_filter_zero_file.py}
(91%)
[nifi-minifi-cpp] branch master updated (b8359e6 -> 5b5e0c6)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from b8359e6 MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming (#630) add 5b5e0c6 MINIFICPP-1024 - Fix CentOS and Fedora docker builds (#641) No new revisions were added by this update. Summary of changes: docker/ContainerBuild.sh | 4 docker/centos/Dockerfile | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-)
[nifi-minifi-cpp] branch master updated: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming (#630)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new b8359e6 MINIFICPP-1006: minor updates for jni building using win
build & correct artifact naming (#630)
b8359e6 is described below
commit b8359e62521143215f4f82a513122bc06c6c70c6
Author: Marc
AuthorDate: Wed Sep 4 07:28:32 2019 -0400
MINIFICPP-1006: minor updates for jni building using win build & correct
artifact naming (#630)
* MINIFICPP-1006: minor updates for jni building using win build
MINIFICPP-1006: Add service account settings to UI
MINIFICPP-1006: Rename artifact
MINIFICPP-1006: Rename artifacts in docker builds
MINIFICPP-1006: UI updates
MINIFICPP-1006: Fix typo and linter errors
MINIFICPP-1006: Remove win64, relying on extra sources
MINIFICPP-1006: Fix build script
MINIFICPP-1006: Remove skip tests
MINIFICPP-1006: fix compression reference
MINIFICPP-1016: Add MetadataWalker and change how metadata is evaluated.
MINIFICPP-1016: Address pr comments and add unit test
MINIFICPP-1006: Fix typo
* MINIFICPP-1006: Fix collision with INSTALLDIR
This closes #630.
Approved by @apiri
Signed-off-by: Marc Parisi
---
CMakeLists.txt | 32 ++-
CMakeSettings.json | 10 +-
LICENSE| 27 ++
cmake/Compression.cmake| 19 +-
conf/minifi.properties | 2 +-
docker/ContainerBuild.sh | 2 +-
docker/Dockerfile | 4 +-
extensions/jni/JVMCreator.h| 3 +-
extensions/windows-event-log/CMakeLists.txt| 21 +-
.../windows-event-log/ConsumeWindowsEventLog.cpp | 185 +++--
.../windows-event-log/ConsumeWindowsEventLog.h | 18 +-
extensions/windows-event-log/tests/CMakeLists.txt | 15 +-
.../tests/MetadataWalkerTests.cpp | 96 +++
.../tests/resources/invalidxml.xml | 17 ++
.../tests/resources/nobodysid.xml | 23 ++
.../windows-event-log/tests/resources/nodata.xml | 19 ++
.../tests/resources/unknownsid.xml | 23 ++
.../windows-event-log/tests/resources/withsids.xml | 23 ++
.../windows-event-log/wel/MetadataWalker.cpp | 135 ++
extensions/windows-event-log/wel/MetadataWalker.h | 94 +++
extensions/windows-event-log/wel/XMLString.h | 60 +
libminifi/include/core/ContentRepository.h | 2 +-
libminifi/src/c2/C2Agent.cpp | 10 +-
libminifi/src/utils/StringUtils.cpp| 14 +-
msi/WixWin.wsi | 291 +
msi/x64.wsi| 24 ++
msi/x86.wsi| 25 ++
win_build_vs.bat | 17 +-
28 files changed, 906 insertions(+), 305 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2023622..0906199 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -303,6 +303,7 @@ endif()
"-DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/thirdparty/curl-install"
-DBUILD_CURL_EXE=OFF
-DBUILD_TESTING=OFF
+-DCMAKE_USE_OPENSSL=ON
-DBUILD_SHARED_LIBS=OFF
-DHTTP_ONLY=ON
-DCMAKE_USE_OPENSSL=ON
@@ -643,14 +644,22 @@ file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/conf/" DESTINATION
"${CMAKE_CURRENT_BINAR
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/README.md" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/NOTICE" DESTINATION
"${CMAKE_CURRENT_BINARY_DIR}")
-set(CPACK_WIX_TEMPLATE "${CMAKE_CURRENT_SOURCE_DIR}/msi/WixWin.wsi")
+
+if(CMAKE_SIZEOF_VOID_P EQUAL 8)
+set(CPACK_WIX_EXTRA_SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/msi/x64.wsi")
+elseif(CMAKE_SIZEOF_VOID_P EQUAL 4)
+set(CPACK_WIX_EXTRA_SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/msi/x86.wsi")
+endif()
+set(CPACK_WIX_TEMPLATE "${CMAKE_CURRENT_SOURCE_DIR}/msi/WixWin.wsi")
+
else()
set(CPACK_SOURCE_GENERATOR "TGZ")
endif(WIN32)
set(CPACK_SOURCE_PACKAGE_FILE_NAME "${ASSEMBLY_BASE_NAME}-source")
set(CPACK_SOURCE_IGNORE_FILES
"/docs/generated/;${CMAKE_SOURCE_DIR}/build/;~$;${CPACK_SOURCE_IGNORE_FILES};${CMAKE_SOURCE_DIR}/.git/;${CMAKE_SOURCE_DIR}/.idea/;${CMAKE_SOURCE_DIR}/cmake-build-debug/;${CMAKE
[nifi-minifi-cpp] branch master updated: MINIFICPP-814 - Fixed ListenHTTP and HTTPClient bugs, created tests for ListenHTTP
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 97a05c1 MINIFICPP-814 - Fixed ListenHTTP and HTTPClient bugs, created tests for ListenHTTP 97a05c1 is described below commit 97a05c1bdb6bfd40f5f33da01cf0893060350ef4 Author: Daniel Bakai AuthorDate: Tue Aug 27 09:50:15 2019 +0200 MINIFICPP-814 - Fixed ListenHTTP and HTTPClient bugs, created tests for ListenHTTP This fixes the following ListenHTTP issues: - HTTPS did not work - SSL Minimum Version enforcement caused hanging client and leaked fds in the server - HEAD requests were not served properly - random port did not work with HTTPS This fixes the following HTTPClient issues: - PUT did not work - HEAD did not work - POST using setPostFields was a potential use-after-free - cURL SSL backend was erroneously deduced - header parsing was faulty This adds the following features: - OpenSSL and cURL are now always built, system versions are not supported - logging for CivetWeb - logging for cURL - tests for ListenHTTP This closes #610. MINIFICPP-814 - Windows fixes Signed-off-by: Marc Parisi --- CMakeLists.txt | 74 +-- CMakeSettings.json | 8 - README.md | 4 +- bootstrap.sh | 15 - centos.sh | 8 +- cmake/LibSSH2.cmake| 16 +- darwin.sh | 12 +- debian.sh | 6 +- extensions/civetweb/processors/ListenHTTP.cpp | 31 +- extensions/civetweb/processors/ListenHTTP.h| 42 +- extensions/civetweb/tests/CMakeLists.txt | 56 +++ extensions/civetweb/tests/ListenHTTPTests.cpp | 538 + .../civetweb/tests/resources/badCA_goodClient.p12 | Bin 0 -> 2413 bytes .../civetweb/tests/resources/badCA_goodClient.pem | 46 ++ extensions/civetweb/tests/resources/generate.sh| 53 ++ extensions/civetweb/tests/resources/goodCA.crt | 20 + .../civetweb/tests/resources/goodCA_badClient.p12 | Bin 0 -> 2413 bytes .../civetweb/tests/resources/goodCA_badClient.pem | 46 ++ .../civetweb/tests/resources/goodCA_goodClient.p12 | Bin 0 -> 2413 bytes .../civetweb/tests/resources/goodCA_goodClient.pem | 46 ++ extensions/civetweb/tests/resources/server.p12 | Bin 0 -> 2405 bytes extensions/civetweb/tests/resources/server.pem | 46 ++ extensions/http-curl/CMakeLists.txt| 5 - extensions/http-curl/client/HTTPClient.cpp | 140 +++--- extensions/http-curl/client/HTTPClient.h | 26 +- extensions/http-curl/tests/CMakeLists.txt | 7 +- extensions/http-curl/tests/unit/CivetwebTests.cpp | 115 - .../http-curl/tests/unit/HTTPClientTests.cpp | 96 extensions/librdkafka/CMakeLists.txt | 4 +- extensions/sftp/CMakeLists.txt | 5 - .../standard-processors/tests/CMakeLists.txt | 5 + fedora.sh | 7 +- libminifi/include/controllers/SSLContextService.h | 52 +- libminifi/include/utils/HTTPClient.h | 91 +++- libminifi/src/controllers/SSLContextService.cpp| 109 - libminifi/test/resources/cn.p12| Bin 0 -> 2437 bytes libminifi/test/unit/SocketTests.cpp| 6 +- main/CMakeLists.txt| 14 +- rheldistro.sh | 8 +- suse.sh| 7 +- thirdparty/civetweb-1.10/src/civetweb.c| 9 +- 41 files changed, 1365 insertions(+), 408 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index cee2f23..2023622 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -34,17 +34,18 @@ option(PORTABLE "Instructs the compiler to remove architecture specific optimiza option(USE_SHARED_LIBS "Builds using shared libraries" ON) option(ENABLE_PYTHON "Instructs the build system to enable building shared objects for the python lib" OFF) cmake_dependent_option(STATIC_BUILD "Attempts to statically link as many dependencies as possible." ON "NOT ENABLE_PYTHON; NOT USE_SHARED_LIBS" OFF) -cmake_dependent_option(USE_SYSTEM_OPENSSL "Instructs the build system to search for and use an SSL library available in the host system" ON "NOT STATIC_BUILD" OFF) option(LIBC_STATIC "Instructs the build system to statically link libstdc++ and glibc into minifiexe. Experiemental" OFF) option(OPENSSL
[nifi-minifi-cpp] branch master updated: MINIFICPP-986 - GetFile processor incorrectly identifies hidden files on Windows
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 0254a3c MINIFICPP-986 - GetFile processor incorrectly identifies
hidden files on Windows
new ae6bfe7 Merge pull request #616 from arpadboda/MINIFICPP-986
0254a3c is described below
commit 0254a3c6cdea6165d3fa5b7f471903ac3d1c404b
Author: Arpad Boda
AuthorDate: Wed Jul 24 16:04:04 2019 +0200
MINIFICPP-986 - GetFile processor incorrectly identifies hidden files on
Windows
---
.../standard-processors/processors/GetFile.cpp | 2 +-
.../standard-processors/tests/unit/GetFileTests.cpp | 20 +++-
libminifi/include/utils/file/FileUtils.h | 9 +
3 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/extensions/standard-processors/processors/GetFile.cpp
b/extensions/standard-processors/processors/GetFile.cpp
index 082b7f7..66fbc4a 100644
--- a/extensions/standard-processors/processors/GetFile.cpp
+++ b/extensions/standard-processors/processors/GetFile.cpp
@@ -240,7 +240,7 @@ bool GetFile::acceptFile(std::string fullName, std::string
name, const GetFileRe
if (request.maxAge > 0 && fileAge > request.maxAge)
return false;
-if (request.ignoreHiddenFile && name.c_str()[0] == '.')
+if (request.ignoreHiddenFile &&
utils::file::FileUtils::is_hidden(fullName))
return false;
if (access(fullName.c_str(), R_OK) != 0)
diff --git a/extensions/standard-processors/tests/unit/GetFileTests.cpp
b/extensions/standard-processors/tests/unit/GetFileTests.cpp
index 0464df9..a857e73 100644
--- a/extensions/standard-processors/tests/unit/GetFileTests.cpp
+++ b/extensions/standard-processors/tests/unit/GetFileTests.cpp
@@ -26,6 +26,10 @@
#include "LogAttribute.h"
#include "GetFile.h"
+#ifdef WIN32
+#include
+#endif
+
/**
* This is an invalidly named test as we can't guarantee order, nor shall we.
*/
@@ -47,12 +51,19 @@ TEST_CASE("GetFile: MaxSize", "[getFileFifo]") { // NOLINT
// Define test input file
std::string in_file = temp_path + utils::file::FileUtils::get_separator() +
"testfifo";
+ // Define test input file
+ std::string hidden_in_file(in_dir);
+ hidden_in_file.append("/.testfifo"); // hidden
+
// Build MiNiFi processing graph
+
auto get_file = plan->addProcessor("GetFile", "Get");
plan->setProperty(get_file, processors::GetFile::Directory.getName(),
temp_path);
plan->setProperty(get_file, processors::GetFile::KeepSourceFile.getName(),
"true");
plan->setProperty(get_file, processors::GetFile::MaxSize.getName(), "50 B");
- plan->addProcessor("LogAttribute", "Log", core::Relationship("success",
"description"), true);
+ plan->setProperty(get_file, processors::GetFile::IgnoreHiddenFile.getName(),
"true");
+ auto log_attr = plan->addProcessor("LogAttribute", "Log",
core::Relationship("success", "description"), true);
+ plan->setProperty(log_attr,
processors::LogAttribute::FlowFilesToLog.getName(), "0");
// Write test input.
std::ofstream in_file_stream(in_file);
@@ -65,9 +76,16 @@ TEST_CASE("GetFile: MaxSize", "[getFileFifo]") { // NOLINT
in_file_stream << "The quick brown fox jumps over the lazy dog who is 2
legit to quit" << std::endl;
in_file_stream.close();
+ std::ofstream hidden_in_file_stream(hidden_in_file);
+ hidden_in_file_stream << "But noone has ever seen it" << std::endl;
+ hidden_in_file_stream.close();
+#ifdef WIN32
+ REQUIRE(SetFileAttributesA(hidden_in_file.c_str(), FILE_ATTRIBUTE_HIDDEN));
+#endif
plan->runNextProcessor(); // Get
plan->runNextProcessor(); // Log
+ REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
// The hidden and the too big files should be ignored
// Check log output on windows std::endl; will produce \r\n can write
manually but might as well just
// account for the size difference here
REQUIRE(LogTestController::getInstance().contains("key:flow.id"));
diff --git a/libminifi/include/utils/file/FileUtils.h
b/libminifi/include/utils/file/FileUtils.h
index 2d7aaa8..db9182e 100644
--- a/libminifi/include/utils/file/FileUtils.h
+++ b/libminifi/include/utils/file/FileUtils.h
@@ -573,6 +573,15 @@ class FileUtils {
return child_path;
}
+ static bool is_hidden(const std::string& path){
+#ifdef WIN32
+DWORD attributes = GetFileAttributesA(path.c_str());
+return ((attributes != INVALID_FILE_ATTRIBUTES) && ((attributes &
FILE_ATTRIBUTE_HIDDEN) != 0));
+#else
+return std::get<1>(split_path(path)).rfind(".", 0) == 0;
+#endif
+ }
+
/*
* Returns the absolute path of the current executable
*/
[nifi-minifi-cpp] branch master updated: MINIFICPP-1017 - Improve logging of docker tests
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new c607f35 MINIFICPP-1017 - Improve logging of docker tests
new 5c6fb1c Merge pull request #634 from arpadboda/MINIFICPP-1017
c607f35 is described below
commit c607f35ca23c6dbad03861a0bcc31184be440a0c
Author: Arpad Boda
AuthorDate: Mon Aug 26 17:13:29 2019 +0200
MINIFICPP-1017 - Improve logging of docker tests
---
docker/DockerBuild.sh | 3 +++
docker/test/integration/minifi/test/__init__.py | 29 +
2 files changed, 18 insertions(+), 14 deletions(-)
diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh
index 20c5070..6217a82 100755
--- a/docker/DockerBuild.sh
+++ b/docker/DockerBuild.sh
@@ -34,6 +34,9 @@ echo "MiNiFi Package: $MINIFI_SOURCE_CODE"
mkdir -p $CMAKE_SOURCE_DIR/docker/minificppsource
rsync -avr \
--exclude '/*build*' \
+ --exclude '/*_repository*' \
+ --exclude '/logs' \
+ --exclude '/cmake' \
--exclude '/docker' \
--exclude '.git' \
--exclude '/extensions/expression-language/Parser.cpp' \
diff --git a/docker/test/integration/minifi/test/__init__.py
b/docker/test/integration/minifi/test/__init__.py
index d25e538..c93f168 100644
--- a/docker/test/integration/minifi/test/__init__.py
+++ b/docker/test/integration/minifi/test/__init__.py
@@ -118,27 +118,28 @@ class DockerTestCluster(SingleNodeDockerCluster):
for container in self.containers:
container = self.client.containers.get(container.id)
-logging.info('Container logs for container \'%s\':\n%s',
container.name, container.logs())
+logging.info('Container logs for container \'%s\':\n%s',
container.name, container.logs().decode("utf-8"))
if b'Segmentation fault' in container.logs():
+logging.warn('Container segfaulted: %s', container.name)
self.segfault=True
if container.status == 'running':
-minifi_app_logs = container.exec_run('/bin/sh -c \'test -f ' +
self.minifi_root + '/logs/minifi-app.log '
-
'&& cat ' +
- self.minifi_root +
'/logs/minifi-app.log\'')
-if len(minifi_app_logs) > 0:
-logging.info('MiNiFi app logs for container \'%s\':\n%s',
container.name, minifi_app_logs)
-
-nifi_app_logs = container.exec_run('/bin/sh -c \'test -f ' +
self.nifi_root + '/logs/nifi-app.log '
-
'&& cat ' +
- self.nifi_root +
'/logs/nifi-app.log\'')
-if len(nifi_app_logs) > 0:
-logging.info('NiFi app logs for container \'%s\':\n%s',
container.name, nifi_app_logs)
+apps = [("MiNiFi", self.minifi_root + '/logs/minifi-app.log'),
("NiFi", self.nifi_root + '/logs/nifi-app.log')]
+for app in apps:
+app_log_status, app_log = container.exec_run('/bin/sh -c
\'cat ' + app[1] + '\'')
+if app_log_status == 0:
+logging.info('%s app logs for container \'%s\':\n',
app[0], container.name)
+for line in app_log.decode("utf-8").splitlines():
+logging.info(line)
+break
+else:
+logging.warning("The container is running, but none of %s
logs were found", " or ".join([x[0] for x in apps]))
+
else:
logging.info(container.status)
logging.info('Could not cat app logs for container \'%s\'
because it is not running',
container.name)
-stats = container.stats(decode=True, stream=True)
-logging.info('Container stats:\n%s', repr(stats))
+stats = container.stats(stream=False)
+logging.info('Container stats:\n%s', stats)
def check_output(self, timeout=5):
"""
[nifi-minifi-cpp] branch master updated: MINIFICPP-1014 - Fix SFTP extension build on MacOS
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new c916d15 MINIFICPP-1014 - Fix SFTP extension build on MacOS
new f2fb81a Merge pull request #633 from bakaid/MINIFICPP-1014
c916d15 is described below
commit c916d15b8ddd3c45646ccb7f66765b7643f46626
Author: Daniel Bakai
AuthorDate: Mon Aug 26 13:10:49 2019 +0200
MINIFICPP-1014 - Fix SFTP extension build on MacOS
---
extensions/sftp/tests/CMakeLists.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/extensions/sftp/tests/CMakeLists.txt
b/extensions/sftp/tests/CMakeLists.txt
index afff438..d350dbd 100644
--- a/extensions/sftp/tests/CMakeLists.txt
+++ b/extensions/sftp/tests/CMakeLists.txt
@@ -45,7 +45,7 @@ if (NOT SKIP_TESTS AND Java_FOUND AND Maven_FOUND AND NOT
DISABLE_EXPRESSION_LAN
# need to add reference to minifi to get EL support for these
tests.
# these tests aren't valid without EL
if (APPLE)
- target_link_libraries (${testfilename} -Wl,-all_load
${LIBSSH2_LIBRARY} ${ZLIB_LIBRARY} ${OPENSSL_LIBRARIES}
minifi-expression-language-extensions inifi-sftp minifi-standard-processors
sftp-test-tools)
+ target_link_libraries (${testfilename} -Wl,-all_load
${LIBSSH2_LIBRARY} ${ZLIB_LIBRARY} ${OPENSSL_LIBRARIES}
minifi-expression-language-extensions minifi-sftp minifi-standard-processors
sftp-test-tools)
else ()
target_link_libraries (${testfilename}
-Wl,--whole-archive ${LIBSSH2_LIBRARY} ${ZLIB_LIBRARY} ${OPENSSL_LIBRARIES}
minifi-expression-language-extensions minifi-sftp minifi-standard-processors
sftp-test-tools -Wl,--no-whole-archive)
endif ()
[nifi-minifi-cpp] branch master updated (45e30dc -> eb14ef6)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 45e30dc Merge pull request #628 from arpadboda/MINIFICPP-1003 new bab9cd6 MINIFICPP-1011 Update Docker to make use of the minifi-agent naming for binary package. new fcc232a MINIFICPP-1012 Provide a minifi-current symlink to the versioned directory of the minifi working directory new eb14ef6 Merge pull request #632 from apiri/MINIFICPP-1012-docker-current-symlink The 601 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docker/ContainerBuild.sh | 2 +- docker/Dockerfile| 37 + 2 files changed, 22 insertions(+), 17 deletions(-)
[nifi-minifi-cpp] branch master updated: MINIFICPP-1003 - persisting properties should leave env vars in config file intact
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 4ca2e5a MINIFICPP-1003 - persisting properties should leave env vars
in config file intact
new 45e30dc Merge pull request #628 from arpadboda/MINIFICPP-1003
4ca2e5a is described below
commit 4ca2e5af73afbc56b067ec33f874183f6bde07cd
Author: Arpad Boda
AuthorDate: Thu Aug 15 18:05:30 2019 +0200
MINIFICPP-1003 - persisting properties should leave env vars in config file
intact
---
libminifi/src/Properties.cpp | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/libminifi/src/Properties.cpp b/libminifi/src/Properties.cpp
index 95cb18e..7dad4e6 100644
--- a/libminifi/src/Properties.cpp
+++ b/libminifi/src/Properties.cpp
@@ -223,13 +223,11 @@ bool Properties::persistProperties() {
continue;
}
-std::string value = equal;
-value =
org::apache::nifi::minifi::utils::StringUtils::replaceEnvironmentVariables(value);
key = org::apache::nifi::minifi::utils::StringUtils::trimRight(key);
-value = org::apache::nifi::minifi::utils::StringUtils::trimRight(value);
+std::string value =
org::apache::nifi::minifi::utils::StringUtils::trimRight(equal);
auto hasIt = properties_copy.find(key);
if (hasIt != properties_copy.end() && !value.empty()) {
- output_file << key << "=" << hasIt->second << std::endl;
+ output_file << key << "=" << value << std::endl;
}
properties_copy.erase(key);
}
[nifi-minifi-cpp] branch master updated: MINIFICPP-984 - Fix linter errors
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 54d2f24 MINIFICPP-984 - Fix linter errors
new 471b5cb Merge pull request #626 from bakaid/MINIFICPP-984-linter
54d2f24 is described below
commit 54d2f24905e61bc96018d769ade5293e7486f8c0
Author: Daniel Bakai
AuthorDate: Wed Aug 14 11:59:44 2019 +0200
MINIFICPP-984 - Fix linter errors
---
extensions/standard-processors/processors/LogAttribute.cpp | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/extensions/standard-processors/processors/LogAttribute.cpp
b/extensions/standard-processors/processors/LogAttribute.cpp
index 1dbaf4a..458b6e5 100644
--- a/extensions/standard-processors/processors/LogAttribute.cpp
+++ b/extensions/standard-processors/processors/LogAttribute.cpp
@@ -56,9 +56,13 @@ core::Property LogAttribute::AttributesToIgnore(
core::Property
LogAttribute::LogPayload(core::PropertyBuilder::createProperty("Log
Payload")->withDescription("If true, the FlowFile's payload will be logged, in
addition to its attributes."
"otherwise, just the Attributes will be
logged")->withDefaultValue(false)->build());
-core::Property
LogAttribute::HexencodePayload(core::PropertyBuilder::createProperty("Hexencode
Payload")->withDescription("If true, the FlowFile's payload will be logged in a
hexencoded format")->withDefaultValue(false)->build());
+core::Property LogAttribute::HexencodePayload(
+core::PropertyBuilder::createProperty("Hexencode
Payload")->withDescription(
+"If true, the FlowFile's payload will be logged in a hexencoded
format")->withDefaultValue(false)->build());
-core::Property
LogAttribute::MaxPayloadLineLength(core::PropertyBuilder::createProperty("Maximum
Payload Line Length")->withDescription("The logged payload will be broken into
lines this long. 0 means no newlines will be
added.")->withDefaultValue(80U)->build());
+core::Property LogAttribute::MaxPayloadLineLength(
+core::PropertyBuilder::createProperty("Maximum Payload Line
Length")->withDescription(
+"The logged payload will be broken into lines this long. 0 means no
newlines will be added.")->withDefaultValue(80U)->build());
core::Property LogAttribute::LogPrefix(
core::PropertyBuilder::createProperty("Log Prefix")->withDescription("Log
prefix appended to the log lines. It helps to distinguish the output of
multiple LogAttribute processors.")->build());
[nifi-minifi-cpp] branch master updated: minificpp-34 Implemented.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 11cefc8 minificpp-34 Implemented.
11cefc8 is described below
commit 11cefc8342aefcd843d48faeef01e025db8c5b9f
Author: amarmer
AuthorDate: Wed Aug 7 13:43:07 2019 -0700
minificpp-34 Implemented.
minificpp-34 Formatting.
minificpp-34 Removed unused variable.
minificpp-34 1. Increased wait time for exe termination to 5sec. 2. Small
refactoring.
minificpp-34 Small refactoring.
minificpp-34 Small refactoring.
minificpp-34 Refactoring.
minificpp-34 No changes, to re-trigger appveyor.
This closes #622.
Signed-off-by: Marc Parisi
---
CMakeLists.txt| 4 +
main/CMakeLists.txt | 3 +-
main/MiNiFiMain.cpp | 17 ++
main/MiNiFiWindowsService.cpp | 372 ++
main/MiNiFiWindowsService.h | 11 ++
5 files changed, 406 insertions(+), 1 deletion(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 4965605..f87f074 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -149,6 +149,10 @@ endif()
endif()
+if (WIN32)
+ add_definitions(-DSERVICE_NAME="MiNiFi")
+endif()
+
#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
if (NOT PORTABLE)
diff --git a/main/CMakeLists.txt b/main/CMakeLists.txt
index 062449b..ecd5f6b 100644
--- a/main/CMakeLists.txt
+++ b/main/CMakeLists.txt
@@ -55,7 +55,8 @@ endif()
endif()
-add_executable(minifiexe MiNiFiMain.cpp)
+add_executable(minifiexe MiNiFiMain.cpp MiNiFiWindowsService.cpp)
+
if (NOT USE_SHARED_LIBS)
if (LIBC_STATIC)
set_target_properties(minifiexe PROPERTIES
LINK_SEARCH_START_STATIC 1)
diff --git a/main/MiNiFiMain.cpp b/main/MiNiFiMain.cpp
index 6c39b5e..5028eb2 100644
--- a/main/MiNiFiMain.cpp
+++ b/main/MiNiFiMain.cpp
@@ -27,6 +27,7 @@
#include
#include
#include
+#include "MiNiFiWindowsService.h"
#pragma comment(lib, "Ws2_32.lib")
#pragma comment(lib, "legacy_stdio_definitions.lib")
#ifdef ENABLE_JNI
@@ -53,6 +54,7 @@
#include "core/RepositoryFactory.h"
#include "FlowController.h"
#include "Main.h"
+
// Variables that allow us to avoid a timed wait.
sem_t *running;
//! Flow Controller
@@ -79,7 +81,12 @@ BOOL WINAPI consoleSignalHandler(DWORD signal) {
return TRUE;
}
+
+void SignalExitProcess() {
+ sem_post(running);
+}
#endif
+
void sigHandler(int signal) {
if (signal == SIGINT || signal == SIGTERM) {
// avoid stopping the controller here.
@@ -88,8 +95,18 @@ void sigHandler(int signal) {
}
int main(int argc, char **argv) {
+#ifdef WIN32
+ CheckRunAsService();
+#endif
+
std::shared_ptr logger =
logging::LoggerConfiguration::getConfiguration().getLogger("main");
+#ifdef WIN32
+ if (!CreateServiceTerminationThread(logger)) {
+return -1;
+ }
+#endif
+
uint16_t stop_wait_time = STOP_WAIT_TIME_MS;
// initialize static functions that were defined apriori
diff --git a/main/MiNiFiWindowsService.cpp b/main/MiNiFiWindowsService.cpp
new file mode 100644
index 000..be0e8db
--- /dev/null
+++ b/main/MiNiFiWindowsService.cpp
@@ -0,0 +1,372 @@
+#ifdef WIN32
+
+#include "MiNiFiWindowsService.h"
+
+#include
+#include
+#include
+#include
+
+#include "core/FlowConfiguration.h"
+
+//#define DEBUG_SERVICE
+
+#ifdef DEBUG_SERVICE
+ #define LOG_INFO(...) OutputDebug(__VA_ARGS__)
+ #define LOG_ERROR(...) OutputDebug(__VA_ARGS__)
+ #define LOG_LASTERROR(str) OutputDebug(str " lastError %x", GetLastError())
+#else
+ #define LOG_INFO(...) Log()->log_info(__VA_ARGS__)
+ #define LOG_ERROR(...) Log()->log_error(__VA_ARGS__)
+ #define LOG_LASTERROR(str) Log()->log_error(str " lastError %x",
GetLastError())
+#endif
+
+#undef DEBUG_SERVICE
+
+// Implemented in MiNiFiMain.cpp
+void SignalExitProcess();
+
+static char* SERVICE_TERMINATION_EVENT_NAME =
"Global\\MiNiFiServiceTermination";
+
+static void OutputDebug(const char* format, ...) {
+ va_list args;
+ va_start(args, format);
+
+ char buf[256];
+ sprintf_s(buf, _countof(buf), "%s: %s", SERVICE_NAME, format);
+
+ char out[1024];
+ StringCbVPrintfA(out, sizeof(out), buf, args);
+
+ OutputDebugStringA(out);
+
+ va_end(args);
+};
+
+void CheckRunAsService() {
+ static const int WAIT_TIME_EXE_TERMINATION = 5000;
+ static const int WAIT_TIME_EXE_RESTART = 6;
+
+ static SERVICE_STATUS s_serviceStatus;
+ static SERVICE_STATUS_HANDLE s_statusHandle;
+ static HANDLE s_hProcess;
+ static HANDLE s_hEvent;
+
+ static auto Log = []() {
+static std::shared_ptr s_logger =
logging::LoggerConfiguration::
[nifi-minifi-cpp] 01/01: Merge pull request #619 from apache/phrocker-patch-1
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 4f13e523fb40ff709e7641495d7d20eac19dbd9f Merge: fb92490 6e217bc Author: Marc AuthorDate: Tue Aug 13 08:39:16 2019 -0400 Merge pull request #619 from apache/phrocker-patch-1 Ran gen-processors-docs PROCESSORS.md | 1517 + 1 file changed, 781 insertions(+), 736 deletions(-)
[nifi-minifi-cpp] branch master updated (fb92490 -> 4f13e52)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from fb92490 MINIFICPP-1000 - Add proper logging for configuration loading add 6e217bc Ran gen-processors-docs new 4f13e52 Merge pull request #619 from apache/phrocker-patch-1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: PROCESSORS.md | 1517 + 1 file changed, 781 insertions(+), 736 deletions(-)
[nifi-minifi-cpp] branch master updated: MINIFICPP-1000 - Add proper logging for configuration loading
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new fb92490 MINIFICPP-1000 - Add proper logging for configuration loading
fb92490 is described below
commit fb9249086d4a9512ccbb129b493d8acd6f93580a
Author: Arpad Boda
AuthorDate: Thu Aug 8 15:18:32 2019 -0400
MINIFICPP-1000 - Add proper logging for configuration loading
This closes #623.
Signed-off-by: Marc Parisi
---
.../include/core/logging/LoggerConfiguration.h | 1 +
libminifi/include/properties/Configure.h | 1 +
libminifi/include/properties/Properties.h | 10 -
libminifi/src/Properties.cpp | 45 ++
main/MiNiFiMain.cpp| 2 +-
5 files changed, 31 insertions(+), 28 deletions(-)
diff --git a/libminifi/include/core/logging/LoggerConfiguration.h
b/libminifi/include/core/logging/LoggerConfiguration.h
index d015862..7f9fdd2 100644
--- a/libminifi/include/core/logging/LoggerConfiguration.h
+++ b/libminifi/include/core/logging/LoggerConfiguration.h
@@ -57,6 +57,7 @@ struct LoggerNamespace {
class LoggerProperties : public Properties {
public:
+ LoggerProperties() : Properties("Logger properties") {}
/**
* Gets all keys that start with the given prefix and do not have a "."
after the prefix and "." separator.
*
diff --git a/libminifi/include/properties/Configure.h
b/libminifi/include/properties/Configure.h
index 9fc639f..f4a3a4e 100644
--- a/libminifi/include/properties/Configure.h
+++ b/libminifi/include/properties/Configure.h
@@ -30,6 +30,7 @@ namespace minifi {
class Configure : public Properties {
public:
+ Configure() : Properties("MiNiFi configuration") {}
void setAgentIdentifier(const std::string ) {
std::lock_guard lock(mutex_);
diff --git a/libminifi/include/properties/Properties.h
b/libminifi/include/properties/Properties.h
index 9047d11..420f26c 100644
--- a/libminifi/include/properties/Properties.h
+++ b/libminifi/include/properties/Properties.h
@@ -45,12 +45,16 @@ namespace minifi {
class Properties {
public:
- Properties();
+ Properties(const std::string& name = "");
virtual ~Properties() {
}
+ virtual const std::string& getName() {
+return name_;
+ }
+
// Clear the load config
void clear() {
std::lock_guard lock(mutex_);
@@ -110,7 +114,7 @@ class Properties {
}
// Get the determined MINIFI_HOME
- std::string getHome() {
+ std::string getHome() const {
return minifi_home_;
}
// Parse Command Line
@@ -137,6 +141,8 @@ class Properties {
std::shared_ptr logger_;
// Home location for this executable
std::string minifi_home_;
+
+ std::string name_;
};
} /* namespace minifi */
diff --git a/libminifi/src/Properties.cpp b/libminifi/src/Properties.cpp
index 0e23b45..95cb18e 100644
--- a/libminifi/src/Properties.cpp
+++ b/libminifi/src/Properties.cpp
@@ -22,14 +22,6 @@
#include "core/Core.h"
#include "core/logging/LoggerConfiguration.h"
-#ifndef FILE_SEPARATOR_C
-#ifdef WIN32
-#define FILE_SEPARATOR_C '\\'
-#else
-#define FILE_SEPARATOR_C '/'
-#endif
-#endif
-
namespace org {
namespace apache {
namespace nifi {
@@ -37,8 +29,9 @@ namespace minifi {
#define TRACE_BUFFER_SIZE 512
-Properties::Properties()
-: logger_(logging::LoggerFactory::getLogger()) {
+Properties::Properties(const std::string& name)
+: logger_(logging::LoggerFactory::getLogger()),
+name_(name) {
}
// Get the config value
@@ -122,26 +115,27 @@ bool Properties::parseConfigureFileLine(char *buf,
std::string _key, std::s
// Load Configure File
void Properties::loadConfigureFile(const char *fileName) {
- std::string adjustedFilename;
- if (fileName) {
-// perform a naive determination if this is a relative path
-if (fileName[0] != FILE_SEPARATOR_C) {
- adjustedFilename = adjustedFilename + getHome() + FILE_SEPARATOR_C +
fileName;
-} else {
- if (adjustedFilename.empty()) {
-adjustedFilename = getHome();
- }
- adjustedFilename += fileName;
-}
+ if (NULL == fileName) {
+logger_->log_error("Configuration file path for %s is a nullptr!",
getName().c_str());
+return;
+ }
+
+ std::string adjustedFilename = getHome();
+ // perform a naive determination if this is a relative path
+ if (fileName[0] != utils::file::FileUtils::get_separator()) {
+adjustedFilename += utils::file::FileUtils::get_separator();
}
- char *path = NULL;
+
+ adjustedFilename += fileName;
+
+ const char *path = NULL;
#ifndef WIN32
char full_path[PATH_MAX];
path = realpath(adjustedFilename.c_str(), full_path);
#else
- path = const_cast(adjustedFilename.c_str());
+ path = adjustedFilenam
[nifi-minifi-cpp] branch master updated: MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload Line Length" options to LogAttribute with defaults to emulate the previous behaviour
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 2cde150 MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload
Line Length" options to LogAttribute with defaults to emulate the previous
behaviour
2cde150 is described below
commit 2cde15078778bbe73d35a666a74f8e0f65c869a8
Author: Daniel Bakai
AuthorDate: Mon Aug 12 18:42:47 2019 +0200
MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload Line Length"
options to LogAttribute with defaults to emulate the previous behaviour
This closes #625.
Signed-off-by: Marc Parisi
---
.../processors/LogAttribute.cpp| 30 ++---
.../standard-processors/processors/LogAttribute.h | 6 +
.../tests/integration/TailFileCronTest.cpp | 2 +-
.../tests/integration/TailFileTest.cpp | 2 +-
.../tests/unit/TailFileTests.cpp | 31 +++---
5 files changed, 62 insertions(+), 9 deletions(-)
diff --git a/extensions/standard-processors/processors/LogAttribute.cpp
b/extensions/standard-processors/processors/LogAttribute.cpp
index cc44f25..1dbaf4a 100644
--- a/extensions/standard-processors/processors/LogAttribute.cpp
+++ b/extensions/standard-processors/processors/LogAttribute.cpp
@@ -56,6 +56,10 @@ core::Property LogAttribute::AttributesToIgnore(
core::Property
LogAttribute::LogPayload(core::PropertyBuilder::createProperty("Log
Payload")->withDescription("If true, the FlowFile's payload will be logged, in
addition to its attributes."
"otherwise, just the Attributes will be
logged")->withDefaultValue(false)->build());
+core::Property
LogAttribute::HexencodePayload(core::PropertyBuilder::createProperty("Hexencode
Payload")->withDescription("If true, the FlowFile's payload will be logged in a
hexencoded format")->withDefaultValue(false)->build());
+
+core::Property
LogAttribute::MaxPayloadLineLength(core::PropertyBuilder::createProperty("Maximum
Payload Line Length")->withDescription("The logged payload will be broken into
lines this long. 0 means no newlines will be
added.")->withDefaultValue(80U)->build());
+
core::Property LogAttribute::LogPrefix(
core::PropertyBuilder::createProperty("Log Prefix")->withDescription("Log
prefix appended to the log lines. It helps to distinguish the output of
multiple LogAttribute processors.")->build());
@@ -68,6 +72,8 @@ void LogAttribute::initialize() {
properties.insert(AttributesToLog);
properties.insert(AttributesToIgnore);
properties.insert(LogPayload);
+ properties.insert(HexencodePayload);
+ properties.insert(MaxPayloadLineLength);
properties.insert(FlowFilesToLog);
properties.insert(LogPrefix);
setSupportedProperties(properties);
@@ -87,6 +93,14 @@ void LogAttribute::onSchedule(const
std::shared_ptr
throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Invalid value for flowfiles
to log: " + flowsToLog.getValue().to_string());
flowfiles_to_log_ = flowsToLog.getValue();
}
+
+ std::string value;
+ if (context->getProperty(HexencodePayload.getName(), value)) {
+utils::StringUtils::StringToBool(value, hexencode_);
+ }
+ if (context->getProperty(MaxPayloadLineLength.getName(), value)) {
+core::Property::StringToInt(value, max_line_length_);
+ }
}
// OnTrigger method, implemented by NiFi LogAttribute
void LogAttribute::onTrigger(const std::shared_ptr
, const std::shared_ptr ) {
@@ -138,9 +152,19 @@ void LogAttribute::onTrigger(const
std::shared_ptr
ReadCallback callback(logger_, flow->getSize());
session->read(flow, );
- auto payload_hex = utils::StringUtils::to_hex(callback.buffer_.data(),
callback.buffer_.size());
- for (size_t i = 0; i < payload_hex.size(); i += 80) {
-message << payload_hex.substr(i, 80) << '\n';
+ std::string printable_payload;
+ if (hexencode_) {
+printable_payload =
utils::StringUtils::to_hex(callback.buffer_.data(), callback.buffer_.size());
+ } else {
+printable_payload =
std::string(reinterpret_cast(callback.buffer_.data()),
callback.buffer_.size());
+ }
+
+ if (max_line_length_ == 0U) {
+message << printable_payload << "\n";
+ } else {
+for (size_t i = 0; i < printable_payload.size(); i +=
max_line_length_) {
+ message << printable_payload.substr(i, max_line_length_) << '\n';
+}
}
} else {
message << "\n";
diff -
[nifi-minifi-cpp] 01/01: Ran gen-processors-docs
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch phrocker-patch-1 in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 6e217bc727e364c6718d2a00bba05128b5d28fc2 Author: Marc AuthorDate: Fri Aug 2 11:51:47 2019 -0400 Ran gen-processors-docs To generate new docs --- PROCESSORS.md | 1517 + 1 file changed, 781 insertions(+), 736 deletions(-) diff --git a/PROCESSORS.md b/PROCESSORS.md index 079e055..3f0a1af 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -1,41 +1,33 @@ - - +
[nifi-minifi-cpp] branch phrocker-patch-1 created (now 6e217bc)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch phrocker-patch-1 in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. at 6e217bc Ran gen-processors-docs This branch includes the following new commits: new 6e217bc Ran gen-processors-docs The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[nifi-minifi-cpp] branch master updated: MINIFICPP-939: Install deps up front on certian systems. Change centos LIBDIR as the version change of cmake enforced a different dir
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new c9d9b78 MINIFICPP-939: Install deps up front on certian systems.
Change centos LIBDIR as the version change of cmake enforced a different dir
c9d9b78 is described below
commit c9d9b788f0856e86f60665664d22d329da883979
Author: Marc Parisi
AuthorDate: Fri Jun 28 09:28:12 2019 -0400
MINIFICPP-939: Install deps up front on certian systems. Change centos
LIBDIR as the version change of cmake enforced a different dir
This closes #602.
Signed-off-by: Marc Parisi
---
bootstrap.sh | 2 +-
docker/bionic/Dockerfile | 2 +-
docker/centos/Dockerfile | 3 +--
docker/fedora/Dockerfile | 2 +-
docker/xenial/Dockerfile | 2 +-
extensions/librdkafka/CMakeLists.txt | 15 +--
6 files changed, 6 insertions(+), 20 deletions(-)
diff --git a/bootstrap.sh b/bootstrap.sh
index df553ac..9bc864d 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -270,7 +270,7 @@ add_dependency GPS_ENABLED "gpsd"
add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
-add_disabled_option KAFKA_ENABLED ${FALSE} "ENABLE_LIBRDKAFKA" "3.4.0"
+add_disabled_option KAFKA_ENABLED ${FALSE} "ENABLE_LIBRDKAFKA"
add_disabled_option MQTT_ENABLED ${FALSE} "ENABLE_MQTT"
diff --git a/docker/bionic/Dockerfile b/docker/bionic/Dockerfile
index 0154f5b..e33f0fc 100644
--- a/docker/bionic/Dockerfile
+++ b/docker/bionic/Dockerfile
@@ -35,7 +35,7 @@ ENV MINIFI_BASE_DIR /opt/minifi
RUN mkdir -p $MINIFI_BASE_DIR
USER $USER
-RUN apt-get update && apt-get install -y openjdk-8-jdk openjdk-8-source sudo
git maven
+RUN apt-get update && apt-get install -y openjdk-8-jdk openjdk-8-source sudo
git maven autogen autoconf automake
ADD $MINIFI_SOURCE_CODE $MINIFI_BASE_DIR
diff --git a/docker/centos/Dockerfile b/docker/centos/Dockerfile
index 864ce03..e54631a 100644
--- a/docker/centos/Dockerfile
+++ b/docker/centos/Dockerfile
@@ -35,7 +35,7 @@ ENV MINIFI_BASE_DIR /opt/minifi
RUN mkdir -p $MINIFI_BASE_DIR
USER $USER
-RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel sudo git which
maven
+RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel gcc g++ sudo
git which maven
ADD $MINIFI_SOURCE_CODE $MINIFI_BASE_DIR
@@ -55,4 +55,3 @@ RUN cd $MINIFI_BASE_DIR \
&& cmake3 -DUSE_SHARED_LIBS= -DENABLE_MQTT=ON -DENABLE_LIBRDKAFKA=ON
-DPORTABLE=ON -DENABLE_COAP=ON -DCMAKE_BUILD_TYPE=Release -DSKIP_TESTS=true
-DENABLE_JNI=$ENABLE_JNI .. \
&& make -j8 package
-#COPY $MINIFI_BASE_DIR/build/nifi-minifi-cpp-$MINIFI_VERSION-bin.tar.gz
$DUMP_LOCATION/nifi-minifi-cpp-centos7-$MINIFI_VERSION-bin.tar.gz
\ No newline at end of file
diff --git a/docker/fedora/Dockerfile b/docker/fedora/Dockerfile
index 612e897..d102349 100644
--- a/docker/fedora/Dockerfile
+++ b/docker/fedora/Dockerfile
@@ -35,7 +35,7 @@ ENV MINIFI_BASE_DIR /opt/minifi
RUN mkdir -p $MINIFI_BASE_DIR
USER $USER
-RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel flex bison make
patch sudo git which maven
+RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel flex bison make
patch sudo git which maven libtool autoconf automake
ADD $MINIFI_SOURCE_CODE $MINIFI_BASE_DIR
diff --git a/docker/xenial/Dockerfile b/docker/xenial/Dockerfile
index 152cabc..248dd7c 100644
--- a/docker/xenial/Dockerfile
+++ b/docker/xenial/Dockerfile
@@ -34,7 +34,7 @@ ENV MINIFI_BASE_DIR /opt/minifi
RUN mkdir -p $MINIFI_BASE_DIR
USER $USER
-RUN apt-get update && apt-get install -y openjdk-8-jdk openjdk-8-source sudo
git maven
+RUN apt-get update && apt-get install -y openjdk-8-jdk openjdk-8-source sudo
git maven autogen autoconf automake
ADD $MINIFI_SOURCE_CODE $MINIFI_BASE_DIR
diff --git a/extensions/librdkafka/CMakeLists.txt
b/extensions/librdkafka/CMakeLists.txt
index e1f4247..8d9 100644
--- a/extensions/librdkafka/CMakeLists.txt
+++ b/extensions/librdkafka/CMakeLists.txt
@@ -55,6 +55,7 @@ ExternalProject_Add(
"-DENABLE_LZ4_EXT=OFF"
"-DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH_PASSTHROUGH_LIST}"
"-DCMAKE_C_FLAGS=${CURL_C_FLAGS}"
+ "-DCMAKE_INSTALL_LIBDIR=lib"
"-DCMAKE_CXX_FLAGS=${CURL_CXX_FLAGS}"
EXCLUDE_FROM_ALL TRUE
)
@@ -66,20 +67,6 @@ add_dependencies(minifi-rdkafka-extensions kafka-external)
include_directories(${ZLIB_INCLUDE_DIRS})
include_directories(${KAFKA_INCLUDE})
target_link_libraries (minifi-rdkafka-extensions ${BYPRODUCT})
-if (WIN32)
-set_target_properties(minifi-rdkafka-extensions PROPERTIES
-LINK
[nifi-minifi-cpp] branch master updated: MINIFICPP-936: Add canonical name check back into processor utils
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 581a9b6 MINIFICPP-936: Add canonical name check back into processor
utils
581a9b6 is described below
commit 581a9b65432ec89cf6edfdd80dbfa165d7627100
Author: Marc Parisi
AuthorDate: Wed Jun 26 19:00:02 2019 -0400
MINIFICPP-936: Add canonical name check back into processor utils
This closes #600.
Signed-off-by: Marc Parisi
---
libminifi/include/processors/ProcessorUtils.h | 29 ++-
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git a/libminifi/include/processors/ProcessorUtils.h
b/libminifi/include/processors/ProcessorUtils.h
index d6ad3e3..40b87c2 100644
--- a/libminifi/include/processors/ProcessorUtils.h
+++ b/libminifi/include/processors/ProcessorUtils.h
@@ -22,20 +22,31 @@ class ProcessorUtils {
static inline std::shared_ptr createProcessor(const
std::string _short_name, const std::string ,
utils::Identifier ,
const
std::shared_ptr _factory) {
auto ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(class_short_name, uuid);
+// fallback to the canonical name if the short name does not work, then
attempt JNI bindings if they exist
if (ptr == nullptr) {
- ptr =
core::ClassLoader::getDefaultClassLoader().instantiate("ExecuteJavaClass",
uuid);
- if (ptr != nullptr) {
-std::shared_ptr processor =
std::static_pointer_cast(ptr);
-processor->initialize();
-processor->setProperty("NiFi Processor", canonicalName);
-processor->setStreamFactory(stream_factory);
-return processor;
+ ptr =
core::ClassLoader::getDefaultClassLoader().instantiate(canonicalName, uuid);
+ if (ptr == nullptr) {
+ptr =
core::ClassLoader::getDefaultClassLoader().instantiate("ExecuteJavaClass",
uuid);
+if (ptr != nullptr) {
+ std::shared_ptr processor =
std::dynamic_pointer_cast(ptr);
+ if (processor == nullptr) {
+throw std::runtime_error("Invalid return from the classloader");
+ }
+ processor->initialize();
+ processor->setProperty("NiFi Processor", canonicalName);
+ processor->setStreamFactory(stream_factory);
+ return processor;
+}
}
}
-if (nullptr == ptr) {
+if (ptr == nullptr) {
return nullptr;
}
-auto returnPtr = std::static_pointer_cast(ptr);
+auto returnPtr = std::dynamic_pointer_cast(ptr);
+
+if (returnPtr == nullptr) {
+ throw std::runtime_error("Invalid return from the classloader");
+}
returnPtr->initialize();
[nifi-minifi-cpp] branch master updated: MINIFICPP-933: Fix raw site to site with secure comms
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 5a4d145 MINIFICPP-933: Fix raw site to site with secure comms
5a4d145 is described below
commit 5a4d145a29405e1ef136492aa991bb9599c0508b
Author: Marc Parisi
AuthorDate: Tue Jun 25 17:31:59 2019 -0400
MINIFICPP-933: Fix raw site to site with secure comms
MINIFICPP-933: Add tests to cover a few corner cases. Break out password to
utility file
MINIFICPP-933: remove sleeps and undo unintentional changes
This closes #597.
Signed-off-by: Marc Parisi
---
.../standard-processors/tests/CMakeLists.txt | 3 +
.../tests/integration/SecureSocketGetTCPTest.cpp | 17 ++--
libminifi/include/controllers/SSLContextService.h | 16 +---
libminifi/include/io/tls/TLSSocket.h | 12 ---
libminifi/include/io/tls/TLSUtils.h| 40 ++
libminifi/src/controllers/SSLContextService.cpp| 2 +-
libminifi/src/io/tls/TLSSocket.cpp | 22 +++---
libminifi/src/io/tls/TLSUtils.cpp | 56 ++
.../test/resources/TestGetTCPSecureEmptyPass.yml | 90 ++
.../resources/TestGetTCPSecureWithFilePass.yml | 90 ++
.../test/resources/TestGetTCPSecureWithPass.yml| 90 ++
libminifi/test/resources/cn.pass | 5 ++
.../test/resources/{cn.pass => empty.cn.pass} | 0
libminifi/test/resources/encrypted.cn.pass | 5 ++
libminifi/test/resources/encrypted.key.pem | 30
15 files changed, 432 insertions(+), 46 deletions(-)
diff --git a/extensions/standard-processors/tests/CMakeLists.txt
b/extensions/standard-processors/tests/CMakeLists.txt
index f87ecf4..51d95ce 100644
--- a/extensions/standard-processors/tests/CMakeLists.txt
+++ b/extensions/standard-processors/tests/CMakeLists.txt
@@ -70,6 +70,9 @@ message("-- Finished building ${INT_TEST_COUNT} integration
test file(s)...")
add_test(NAME TestExecuteProcess COMMAND TestExecuteProcess )
add_test(NAME SecureSocketGetTCPTest COMMAND SecureSocketGetTCPTest
"${TEST_RESOURCES}/TestGetTCPSecure.yml" "${TEST_RESOURCES}/")
+add_test(NAME SecureSocketGetTCPTestEmptyPass COMMAND SecureSocketGetTCPTest
"${TEST_RESOURCES}/TestGetTCPSecureEmptyPass.yml" "${TEST_RESOURCES}/")
+add_test(NAME SecureSocketGetTCPTestWithPassword COMMAND
SecureSocketGetTCPTest "${TEST_RESOURCES}/TestGetTCPSecureWithPass.yml"
"${TEST_RESOURCES}/")
+add_test(NAME SecureSocketGetTCPTestWithPasswordFile COMMAND
SecureSocketGetTCPTest "${TEST_RESOURCES}/TestGetTCPSecureWithFilePass.yml"
"${TEST_RESOURCES}/")
add_test(NAME TailFileTest COMMAND TailFileTest
"${TEST_RESOURCES}/TestTailFile.yml" "${TEST_RESOURCES}/")
diff --git
a/extensions/standard-processors/tests/integration/SecureSocketGetTCPTest.cpp
b/extensions/standard-processors/tests/integration/SecureSocketGetTCPTest.cpp
index e3a0478..73b2005 100644
---
a/extensions/standard-processors/tests/integration/SecureSocketGetTCPTest.cpp
+++
b/extensions/standard-processors/tests/integration/SecureSocketGetTCPTest.cpp
@@ -98,8 +98,13 @@ class SecureSocketTest : public IntegrationBase {
path = key_dir + "nifi-cert.pem";
configuration->set("nifi.security.client.ca.certificate", path);
configuration->set("nifi.c2.enable", "false");
+std::string endpoint;
+inv->getProperty(minifi::processors::GetTCP::EndpointList.getName(),
endpoint);
+auto endpoints = utils::StringUtils::split(endpoint, ",");
+assert(1 == endpoints.size());
+auto hostAndPort = utils::StringUtils::split(endpoint, ":");
std::shared_ptr socket_context
= std::make_shared(configuration);
-server_socket =
std::make_shared(socket_context,
"localhost", 8776, 3);
+server_socket =
std::make_shared(socket_context,
hostAndPort.at(0), std::stoi(hostAndPort.at(1)), 3);
server_socket->initialize();
isRunning_ = true;
@@ -155,10 +160,6 @@ class SecureSocketTest : public IntegrationBase {
cleanup();
}
- virtual void waitToVerifyProcessor() {
- std::this_thread::sleep_for(std::chrono::seconds(3));
-}
-
protected:
std::function check;
std::function*b, int *size)> handler;
@@ -170,11 +171,11 @@ class SecureSocketTest : public IntegrationBase {
std::shared_ptr
server_socket;
};
-static void sigpipe_handle(int x) {}
+static void sigpipe_handle(int x) {
+}
int main(int argc, char **argv) {
- std::string key_dir, test_file_location, url;
- url = "http://localhost:/api/heartbeat;;
+ std::string key_dir, test_file_locati
[nifi-minifi-cpp] branch master updated: MINIFICPP-910 - Extend StringUtils with string join capability
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new ca2a68d MINIFICPP-910 - Extend StringUtils with string join capability
ca2a68d is described below
commit ca2a68d6ec53924351ae1262c09cb47f28c0579c
Author: Arpad Boda
AuthorDate: Wed Jun 19 01:08:01 2019 +0200
MINIFICPP-910 - Extend StringUtils with string join capability
This closes #585.
Signed-off-by: Marc Parisi
---
libminifi/include/utils/StringUtils.h| 85
libminifi/test/unit/StringUtilsTests.cpp | 19 +++
2 files changed, 104 insertions(+)
diff --git a/libminifi/include/utils/StringUtils.h
b/libminifi/include/utils/StringUtils.h
index daf3a20..326a048 100644
--- a/libminifi/include/utils/StringUtils.h
+++ b/libminifi/include/utils/StringUtils.h
@@ -47,6 +47,26 @@ namespace nifi {
namespace minifi {
namespace utils {
+namespace {
+ template
+ struct string_traits;
+ template<>
+ struct string_traits{
+template
+static std::string convert_to_string(T&& t){
+ return std::to_string(std::forward(t));
+}
+ };
+
+ template<>
+ struct string_traits{
+template
+static std::wstring convert_to_string(T&& t){
+ return std::to_wstring(std::forward(t));
+}
+ };
+}
+
/**
* Stateless String utility class.
*
@@ -262,6 +282,71 @@ class StringUtils {
return result_string;
}
+ /**
+ * Concatenates strings stored in an arbitrary container using the provided
separator.
+ * @tparam TChar char type of the string (char or wchar_t)
+ * @tparam U arbitrary container which has string or wstring value type
+ * @param separator that is inserted between each elements. Type should match
the type of strings in container.
+ * @param container that contains the strings to be concatenated
+ * @return the result string
+ */
+ template>::value>::type* = nullptr>
+ static std::basic_string join(const std::basic_string&
separator, const U& container) {
+typedef typename U::const_iterator ITtype;
+ITtype it = container.cbegin();
+std::basic_stringstream sstream;
+while(it != container.cend()) {
+ sstream << (*it);
+ ++it;
+ if(it != container.cend()) {
+sstream << separator;
+ }
+}
+return sstream.str();
+ };
+
+ /**
+ * Just a wrapper for the above function to be able to create separator from
const char* or const wchar_t*
+ */
+ template>::value>::type* = nullptr>
+ static std::basic_string join(const TChar* separator, const U&
container) {
+return join(std::basic_string(separator), container);
+ };
+
+
+ /**
+ * Concatenates string representation of integrals stored in an arbitrary
container using the provided separator.
+ * @tparam TChar char type of the string (char or wchar_t)
+ * @tparam U arbitrary container which has any integral value type
+ * @param separator that is inserted between each elements. Type of this
determines the result type. (wstring separator -> wstring)
+ * @param container that contains the integrals to be concatenated
+ * @return the result string
+ */
+ template::value>::type* =
nullptr,
+ typename std::enable_if>::value>::type* = nullptr>
+ static std::basic_string join(const std::basic_string&
separator, const U& container) {
+typedef typename U::const_iterator ITtype;
+ITtype it = container.cbegin();
+std::basic_stringstream sstream;
+while(it != container.cend()) {
+ sstream << string_traits::convert_to_string(*it);
+ ++it;
+ if(it != container.cend()) {
+sstream << separator;
+ }
+}
+return sstream.str();
+ }
+
+ /**
+ * Just a wrapper for the above function to be able to create separator from
const char* or const wchar_t*
+ */
+ template::value>::type* =
nullptr,
+ typename std::enable_if>::value>::type* = nullptr>
+ static std::basic_string join(const TChar* separator, const U&
container) {
+return join(std::basic_string(separator), container);
+ };
+
};
} /* namespace utils */
diff --git a/libminifi/test/unit/StringUtilsTests.cpp
b/libminifi/test/unit/StringUtilsTests.cpp
index 3abf339..6486f29 100644
--- a/libminifi/test/unit/StringUtilsTests.cpp
+++ b/libminifi/test/unit/StringUtilsTests.cpp
@@ -17,6 +17,7 @@
*/
#include
+#include
#include
#include
#include "../TestBase.h"
@@ -93,3 +94,21 @@ TEST_CASE("TestStringUtils::testEnv5", "[test split
classname]") {
REQUIRE(expected == StringUtils::replaceEnvironmentVariables(test_string));
}
+
+TEST_CASE("TestStringUtils::testJoin", "[test string join]") {
+ std::set strings = {"3", "2", "
[nifi-minifi-cpp] branch master updated: MINIFICPP-30: Add support for regex with Multiple file mode
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 7a2e196 MINIFICPP-30: Add support for regex with Multiple file mode
7a2e196 is described below
commit 7a2e196da9bf7cd2638c0f6b616505b53bb774f8
Author: Marc Parisi
AuthorDate: Mon Jun 3 19:11:18 2019 -0400
MINIFICPP-30: Add support for regex with Multiple file mode
MINIFICPP-30: Update docs to make clear the impact of rollover on Multi
file mode
MINIFICPP-30: Fix test issue
Approved by aboda on github
This closes #580.
Signed-off-by: Marc Parisi
---
extensions/standard-processors/CMakeLists.txt | 2 +-
.../standard-processors/processors/TailFile.cpp| 339 ++---
.../standard-processors/processors/TailFile.h | 59 ++--
.../tests/unit/TailFileTests.cpp | 334 ++--
libminifi/CMakeLists.txt | 3 +-
libminifi/include/utils/StringUtils.h | 6 +-
libminifi/include/utils/file/PathUtils.h | 48 +++
libminifi/src/utils/file/PathUtils.cpp | 57
libminifi/test/unit/FileUtilsTests.cpp | 39 +++
9 files changed, 582 insertions(+), 305 deletions(-)
diff --git a/extensions/standard-processors/CMakeLists.txt
b/extensions/standard-processors/CMakeLists.txt
index 7691c78..ddaf19f 100644
--- a/extensions/standard-processors/CMakeLists.txt
+++ b/extensions/standard-processors/CMakeLists.txt
@@ -31,7 +31,7 @@ if(CMAKE_THREAD_LIBS_INIT)
target_link_libraries(minifi-standard-processors "${CMAKE_THREAD_LIBS_INIT}")
endif()
-
+target_link_libraries(minifi-standard-processors core-minifi)
SET (STANDARD-PROCESSORS minifi-standard-processors PARENT_SCOPE)
register_extension(minifi-standard-processors)
diff --git a/extensions/standard-processors/processors/TailFile.cpp
b/extensions/standard-processors/processors/TailFile.cpp
index 8302aeb..c282df2 100644
--- a/extensions/standard-processors/processors/TailFile.cpp
+++ b/extensions/standard-processors/processors/TailFile.cpp
@@ -37,8 +37,14 @@
#include
#include
#include "utils/file/FileUtils.h"
+#include "utils/file/PathUtils.h"
#include "utils/TimeUtil.h"
#include "utils/StringUtils.h"
+#ifdef HAVE_REGEX_CPP
+#include
+#else
+#include
+#endif
#include "TailFile.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
@@ -61,21 +67,36 @@ namespace nifi {
namespace minifi {
namespace processors {
-core::Property TailFile::FileName("File to Tail", "Fully-qualified filename of
the file that should be tailed", "");
+core::Property TailFile::FileName("File to Tail", "Fully-qualified filename of
the file that should be tailed when using single file mode, or a file regex
when using multifile mode", "");
core::Property TailFile::StateFile("State File", "Specifies the file that
should be used for storing state about"
" what data has been ingested so that upon
restart NiFi can resume from where it left off",
"TailFileState");
core::Property TailFile::Delimiter("Input Delimiter", "Specifies the character
that should be used for delimiting the data being tailed"
"from the incoming file.",
"");
+
+core::Property TailFile::TailMode(
+core::PropertyBuilder::createProperty("tail-mode", "Tailing
Mode")->withDescription(
+"Specifies the tail file mode. In 'Single file' mode only a single
file will be watched. "
+"In 'Multiple file' mode a regex may be used. Note that in multiple
file mode we will still continue to watch for rollover on the initial set of
watched files. "
+"The Regex used to locate multiple files will be run during the
schedule phrase. Note that if rotated files are matched by the regex, those
files will be tailed.")->isRequired(true)
+->withAllowableValue("Single
file")->withAllowableValue("Multiple file")->withDefaultValue("Single
file")->build());
+
+core::Property
TailFile::BaseDirectory(core::PropertyBuilder::createProperty("tail-base-directory",
"Base Directory")->isRequired(false)->build());
+
core::Relationship TailFile::Success("success", "All files are routed to
success");
+const char *TailFile::CURRENT_STR = "CURRENT.";
+const char *TailFile::POSITION_STR = "POSITION.";
+
void TailFile::initial
[nifi-minifi-cpp] branch master updated: MINIFICPP-621 Add initial tailfile nanofi example
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new e7ac27d MINIFICPP-621 Add initial tailfile nanofi example
e7ac27d is described below
commit e7ac27d00eb160b28567cea2201deda2af10593a
Author: Murtuza Shareef
AuthorDate: Mon Jun 10 18:15:48 2019 -0400
MINIFICPP-621 Add initial tailfile nanofi example
This closes #590.
Signed-off-by: Marc Parisi
---
libminifi/include/ResourceClaim.h | 1 +
nanofi/examples/CMakeLists.txt | 4 +
nanofi/examples/tail_file.c| 257 +
nanofi/include/api/nanofi.h| 19 +++
nanofi/include/core/cstructs.h | 11 ++
nanofi/include/core/string_utils.h | 53
nanofi/src/api/nanofi.cpp | 37 --
nanofi/src/core/string_utils.c | 86 +
8 files changed, 457 insertions(+), 11 deletions(-)
diff --git a/libminifi/include/ResourceClaim.h
b/libminifi/include/ResourceClaim.h
index 8cae8fc..e4523e4 100644
--- a/libminifi/include/ResourceClaim.h
+++ b/libminifi/include/ResourceClaim.h
@@ -59,6 +59,7 @@ class ResourceClaim : public
std::enable_shared_from_this {
explicit ResourceClaim(std::shared_ptr>
claim_manager);
explicit ResourceClaim(const std::string path,
std::shared_ptr> claim_manager, bool deleted
= false);
+
// Destructor
~ResourceClaim() {
}
diff --git a/nanofi/examples/CMakeLists.txt b/nanofi/examples/CMakeLists.txt
index 6a9779c..4301a7d 100644
--- a/nanofi/examples/CMakeLists.txt
+++ b/nanofi/examples/CMakeLists.txt
@@ -83,4 +83,8 @@ add_executable(monitor_directory monitor_directory.c)
target_link_libraries(monitor_directory nanofi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi-http-curl ${LINK_END_FLAGS})
+add_executable(tail_file tail_file.c)
+
+target_link_libraries(tail_file nanofi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi-http-curl ${LINK_END_FLAGS})
+
endif()
diff --git a/nanofi/examples/tail_file.c b/nanofi/examples/tail_file.c
new file mode 100644
index 000..2200df4
--- /dev/null
+++ b/nanofi/examples/tail_file.c
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+#include "api/nanofi.h"
+#include "core/string_utils.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+typedef struct flow_file_records {
+flow_file_record ** records;
+uint64_t len;
+} flow_file_records;
+
+struct flow_file_records * flowfiles = NULL;
+nifi_instance * instance = NULL;
+standalone_processor * proc = NULL;
+int file_offset = 0;
+int stopped = 0;
+
+void signal_handler(int signum) {
+if (signum == SIGINT || signum == SIGTERM) {
+stopped = 1;
+}
+}
+
+void transmit_flow_files(nifi_instance * instance) {
+NULL_CHECK( ,flowfiles);
+int i;
+for (i = 0; i < flowfiles->len; ++i) {
+NULL_CHECK( ,flowfiles->records[i]);
+transmit_flowfile(flowfiles->records[i], instance);
+}
+}
+
+void free_flow_file_records() {
+NULL_CHECK( ,flowfiles);
+int i;
+for (i = 0; i < flowfiles->len; ++i) {
+free_flowfile(flowfiles->records[i]);
+}
+free(flowfiles);
+flowfiles = NULL;
+}
+
+void set_offset(int offset) {
+file_offset = offset;
+}
+
+int get_offset() {
+return file_offset;
+}
+
+void free_all_strings(char ** strings, int num_strings) {
+int i;
+for (i = 0; i < num_strings; ++i) {
+free(strings[i]);
+}
+}
+
+void on_trigger_callback(processor_session * ps, processor_context * ctx) {
+
+char file_path[4096];
+char delimiter[2];
+
+if (get_property(ctx, "file_path", file_path, 50) != 0) {
+return;
+}
+
+if (get_property(ctx, "delimiter", delimiter, 2) != 0) {
+return;
+}
+
+if (strlen(delimiter) == 0) {
+printf("Delimiter not specified or it is empty\n");
+return;
+}
+char delim = '\0';
+if (strlen(delimiter) > 0) {
+delim = de
[nifi-minifi-cpp] branch master updated: MINIFICPP-919 - Site2Site transaction CRC checksum randomly fails to match on Windows
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 3d7b771 MINIFICPP-919 - Site2Site transaction CRC checksum randomly
fails to match on Windows
3d7b771 is described below
commit 3d7b7717ac509486663f7a59a27ddca1ef780816
Author: Arpad Boda
AuthorDate: Thu Jun 13 14:53:02 2019 +0200
MINIFICPP-919 - Site2Site transaction CRC checksum randomly fails to match
on Windows
This closes #593.
Signed-off-by: Marc Parisi
---
libminifi/include/sitetosite/SiteToSite.h | 2 +-
libminifi/src/sitetosite/SiteToSiteClient.cpp | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/libminifi/include/sitetosite/SiteToSite.h
b/libminifi/include/sitetosite/SiteToSite.h
index 2cc0823..6ba2dbb 100644
--- a/libminifi/include/sitetosite/SiteToSite.h
+++ b/libminifi/include/sitetosite/SiteToSite.h
@@ -281,7 +281,7 @@ class Transaction {
return _direction;
}
// getCRC
- long getCRC() {
+ uint64_t getCRC() {
return crcStream.getCRC();
}
// updateCRC
diff --git a/libminifi/src/sitetosite/SiteToSiteClient.cpp
b/libminifi/src/sitetosite/SiteToSiteClient.cpp
index 3bc91fe..db145d7 100644
--- a/libminifi/src/sitetosite/SiteToSiteClient.cpp
+++ b/libminifi/src/sitetosite/SiteToSiteClient.cpp
@@ -273,7 +273,7 @@ bool SiteToSiteClient::confirm(std::string transactionID) {
if (code == CONFIRM_TRANSACTION) {
logger_->log_debug("Site2Site transaction %s peer confirm transaction
with CRC %s", transactionID, message);
if (this->_currentVersion > 3) {
-int64_t crcValue = transaction->getCRC();
+uint64_t crcValue = transaction->getCRC();
std::string crc = std::to_string(crcValue);
if (message == crc) {
logger_->log_debug("Site2Site transaction %s CRC matched",
transactionID);
[nifi-minifi-cpp] 02/03: MINIFICPP-917 Update TensorFlow extension to the latest minifi-cpp CMake best practices
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 848408dee4cb32ced5402a5a57ec319c7f29c9f1
Author: Andrew I. Christianson
AuthorDate: Wed Jun 12 15:02:53 2019 -0400
MINIFICPP-917 Update TensorFlow extension to the latest minifi-cpp CMake
best practices
This closes #591.
Signed-off-by: Marc Parisi
---
.gitignore | 2 ++
cmake/FindTensorFlow.cmake | 1 +
extensions/tensorflow/CMakeLists.txt | 27 ++
libminifi/test/tensorflow-tests/CMakeLists.txt | 3 ++-
4 files changed, 7 insertions(+), 26 deletions(-)
diff --git a/.gitignore b/.gitignore
index 32230e7..b135a55 100644
--- a/.gitignore
+++ b/.gitignore
@@ -73,3 +73,5 @@ docker/fedora/minificppsource
/cmake-build-*
.vs/VSWorkspaceState.json
.vs/slnx.sqlite
+/.ccls-cache
+/.vscode
\ No newline at end of file
diff --git a/cmake/FindTensorFlow.cmake b/cmake/FindTensorFlow.cmake
index be1023f..732e3ae 100644
--- a/cmake/FindTensorFlow.cmake
+++ b/cmake/FindTensorFlow.cmake
@@ -29,6 +29,7 @@ find_path(TENSORFLOW_INCLUDE_DIR
third_party
HINTS
${TENSORFLOW_PATH}
+ /usr/include/tensorflow
/usr/local/include/google/tensorflow
/usr/local/include/tensorflow
/usr/include/google/tensorflow)
diff --git a/extensions/tensorflow/CMakeLists.txt
b/extensions/tensorflow/CMakeLists.txt
index 970f6cc..a641a18 100644
--- a/extensions/tensorflow/CMakeLists.txt
+++ b/extensions/tensorflow/CMakeLists.txt
@@ -17,6 +17,8 @@
# under the License.
#
+include(${CMAKE_SOURCE_DIR}/extensions/ExtensionHeader.txt)
+
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -24,8 +26,6 @@ find_package(TensorFlow REQUIRED)
message("-- Found TensorFlow: ${TENSORFLOW_INCLUDE_DIRS}")
-include_directories(../../libminifi/include ../../libminifi/include/core
../../thirdparty/spdlog-20170710/include ../../thirdparty/concurrentqueue
../../thirdparty/yaml-cpp-yaml-cpp-0.5.3/include
${CIVET_THIRDPARTY_ROOT}/include ../../thirdparty/)
-
include_directories(${TENSORFLOW_INCLUDE_DIRS})
file(GLOB SOURCES "*.cpp")
@@ -39,30 +39,7 @@ if(CMAKE_THREAD_LIBS_INIT)
target_link_libraries(minifi-tensorflow-extensions
"${CMAKE_THREAD_LIBS_INIT}")
endif()
-find_package(UUID REQUIRED)
-target_link_libraries(minifi-tensorflow-extensions ${LIBMINIFI}
${UUID_LIBRARIES})
-find_package(OpenSSL REQUIRED)
-include_directories(${OPENSSL_INCLUDE_DIR})
-target_link_libraries(minifi-tensorflow-extensions ${CMAKE_DL_LIBS})
target_link_libraries(minifi-tensorflow-extensions ${TENSORFLOW_LIBRARIES})
-find_package(ZLIB REQUIRED)
-include_directories(${ZLIB_INCLUDE_DIRS})
-target_link_libraries (minifi-tensorflow-extensions ${ZLIB_LIBRARIES})
-
-if (WIN32)
-set_target_properties(minifi-tensorflow-extensions PROPERTIES
-LINK_FLAGS "/WHOLEARCHIVE"
-)
-elseif (APPLE)
-set_target_properties(minifi-tensorflow-extensions PROPERTIES
-LINK_FLAGS "-Wl,-all_load"
-)
-else ()
-set_target_properties(minifi-tensorflow-extensions PROPERTIES
-LINK_FLAGS "-Wl,--whole-archive"
-)
-endif ()
-
SET (TENSORFLOW-EXTENSIONS minifi-tensorflow-extensions PARENT_SCOPE)
diff --git a/libminifi/test/tensorflow-tests/CMakeLists.txt
b/libminifi/test/tensorflow-tests/CMakeLists.txt
index e0b3ffa..906b325 100644
--- a/libminifi/test/tensorflow-tests/CMakeLists.txt
+++ b/libminifi/test/tensorflow-tests/CMakeLists.txt
@@ -26,12 +26,13 @@ FOREACH(testfile ${TENSORFLOW_INTEGRATION_TESTS})
get_filename_component(testfilename "${testfile}" NAME_WE)
add_executable("${testfilename}" "${testfile}" ${SPD_SOURCES}
"${TEST_DIR}/TestBase.cpp")
target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/tensorflow")
+ target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors")
target_include_directories(${testfilename} PRIVATE BEFORE
${TENSORFLOW_INCLUDE_DIRS})
createTests("${testfilename}")
if (APPLE)
target_link_libraries (${testfilename} -Wl,-all_load
minifi-tensorflow-extensions)
else ()
- target_link_libraries (${testfilename} -Wl,--whole-archive
minifi-tensorflow-extensions -Wl,--no-whole-archive)
+ target_link_libraries (${testfilename} -Wl,--whole-archive
minifi-tensorflow-extensions minifi-standard-processors -Wl,--no-whole-archive)
endif ()
MATH(EXPR EXTENSIONS_TEST_COUNT "${EXTENSIONS_TEST_COUNT}+1")
add_test(NAME "${testfilename}" COMMAND "${testfilename}"
WORKING_DIRECTORY ${TEST_DIR})
[nifi-minifi-cpp] 03/03: MINIFICPP-918 Update TensorFlow processors to use property builder
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit dedeeffc3f78fe794b52d2870acf7ba69f3c2a30
Author: Andrew I. Christianson
AuthorDate: Wed Jun 12 16:13:09 2019 -0400
MINIFICPP-918 Update TensorFlow processors to use property builder
This closes #592.
Signed-off-by: Marc Parisi
---
.gitignore | 1 +
extensions/tensorflow/TFApplyGraph.cpp | 24 --
extensions/tensorflow/TFConvertImageToTensor.cpp | 100 +++
3 files changed, 85 insertions(+), 40 deletions(-)
diff --git a/.gitignore b/.gitignore
index b135a55..d014fd5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -56,6 +56,7 @@ libminifi/test/**/*.a
libminifi/include/agent/agent_version.h
docs/generated
thirdparty/apache-rat/apache-rat*
+/compile_commands.json
# Ignore source files that have been placed in the docker directory during
build
docker/minificppsource
diff --git a/extensions/tensorflow/TFApplyGraph.cpp
b/extensions/tensorflow/TFApplyGraph.cpp
index ce16d97..8d0092d 100644
--- a/extensions/tensorflow/TFApplyGraph.cpp
+++ b/extensions/tensorflow/TFApplyGraph.cpp
@@ -16,8 +16,9 @@
*/
#include "TFApplyGraph.h"
-
-#include "tensorflow/cc/ops/standard_ops.h"
+#include
+#include
+#include
namespace org {
namespace apache {
@@ -25,12 +26,19 @@ namespace nifi {
namespace minifi {
namespace processors {
-core::Property TFApplyGraph::InputNode( // NOLINT
-"Input Node",
-"The node of the TensorFlow graph to feed tensor inputs to", "");
-core::Property TFApplyGraph::OutputNode( // NOLINT
-"Output Node",
-"The node of the TensorFlow graph to read tensor outputs from", "");
+core::Property TFApplyGraph::InputNode(
+core::PropertyBuilder::createProperty("Input Node")
+->withDescription(
+"The node of the TensorFlow graph to feed tensor inputs to")
+->withDefaultValue("")
+->build());
+
+core::Property TFApplyGraph::OutputNode(
+core::PropertyBuilder::createProperty("Output Node")
+->withDescription(
+"The node of the TensorFlow graph to read tensor outputs from")
+->withDefaultValue("")
+->build());
core::Relationship TFApplyGraph::Success( // NOLINT
"success",
diff --git a/extensions/tensorflow/TFConvertImageToTensor.cpp
b/extensions/tensorflow/TFConvertImageToTensor.cpp
index b6a1183..5b2ace0 100644
--- a/extensions/tensorflow/TFConvertImageToTensor.cpp
+++ b/extensions/tensorflow/TFConvertImageToTensor.cpp
@@ -16,8 +16,9 @@
*/
#include "TFConvertImageToTensor.h"
-
-#include "tensorflow/cc/ops/standard_ops.h"
+#include
+#include
+#include
namespace org {
namespace apache {
@@ -25,36 +26,71 @@ namespace nifi {
namespace minifi {
namespace processors {
-core::Property TFConvertImageToTensor::ImageFormat( // NOLINT
-"Input Format",
-"The format of the input image (PNG or RAW). RAW is RGB24.", "");
-core::Property TFConvertImageToTensor::InputWidth( // NOLINT
-"Input Width",
-"The width, in pixels, of the input image.", "");
-core::Property TFConvertImageToTensor::InputHeight( // NOLINT
-"Input Height",
-"The height, in pixels, of the input image.", "");
-core::Property TFConvertImageToTensor::OutputWidth( // NOLINT
-"Output Width",
-"The width, in pixels, of the output image.", "");
-core::Property TFConvertImageToTensor::OutputHeight( // NOLINT
-"Output Height",
-"The height, in pixels, of the output image.", "");
-core::Property TFConvertImageToTensor::NumChannels( // NOLINT
-"Channels",
-"The number of channels (e.g. 3 for RGB, 4 for RGBA) in the input image",
"3");
-core::Property TFConvertImageToTensor::CropOffsetX( // NOLINT
-"Crop Offset X",
-"The X (horizontal) offset, in pixels, to crop the input image (relative
to top-left corner).", "");
-core::Property TFConvertImageToTensor::CropOffsetY( // NOLINT
-"Crop Offset Y",
-"The Y (vertical) offset, in pixels, to crop the input image (relative to
top-left corner).", "");
-core::Property TFConvertImageToTensor::CropSizeX( // NOLINT
-"Crop Size X",
-"The X (horizontal) size, in pixels, to crop the input image.", "");
-core::Property TFConvertImageToTensor::CropSizeY( // NOLINT
-"Crop Size Y",
-"The Y (vertical) size, in pixels, to crop the input image.", "
[nifi-minifi-cpp] branch master updated (a88fc76 -> dedeeff)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from a88fc76 MINIFICPP-913: Add configuration item
new e28deb3 MINIFICPP-911 - Added ListSFTP and FetchSFTP processors
new 848408d MINIFICPP-917 Update TensorFlow extension to the latest
minifi-cpp CMake best practices
new dedeeff MINIFICPP-918 Update TensorFlow processors to use property
builder
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore |3 +
NOTICE |1 +
cmake/FindTensorFlow.cmake |1 +
extensions/sftp/CMakeLists.txt |3 +
extensions/sftp/SFTPLoader.h |8 +-
extensions/sftp/client/SFTPClient.cpp | 173 ++-
extensions/sftp/client/SFTPClient.h| 40 +-
extensions/sftp/processors/FetchSFTP.cpp | 273 +
extensions/sftp/processors/FetchSFTP.h | 108 ++
extensions/sftp/processors/ListSFTP.cpp| 1173
extensions/sftp/processors/ListSFTP.h | 218
extensions/sftp/processors/PutSFTP.cpp | 506 ++---
extensions/sftp/processors/PutSFTP.h | 57 +-
extensions/sftp/processors/SFTPProcessorBase.cpp | 475
.../processors/{PutSFTP.h => SFTPProcessorBase.h} | 127 +--
extensions/sftp/tests/CMakeLists.txt |2 +-
extensions/sftp/tests/FetchSFTPTests.cpp | 425 +++
extensions/sftp/tests/ListSFTPTests.cpp| 907 +++
extensions/sftp/tests/ListThenFetchSFTPTests.cpp | 269 +
extensions/sftp/tests/PutSFTPTests.cpp |8 +-
extensions/sftp/tests/tools/SFTPTestServer.cpp |9 +-
extensions/tensorflow/CMakeLists.txt | 27 +-
extensions/tensorflow/TFApplyGraph.cpp | 24 +-
extensions/tensorflow/TFConvertImageToTensor.cpp | 100 +-
libminifi/include/core/Property.h |2 +-
libminifi/include/utils/TimeUtil.h | 22 +-
libminifi/include/utils/file/FileUtils.h | 139 ++-
libminifi/src/core/ConfigurableComponent.cpp | 12 +-
libminifi/test/TestBase.cpp| 52 +-
libminifi/test/TestBase.h |6 +-
libminifi/test/tensorflow-tests/CMakeLists.txt |3 +-
libminifi/test/unit/FileUtilsTests.cpp | 22 +
32 files changed, 4463 insertions(+), 732 deletions(-)
create mode 100644 extensions/sftp/processors/FetchSFTP.cpp
create mode 100644 extensions/sftp/processors/FetchSFTP.h
create mode 100644 extensions/sftp/processors/ListSFTP.cpp
create mode 100644 extensions/sftp/processors/ListSFTP.h
create mode 100644 extensions/sftp/processors/SFTPProcessorBase.cpp
copy extensions/sftp/processors/{PutSFTP.h => SFTPProcessorBase.h} (56%)
create mode 100644 extensions/sftp/tests/FetchSFTPTests.cpp
create mode 100644 extensions/sftp/tests/ListSFTPTests.cpp
create mode 100644 extensions/sftp/tests/ListThenFetchSFTPTests.cpp
[nifi-minifi-cpp] branch master updated: MINIFICPP-913: Add configuration item
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new a88fc76 MINIFICPP-913: Add configuration item
a88fc76 is described below
commit a88fc76ccd2681fa44836d4c8210bc0fa8a22bab
Author: Marc
AuthorDate: Sat Jun 8 21:39:36 2019 -0400
MINIFICPP-913: Add configuration item
This closes #588.
Update documentation to clarify that we should have nifi.c2.flow.base.url
Signed-off-by: Marc Parisi
---
C2.md | 1 +
conf/minifi.properties | 3 +++
2 files changed, 4 insertions(+)
diff --git a/C2.md b/C2.md
index 0fcf1aa..c1b411e 100644
--- a/C2.md
+++ b/C2.md
@@ -73,6 +73,7 @@ an alternate key, but you are encouraged to switch your
configuration options as
# specify the rest URIs if using RESTSender
nifi.c2.rest.url=http://localhost:10080/minifi-c2-api/c2-protocol/heartbeat
nifi.c2.rest.url.ack=http://localhost:10080/minifi-c2-api/c2-protocol/acknowledge
+ nifi.c2.flow.base.url=http://localhost:10080/minifi-c2-api/c2-protocol/
# c2 agent identifier
nifi.c2.agent.identifier=
diff --git a/conf/minifi.properties b/conf/minifi.properties
index 6c4ccd5..46af4f4 100644
--- a/conf/minifi.properties
+++ b/conf/minifi.properties
@@ -46,6 +46,9 @@
nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_reposi
#nifi.c2.agent.protocol.class=CoapProtocol
#nifi.c2.agent.coap.host=
#nifi.c2.agent.coap.port=
+## base URL of the c2 server,
+## very likely the same base url of rest urls
+#nifi.c2.flow.base.url=
#nifi.c2.rest.url=
#nifi.c2.rest.url.ack=
#nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation
[nifi-minifi-cpp] branch master updated: MINIFICPP-914: Incur wait while awaiting minifi shutdown during restart
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 1557d52 MINIFICPP-914: Incur wait while awaiting minifi shutdown
during restart
1557d52 is described below
commit 1557d522eb09e7698a7ab65419b1b06f50193805
Author: Marc Parisi
AuthorDate: Mon Jun 10 13:44:55 2019 -0400
MINIFICPP-914: Incur wait while awaiting minifi shutdown during restart
This closes #589.
Signed-off-by: Marc Parisi
---
bin/minifi.sh | 42 +-
1 file changed, 37 insertions(+), 5 deletions(-)
diff --git a/bin/minifi.sh b/bin/minifi.sh
index 1a62838..68eb0b7 100755
--- a/bin/minifi.sh
+++ b/bin/minifi.sh
@@ -89,6 +89,12 @@ active_pid() {
fi
}
+endnow() {
+ echo "Killing MiNiFi..."
+ kill -9 ${saved_pid} > /dev/null 2>&1
+
+}
+
install() {
detectOS
@@ -250,10 +256,23 @@ case "\$1" in
fi
;;
restart)
-echo Restarting MiNiFi service
-\${bin_dir}/minifi.sh stop
-\${bin_dir}/minifi.sh start
-;;
+ echo "Restarting the MiNiFi service. Hit CTRL+C at any time to forcibly
terminate MiNiFi"
+ ${bin_dir}/minifi.sh stop
+ ticks=1
+ printf "Waiting for process to terminate."
+ trap endnow INT
+ if [ "${saved_pid}" -gt 0 ]; then
+while [ $(active_pid ${saved_pid}) -eq 0 ]; do
+sleep 1
+ticks=$((ticks+1))
+numticks=$(($ticks % 5))
+if [ "${numticks}" -eq 0 ]; then
+printf "."
+fi
+done
+ fi
+ ${bin_dir}/minifi.sh start
+ ;;
*)
echo "Usage: service minifi {start|stop|restart|status}"
;;
@@ -362,8 +381,21 @@ case "$1" in
fi
;;
restart)
- echo Restarting MiNiFi service
+ echo "Restarting MiNiFi. Hit CTRL+C at any time to forcibly terminate
MiNiFi"
${bin_dir}/minifi.sh stop
+ ticks=1
+ printf "Waiting for process to terminate."
+ trap endnow INT
+ if [ "${saved_pid}" -gt 0 ]; then
+ while [ $(active_pid ${saved_pid}) -eq 0 ]; do
+ sleep 1
+ ticks=$((ticks+1))
+ numticks=$(($ticks % 5))
+ if [ "${numticks}" -eq 0 ]; then
+ printf "."
+ fi
+done
+ fi
${bin_dir}/minifi.sh start
;;
install)
[nifi-minifi-cpp] branch master updated: MINIFICPP-902: Attempt improved caching with travis
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new d82cdb4 MINIFICPP-902: Attempt improved caching with travis d82cdb4 is described below commit d82cdb43aca9ca6d68a50e75c6884baa6579bd01 Author: Marc Parisi AuthorDate: Thu May 30 12:29:57 2019 -0400 MINIFICPP-902: Attempt improved caching with travis This closes #576. +1 from committers based on community approval. Signed-off-by: Marc Parisi --- .travis.yml| 30 ++ CMakeLists.txt | 9 + bootstrap.sh | 19 --- 3 files changed, 43 insertions(+), 15 deletions(-) diff --git a/.travis.yml b/.travis.yml index be3c2b4..6c7f316 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,10 +14,11 @@ # limitations under the License. language: cpp -cache: +cache: + ccache: true directories: - - $HOME/.ccache - $HOME/Library/Caches/Homebrew + - $TRAVIS_BUILD_DIR/bison-3.0.5 timeout: 300 env: @@ -54,6 +55,8 @@ matrix: # Establish updated toolchain as default - sudo unlink /usr/bin/gcc && sudo ln -s /usr/bin/gcc-4.8 /usr/bin/gcc - sudo unlink /usr/bin/g++ && sudo ln -s /usr/bin/g++-4.8 /usr/bin/g++ + script: +- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_RULE_MESSAGES=OFF .. && cmake --build . --parallel 4 && sudo make test ARGS="-j2 --output-on-failure" - os: linux dist: trusty sudo: required @@ -82,18 +85,20 @@ matrix: # Establish updated toolchain as default - sudo unlink /usr/bin/gcc && sudo ln -s /usr/bin/gcc-4.8 /usr/bin/gcc - sudo unlink /usr/bin/g++ && sudo ln -s /usr/bin/g++-4.8 /usr/bin/g++ + script: +- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= .. && make -j4 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure" - os: linux name: trusty static build with tests dist: trusty sudo: required script: -- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= .. && make -j2 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure" +- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= .. && make -j4 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure" - os: linux name: xenial shared build with tests dist: xenial sudo: required script: -- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= .. && make -j2 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure" +- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= .. && cmake --build . --parallel 4 && sudo make test ARGS="-j4 --output-on-failure" - os: linux name: debian static package build dist: trusty @@ -101,7 +106,7 @@ matrix: services: - docker script: -- mkdir build && cd build && cmake .. && make debian +- mkdir -p build && cd build && cmake .. && make debian - os: linux name: centos static package biuld dist: trusty @@ -109,7 +114,7 @@ matrix: services: - docker script: -- mkdir build && cd build && cmake .. && make centos +- mkdir -p build && cd build && cmake .. && make centos - os: linux name: fedora static package build dist: trusty @@ -117,7 +122,7 @@ matrix: services: - docker script: -- mkdir build && cd build && cmake .. && make fedora +- mkdir -p build && cd build && cmake .. && make fedora - os: linux name: bionic static package build dist: trusty @@ -125,12 +130,12 @@ matrix: services: - docker script: -- mkdir build && cd build && cmake .. && make u18 +- mkdir -p build && cd build && cmake .. && make u18 - os: linux dist: xenial sudo: required script: -- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=ON .. && make -j2 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure" +- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=ON .. && cmake --build . --parallel 4 && sudo make tes
[nifi-minifi-cpp] branch master updated: Important documentation update
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 6167a34 Important documentation update new 7fc5451 Merge pull request #587 from LopezChris/master 6167a34 is described below commit 6167a3476453ce481b0f01ce0ab9d34767d9365e Author: Christian Lopez AuthorDate: Fri Jun 7 16:47:26 2019 -0700 Important documentation update --- C2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/C2.md b/C2.md index 1ba1eba..0fcf1aa 100644 --- a/C2.md +++ b/C2.md @@ -33,7 +33,7 @@ options defined are located in minifi.properties. ## Description Apache NiFi MiNiFi C++ can communicates with a C2 Server via a number of protocols. These protocols -deliver a C2 response the server, expecting requests in a hearbeat response. The protocol transforms +deliver a C2 response the server, expecting requests in a heartbeat response. The protocol transforms the C2 messages into a protocol specific representation. The internal representation is an AST therefore you must define the root classes, which configure the classes that branch from the root. You can define arbitrary nodes and sub-trees, but this isn't necessary and only advantageous for custom C2 servers. That
[nifi-minifi-cpp] branch MINIFICPP-913 created (now 1e3d87a)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch MINIFICPP-913 in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. at 1e3d87a MINIFICPP-913 No new revisions were added by this update.
[nifi-minifi-cpp] branch master updated: minificpp-773 Implemented Text rendering.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 47b0413 minificpp-773 Implemented Text rendering.
47b0413 is described below
commit 47b041396d127ba08e8651a64f39b9c33fa3d593
Author: amarmer
AuthorDate: Tue Jun 4 14:07:00 2019 -0700
minificpp-773 Implemented Text rendering.
This closes #584.
Signed-off-by: Marc Parisi
---
.../windows-event-log/ConsumeWindowsEventLog.cpp | 131 ++---
.../windows-event-log/ConsumeWindowsEventLog.h | 14 ++-
2 files changed, 124 insertions(+), 21 deletions(-)
diff --git a/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
index afc6d87..fc07e0f 100644
--- a/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
+++ b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
@@ -29,6 +29,7 @@
#include
#include
#include
+#include
#include "io/DataStream.h"
#include "core/ProcessContext.h"
@@ -42,6 +43,10 @@ namespace nifi {
namespace minifi {
namespace processors {
+static std::string to_string(const wchar_t* pChar) {
+ return std::wstring_convert>().to_bytes(pChar);
+}
+
const std::string
ConsumeWindowsEventLog::ProcessorName("ConsumeWindowsEventLog");
core::Property ConsumeWindowsEventLog::Channel(
@@ -60,6 +65,14 @@ core::Property ConsumeWindowsEventLog::Query(
supportsExpressionLanguage(true)->
build());
+core::Property ConsumeWindowsEventLog::RenderFormatXML(
+ core::PropertyBuilder::createProperty("Render Format XML?")->
+ isRequired(true)->
+ withDefaultValue(true)->
+ withDescription("Render format XML or Text.)")->
+ supportsExpressionLanguage(true)->
+ build());
+
core::Property ConsumeWindowsEventLog::MaxBufferSize(
core::PropertyBuilder::createProperty("Max Buffer Size")->
isRequired(true)->
@@ -84,6 +97,9 @@ core::Relationship ConsumeWindowsEventLog::Success("success",
"Relationship for
ConsumeWindowsEventLog::ConsumeWindowsEventLog(const std::string& name,
utils::Identifier uuid)
: core::Processor(name, uuid),
logger_(logging::LoggerFactory::getLogger()) {
+ // Initializes COM for current thread, it is needed to MSXML parser.
+ CoInitializeEx(0, COINIT_APARTMENTTHREADED);
+
char buff[MAX_COMPUTERNAME_LENGTH + 1];
DWORD size = sizeof(buff);
if (GetComputerName(buff, )) {
@@ -93,9 +109,16 @@ ConsumeWindowsEventLog::ConsumeWindowsEventLog(const
std::string& name, utils::I
}
}
+ConsumeWindowsEventLog::~ConsumeWindowsEventLog() {
+ if (xmlDoc_) {
+xmlDoc_.Release();
+ }
+ CoUninitialize();
+}
+
void ConsumeWindowsEventLog::initialize() {
//! Set the supported properties
- setSupportedProperties({Channel, Query, MaxBufferSize,
InactiveDurationToReconnect});
+ setSupportedProperties({Channel, Query, RenderFormatXML, MaxBufferSize,
InactiveDurationToReconnect});
//! Set the supported relationships
setSupportedRelationships({Success});
@@ -121,7 +144,7 @@ void ConsumeWindowsEventLog::onTrigger(const
std::shared_ptr 0) {
lastActivityTimestamp_ = now;
@@ -134,14 +157,80 @@ void ConsumeWindowsEventLog::onTrigger(const
std::shared_ptr )
-{
+void ConsumeWindowsEventLog::createTextOutput(const MSXML2::IXMLDOMElementPtr
pRoot, std::wstringstream& stream, std::vector& ancestors) {
+ const auto pNodeChildren = pRoot->childNodes;
+
+ auto writeAncestors = [](const std::vector& ancestors,
std::wstringstream& stream) {
+for (size_t j = 0; j < ancestors.size() - 1; j++) {
+ stream << ancestors[j] + L'/';
+}
+stream << ancestors.back();
+ };
+
+ if (0 == pNodeChildren->length) {
+writeAncestors(ancestors, stream);
+
+stream << std::endl;
+ } else {
+for (long i = 0; i < pNodeChildren->length; i++) {
+ std::wstringstream curStream;
+
+ const auto pNode = pNodeChildren->item[i];
+
+ const auto nodeType = pNode->GetnodeType();
+
+ if (DOMNodeType::NODE_TEXT == nodeType) {
+const auto nodeValue = pNode->text;
+if (nodeValue.length()) {
+ writeAncestors(ancestors, stream);
+
+ std::wstring strNodeValue = static_cast(nodeValue);
+
+ // Remove '\n', '\r' - just substitute all whitespaces with ' '.
+ strNodeValue = std::regex_replace(strNodeValue,
std::wregex(L"\\s+"), L" ");
+
+ curStream << L" = " << strNodeValue;
+}
+
+stream << curStream.str() << std::endl;
+ } else if (DOMNodeType::NODE_ELEMENT == nodeType) {
+curStream << pNode->nodeName;
+
+const auto pAttributes = pNode->attribu
[nifi-minifi-cpp] 02/03: MINIFICPP-715 - RawSiteToSite: initiateResourceNegotiation functions returns success in case of uknown protocol
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 50091bfbccab140f7051b16c06ec3a7b305899b7
Author: Arpad Boda
AuthorDate: Tue Jun 4 14:59:39 2019 +0200
MINIFICPP-715 - RawSiteToSite: initiateResourceNegotiation functions
returns success in case of uknown protocol
This closes #581.
Signed-off-by: Marc Parisi
---
libminifi/src/sitetosite/RawSocketProtocol.cpp | 18 ++
1 file changed, 6 insertions(+), 12 deletions(-)
diff --git a/libminifi/src/sitetosite/RawSocketProtocol.cpp
b/libminifi/src/sitetosite/RawSocketProtocol.cpp
index 008bf91..4a700c2 100644
--- a/libminifi/src/sitetosite/RawSocketProtocol.cpp
+++ b/libminifi/src/sitetosite/RawSocketProtocol.cpp
@@ -157,18 +157,15 @@ bool RawSiteToSiteClient::initiateResourceNegotiation() {
return initiateResourceNegotiation();
}
}
- ret = -1;
+ logger_->log_error("Site2Site Negotiate protocol failed to find a common
version with server");
return false;
case NEGOTIATED_ABORT:
- logger_->log_warn("Site2Site Negotiate protocol response ABORT");
- ret = -1;
+ logger_->log_error("Site2Site Negotiate protocol response ABORT");
return false;
default:
- logger_->log_warn("Negotiate protocol response unknown code %d",
statusCode);
- return true;
+ logger_->log_error("Negotiate protocol response unknown code %d",
statusCode);
+ return false;
}
-
- return true;
}
bool RawSiteToSiteClient::initiateCodecResourceNegotiation() {
@@ -220,18 +217,15 @@ bool
RawSiteToSiteClient::initiateCodecResourceNegotiation() {
return initiateCodecResourceNegotiation();
}
}
- ret = -1;
+ logger_->log_error("Site2Site Negotiate codec failed to find a common
version with server");
return false;
case NEGOTIATED_ABORT:
logger_->log_error("Site2Site Codec Negotiate response ABORT");
- ret = -1;
return false;
default:
logger_->log_error("Negotiate Codec response unknown code %d",
statusCode);
- return true;
+ return false;
}
-
- return true;
}
bool RawSiteToSiteClient::handShake() {
[nifi-minifi-cpp] 03/03: MINIFICPP-909: Pass version number through CMAKE
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 7c8fe0f7c6dec276099c176882df9e22f709a1a6
Author: Marc Parisi
AuthorDate: Tue Jun 4 12:55:50 2019 -0400
MINIFICPP-909: Pass version number through CMAKE
This closes #583.
Signed-off-by: Marc Parisi
---
cmake/DockerConfig.cmake | 2 +-
docker/DockerVerify.sh | 3 ++-
docker/test/integration/minifi/__init__.py | 8
docker/test/{ => integration}/test_hash_content.py | 3 +--
4 files changed, 4 insertions(+), 12 deletions(-)
diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake
index b1ca54d..6f70e6c 100644
--- a/cmake/DockerConfig.cmake
+++ b/cmake/DockerConfig.cmake
@@ -52,4 +52,4 @@ add_custom_target(
add_custom_target(
docker-verify
-COMMAND ${CMAKE_SOURCE_DIR}/docker/DockerVerify.sh)
+COMMAND ${CMAKE_SOURCE_DIR}/docker/DockerVerify.sh
${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH})
diff --git a/docker/DockerVerify.sh b/docker/DockerVerify.sh
index ae48e33..592de3c 100755
--- a/docker/DockerVerify.sh
+++ b/docker/DockerVerify.sh
@@ -19,6 +19,8 @@ set -e
docker_dir="$( cd ${0%/*} && pwd )"
+export MINIFI_VERSION=$1
+
# Create virutal environment for testing
if [[ ! -d ./test-env-py3 ]]; then
echo "Creating virtual environment in ./test-env-py3" 1>&2
@@ -49,7 +51,6 @@ pip install --upgrade \
export JAVA_HOME="/usr/lib/jvm/default-jvm"
export PATH="$PATH:/usr/lib/jvm/default-jvm/bin"
-export MINIFI_VERSION=0.6.0
export PYTHONPATH="${PYTHONPATH}:${docker_dir}/test/integration"
exec pytest -s -v "${docker_dir}"/test/integration
diff --git a/docker/test/integration/minifi/__init__.py
b/docker/test/integration/minifi/__init__.py
index a13ba2f..d8b280b 100644
--- a/docker/test/integration/minifi/__init__.py
+++ b/docker/test/integration/minifi/__init__.py
@@ -421,20 +421,12 @@ class DebugFlow(Processor):
def __init__(self, ):
super(DebugFlow, self).__init__('DebugFlow')
-class HashAttribute(Processor):
-def __init__(self, attributename):
-super(HashAttribute, self).__init__('HashAttribute',
- properties={'Hash Value Attribute
Key': attributename},
- auto_terminate=['failure'])
-
class AttributesToJSON(Processor):
def __init__(self, destination, attributes):
super(AttributesToJSON, self).__init__('AttributesToJSON',
properties={'Destination': destination,
'Attributes List': attributes},
schedule={'scheduling period': '0 sec'},
auto_terminate=['failure'])
-
-
class GetFile(Processor):
def __init__(self, input_dir):
super(GetFile, self).__init__('GetFile',
diff --git a/docker/test/test_hash_content.py
b/docker/test/integration/test_hash_content.py
similarity index 89%
rename from docker/test/test_hash_content.py
rename to docker/test/integration/test_hash_content.py
index 5dab499..1482014 100644
--- a/docker/test/test_hash_content.py
+++ b/docker/test/integration/test_hash_content.py
@@ -21,8 +21,7 @@ def test_hash_invoke():
"""
Verify sending using InvokeHTTP to a receiver using ListenHTTP.
"""
-
-invoke_flow = (GetFile('/tmp/input') >> HashAttribute('hash')
+invoke_flow = (GetFile('/tmp/input') >>
Processor(name='HashContent',clazz='HashContent',properties={'Hash Attribute':
'hash'},auto_terminate=['failure'])
>> InvokeHTTP('http://minifi-listen:8080/contentListener',
method='POST'))
listen_flow = ListenHTTP(8080) >> LogAttribute() >>
PutFile('/tmp/output')
[nifi-minifi-cpp] 01/03: MINIFICPP-908 - Fix some logging issues
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 2ff7442dad764283122693f9cb3c0c53f0ba3024
Author: Arpad Boda
AuthorDate: Tue Jun 4 18:16:57 2019 +0200
MINIFICPP-908 - Fix some logging issues
This closes #582.
Signed-off-by: Marc Parisi
---
libminifi/src/ThreadedSchedulingAgent.cpp | 2 +-
libminifi/src/sitetosite/RawSocketProtocol.cpp | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/libminifi/src/ThreadedSchedulingAgent.cpp
b/libminifi/src/ThreadedSchedulingAgent.cpp
index e2da4f6..88b39f3 100644
--- a/libminifi/src/ThreadedSchedulingAgent.cpp
+++ b/libminifi/src/ThreadedSchedulingAgent.cpp
@@ -63,7 +63,7 @@ void
ThreadedSchedulingAgent::schedule(std::shared_ptr processo
}
if (thread_pool_.isRunning(processor->getUUIDStr())) {
-logger_->log_warn("Can not schedule threads for processor %s because there
are existing threads running");
+logger_->log_warn("Can not schedule threads for processor %s because there
are existing threads running", processor->getName());
return;
}
diff --git a/libminifi/src/sitetosite/RawSocketProtocol.cpp
b/libminifi/src/sitetosite/RawSocketProtocol.cpp
index 316397a..008bf91 100644
--- a/libminifi/src/sitetosite/RawSocketProtocol.cpp
+++ b/libminifi/src/sitetosite/RawSocketProtocol.cpp
@@ -384,7 +384,7 @@ bool
RawSiteToSiteClient::getPeerList(std::vector ) {
}
PeerStatus status(std::make_shared(port_id_, host, port, secure),
count, true);
peers.push_back(std::move(status));
- logging::LOG_TRACE(logger_) << "Site2Site Peer host " << host << " port
" << port << " Secure " << secure;
+ logging::LOG_TRACE(logger_) << "Site2Site Peer host " << host << " port
" << port << " Secure " << std::to_string(secure);
}
tearDown();
[nifi-minifi-cpp] branch master updated (969ae92 -> 7c8fe0f)
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from 969ae92 minificpp-873 Implemented.
new 2ff7442 MINIFICPP-908 - Fix some logging issues
new 50091bf MINIFICPP-715 - RawSiteToSite: initiateResourceNegotiation
functions returns success in case of uknown protocol
new 7c8fe0f MINIFICPP-909: Pass version number through CMAKE
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
cmake/DockerConfig.cmake | 2 +-
docker/DockerVerify.sh | 3 ++-
docker/test/integration/minifi/__init__.py | 8
docker/test/{ => integration}/test_hash_content.py | 3 +--
libminifi/src/ThreadedSchedulingAgent.cpp | 2 +-
libminifi/src/sitetosite/RawSocketProtocol.cpp | 20 +++-
6 files changed, 12 insertions(+), 26 deletions(-)
rename docker/test/{ => integration}/test_hash_content.py (89%)
[nifi-minifi-cpp] branch master updated: minificpp-873 Implemented.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 969ae92 minificpp-873 Implemented.
969ae92 is described below
commit 969ae923801ef4385a65089f3181e5d146806ae6
Author: amarmer
AuthorDate: Wed May 29 02:20:54 2019 -0700
minificpp-873 Implemented.
minificpp-873 Code review.
minificpp-873 Refactoring.
minificpp-873 Very small modification.
This closes #573.
Signed-off-by: Marc Parisi
---
.../CollectorInitiatedSubscription.cpp | 713 +
.../CollectorInitiatedSubscription.h | 112
extensions/windows-event-log/SupportedProperty.h | 108
3 files changed, 933 insertions(+)
diff --git a/extensions/windows-event-log/CollectorInitiatedSubscription.cpp
b/extensions/windows-event-log/CollectorInitiatedSubscription.cpp
new file mode 100644
index 000..01ca26a
--- /dev/null
+++ b/extensions/windows-event-log/CollectorInitiatedSubscription.cpp
@@ -0,0 +1,713 @@
+/**
+ * @file CollectorInitiatedSubscription.cpp
+ CollectorInitiatedSubscription class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CollectorInitiatedSubscription.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "utils/ScopeGuard.h"
+
+#pragma comment(lib, "wevtapi.lib")
+#pragma comment(lib, "Wecapi.lib")
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+#define LOG_SUBSCRIPTION_ERROR(error) logError(__LINE__, error)
+#define LOG_SUBSCRIPTION_WINDOWS_ERROR(info) logWindowsError(__LINE__, info)
+
+static std::string to_string(const wchar_t* pChar) {
+ return std::wstring_convert>().to_bytes(pChar);
+}
+
+const std::string ProcessorName("CollectorInitiatedSubscription");
+
+//! Supported Relationships
+static core::Relationship s_success("success", "Relationship for successfully
consumed events.");
+
+
+CollectorInitiatedSubscription::CollectorInitiatedSubscription(const
std::string& name, utils::Identifier uuid)
+ : core::Processor(name, uuid),
logger_(logging::LoggerFactory::getLogger()) {
+ char buff[MAX_COMPUTERNAME_LENGTH + 1];
+ DWORD size = sizeof(buff);
+ if (GetComputerName(buff, )) {
+computerName_ = buff;
+ }
+ else {
+LOG_SUBSCRIPTION_WINDOWS_ERROR("GetComputerName");
+ }
+
+ supportedProperties_ = {
+subscriptionName_ = {
+ core::PropertyBuilder::createProperty("Subscription Name")->
+ isRequired(true)->
+ withDescription("The name of the subscription. The value provided for
this parameter should be unique within the computer's scope.")->
+ supportsExpressionLanguage(true)->
+ build()
+},
+
+subscriptionDescription_ = {
+ core::PropertyBuilder::createProperty("Subscription Description")->
+ isRequired(true)->
+ withDescription("A description of the subscription.")->
+ supportsExpressionLanguage(true)->
+ build()
+},
+
+sourceAddress_ = {
+ core::PropertyBuilder::createProperty("Source Address")->
+ isRequired(true)->
+ withDescription("The IP address or fully qualified domain name (FQDN) of
the local or remote computer (event source) from which the events are
collected.")->
+ supportsExpressionLanguage(true)->
+ build()
+},
+
+sourceUserName_ = {
+ core::PropertyBuilder::createProperty("Source User Name")->
+ isRequired(true)->
+ withDescription("The user name, which is used by the remote computer
(event source) to authenticate the user.")->
+ supportsExpressionLanguage(true)->
+ build()
+},
+
+source
[nifi-minifi-cpp] branch master updated: MINIFICPP-885: Minor change to remove warning
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new d9912ff MINIFICPP-885: Minor change to remove warning
d9912ff is described below
commit d9912ff62f202cad0b6c9ca2fe44227585e8228b
Author: Marc Parisi
AuthorDate: Thu May 30 12:27:10 2019 -0400
MINIFICPP-885: Minor change to remove warning
This closes #575.
Signed-off-by: Marc Parisi
---
libminifi/include/utils/TimeUtil.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/libminifi/include/utils/TimeUtil.h
b/libminifi/include/utils/TimeUtil.h
index cdf2194..a20059a 100644
--- a/libminifi/include/utils/TimeUtil.h
+++ b/libminifi/include/utils/TimeUtil.h
@@ -83,7 +83,8 @@ inline int64_t pareDateTimeStr(const std::string ) {
if (sscanf(str.c_str(), "%4u-%2hhu-%2hhuT%2hhu:%2hhu:%2hhuZ%n", ,
, , , , , ) != 6) {
return -1;
}
- if (read != str.size()) {
+ // while it is unlikely that read will be < 0, the conditional adds little
cost for a little defensiveness.
+ if (read < 0 || static_cast(read) != str.size()) {
return -1;
}
[nifi-minifi-cpp] 02/02: MINIFICPP-57: Add build profiles
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit f22903e47fa0f28e0a203dc0846fe70cbedc8b5e
Author: Marc Parisi
AuthorDate: Fri May 31 06:38:28 2019 -0400
MINIFICPP-57: Add build profiles
This closes #579.
Signed-off-by: Marc Parisi
---
bootstrap.sh | 6 -
bstrp_functions.sh | 65 ++
2 files changed, 70 insertions(+), 1 deletion(-)
diff --git a/bootstrap.sh b/bootstrap.sh
index fd4dc3f..e80603d 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -294,8 +294,10 @@ TESTS_DISABLED=${FALSE}
add_disabled_option SQLITE_ENABLED ${FALSE} "ENABLE_SQLITE"
-USE_SHARED_LIBS=${TRUE}
+USE_SHARED_LIBS=${TRUE}
+## name, default, values
+add_multi_option BUILD_PROFILE "RelWithDebInfo" "RelWithDebInfo" "Debug"
"MinSizeRel" "Release"
# Since the following extensions have limitations on
@@ -452,6 +454,8 @@ build_cmake_command(){
fi
CMAKE_BUILD_COMMAND="${CMAKE_BUILD_COMMAND}
-DBUILD_IDENTIFIER=${BUILD_IDENTIFIER}"
+
+CMAKE_BUILD_COMMAND="${CMAKE_BUILD_COMMAND}
-DCMAKE_BUILD_TYPE=${BUILD_PROFILE}"
add_os_flags
diff --git a/bstrp_functions.sh b/bstrp_functions.sh
index da226d7..4f34fc9 100755
--- a/bstrp_functions.sh
+++ b/bstrp_functions.sh
@@ -49,6 +49,67 @@ add_disabled_option(){
fi
}
+add_multi_option(){
+ eval "$1=$2"
+ ARRAY=()
+ eval "export $1_OPTIONS=()"
+ for i in "${@:3}"; do
+ARRAY+=($i)
+ done
+ for i in ${!ARRAY[@]}; do
+eval "$1_OPTIONS[$i]=${ARRAY[$i]}"
+ done
+}
+
+print_multi_option_status(){
+ feature="$1"
+ feature_status=${!1}
+ declare -a VAR_OPTS=()
+
+ declare VAR_OPTS=$1_OPTIONS[@]
+ VAR_OPTS=$1_OPTIONS[@]
+
+ for option in "${!VAR_OPTS}" ; do
+if [ "${option}" = "$feature_status" ]; then
+ printf "${RED}"
+fi
+printf "${option}"
+printf "${NO_COLOR} "
+ done
+}
+
+ToggleMultiOption(){
+ feature="$1"
+ feature_status=${!1}
+ declare -a VAR_OPTS=()
+
+ declare VAR_OPTS=$1_OPTIONS[@]
+ #echo -e "${RED}${feature_status}${NO_COLOR} (${VAR_OPTS_VAL})"
+ VAR_OPTS=$1_OPTIONS[@]
+ invariant=""
+ first=""
+ # the alternative is to loop through an array but since we're an indirected
+ # copy, we'll treat this as a manual circular buffer
+ for option in "${!VAR_OPTS}" ; do
+ if [ -z "${first}" ]; then
+ first=${option}
+ fi
+ if [ "${invariant}" = "next" ]; then
+ eval "$1=${option}"
+ invariant=""
+ break
+fi
+if [ "${option}" = "$feature_status" ]; then
+ invariant="next"
+fi
+ done
+ if [ "${invariant}" = "next" ]; then
+ eval "$1=${first}"
+ fi
+}
+
+
+
add_dependency(){
DEPENDENCIES+=("$1:$2")
}
@@ -93,6 +154,7 @@ save_state(){
echo_state_variable BUILD_IDENTIFIER
echo_state_variable BUILD_DIR
echo_state_variable TESTS_DISABLED
+ echo_state_variable BUILD_PROFILE
echo_state_variable USE_SHARED_LIBS
for option in "${OPTIONS[@]}" ; do
echo_state_variable $option
@@ -186,6 +248,7 @@ print_feature_status(){
}
+
show_main_menu() {
clear
echo ""
@@ -269,6 +332,7 @@ show_supported_features() {
echo "2. Enable all extensions"
echo "3. Enable JNI Support ..$(print_feature_status JNI_ENABLED)"
echo "4. Use Shared Dependency Links .$(print_feature_status
USE_SHARED_LIBS)"
+ echo "5. Build Profile ...$(print_multi_option_status
BUILD_PROFILE)"
echo "P. Continue with these options"
if [ "$GUIDED_INSTALL" = "${TRUE}" ]; then
echo "R. Return to Main Menu"
@@ -315,6 +379,7 @@ read_feature_options(){
echo -e "${RED}Python support must be disabled before changing this
value...${NO_COLOR}" && sleep 2
fi
;;
+5) ToggleMultiOption BUILD_PROFILE;;
p) FEATURES_SELECTED="true" ;;
r) if [ "$GUIDED_INSTALL" = "${TRUE}" ]; then
MENU="main"
[nifi-minifi-cpp] branch master updated (6da475b -> f22903e)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 6da475b MINIFICPP-898 - Listen port validator is broken new 7cb81f1 MINIFICPP-792 - TailFile processor should handle rotation of source file new f22903e MINIFICPP-57: Add build profiles The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: bootstrap.sh | 6 +- bstrp_functions.sh | 65 .../standard-processors/processors/TailFile.cpp| 134 ++- .../standard-processors/processors/TailFile.h | 2 + .../tests/unit/TailFileTests.cpp | 179 - 5 files changed, 297 insertions(+), 89 deletions(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-792 - TailFile processor should handle rotation of source file
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 7cb81f17982f3b408be5ba34c0bf07113b68d915
Author: Arpad Boda
AuthorDate: Thu Mar 28 12:13:52 2019 +0100
MINIFICPP-792 - TailFile processor should handle rotation of source file
---
.../standard-processors/processors/TailFile.cpp| 134 ++-
.../standard-processors/processors/TailFile.h | 2 +
.../tests/unit/TailFileTests.cpp | 179 -
3 files changed, 227 insertions(+), 88 deletions(-)
diff --git a/extensions/standard-processors/processors/TailFile.cpp
b/extensions/standard-processors/processors/TailFile.cpp
index 71314b5..8302aeb 100644
--- a/extensions/standard-processors/processors/TailFile.cpp
+++ b/extensions/standard-processors/processors/TailFile.cpp
@@ -36,6 +36,7 @@
#include
#include
#include
+#include "utils/file/FileUtils.h"
#include "utils/TimeUtil.h"
#include "utils/StringUtils.h"
#include "TailFile.h"
@@ -164,90 +165,57 @@ void TailFile::storeState() {
static bool sortTailMatchedFileItem(TailMatchedFileItem i, TailMatchedFileItem
j) {
return (i.modifiedTime < j.modifiedTime);
}
-void TailFile::checkRollOver(const std::string , const
std::string ) {
+void TailFile::checkRollOver(const std::string , const
std::string ) {
struct stat statbuf;
std::vector matchedFiles;
std::string fullPath = fileLocation + "/" + _currentTailFileName;
if (stat(fullPath.c_str(), ) == 0) {
-if (statbuf.st_size > this->_currentTailFilePosition)
- // there are new input for the current tail file
- return;
-
-uint64_t modifiedTimeCurrentTailFile = ((uint64_t) (statbuf.st_mtime) *
1000);
-std::string pattern = fileName;
-std::size_t found = fileName.find_last_of(".");
+logger_->log_trace("Searching for files rolled over");
+std::string pattern = baseFileName;
+std::size_t found = baseFileName.find_last_of(".");
if (found != std::string::npos)
- pattern = fileName.substr(0, found);
-#ifndef WIN32
-DIR *d;
-d = opendir(fileLocation.c_str());
-if (!d)
- return;
-while (1) {
- struct dirent *entry;
- entry = readdir(d);
- if (!entry)
-break;
- std::string d_name = entry->d_name;
- if (!(entry->d_type & DT_DIR)) {
-std::string fileName = d_name;
-std::string fileFullName = fileLocation + "/" + d_name;
-if (fileFullName.find(pattern) != std::string::npos &&
stat(fileFullName.c_str(), ) == 0) {
- if (((uint64_t) (statbuf.st_mtime) * 1000) >=
modifiedTimeCurrentTailFile) {
-TailMatchedFileItem item;
-item.fileName = fileName;
-item.modifiedTime = ((uint64_t) (statbuf.st_mtime) * 1000);
-matchedFiles.push_back(item);
+ pattern = baseFileName.substr(0, found);
+
+// Callback, called for each file entry in the listed directory
+// Return value is used to break (false) or continue (true) listing
+auto lambda = [&](const std::string& path, const std::string& filename) ->
bool {
+ struct stat sb;
+ std::string fileFullName = path +
utils::file::FileUtils::get_separator() + filename;
+ if ((fileFullName.find(pattern) != std::string::npos) &&
stat(fileFullName.c_str(), ) == 0) {
+uint64_t candidateModTime = ((uint64_t) (sb.st_mtime) * 1000);
+if (candidateModTime >= _currentTailFileModificationTime) {
+ if (filename == _currentTailFileName && candidateModTime ==
_currentTailFileModificationTime &&
+ sb.st_size == _currentTailFilePosition) {
+return true; // Skip the current file as a candidate in case it
wasn't updated
}
+ TailMatchedFileItem item;
+ item.fileName = filename;
+ item.modifiedTime = ((uint64_t) (sb.st_mtime) * 1000);
+ matchedFiles.push_back(item);
}
}
-}
-closedir(d);
-#else
-
-HANDLE hFind;
-WIN32_FIND_DATA FindFileData;
-
-if ((hFind = FindFirstFile(fileLocation.c_str(), )) !=
INVALID_HANDLE_VALUE) {
- do {
-struct stat statbuf {};
-if (stat(FindFileData.cFileName, ) != 0) {
- logger_->log_warn("Failed to stat %s", FindFileData.cFileName);
- break;
-}
+ return true;
+};
-std::string fileFullName = fileLocation + "/" + FindFileData.cFileName;
+utils::file::FileUtils::list_dir(fileLocation, lambda, logger_, false);
-if (fileFullName.find(pattern) != std::string::npos &&
stat(fileFullName.c_str(), ) == 0) {
- if (((uint64_t)(statbuf.st_mtime) * 1000) >=
modifiedTimeCurrentTai
[nifi-minifi-cpp] branch master updated: MINIFICPP-898 - Listen port validator is broken
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6da475b MINIFICPP-898 - Listen port validator is broken
6da475b is described below
commit 6da475b935a27c606a7ab49c7aa1d3ec253b56d5
Author: Arpad Boda
AuthorDate: Wed May 29 18:01:29 2019 +0200
MINIFICPP-898 - Listen port validator is broken
This closes #574.
Signed-off-by: Marc Parisi
---
libminifi/include/core/PropertyValidation.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libminifi/include/core/PropertyValidation.h
b/libminifi/include/core/PropertyValidation.h
index 2df5218..b0e59e9 100644
--- a/libminifi/include/core/PropertyValidation.h
+++ b/libminifi/include/core/PropertyValidation.h
@@ -342,7 +342,7 @@ class StandardValidators {
}
static const std::shared_ptr LISTEN_PORT_VALIDATOR(){
-static std::shared_ptr validator =
std::make_shared("LISTEN_PORT_VALIDATOR");
+static std::shared_ptr validator =
std::make_shared("PORT_VALIDATOR");
return validator;
}
[nifi-minifi-cpp] branch master updated: MINIFICPP-884 - Fix LogAttribute logging multiple FlowFiles
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new e1866d7 MINIFICPP-884 - Fix LogAttribute logging multiple FlowFiles
e1866d7 is described below
commit e1866d7ea16533465406f51cf9e027a26767ac77
Author: Daniel Bakai
AuthorDate: Tue May 28 16:36:40 2019 +0200
MINIFICPP-884 - Fix LogAttribute logging multiple FlowFiles
This closes #572.
Signed-off-by: Marc Parisi
---
extensions/standard-processors/processors/LogAttribute.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/extensions/standard-processors/processors/LogAttribute.cpp
b/extensions/standard-processors/processors/LogAttribute.cpp
index 2309d0e..6ffa397 100644
--- a/extensions/standard-processors/processors/LogAttribute.cpp
+++ b/extensions/standard-processors/processors/LogAttribute.cpp
@@ -94,7 +94,6 @@ void LogAttribute::onTrigger(const
std::shared_ptr
std::string dashLine = "--";
LogAttrLevel level = LogAttrLevelInfo;
bool logPayload = false;
- std::ostringstream message;
uint64_t i = 0;
const auto max = flowfiles_to_log_ == 0 ? UINT64_MAX : flowfiles_to_log_;
@@ -115,6 +114,7 @@ void LogAttribute::onTrigger(const
std::shared_ptr
context->getProperty(LogPayload.getName(), logPayload);
+std::ostringstream message;
message << "Logging for flow file " << "\n";
message << dashLine;
message << "\nStandard FlowFile Attributes";
@@ -146,7 +146,7 @@ void LogAttribute::onTrigger(const
std::shared_ptr
}
}
}
-message << "\n" << dashLine << std::ends;
+message << "\n" << dashLine;
std::string output = message.str();
switch (level) {
[nifi-minifi-cpp] branch master updated: MINIFICPP-880: Change menu item
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new efece6f MINIFICPP-880: Change menu item
efece6f is described below
commit efece6fecce0f8de9ca2c82175bd7df0cf3b720d
Author: Marc Parisi
AuthorDate: Fri May 24 12:01:42 2019 -0400
MINIFICPP-880: Change menu item
This closes #570.
Signed-off-by: Marc Parisi
---
bstrp_functions.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bstrp_functions.sh b/bstrp_functions.sh
index 1f194d5..da226d7 100755
--- a/bstrp_functions.sh
+++ b/bstrp_functions.sh
@@ -280,7 +280,7 @@ show_supported_features() {
read_feature_options(){
local choice
- read -p "Enter choice [ A - T or 1-4 ] " choice
+ read -p "Enter choice [ A - V or 1-4 ] " choice
choice=$(echo ${choice} | tr '[:upper:]' '[:lower:]')
case $choice in
a) ToggleFeature ROCKSDB_ENABLED ;;
@@ -321,7 +321,7 @@ read_feature_options(){
fi
;;
q) exit 0;;
-*) echo -e "${RED}Please enter an option A-T or 1-4...${NO_COLOR}" &&
sleep 2
+*) echo -e "${RED}Please enter an option A-V or 1-4...${NO_COLOR}" &&
sleep 2
esac
}
[nifi-minifi-cpp] branch master updated: MINIFICPP-853: Minor updates to readmes
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 81eb50a MINIFICPP-853: Minor updates to readmes 81eb50a is described below commit 81eb50a333f42049af5e2636c392f41bc9e1b3f6 Author: Marc Parisi AuthorDate: Wed May 22 06:40:23 2019 -0400 MINIFICPP-853: Minor updates to readmes This closes #566. Approved on GH by arpadboda. Signed-off-by: Marc Parisi --- CONFIGURE.md | 7 --- OPS.md | 8 +--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/CONFIGURE.md b/CONFIGURE.md index f4151b6..bf76ce3 100644 --- a/CONFIGURE.md +++ b/CONFIGURE.md @@ -298,9 +298,10 @@ The MQTTController Service can be configured for MQTT connectivity and provide t In the configuration below there are two classes defined under "NetworkPrioritizerService", one class "NetworkPrioritizerService2" defines en0, and en1. If en0 is down at any point, then en1 will be given priority before resorting to en2 and en3 of "NetworkPrioritizerService3". If the throughput for "NetworkPrioritizerService2" exceeds the defined throughput or the max payload of 1024, then "NetworkPrioritizerService3" will be used. If Max Payload and - Max Throughput are not defined, then they will not be limiting factors. As of release, 0.5.0, Max Payload will only be used for processors that custom - implement that feature. RPGs will not support max payloads until 0.6.0. Additionally, since connection queues aren't prioritized, you must have a live connection - for your data to send it. Since connection queues can't be re-prioritized, this can create a starvation problem. The configuration is required to account for this. + Max Throughput are not defined, then they will not be limiting factors. + + Since connection queues can't be re-prioritized, this can create a starvation problem if no connection is available. + The configuration is required to account for this. Controller Services: - name: NetworkPrioritizerService diff --git a/OPS.md b/OPS.md index 923de64..9cce9f5 100644 --- a/OPS.md +++ b/OPS.md @@ -26,13 +26,15 @@ This readme defines operational commands for managing instances. ## Description -Apache NiFi MiNiFi C++ can be managed through our [C2 protocol](https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal) +Apache NiFi MiNiFi C++ can be managed through our [C2 protocol](https://cwiki.apache.org/confluence/display/MINIFI/C2+Design) or through a local interface called the MiNiFi Controller. This feature is disabled by default, and requires that C2 be enabled -and configured with an agent class before using the MiNiFi controller features outlined here. +and configured with an agent class before using the MiNiFi controller features outlined here. MiNiFi controller is an example +implementation of our C2 protocol. This featureset is not intended to replace your Command and Control implementation. Instead +it is meant to provide testing and minimal operational capabilities by example. ## Managing MiNiFi -The MiNiFi controller is an executable in the bin directory that can be used to control the MiNiFi C++ agent while it runs -- utilizing the [Command and Control Protocol](https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal). Currently the controller will let you stop subcomponents within a running instance, clear queues, get the status of queues, and update the flow for a warm re-deploy. +The MiNiFi controller is an executable in the bin directory that can be used to control the MiNiFi C++ agent while it runs -- utilizing the [Command and Control Protocol](https://cwiki.apache.org/confluence/display/MINIFI/C2+Design). Currently the controller will let you stop subcomponents within a running instance, clear queues, get the status of queues, and update the flow for a warm re-deploy. The minificontroller can track a single MiNiFi C++ agent through the use of three options. Port is required. The hostname is not and will default to localhost. Additionally, controller.socket.local.any.interface allows
[nifi-minifi-cpp] branch master updated: MINIFICPP-865 - Statically link AWS SDK to libminifi
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 0601747 MINIFICPP-865 - Statically link AWS SDK to libminifi
0601747 is described below
commit 060174725a9a4345bca724907a1ba16936a62d06
Author: Arpad Boda
AuthorDate: Thu May 23 18:24:51 2019 +0200
MINIFICPP-865 - Statically link AWS SDK to libminifi
This closes #569.
Signed-off-by: Marc Parisi
---
extensions/aws/CMakeLists.txt | 49 ++-
1 file changed, 35 insertions(+), 14 deletions(-)
diff --git a/extensions/aws/CMakeLists.txt b/extensions/aws/CMakeLists.txt
index e0b0b07..dc354e2 100644
--- a/extensions/aws/CMakeLists.txt
+++ b/extensions/aws/CMakeLists.txt
@@ -25,6 +25,15 @@ file(GLOB SOURCES "*.cpp" "s3/*.cpp"
"controllerservices/*.cpp")
set(BASE_DIR "${CMAKE_CURRENT_BINARY_DIR}/aws-sdk-cpp")
set(BYPRODUCT "${CMAKE_CURRENT_BINARY_DIR}/aws-sdk-install/")
+
+if(STATIC_BUILD)
+set(MINIFI_AWS_BUILD_SHARED_LIBS "OFF")
+set(MINIFI_AWS_UNITY_BUILD "ON")
+else()
+set(MINIFI_AWS_BUILD_SHARED_LIBS "ON")
+set(MINIFI_AWS_UNITY_BUILD "OFF")
+endif()
+
# AWS SDK 1.7.89
ExternalProject_Add(
awssdk-external
@@ -33,10 +42,11 @@ ExternalProject_Add(
EXCLUDE_FROM_ALL TRUE
INSTALL_DIR ${BYPRODUCT}
CMAKE_ARGS -DBUILD_ONLY=s3
--DENABLE_TESTING=OFF
--DBUILD_SHARED_LIBS=ON
--DCMAKE_BUILD_TYPE=RelWithDebInfo
--DCMAKE_INSTALL_PREFIX=${BYPRODUCT}
+-DENABLE_TESTING=OFF
+-DBUILD_SHARED_LIBS=${MINIFI_AWS_BUILD_SHARED_LIBS}
+-DENABLE_UNITY_BUILD=${MINIFI_AWS_UNITY_BUILD}
+-DCMAKE_BUILD_TYPE=RelWithDebInfo
+-DCMAKE_INSTALL_PREFIX=${BYPRODUCT}
)
add_library(awssdklib STATIC IMPORTED)
@@ -50,17 +60,28 @@ set(AWSSDK_LIBRARY awssdklib CACHE STRING "" FORCE)
include_directories(${AWSSDK_INCLUDE_DIRS})
add_library(minifi-aws STATIC ${SOURCES})
+if(STATIC_BUILD)
+set(SUFFIX "a")
+else()
+if(APPLE)
+set(SUFFIX "dylib")
+else()
+set(SUFFIX "so")
+endif()
+endif()
+
+
if (APPLE)
-target_link_libraries (minifi-aws
${BYPRODUCT}lib/libaws-c-event-stream.dylib)
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-c-common.dylib)
-target_link_libraries (minifi-aws
${BYPRODUCT}lib/libaws-cpp-sdk-core.dylib)
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-cpp-sdk-s3.dylib)
-else ()
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-c-event-stream.so)
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-c-common.so)
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-cpp-sdk-core.so)
-target_link_libraries (minifi-aws ${BYPRODUCT}lib/libaws-cpp-sdk-s3.so)
-endif ()
+if(STATIC_BUILD)
+target_link_libraries(minifi-aws "-framework CoreFoundation")
+endif()
+endif()
+
+target_link_libraries(minifi-aws ${BYPRODUCT}lib/libaws-c-common.${SUFFIX})
+target_link_libraries(minifi-aws ${BYPRODUCT}lib/libaws-checksums.${SUFFIX})
+target_link_libraries(minifi-aws
${BYPRODUCT}lib/libaws-c-event-stream.${SUFFIX})
+target_link_libraries(minifi-aws ${BYPRODUCT}lib/libaws-cpp-sdk-core.${SUFFIX})
+target_link_libraries(minifi-aws ${BYPRODUCT}lib/libaws-cpp-sdk-s3.${SUFFIX})
add_dependencies(minifi-aws awssdk-external)
SET (AWS-EXTENSION minifi-aws PARENT_SCOPE)
[nifi-minifi-cpp] branch master updated: MINIFICPP-877 - Add initializers to ObjectFactories
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 398f17b MINIFICPP-877 - Add initializers to ObjectFactories
398f17b is described below
commit 398f17b4bfe1755ed02ae672b741aa2928c0d12a
Author: Daniel Bakai
AuthorDate: Wed May 22 18:31:18 2019 +0200
MINIFICPP-877 - Add initializers to ObjectFactories
This closes #567.
Signed-off-by: Marc Parisi
---
extensions/http-curl/HTTPCurlLoader.h| 15 +++
extensions/http-curl/client/HTTPClient.cpp | 6 -
extensions/http-curl/client/HTTPClient.h | 21 ---
extensions/http-curl/processors/InvokeHTTP.h | 2 --
libminifi/include/core/ClassLoader.h | 40
libminifi/src/core/ClassLoader.cpp | 9 +++
libminifi/test/TestBase.h| 1 +
7 files changed, 65 insertions(+), 29 deletions(-)
diff --git a/extensions/http-curl/HTTPCurlLoader.h
b/extensions/http-curl/HTTPCurlLoader.h
index d01f822..3d24cd4 100644
--- a/extensions/http-curl/HTTPCurlLoader.h
+++ b/extensions/http-curl/HTTPCurlLoader.h
@@ -35,6 +35,17 @@
#include "sitetosite/HTTPProtocol.h"
#include "utils/StringUtils.h"
+class HttpCurlObjectFactoryInitializer : public core::ObjectFactoryInitializer
{
+ public:
+ virtual bool initialize() {
+return curl_global_init(CURL_GLOBAL_DEFAULT) == CURLE_OK;
+ }
+
+ virtual void deinitialize() {
+curl_global_cleanup();
+ }
+};
+
class HttpCurlObjectFactory : public core::ObjectFactory {
public:
HttpCurlObjectFactory() {
@@ -80,6 +91,10 @@ class HttpCurlObjectFactory : public core::ObjectFactory {
}
}
+ virtual std::unique_ptr getInitializer() {
+return std::unique_ptr(new
HttpCurlObjectFactoryInitializer());
+ }
+
static bool added;
};
diff --git a/extensions/http-curl/client/HTTPClient.cpp
b/extensions/http-curl/client/HTTPClient.cpp
index 6ef19f2..f2be475 100644
--- a/extensions/http-curl/client/HTTPClient.cpp
+++ b/extensions/http-curl/client/HTTPClient.cpp
@@ -47,8 +47,6 @@ HTTPClient::HTTPClient(const std::string , const
std::shared_ptr::getLogger()) {
- HTTPClientInitializer *initializer = HTTPClientInitializer::getInstance();
- initializer->initialize();
http_session_ = curl_easy_init();
}
@@ -69,8 +67,6 @@ HTTPClient::HTTPClient(std::string name, utils::Identifier
uuid)
keep_alive_probe_(-1),
keep_alive_idle_(-1),
logger_(logging::LoggerFactory::getLogger()) {
- HTTPClientInitializer *initializer = HTTPClientInitializer::getInstance();
- initializer->initialize();
http_session_ = curl_easy_init();
}
@@ -91,8 +87,6 @@ HTTPClient::HTTPClient()
keep_alive_probe_(-1),
keep_alive_idle_(-1),
logger_(logging::LoggerFactory::getLogger()) {
- HTTPClientInitializer *initializer = HTTPClientInitializer::getInstance();
- initializer->initialize();
http_session_ = curl_easy_init();
}
diff --git a/extensions/http-curl/client/HTTPClient.h
b/extensions/http-curl/client/HTTPClient.h
index 84cb1a6..d27a688 100644
--- a/extensions/http-curl/client/HTTPClient.h
+++ b/extensions/http-curl/client/HTTPClient.h
@@ -46,27 +46,6 @@ namespace minifi {
namespace utils {
/**
- * Purpose and Justification: Initializes and cleans up curl once. Cleanup
will only occur at the end of our execution since we are relying on a static
variable.
- */
-class HTTPClientInitializer {
- public:
- static HTTPClientInitializer *getInstance() {
-static HTTPClientInitializer initializer;
-return
- }
- void initialize() {
-
- }
- private:
- ~HTTPClientInitializer() {
-curl_global_cleanup();
- }
- HTTPClientInitializer() {
-curl_global_init(CURL_GLOBAL_DEFAULT);
- }
-};
-
-/**
* Purpose and Justification: Pull the basics for an HTTPClient into a self
contained class. Simply provide
* the URL and an SSLContextService ( can be null).
*
diff --git a/extensions/http-curl/processors/InvokeHTTP.h
b/extensions/http-curl/processors/InvokeHTTP.h
index 3da3c75..7c62aa2 100644
--- a/extensions/http-curl/processors/InvokeHTTP.h
+++ b/extensions/http-curl/processors/InvokeHTTP.h
@@ -60,8 +60,6 @@ class InvokeHTTP : public core::Processor {
penalize_no_retry_(false),
disable_peer_verification_(false),
logger_(logging::LoggerFactory::getLogger()) {
-static utils::HTTPClientInitializer *initializer =
utils::HTTPClientInitializer::getInstance();
-initializer->initialize();
}
// Destructor
virtual ~InvokeHTTP();
diff --git a/libminifi/include/core/ClassLoader.h
b/libminifi/include/core/ClassLoader.h
index ad82e71..41d683e 100644
--- a/libminifi/include/core/ClassLoader.h
+++ b/libminifi/include/core/ClassLoader.h
@@ -21,6 +21,7 @@
#include
#include
#incl
[nifi-minifi-cpp] branch master updated: MINIFICPP-878 - Fix AWS build
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6cbcac2 MINIFICPP-878 - Fix AWS build
6cbcac2 is described below
commit 6cbcac28a83109efc41a7b141f8b85ad23892fa6
Author: Arpad Boda
AuthorDate: Thu May 23 14:42:00 2019 +0200
MINIFICPP-878 - Fix AWS build
This closes #568.
Signed-off-by: Marc Parisi
---
bootstrap.sh | 3 +--
extensions/aws/CMakeLists.txt | 2 +-
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/bootstrap.sh b/bootstrap.sh
index fe61f9e..4823df9 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -264,8 +264,7 @@ add_dependency USB_ENABLED "libpng"
add_disabled_option GPS_ENABLED ${FALSE} "ENABLE_GPS"
add_dependency GPS_ENABLED "gpsd"
-MINIFICPP-874
-#add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
+add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
add_disabled_option KAFKA_ENABLED ${FALSE} "ENABLE_LIBRDKAFKA" "3.4.0"
diff --git a/extensions/aws/CMakeLists.txt b/extensions/aws/CMakeLists.txt
index a80f862..e0b0b07 100644
--- a/extensions/aws/CMakeLists.txt
+++ b/extensions/aws/CMakeLists.txt
@@ -29,7 +29,7 @@ set(BYPRODUCT "${CMAKE_CURRENT_BINARY_DIR}/aws-sdk-install/")
ExternalProject_Add(
awssdk-external
GIT_REPOSITORY "https://github.com/aws/aws-sdk-cpp.git;
-GIT_TAG "09b65deba03cfbef9a1e5d5986aa4de71bc03cd8"
+GIT_TAG "6d9437429c28927576cb5c898e490f8c8cf9"
EXCLUDE_FROM_ALL TRUE
INSTALL_DIR ${BYPRODUCT}
CMAKE_ARGS -DBUILD_ONLY=s3
[nifi-minifi-cpp] branch master updated: MINIFICPP-874: Remove AWS from travis
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 4a111b2 MINIFICPP-874: Remove AWS from travis
4a111b2 is described below
commit 4a111b2210afbe93b42b4724e888ccd8baf7a260
Author: Marc Parisi
AuthorDate: Tue May 21 18:42:47 2019 -0400
MINIFICPP-874: Remove AWS from travis
This closes #565.
Approved on GH by jdyer64.
Signed-off-by: Marc Parisi
---
bootstrap.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/bootstrap.sh b/bootstrap.sh
index 4823df9..fe61f9e 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -264,7 +264,8 @@ add_dependency USB_ENABLED "libpng"
add_disabled_option GPS_ENABLED ${FALSE} "ENABLE_GPS"
add_dependency GPS_ENABLED "gpsd"
-add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
+MINIFICPP-874
+#add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
add_disabled_option KAFKA_ENABLED ${FALSE} "ENABLE_LIBRDKAFKA" "3.4.0"
[nifi-minifi-cpp] branch master updated: MINIFICPP-850: Add operating system to c2 response
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 443dee2 MINIFICPP-850: Add operating system to c2 response
443dee2 is described below
commit 443dee281ac14a2774fa6a22e8c2f627cb5a385d
Author: Marc Parisi
AuthorDate: Thu May 16 13:10:17 2019 -0400
MINIFICPP-850: Add operating system to c2 response
This closes #557.
Approved on GH by kevdoran.
Signed-off-by: Marc Parisi
---
.gitignore | 2 +
.../standard-processors/tests/CMakeLists.txt | 2 +
.../tests/unit/ManifestTests.cpp | 27 +-
.../include/core/state/nodes/DeviceInformation.h | 341 +++--
4 files changed, 210 insertions(+), 162 deletions(-)
diff --git a/.gitignore b/.gitignore
index 26d8bfe..32230e7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -31,6 +31,8 @@
nifi-minifi-cpp.kdev4
.kdev4
+.device_id
+
# Filter out generated files from the included libuuid
thirdparty/uuid/tst_uuid*
assemblies
diff --git a/extensions/standard-processors/tests/CMakeLists.txt
b/extensions/standard-processors/tests/CMakeLists.txt
index ba685b2..f87ecf4 100644
--- a/extensions/standard-processors/tests/CMakeLists.txt
+++ b/extensions/standard-processors/tests/CMakeLists.txt
@@ -39,9 +39,11 @@ FOREACH(testfile ${PROCESSOR_UNIT_TESTS})
else ()
target_link_libraries ("${testfilename}" -Wl,--whole-archive
${ZLIB_LIBRARY} ${OPENSSL_LIBRARIES} minifi-standard-processors
minifi-civet-extensions -Wl,--no-whole-archive)
endif()
+ add_test(NAME "${testfilename}" COMMAND "${testfilename}"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
MATH(EXPR PROCESSOR_INT_TEST_COUNT "${PROCESSOR_INT_TEST_COUNT}+1")
ENDFOREACH()
+message("-- Finished building ${PROCESSOR_INT_TEST_COUNT} processor unit test
file(s)...")
SET(INT_TEST_COUNT 0)
diff --git a/extensions/standard-processors/tests/unit/ManifestTests.cpp
b/extensions/standard-processors/tests/unit/ManifestTests.cpp
index d261e4d..e111636 100644
--- a/extensions/standard-processors/tests/unit/ManifestTests.cpp
+++ b/extensions/standard-processors/tests/unit/ManifestTests.cpp
@@ -17,8 +17,9 @@
*/
#include
-
+#include
#include "core/Processor.h"
+#include "core/state/nodes/DeviceInformation.h"
#include "core/state/nodes/AgentInformation.h"
#include "TestBase.h"
#include "io/ClientSocket.h"
@@ -68,7 +69,7 @@ TEST_CASE("Test Valid Regex", "[validRegex]") {
}
TEST_CASE("Test Relationships", "[rel1]") {
- minifi::state::response::ComponentManifest manifest("minifi-system");
+ minifi::state::response::ComponentManifest
manifest("minifi-standard-processors");
auto serialized = manifest.serialize();
REQUIRE(serialized.size() > 0);
const auto = serialized[0];
@@ -95,7 +96,7 @@ TEST_CASE("Test Relationships", "[rel1]") {
}
TEST_CASE("Test Dependent", "[dependent]") {
- minifi::state::response::ComponentManifest manifest("minifi-system");
+ minifi::state::response::ComponentManifest
manifest("minifi-standard-processors");
auto serialized = manifest.serialize();
REQUIRE(serialized.size() > 0);
const auto = serialized[0];
@@ -149,3 +150,23 @@ TEST_CASE("Test Scheduling Defaults", "[schedDef]") {
}
}
}
+
+TEST_CASE("Test operatingSystem Defaults", "[opsys]") {
+ minifi::state::response::DeviceInfoNode manifest("minifi-system");
+ auto serialized = manifest.serialize();
+ REQUIRE(serialized.size() > 0);
+ minifi::state::response::SerializedResponseNode proc_0;
+ for (const auto : serialized) {
+if ("systemInfo" == node.name) {
+ for (const auto : node.children) {
+if ("operatingSystem" == sinfo.name) {
+ proc_0 = sinfo;
+ break;
+}
+ }
+}
+ }
+ REQUIRE(!proc_0.value.empty());
+ std::set expected({"Linux", "Windows 32", "Windows 64", "Mac
OSX", "Unix"});
+ REQUIRE(expected.find(proc_0.value.to_string()) != std::end(expected));
+}
diff --git a/libminifi/include/core/state/nodes/DeviceInformation.h
b/libminifi/include/core/state/nodes/DeviceInformation.h
index 59f3888..cabdeab 100644
--- a/libminifi/include/core/state/nodes/DeviceInformation.h
+++ b/libminifi/include/core/state/nodes/DeviceInformation.h
@@ -98,7 +98,7 @@ class Device {
canonical_hostname_ = hostname;
std::stringstream ips;
- auto ipaddressess = getIpAddresses();
+auto ipaddressess = getIpAddresses();
for (auto i
[nifi-minifi-cpp] branch master updated: MINIFICPP-866 - GetTCPTests should use random port
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new e672682 MINIFICPP-866 - GetTCPTests should use random port
e672682 is described below
commit e67268268b32c6cd6e13e0075898a3515a27465e
Author: Arpad Boda
AuthorDate: Tue May 21 16:43:18 2019 +0200
MINIFICPP-866 - GetTCPTests should use random port
This closes #564.
Signed-off-by: Marc Parisi
---
.../standard-processors/tests/unit/GetTCPTests.cpp | 22 +++---
1 file changed, 7 insertions(+), 15 deletions(-)
diff --git a/extensions/standard-processors/tests/unit/GetTCPTests.cpp
b/extensions/standard-processors/tests/unit/GetTCPTests.cpp
index a5d9962..9b7b29d 100644
--- a/extensions/standard-processors/tests/unit/GetTCPTests.cpp
+++ b/extensions/standard-processors/tests/unit/GetTCPTests.cpp
@@ -24,6 +24,7 @@
#include
#include "unit/ProvenanceTestHelper.h"
#include "TestBase.h"
+#include "RandomServerSocket.h"
#include "Scheduling.h"
#include "LogAttribute.h"
#include "GetTCP.h"
@@ -49,11 +50,8 @@ TEST_CASE("GetTCPWithoutEOM", "[GetTCP1]") {
content_repo->initialize(std::make_shared());
- std::shared_ptr socket_context
=
std::make_shared(std::make_shared());
std::shared_ptr stream_factory
= minifi::io::StreamFactory::getInstance(std::make_shared());
- org::apache::nifi::minifi::io::ServerSocket server(socket_context,
"localhost", 9184, 1);
-
- REQUIRE(-1 != server.initialize());
+ org::apache::nifi::minifi::io::RandomServerSocket server("localhost");
LogTestController::getInstance().setDebug();
LogTestController::getInstance().setDebug();
@@ -103,7 +101,7 @@ TEST_CASE("GetTCPWithoutEOM", "[GetTCP1]") {
std::shared_ptr
controller_services_provider = nullptr;
std::shared_ptr context =
std::make_shared(node, controller_services_provider,
repo, repo, content_repo);
std::shared_ptr context2 =
std::make_shared(node2, controller_services_provider,
repo, repo, content_repo);
-
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::EndpointList,
"localhost:9184");
+
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::EndpointList,
"localhost:" + std::to_string(server.getPort()));
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::ReconnectInterval,
"100 msec");
auto session = std::make_shared(context);
auto session2 = std::make_shared(context2);
@@ -159,14 +157,11 @@ TEST_CASE("GetTCPWithOEM", "[GetTCP2]") {
content_repo->initialize(std::make_shared());
- std::shared_ptr socket_context
=
std::make_shared(std::make_shared());
std::shared_ptr stream_factory
= minifi::io::StreamFactory::getInstance(std::make_shared());
TestController testController;
- org::apache::nifi::minifi::io::ServerSocket server(socket_context,
"localhost", 9182, 1);
-
- REQUIRE(-1 != server.initialize());
+ org::apache::nifi::minifi::io::RandomServerSocket server("localhost");
LogTestController::getInstance().setDebug();
LogTestController::getInstance().setTrace();
@@ -216,7 +211,7 @@ TEST_CASE("GetTCPWithOEM", "[GetTCP2]") {
std::shared_ptr
controller_services_provider = nullptr;
std::shared_ptr context =
std::make_shared(node, controller_services_provider,
repo, repo, content_repo);
std::shared_ptr context2 =
std::make_shared(node2, controller_services_provider,
repo, repo, content_repo);
-
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::EndpointList,
"localhost:9182");
+
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::EndpointList,
"localhost:" + std::to_string(server.getPort()));
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::ReconnectInterval,
"100 msec");
// we're using new lines above
context->setProperty(org::apache::nifi::minifi::processors::GetTCP::EndOfMessageByte,
"10");
@@ -284,16 +279,13 @@ TEST_CASE("GetTCPWithOnlyOEM", "[GetTCP3]") {
content_repo->initialize(std::make_shared());
- std::shared_ptr socket_context
=
std::make_shared(std::make_shared());
std::shared_ptr stream_factory
= minifi::io::StreamFactory::getInstance(std::make_shared());
TestController testController;
LogTestController::getInstance().setDebug();
- org::apache::nifi::minifi::io::ServerSocket server(socket_context,
"localhost", 9182, 1);
-
- REQUIRE(-1 != server.initialize());
+ org::apache::nifi::minifi::io::RandomServerSocket server("localhost");
LogT
[nifi-minifi-cpp] branch master updated: MINIFICPP-854 CaptureRTSPFrame with OpenCV
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 1ae1def MINIFICPP-854 CaptureRTSPFrame with OpenCV
1ae1def is described below
commit 1ae1def4e2735ffafad3d238945c225a2950de0d
Author: Jeremy Dyer
AuthorDate: Tue Mar 5 15:37:12 2019 -0500
MINIFICPP-854 CaptureRTSPFrame with OpenCV
This closes #561.
Signed-off-by: Marc Parisi
---
CMakeLists.txt | 32 ++--
README.md | 1 +
bootstrap.sh | 4 +
bstrp_functions.sh | 4 +-
cmake/Extensions.cmake | 33 +++-
extensions/opencv/CMakeLists.txt | 65
extensions/opencv/CaptureRTSPFrame.cpp | 170 +
extensions/opencv/CaptureRTSPFrame.h | 141 +
libminifi/test/opencv-tests/CMakeLists.txt | 39 +
.../test/opencv-tests/CaptureRTSPFrameTest.cpp | 72 +
10 files changed, 545 insertions(+), 16 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index c3ab96b..1bc647f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -37,7 +37,7 @@ option(USE_SHARED_LIBS "Builds using shared libraries" ON)
option(ENABLE_PYTHON "Instructs the build system to enable building shared
objects for the python lib" OFF)
-cmake_dependent_option(STATIC_BUILD "Attempts to statically link as many
dependencies as possible." ON "NOT ENABLE_PYTHON; NOT USE_SHARED_LIBS" OFF)
+cmake_dependent_option(STATIC_BUILD "Attempts to statically link as many
dependencies as possible." ON "NOT ENABLE_PYTHON; NOT USE_SHARED_LIBS" OFF)
cmake_dependent_option(USE_SYSTEM_OPENSSL "Instructs the build system to
search for and use an SSL library available in the host system" ON "NOT
STATIC_BUILD" OFF)
option(LIBC_STATIC "Instructs the build system to statically link libstdc++
and glibc into minifiexe. Experiemental" OFF)
@@ -47,9 +47,8 @@ option(ENABLE_OPS "Enable Operations/zlib Tools" ON)
option(USE_SYSTEM_UUID "Instructs the build system to search for and use a
UUID library available in the host system" OFF)
option(ENABLE_JNI "Instructs the build system to enable the JNI extension" OFF)
-
+option(ENABLE_OPENCV "Instructs the build system to enable the OpenCV
extension" OFF)
cmake_dependent_option(USE_SYSTEM_CURL "Instructs the build system to search
for and use a cURL library available in the host system" ON "NOT STATIC_BUILD"
OFF)
-
option(BUILD_SHARED_LIBS "Build yaml cpp shared lib" OFF)
cmake_dependent_option(USE_SYSTEM_ZLIB "Instructs the build system to search
for and use a zlib library available in the host system" ON "NOT STATIC_BUILD"
OFF)
@@ -187,7 +186,7 @@ list(APPEND CMAKE_MODULE_PATH
"${CMAKE_CURRENT_SOURCE_DIR}/cmake")
if (NOT OPENSSL_OFF)
if(USE_SYSTEM_OPENSSL)
-
+
# Set the right openssl root path
if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
set(OPENSSL_ROOT_DIR "/usr/local/opt/openssl/")
@@ -239,19 +238,19 @@ if(WIN32 OR NOT USE_SYSTEM_ZLIB)
BUILD_BYPRODUCTS ${BYPRODUCT}
)
-
+
add_library(z STATIC IMPORTED)
set_target_properties(z PROPERTIES IMPORTED_LOCATION
"${CMAKE_CURRENT_BINARY_DIR}/${BYPRODUCT}")
-
+
set(ZLIB_BYPRODUCT "${CMAKE_CURRENT_BINARY_DIR}/${BYPRODUCT}" CACHE STRING
"" FORCE)
- set(ZLIB_BYPRODUCT_INCLUDE
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/zlib/include" CACHE STRING "" FORCE)
+ set(ZLIB_BYPRODUCT_INCLUDE
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/zlib/include" CACHE STRING "" FORCE)
set(ZLIB_BIN_DIR "${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/"
CACHE STRING "" FORCE)
-
+
add_dependencies(z zlib-external)
set(ZLIB_FOUND "YES" CACHE STRING "" FORCE)
set(ZLIB_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/zlib/include"
CACHE STRING "" FORCE)
set(ZLIB_INCLUDE_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/zlib/include"
CACHE STRING "" FORCE)
-
+
set(ZLIB_LIBRARY "${CMAKE_CURRENT_BINARY_DIR}/${BYPRODUCT}" CACHE STRING ""
FORCE)
set(ZLIB_LIBRARIES "${ZLIB_LIBRARY}" CACHE STRING "" FORCE)
set(ZLIB_LIBRARY_RELEASE "${CMAKE_CURRENT_BINARY_DIR}/${BYPRODUCT}" CACHE
STRING "" FORCE)
@@ -351,13 +350,13 @@ endif()
set(CURL_SRC_DIR "${CMAKE_CURRENT_SOURCE_DIR
[nifi-minifi-cpp] branch master updated: MINIFICPP-856 Introduce AWS SDK to the MiNiFi Cmake build
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 1250d3c MINIFICPP-856 Introduce AWS SDK to the MiNiFi Cmake build
1250d3c is described below
commit 1250d3c8f98d9df38d745f2d170d191e2bf677f4
Author: Jeremy Dyer
AuthorDate: Mon Apr 22 10:22:35 2019 -0400
MINIFICPP-856 Introduce AWS SDK to the MiNiFi Cmake build
Updated documentation as pointed out by @phrocker
This closes #562.
Signed-off-by: Marc Parisi
---
CMakeLists.txt | 5 +
CONTROLLERS.md | 39
README.md | 1 +
bootstrap.sh | 2 +
bstrp_functions.sh | 2 +
extensions/aws/AWSLoader.cpp | 27 ++
extensions/aws/AWSLoader.h | 70 ++
extensions/aws/CMakeLists.txt | 67 +
.../controllerservices/AWSCredentialsService.cpp | 57 +++
.../aws/controllerservices/AWSCredentialsService.h | 104 +
extensions/aws/s3/S3.h | 73 +++
11 files changed, 447 insertions(+)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 3f74bfa..c3ab96b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -553,6 +553,11 @@ if (ENABLE_TENSORFLOW)
createExtension(TENSORFLOW-EXTENSIONS "TENSORFLOW EXTENSIONS" "This
enables TensorFlow support" "extensions/tensorflow"
"${TEST_DIR}/tensorflow-tests")
endif()
+## AWS Extentions
+if (ENABLE_AWS)
+ createExtension(AWS-EXTENSIONS "AWS EXTENSIONS" "This enables AWS
support" "extensions/aws" )
+endif()
+
## Bustache/template extensions
option(ENABLE_BUSTACHE "Enables Bustache (ApplyTemplate) support." OFF)
diff --git a/CONTROLLERS.md b/CONTROLLERS.md
new file mode 100644
index 000..f1482f8
--- /dev/null
+++ b/CONTROLLERS.md
@@ -0,0 +1,39 @@
+
+
+# Controller Services
+
+## Table of Contents
+
+- [AWSCredentialsService](#awsCredentialsService)
+
+## AWSCredentialsService
+
+### Description
+
+Manages the Amazon Web Services (AWS) ```Access Key``` and ```Secret Access
Key``` for a AWS account. This allows for multiple
+AWS credential services to be defined. This also allows for multiple AWS
related processors to reference this single
+controller service so that AWS credentials can be managed and controlled in a
central location.
+
+### Properties
+
+In the list below, the names of required properties appear in bold. Any other
+properties (not in bold) are considered optional. The table also indicates any
+default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Expression Language Supported? |
Description |
+| - | - | - | - |
+| **Access Key** | | | Yes | Specifies the AWS Access Key |
+| **Secret Key** | | | Yes | Specifies the AWS Secret Key |
\ No newline at end of file
diff --git a/README.md b/README.md
index 18ee178..898be3c 100644
--- a/README.md
+++ b/README.md
@@ -70,6 +70,7 @@ Through JNI extensions you can run NiFi processors using
NARs. The JNI extension
| Extension Set| Processors | CMAKE Flag |
| - |:-| :-|
| Archive Extensions|
[ApplyTemplate](PROCESSORS.md#applytemplate)[CompressContent](PROCESSORS.md#compresscontent)[ManipulateArchive](PROCESSORS.md#manipulatearchive)[MergeContent](PROCESSORS.md#mergecontent)[FocusArchiveEntry](PROCESSORS.md#focusarchiveentry)[UnfocusArchiveEntry](PROCESSORS.md#unfocusarchiveentry)
| -DBUILD_LIBARCHIVE=ON |
+| AWS | [AWSCredentialsService](CONTROLLERS.md#awsCredentialsService) |
-DENABLE_AWS=ON |
| CURL | [InvokeHTTP](PROCESSORS.md#invokehttp) |-DDISABLE_CURL=ON |
| GPS | GetGPS |-DENABLE_GPS=ON |
| Kafka | [PublishKafka](PROCESSORS.md#publishkafka) |
-DENABLE_LIBRDKAFKA=ON |
diff --git a/bootstrap.sh b/bootstrap.sh
index df6eb65..b85b18d 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -264,6 +264,8 @@ add_dependency USB_ENABLED "libpng"
add_disabled_option GPS_ENABLED ${FALSE} "ENABLE_GPS"
add_dependency GPS_ENABLED "gpsd"
+add_disabled_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
+
add_disabled_option KAFKA_ENABLED ${FALSE} "ENABLE_LIBRDKAFKA" "3.4.0"
add_disabled_option MQTT_ENABLED ${FALSE} "ENABLE_MQTT"
diff --git a/bstrp_functions.sh b/bstrp_functions.sh
index aecffe6..2a5bfaf 100755
--- a/bstrp_functions.sh
+++ b/bstrp_functions.sh
@@ -259,6 +259,7 @@ show_supported_features() {
echo "M. SQLite Support
[nifi-minifi-cpp] branch master updated: MINIFICPP-852: Add details to restful response. CoAP will not include details for the time being
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new dfbd3bd MINIFICPP-852: Add details to restful response. CoAP will not
include details for the time being
dfbd3bd is described below
commit dfbd3bd2417d5c6e1ffd2cfe444a909ec63d19b4
Author: Marc Parisi
AuthorDate: Fri May 17 15:17:36 2019 -0400
MINIFICPP-852: Add details to restful response. CoAP will not include
details for the time being
This closes #560.
Approved on GH by kevdoran.
Signed-off-by: Marc Parisi
---
extensions/http-curl/tests/C2FailedUpdateTest.cpp | 30 +++--
libminifi/include/core/logging/Logger.h | 3 +-
libminifi/include/core/yaml/YamlConfiguration.h | 52 ++-
libminifi/src/c2/C2Agent.cpp | 1 +
libminifi/src/c2/protocols/RESTProtocol.cpp | 25 ++-
thirdparty/google-styleguide/run_linter.sh| 2 +-
6 files changed, 78 insertions(+), 35 deletions(-)
diff --git a/extensions/http-curl/tests/C2FailedUpdateTest.cpp
b/extensions/http-curl/tests/C2FailedUpdateTest.cpp
index 1df1219..ec98cf7 100644
--- a/extensions/http-curl/tests/C2FailedUpdateTest.cpp
+++ b/extensions/http-curl/tests/C2FailedUpdateTest.cpp
@@ -57,8 +57,31 @@ class ConfigHandler : public CivetHandler {
ConfigHandler() {
calls_ = 0;
}
- bool handlePost(CivetServer *server, struct mg_connection *conn) {
+ virtual bool handlePost(CivetServer *server, struct mg_connection *conn)
override {
calls_++;
+const struct mg_request_info *req_info = mg_get_request_info(conn);
+long long remainlen;
+long long readlen = 0;
+long long contentlen = req_info->content_length;
+char buf[1024];
+
+std::string data;
+while (readlen < contentlen) {
+ remainlen = contentlen - readlen;
+ if (remainlen > sizeof(buf)) {
+remainlen = sizeof(buf);
+ }
+ remainlen = mg_read(conn, buf, (size_t) remainlen);
+ if (remainlen <= 0) {
+break;
+ }
+ readlen += remainlen;
+ data += std::string(buf, remainlen);
+}
+if (data.find("operationState") != std::string::npos) {
+ assert(data.find("state\": \"NOT_APPLIED") != std::string::npos);
+}
+
if (responses.size() > 0) {
std::string top_str = responses.back();
responses.pop_back();
@@ -73,7 +96,7 @@ class ConfigHandler : public CivetHandler {
return true;
}
- bool handleGet(CivetServer *server, struct mg_connection *conn) {
+ virtual bool handleGet(CivetServer *server, struct mg_connection *conn)
override {
std::ifstream myfile(test_file_location_.c_str());
if (myfile.is_open()) {
@@ -146,6 +169,7 @@ int main(int argc, char **argv) {
configuration->set("nifi.c2.enable", "true");
configuration->set("nifi.c2.agent.class", "test");
configuration->set("nifi.c2.rest.url", "http://localhost:7071/update;);
+ configuration->set("nifi.c2.rest.url.ack", "http://localhost:7071/update;);
configuration->set("nifi.c2.agent.heartbeat.period", "1000");
mkdir("content_repository", S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
@@ -161,7 +185,7 @@ int main(int argc, char **argv) {
std::shared_ptr repo =
std::static_pointer_cast(test_repo);
std::shared_ptr controller =
std::make_shared(test_repo, test_flow_repo,
configuration, std::move(yaml_ptr), content_repo, DEFAULT_ROOT_GROUP_NAME,
- true);
+
true);
core::YamlConfiguration yaml_config(test_repo, test_repo, content_repo,
stream_factory, configuration, test_file_location);
diff --git a/libminifi/include/core/logging/Logger.h
b/libminifi/include/core/logging/Logger.h
index 37ef390..4b8be46 100644
--- a/libminifi/include/core/logging/Logger.h
+++ b/libminifi/include/core/logging/Logger.h
@@ -271,7 +271,8 @@ class Logger : public BaseLogger {
if (!delegate_->should_log(level)) {
return;
}
-delegate_->log(level, format_string(format,
conditional_conversion(args)...));
+const auto str = format_string(format, conditional_conversion(args)...);
+delegate_->log(level, str);
}
Logger(Logger const&);
diff --git a/libminifi/include/core/yaml/YamlConfiguration.h
b/libminifi/include/core/yaml/YamlConfiguration.h
index cf8209e..318cb27 100644
--- a/libminifi/include/core/yaml/YamlConfiguration.h
+++ b/libminifi/include/core/yaml/YamlConfiguration.h
@@ -55,12 +55,8 @@ namespace core {
class YamlConfiguration : public FlowConfiguration {
public:
- explicit YamlConfiguration(std::shared_ptr repo,
-
[nifi-minifi-cpp] branch master updated: MINIFICPP-842 - CivetWeb tests should use random port
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 102df8d MINIFICPP-842 - CivetWeb tests should use random port
102df8d is described below
commit 102df8d3f61fe5e91573da378855d54111b97196
Author: Arpad Boda
AuthorDate: Wed May 15 12:27:50 2019 +0200
MINIFICPP-842 - CivetWeb tests should use random port
This closes #554.
Signed-off-by: Marc Parisi
---
extensions/civetweb/processors/ListenHTTP.cpp | 51 -
extensions/civetweb/processors/ListenHTTP.h | 3 ++
libminifi/include/core/PropertyValidation.h | 16
libminifi/test/civetweb-tests/CivetwebTests.cpp | 13 ---
4 files changed, 68 insertions(+), 15 deletions(-)
diff --git a/extensions/civetweb/processors/ListenHTTP.cpp
b/extensions/civetweb/processors/ListenHTTP.cpp
index dbb00a4..a6c2ef7 100644
--- a/extensions/civetweb/processors/ListenHTTP.cpp
+++ b/extensions/civetweb/processors/ListenHTTP.cpp
@@ -27,11 +27,16 @@ namespace minifi {
namespace processors {
core::Property ListenHTTP::BasePath(
-core::PropertyBuilder::createProperty("Base Path")->withDescription("Base
path for incoming
connections")->isRequired(false)->withDefaultValue("contentListener")->build());
+core::PropertyBuilder::createProperty("Base Path")
+->withDescription("Base path for incoming connections")
+->isRequired(false)
+->withDefaultValue("contentListener")->build());
core::Property ListenHTTP::Port(
-core::PropertyBuilder::createProperty("Listening
Port")->withDescription("The Port to listen on for incoming
connections")->isRequired(true)->withDefaultValue(
-80, core::StandardValidators::PORT_VALIDATOR())->build());
+core::PropertyBuilder::createProperty("Listening Port")
+->withDescription("The Port to listen on for incoming connections. 0
means port is going to be selected randomly.")
+->isRequired(true)
+->withDefaultValue(80,
core::StandardValidators::LISTEN_PORT_VALIDATOR())->build());
core::Property ListenHTTP::AuthorizedDNPattern("Authorized DN Pattern", "A
Regular Expression to apply against the Distinguished Name of incoming"
" connections. If the Pattern
does not match the DN, the connection will be refused.",
@@ -40,12 +45,18 @@ core::Property ListenHTTP::SSLCertificate("SSL
Certificate", "File containing PE
core::Property ListenHTTP::SSLCertificateAuthority("SSL Certificate
Authority", "File containing trusted PEM-formatted certificates", "");
core::Property ListenHTTP::SSLVerifyPeer(
-core::PropertyBuilder::createProperty("SSL Verify
Peer")->withDescription("Whether or not to verify the client's certificate
(yes/no)")->isRequired(false)->withAllowableValue("yes")
-->withAllowableValue("no")->withDefaultValue("no")->build());
+core::PropertyBuilder::createProperty("SSL Verify Peer")
+->withDescription("Whether or not to verify the client's certificate
(yes/no)")
+->isRequired(false)
+->withAllowableValues({"yes", "no"})
+->withDefaultValue("no")->build());
core::Property ListenHTTP::SSLMinimumVersion(
-core::PropertyBuilder::createProperty("SSL Minimum
Version")->withDescription("Minimum TLS/SSL version allowed (SSL2, SSL3,
TLS1.0, TLS1.1, TLS1.2)")->isRequired(false)
-
->withAllowableValue("SSL2")->withAllowableValue("SSL3")->withAllowableValue("TLS1.0")->withAllowableValue("TLS1.1")->withAllowableValue("TLS1.2")->withDefaultValue("SSL2")->build());
+core::PropertyBuilder::createProperty("SSL Minimum Version")
+-> withDescription("Minimum TLS/SSL version allowed (SSL2, SSL3,
TLS1.0, TLS1.1, TLS1.2)")
+->isRequired(false)
+->withAllowableValues({"SSL2", "SSL3", "TLS1.0",
"TLS1.1", "TLS1.2"})
+->withDefaultValue("SSL2")->build());
core::Property ListenHTTP::HeadersAsAttributesRegex("HTTP Headers to receive
as Attributes (Regex)", "Specifies the Regular Expression that determines the
names of HTTP Headers that"
" should be passed along
as FlowFile attributes",
@@ -83,13 +94,14 @@ void ListenHTTP::onSche
[nifi-minifi-cpp] branch master updated: MINIFICPP-822 - Nanofi raw S2S implementation shouldn't depend on any C++ code.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 42aee28 MINIFICPP-822 - Nanofi raw S2S implementation shouldn't
depend on any C++ code.
42aee28 is described below
commit 42aee28c29bb9e0c448ca3272ea80cfe6477d00e
Author: Arpad Boda
AuthorDate: Thu Apr 25 15:11:17 2019 +0200
MINIFICPP-822 - Nanofi raw S2S implementation shouldn't depend on any C++
code.
Fix return value of DescriptionStream::readData in case buffer is not fully
filled
Address review comments
This closes #550.
Signed-off-by: Marc Parisi
---
cmake/BuildTests.cmake | 4 +-
libminifi/opsys/posix/io/ClientSocket.h| 10 +-
libminifi/src/io/DescriptorStream.cpp | 5 +-
libminifi/src/io/posix/ClientSocket.cpp| 4 +-
libminifi/test/CPPLINT.cfg | 1 +
libminifi/test/RandomServerSocket.cpp | 58
.../test/RandomServerSocket.h | 43 ++-
nanofi/include/core/cstructs.h | 13 +-
nanofi/include/core/cxxstructs.h | 4 -
nanofi/include/sitetosite/CPeer.h | 11 +-
nanofi/include/sitetosite/CRawSocketProtocol.h | 2 +-
nanofi/include/sitetosite/CSiteToSite.h| 13 +-
nanofi/src/core/cstream.c | 236
nanofi/src/core/cstream.cpp| 164
nanofi/src/sitetosite/CPeer.c | 7 +-
nanofi/src/sitetosite/CRawSocketProtocol.c | 6 +-
nanofi/tests/CSite2SiteTests.cpp | 297 +++--
17 files changed, 504 insertions(+), 374 deletions(-)
diff --git a/cmake/BuildTests.cmake b/cmake/BuildTests.cmake
index 591516d..668564e 100644
--- a/cmake/BuildTests.cmake
+++ b/cmake/BuildTests.cmake
@@ -97,7 +97,7 @@ SET(SPD_LIB spd_lib)
add_library(${SPD_LIB} STATIC ${SPD_SOURCES})
SET(TEST_BASE_LIB test_base)
-add_library(${TEST_BASE_LIB} STATIC "${TEST_DIR}/TestBase.cpp")
+add_library(${TEST_BASE_LIB} STATIC "${TEST_DIR}/TestBase.cpp"
"${TEST_DIR}/RandomServerSocket.cpp")
target_include_directories(${TEST_BASE_LIB} BEFORE PRIVATE
"${CMAKE_SOURCE_DIR}/thirdparty/catch")
target_include_directories(${TEST_BASE_LIB} BEFORE PRIVATE
"${CMAKE_SOURCE_DIR}/libminifi/include/")
target_include_directories(${TEST_BASE_LIB} BEFORE PRIVATE
"${CMAKE_SOURCE_DIR}/thirdparty/cron")
@@ -139,7 +139,7 @@ FOREACH(testfile ${NANOFI_UNIT_TESTS})
target_include_directories(${testfilename} BEFORE PRIVATE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors/processors/")
target_include_directories(${testfilename} BEFORE PRIVATE
"${CMAKE_SOURCE_DIR}/libminifi/test")
appendIncludes("${testfilename}")
-target_link_libraries(${testfilename} ${CMAKE_THREAD_LIBS_INIT}
${CATCH_MAIN_LIB} nanofi)
+target_link_libraries(${testfilename} ${CMAKE_THREAD_LIBS_INIT}
${CATCH_MAIN_LIB} ${TEST_BASE_LIB} nanofi)
if (APPLE)
# minifi-standard-processors
diff --git a/libminifi/opsys/posix/io/ClientSocket.h
b/libminifi/opsys/posix/io/ClientSocket.h
index 4ed8b99..739f7f9 100644
--- a/libminifi/opsys/posix/io/ClientSocket.h
+++ b/libminifi/opsys/posix/io/ClientSocket.h
@@ -108,7 +108,9 @@ class Socket : public BaseStream {
* Return the port for this socket
* @returns port
*/
- uint16_t getPort();
+ uint16_t getPort() const {
+return port_;
+ }
// data stream extensions
/**
@@ -276,6 +278,12 @@ class Socket : public BaseStream {
bool nonBlocking_;
+
+ protected:
+ void setPort(uint16_t port) {
+port_ = port;
+ }
+
private:
std::shared_ptr logger_;
static std::string init_hostname() {
diff --git a/libminifi/src/io/DescriptorStream.cpp
b/libminifi/src/io/DescriptorStream.cpp
index 38e1c85..6431ef5 100644
--- a/libminifi/src/io/DescriptorStream.cpp
+++ b/libminifi/src/io/DescriptorStream.cpp
@@ -84,11 +84,10 @@ int DescriptorStream::readData(uint8_t *buf, int buflen) {
if (!IsNullOrEmpty(buf)) {
auto size_read = ::read(fd_, buf, buflen);
-if (size_read != buflen) {
+if (size_read < 0) {
return -1;
-} else {
- return buflen;
}
+return size_read;
} else {
return -1;
diff --git a/libminifi/src/io/posix/ClientSocket.cpp
b/libminifi/src/io/posix/ClientSocket.cpp
index 113f914..7fa4d22 100644
--- a/libminifi/src/io/posix/ClientSocket.cpp
+++ b/libminifi/src/io/posix/ClientSocket.cpp
@@ -42,7 +42,7 @@ namespace nifi {
namespace minifi {
namespace io {
-Socket::Socket(const std::shared_ptr , const
std::string , const uint16_t port, const uint16_t listeners = -1)
+Socket::Socket(const std
[nifi-minifi-cpp] branch master updated: MINIFICPP-287: Add property for flow files to log
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new bcf8921 MINIFICPP-287: Add property for flow files to log
bcf8921 is described below
commit bcf8921b839c967a31c764ff29027b320c462634
Author: Marc Parisi
AuthorDate: Tue May 14 15:26:05 2019 -0400
MINIFICPP-287: Add property for flow files to log
This closes #553.
Approved on GH By jdye64.
Signed-off-by: Marc Parisi
---
.../processors/LogAttribute.cpp| 165 +++--
.../standard-processors/processors/LogAttribute.h | 10 +-
.../tests/unit/ProcessorTests.cpp | 91
libminifi/include/core/PropertyValidation.h| 5 +
libminifi/include/core/state/Value.h | 13 ++
libminifi/src/core/PropertyValidation.cpp | 2 +
6 files changed, 208 insertions(+), 78 deletions(-)
diff --git a/extensions/standard-processors/processors/LogAttribute.cpp
b/extensions/standard-processors/processors/LogAttribute.cpp
index 678d12c..a02bafe 100644
--- a/extensions/standard-processors/processors/LogAttribute.cpp
+++ b/extensions/standard-processors/processors/LogAttribute.cpp
@@ -39,12 +39,17 @@ namespace nifi {
namespace minifi {
namespace processors {
-core::Property
LogAttribute::LogLevel(core::PropertyBuilder::createProperty("Log
Level")->withDescription("The Log Level to use when logging the
Attributes")->withAllowableValues({
+core::Property
LogAttribute::LogLevel(core::PropertyBuilder::createProperty("Log
Level")->withDescription("The Log Level to use when logging the
Attributes")->withAllowableValues( {
"info", "trace", "error", "warn", "debug" })->build());
core::Property LogAttribute::AttributesToLog(
core::PropertyBuilder::createProperty("Attributes to
Log")->withDescription("A comma-separated list of Attributes to Log. If not
specified, all attributes will be logged.")->build());
+core::Property LogAttribute::FlowFilesToLog(
+core::PropertyBuilder::createProperty("FlowFiles To Log")->withDescription(
+"Number of flow files to log. If set to zero all flow files will be
logged. Please note that this may block other threads from running if not used
judiciously.")->withDefaultValue(1)
+->build());
+
core::Property LogAttribute::AttributesToIgnore(
core::PropertyBuilder::createProperty("Attributes to
Ignore")->withDescription("A comma-separated list of Attributes to ignore. If
not specified, no attributes will be ignored.")->build());
@@ -63,6 +68,7 @@ void LogAttribute::initialize() {
properties.insert(AttributesToLog);
properties.insert(AttributesToIgnore);
properties.insert(LogPayload);
+ properties.insert(FlowFilesToLog);
properties.insert(LogPrefix);
setSupportedProperties(properties);
// Set the supported relationships
@@ -71,91 +77,100 @@ void LogAttribute::initialize() {
setSupportedRelationships(relationships);
}
-void LogAttribute::onTrigger(core::ProcessContext *context,
core::ProcessSession *session) {
- logger_->log_trace("enter log attribute");
+void LogAttribute::onSchedule(const std::shared_ptr
, const std::shared_ptr ) {
+ core::Property flowsToLog = FlowFilesToLog;
+
+ if (getProperty(FlowFilesToLog.getName(), flowsToLog)) {
+// we are going this route since to avoid breaking backwards compatibility
the get property function doesn't perform validation ( That's done
+// in configuration. In future releases we can add that exception handling
there.
+if (!flowsToLog.getValue().validate("Validating FlowFilesToLog").valid())
+ throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Invalid value for flowfiles
to log: " + flowsToLog.getValue().to_string());
+flowfiles_to_log_ = flowsToLog.getValue();
+ }
+}
+// OnTrigger method, implemented by NiFi LogAttribute
+void LogAttribute::onTrigger(const std::shared_ptr
, const std::shared_ptr ) {
+ logger_->log_trace("enter log attribute, attempting to retrieve %u flow
files", flowfiles_to_log_);
std::string dashLine = "--";
LogAttrLevel level = LogAttrLevelInfo;
bool logPayload = false;
std::ostringstream message;
- std::shared_ptr flow = session->get();
+ uint64_t i = 0;
+ const auto max = flowfiles_to_log_ == 0 ? UINT64_MAX : flowfiles_to_log_;
+ for (; i < max; ++i) {
+std::shared_ptr flow = session->get();
- if (!flow) {
-return;
- }
+if (!flow) {
+ break;
+}
- std::string value;
- if (context->getProperty(LogLevel
[nifi-minifi-cpp] branch master updated: MINIFICPP-773 Implemented.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new b635f33 MINIFICPP-773 Implemented.
b635f33 is described below
commit b635f33931a7367820517e584ebb1bc5ca8c56b7
Author: amarmer
AuthorDate: Wed May 8 15:51:41 2019 -0700
MINIFICPP-773 Implemented.
MINIFICPP-773 Fixed EvtSubscribe parameter.
MINIFICPP-773 Fixed formatting.
MINIFICPP-773 Code review modifications.
This closes #551.
Signed-off-by: Marc Parisi
---
.../windows-event-log/ConsumeWindowsEventLog.cpp | 306 +
.../windows-event-log/ConsumeWindowsEventLog.h | 102 +++
2 files changed, 408 insertions(+)
diff --git a/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
new file mode 100644
index 000..afc6d87
--- /dev/null
+++ b/extensions/windows-event-log/ConsumeWindowsEventLog.cpp
@@ -0,0 +1,306 @@
+/**
+ * @file ConsumeWindowsEventLog.cpp
+ * ConsumeWindowsEventLog class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#pragma comment(lib, "wevtapi.lib")
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string
ConsumeWindowsEventLog::ProcessorName("ConsumeWindowsEventLog");
+
+core::Property ConsumeWindowsEventLog::Channel(
+ core::PropertyBuilder::createProperty("Channel")->
+ isRequired(true)->
+ withDefaultValue("System")->
+ withDescription("The Windows Event Log Channel to listen to.")->
+ supportsExpressionLanguage(true)->
+ build());
+
+core::Property ConsumeWindowsEventLog::Query(
+ core::PropertyBuilder::createProperty("Query")->
+ isRequired(true)->
+ withDefaultValue("*")->
+ withDescription("XPath Query to filter events. (See
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx
for examples.)")->
+ supportsExpressionLanguage(true)->
+ build());
+
+core::Property ConsumeWindowsEventLog::MaxBufferSize(
+ core::PropertyBuilder::createProperty("Max Buffer Size")->
+ isRequired(true)->
+ withDefaultValue("1 MB")->
+ withDescription(
+"The individual Event Log XMLs are rendered to a buffer."
+" This specifies the maximum size in bytes that the buffer will be allowed
to grow to. (Limiting the maximum size of an individual Event XML.)")->
+ build());
+
+core::Property ConsumeWindowsEventLog::InactiveDurationToReconnect(
+ core::PropertyBuilder::createProperty("Inactive Duration To Reconnect")->
+ isRequired(true)->
+ withDefaultValue("10 min")->
+ withDescription(
+"If no new event logs are processed for the specified time period, "
+" this processor will try reconnecting to recover from a state where any
further messages cannot be consumed."
+" Such situation can happen if Windows Event Log service is restarted, or
ERROR_EVT_QUERY_RESULT_STALE (15011) is returned."
+" Setting no duration, e.g. '0 ms' disables auto-reconnection.")->
+ build());
+
+core::Relationship ConsumeWindowsEventLog::Success("success", "Relationship
for successfully consumed events.");
+
+ConsumeWindowsEventLog::ConsumeWindowsEventLog(const std::string& name,
utils::Identifier uuid)
+ : core::Processor(name, uuid),
logger_(logging::LoggerFactory::getLogger()) {
+ char buff[MAX_COMPUTERNAME_LENGTH + 1];
+ DWORD size = sizeof(buff);
+ if (GetComputerName(buff, )) {
+computerName_ = buff;
+ } else {
+LogWindowsError();
+ }
+}
+
+void Consu
[nifi-minifi-cpp] branch master updated: MINIFICPP-831 - Remove Debug in favor of RelWithDebInfo and use Ninja for this target.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new df13501 MINIFICPP-831 - Remove Debug in favor of RelWithDebInfo and
use Ninja for this target.
df13501 is described below
commit df13501f7de8ac51244095b2cbc7be266c29c30c
Author: Marc Parisi
AuthorDate: Sun Apr 28 17:07:36 2019 -0400
MINIFICPP-831 - Remove Debug in favor of RelWithDebInfo and use Ninja for
this target.
This closes #548.
Signed-off-by: Marc Parisi
---
CMakeSettings.json | 36
1 file changed, 24 insertions(+), 12 deletions(-)
diff --git a/CMakeSettings.json b/CMakeSettings.json
index 2064f29..7582dc5 100644
--- a/CMakeSettings.json
+++ b/CMakeSettings.json
@@ -2,14 +2,14 @@
// See https://go.microsoft.com//fwlink//?linkid=834763 for more information
about this file.
"configurations": [
{
- "name": "x64-Debug",
- "generator": "Visual Studio 14 2015 Win64",
- "configurationType": "Debug",
+ "name": "x64-RelWithDebInfo",
+ "generator": "Ninja",
+ "configurationType": "RelWithDebInfo",
"inheritEnvironments": [ "msvc_x64_x64" ],
"buildRoot":
"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot":
"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
- "buildCommandArgs": "-v:minimal -m:8",
+ "buildCommandArgs": "-v -j8",
"variables": [
{
"name": "OPENSSL_OFF",
@@ -20,6 +20,10 @@
"value": "ON"
},
{
+ "name": "PORTABLE",
+ "value": "ON"
+},
+{
"name": "OPENSSL_ROOT_DIR",
"value": "C:\\OpenSSL-Win64"
},
@@ -28,11 +32,11 @@
"value": "OFF"
},
{
- "name": "USE_SYSTEM_CURL",
- "value": "OFF"
+ "name": "DISABLE_CONTROLLER",
+ "value": "TRUE"
},
{
- "name": "USE_SYSTEM_UUID",
+ "name": "USE_SYSTEM_CURL",
"value": "OFF"
},
{
@@ -40,6 +44,10 @@
"value": "ON"
},
{
+ "name": "USE_SYSTEM_UUID",
+ "value": "OFF"
+},
+{
"name": "FORCE_WINDOWS",
"value": "ON"
},
@@ -48,10 +56,6 @@
"value": "FALSE"
},
{
-"name": "ENABLE_COAP",
-"value": "OFF"
-},
-{
"name": "ENABLE_WEL",
"value": "TRUE"
},
@@ -64,6 +68,14 @@
"value": "OFF"
},
{
+ "name": "FORCE_WINDOWS",
+ "value": "ON"
+},
+{
+ "name": "ENABLE_COAP",
+ "value": "OFF"
+},
+{
"name": "DISABLE_LIBARCHIVE",
"value": "TRUE"
},
@@ -85,7 +97,7 @@
{
"name": "x64-Release",
"generator": "Ninja",
- "configurationType": "RelWithDebInfo",
+ "configurationType": "Release",
"inheritEnvironments": [ "msvc_x64_x64" ],
"buildRoot":
"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot":
"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
[nifi-minifi-cpp] branch master updated: MINIFICPP-835: Use release tarball
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 8d297ac MINIFICPP-835: Use release tarball
8d297ac is described below
commit 8d297acc7248d69d266a1ca906921f06509a759f
Author: Marc Parisi
AuthorDate: Tue Apr 30 11:32:10 2019 -0400
MINIFICPP-835: Use release tarball
MINIFICPP-835: Change params for windows.
MINIFICPP-835: Ensure OpenSSL properly pulled with find_package
This closes #549.
Signed-off-by: Marc Parisi
---
CMakeLists.txt | 59 ++-
CMakeSettings.json | 6 +---
cmake/LibreSSL.cmake| 62 +
cmake/curl/dummy/FindCURL.cmake | 5 +++-
cmake/ssl/FindOpenSSL.cmake | 6 ++--
libminifi/CMakeLists.txt| 2 --
main/CMakeLists.txt | 12
7 files changed, 97 insertions(+), 55 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0b35539..8066a33 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -40,6 +40,7 @@ option(ENABLE_PYTHON "Instructs the build system to enable
building shared objec
cmake_dependent_option(STATIC_BUILD "Attempts to statically link as many
dependencies as possible." ON "NOT ENABLE_PYTHON; NOT USE_SHARED_LIBS" OFF)
cmake_dependent_option(USE_SYSTEM_OPENSSL "Instructs the build system to
search for and use an SSL library available in the host system" ON "NOT
STATIC_BUILD" OFF)
+option(LIBC_STATIC "Instructs the build system to statically link libstdc++
and glibc into minifiexe. Experiemental" OFF)
option(OPENSSL_OFF "Disables OpenSSL" OFF)
option(ENABLE_OPS "Enable Operations/zlib Tools" ON)
@@ -200,45 +201,10 @@ if (NOT OPENSSL_OFF)
find_package (OpenSSL REQUIRED)
else()
- message("Using bundled LibreSSL")
- set(BUILD_ARGS "")
- if (WIN32)
- set(BUILD_ARGS " -GVisual Studio 15 2017")
- endif(WIN32)
- ExternalProject_Add(
- libressl-portable
- GIT_REPOSITORY
"https://github.com/libressl-portable/portable.git;
- GIT_TAG "v2.8.3"
- SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-src"
- CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
-
"-DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install"
- "-DLIBRESSL_APPS=OFF"
- "-DLIBRESSL_TESTS=OFF"
- "${BUILD_ARGS}"
- PATCH_COMMAND ./autogen.sh
- )
-
+ include(LibreSSL)
+ use_libre_ssl(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
list(APPEND CMAKE_MODULE_PATH
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/ssl")
- add_library(crypto STATIC IMPORTED)
- set_target_properties(crypto PROPERTIES IMPORTED_LOCATION
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/lib/libcrypto.a")
- add_dependencies(crypto libressl-portable)
-
- add_library(ssl STATIC IMPORTED)
- set_target_properties(ssl PROPERTIES IMPORTED_LOCATION
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/lib/libssl.a")
- set_target_properties(ssl PROPERTIES INTERFACE_LINK_LIBRARIES
crypto)
- add_dependencies(ssl libressl-portable)
-
- add_library(tls STATIC IMPORTED)
- set_target_properties(tls PROPERTIES IMPORTED_LOCATION
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/lib/libtls.a")
- set_target_properties(tls PROPERTIES INTERFACE_LINK_LIBRARIES
crypto)
- add_dependencies(tls libressl-portable)
-
- set(LIBRESSL_SRC_DIR
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/libressl/" CACHE STRING "" FORCE)
- set(LIBRESSL_BIN_DIR
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/" CACHE STRING ""
FORCE)
-
- set(OPENSSL_FOUND "YES" CACHE STRING "" FORCE)
- set(OPENSSL_INCLUDE_DIR
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/libressl/include" CACHE STRING "" FORCE)
- set(OPENSSL_LIBRARIES
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/libressl-install/lib/libtls.a"
"${CMAKE_CURRENT_BINARY_DIR}/thirdparty/
[nifi-minifi-cpp] branch master updated: MINIFICPP-809: Better support python capabilities and recordreader/writers
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 58905c2 MINIFICPP-809: Better support python capabilities and
recordreader/writers
58905c2 is described below
commit 58905c29ab563af4c579eb6467d17b80984e5ea2
Author: Marc Parisi
AuthorDate: Fri Apr 19 11:27:01 2019 -0400
MINIFICPP-809: Better support python capabilities and recordreader/writers
MINIFICPP-809: better support append
MINIFICPP-809: Add google sentiment analyzer example
MINIFICPP-809: Fix linter issues
MINIFICPP-809: Resolving attribute issue seen during demo
This closes #540.
Signed-off-by: Marc Parisi
---
extensions/jni/ExecuteJavaControllerService.cpp| 6 +-
extensions/jni/ExecuteJavaProcessor.cpp| 7 +-
extensions/jni/jvm/JavaControllerService.h | 9 ++
extensions/jni/jvm/JniProcessSession.cpp | 15 ++-
extensions/jni/jvm/NarClassLoader.h| 52
extensions/jni/nifi-framework-jni/pom.xml | 1 -
.../org/apache/nifi/processor/JniClassLoader.java | 136 -
.../apache/nifi/processor/JniProcessSession.java | 68 +--
.../{ => examples}/SentimentAnalysis.py| 0
.../pythonprocessors/google/SentimentAnalyzer.py | 64 ++
.../rocksdb-repos/DatabaseContentRepository.cpp| 3 +-
.../rocksdb-repos/DatabaseContentRepository.h | 2 +-
.../script/python/ExecutePythonProcessor.cpp | 3 +-
extensions/script/python/PythonCreator.h | 11 +-
libminifi/include/core/StreamManager.h | 2 +-
.../include/core/repository/FileSystemRepository.h | 2 +-
.../core/repository/VolatileContentRepository.h| 2 +-
libminifi/include/io/FileStream.h | 4 +-
libminifi/src/core/ProcessSession.cpp | 14 ++-
.../src/core/repository/FileSystemRepository.cpp | 4 +-
.../core/repository/VolatileContentRepository.cpp | 2 +-
libminifi/src/io/FileStream.cpp| 7 +-
22 files changed, 343 insertions(+), 71 deletions(-)
diff --git a/extensions/jni/ExecuteJavaControllerService.cpp
b/extensions/jni/ExecuteJavaControllerService.cpp
index 50210ea..7ea1585 100644
--- a/extensions/jni/ExecuteJavaControllerService.cpp
+++ b/extensions/jni/ExecuteJavaControllerService.cpp
@@ -97,10 +97,12 @@ void ExecuteJavaControllerService::onEnable() {
clazzInstance = java_servicer_->newInstance(class_name_);
- auto onEnabledName = java_servicer_->getAnnotation(class_name_, "OnEnabled");
+ auto methods_with_signatures = java_servicer_->getAnnotations(class_name_,
"OnEnabled");
current_cs_class = java_servicer_->getObjectClass(class_name_,
clazzInstance);
try {
-current_cs_class.callVoidMethod(env, clazzInstance,
onEnabledName.first.c_str(), onEnabledName.second, contextInstance);
+for (const auto : methods_with_signatures) {
+ current_cs_class.callVoidMethod(env, clazzInstance, mwithsig.first,
mwithsig.second, contextInstance);
+}
} catch (std::runtime_error ) {
// this can be ignored.
}
diff --git a/extensions/jni/ExecuteJavaProcessor.cpp
b/extensions/jni/ExecuteJavaProcessor.cpp
index 12f7e39..3a7d154 100644
--- a/extensions/jni/ExecuteJavaProcessor.cpp
+++ b/extensions/jni/ExecuteJavaProcessor.cpp
@@ -139,7 +139,7 @@ void ExecuteJavaProcessor::onSchedule(const
std::shared_ptrnewInstance(class_name_);
- auto onScheduledName = java_servicer_->getAnnotation(class_name_,
"OnScheduled");
+ auto onScheduledNames = java_servicer_->getAnnotations(class_name_,
"OnScheduled");
current_processor_class = java_servicer_->getObjectClass(class_name_,
clazzInstance);
// attempt to schedule here
@@ -158,7 +158,10 @@ void ExecuteJavaProcessor::onSchedule(const
std::shared_ptrsetReference(env,
context_instance_, );
try {
-current_processor_class.callVoidMethod(env, clazzInstance,
onScheduledName.first.c_str(), onScheduledName.second, context_instance_);
+
+for (const auto : onScheduledNames) {
+ current_processor_class.callVoidMethod(env, clazzInstance,
onScheduledName.first.c_str(), onScheduledName.second, context_instance_);
+}
} catch (std::runtime_error ) {
// this can be ignored.
}
diff --git a/extensions/jni/jvm/JavaControllerService.h
b/extensions/jni/jvm/JavaControllerService.h
index 9721440..4843b77 100644
--- a/extensions/jni/jvm/JavaControllerService.h
+++ b/extensions/jni/jvm/JavaControllerService.h
@@ -133,6 +133,15 @@ class JavaControllerService : public
core::controller::ControllerService, public
std::pair getAnnotation(const std::string
_name, const std::string _name) {
return nar_loader_->getAnnotation(requested_name,
[nifi-minifi-cpp] branch master updated: MINIFICPP-818 - Remove fileutils dependency from putfile tests
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 93420df MINIFICPP-818 - Remove fileutils dependency from putfile tests
93420df is described below
commit 93420dfe1c30feed63ac7bbacb272a529ca7922a
Author: Arpad Boda
AuthorDate: Wed Apr 24 12:11:46 2019 +0200
MINIFICPP-818 - Remove fileutils dependency from putfile tests
This closes #543.
Signed-off-by: Marc Parisi
---
extensions/standard-processors/tests/unit/PutFileTests.cpp | 13 +
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/extensions/standard-processors/tests/unit/PutFileTests.cpp
b/extensions/standard-processors/tests/unit/PutFileTests.cpp
index cc34f4b..e21c6a7 100644
--- a/extensions/standard-processors/tests/unit/PutFileTests.cpp
+++ b/extensions/standard-processors/tests/unit/PutFileTests.cpp
@@ -369,11 +369,16 @@ TEST_CASE("PutFileMaxFileCountTest",
"[getfileputpfilemaxcount]") {
// Only 1 of the 2 files should make it to the target dir
// Non-determistic, so let's just count them
int files_in_dir = 0;
- auto lambda = [_in_dir](const std::string&, const std::string&) ->
bool {
-return ++files_in_dir < 2;
- };
- utils::file::FileUtils::list_dir(putfiledir, lambda,
testController.getLogger(), false);
+ for (int i = 0; i < 2; ++i) {
+std::stringstream ss;
+ss << putfiledir << "/" << "tstFile" << i << ".ext";
+std::ifstream file(ss.str());
+if (file.is_open() && file.good()) {
+ files_in_dir++;
+ file.close();
+}
+ }
REQUIRE(files_in_dir == 1);
[nifi-minifi-cpp] branch master updated: MINIFICPP-830 - update Xcode versions in Travis build
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 9ef45e7 MINIFICPP-830 - update Xcode versions in Travis build
9ef45e7 is described below
commit 9ef45e7071e4756bd050c801106edc3c598270e9
Author: Dustin Rodrigues
AuthorDate: Fri Apr 26 22:39:22 2019 -0400
MINIFICPP-830 - update Xcode versions in Travis build
This closes #547.
Signed-off-by: Marc Parisi
---
.travis.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index af65036..0128c0f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -130,7 +130,7 @@ matrix:
script:
- ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS=ON ..
&& make -j2 VERBOSE=1 && sudo make test ARGS="-j2 --output-on-failure"
- os: osx
- osx_image: xcode8.3
+ osx_image: xcode9.2
env:
- CMAKE_BUILD_OPTIONS="${CMAKE_BUILD_OPTIONS} -DENABLE_LUA_SCRIPTING=1"
# https://docs.travis-ci.com/user/osx-ci-environment/#Homebrew
@@ -151,7 +151,7 @@ matrix:
- package='libpcap'; [[ $(brew ls --versions ${package}) ]] && { brew
outdated ${package} || brew upgrade ${package}; } || brew install ${package}
- package='xz'; [[ $(brew ls --versions ${package}) ]] && { brew
outdated ${package} || brew upgrade ${package}; } || brew install ${package}
- os: osx
- osx_image: xcode9.1
+ osx_image: xcode10
env:
- CMAKE_BUILD_OPTIONS="${CMAKE_BUILD_OPTIONS} -DENABLE_LUA_SCRIPTING=1"
# https://docs.travis-ci.com/user/osx-ci-environment/#Homebrew
[nifi-minifi-cpp] branch master updated: MINIFICPP-817 - Update cxxopts version to 2.1.2
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new fdb46dd MINIFICPP-817 - Update cxxopts version to 2.1.2
fdb46dd is described below
commit fdb46dd72ecbf596eb4588714304566151096b68
Author: Daniel Bakai
AuthorDate: Wed Apr 24 11:20:04 2019 +0200
MINIFICPP-817 - Update cxxopts version to 2.1.2
This closes #544.
Signed-off-by: Marc Parisi
---
thirdparty/cxxopts/CHANGELOG.md| 27 +--
thirdparty/cxxopts/CMakeLists.txt | 12 ++--
thirdparty/cxxopts/README.md | 17 -
thirdparty/cxxopts/include/cxxopts.hpp | 31 ---
thirdparty/cxxopts/src/example.cpp | 1 +
5 files changed, 76 insertions(+), 12 deletions(-)
diff --git a/thirdparty/cxxopts/CHANGELOG.md b/thirdparty/cxxopts/CHANGELOG.md
index 01b67b6..792c2e2 100644
--- a/thirdparty/cxxopts/CHANGELOG.md
+++ b/thirdparty/cxxopts/CHANGELOG.md
@@ -3,6 +3,19 @@
This is the changelog for `cxxopts`, a C++11 library for parsing command line
options. The project adheres to semantic versioning.
+## 2.1.2
+
+### Bug Fixes
+
+* Use `std::forward` instead of returning a copy in `toLocalString`.
+
+## 2.1.1
+
+### Bug Fixes
+
+* Revert the change adding `const` type for `argv`, because most users expect
+ to pass a non-const `argv` from `main`.
+
## 2.1
### Changed
@@ -12,12 +25,22 @@ options. The project adheres to semantic versioning.
when a positional argument could follow an option with an implicit value.
For example, `--foo value`, where `foo` has an implicit value, will be
parsed as `--foo=implicit` and a positional argument `value`.
-* Fixed an ambiguous overload in the `parse_positional` function when an
- `initializer_list` was directly passed.
+* Boolean values are no longer special, but are just an option with a default
+ and implicit value.
+
+### Added
+
+* Added support for `std::optional` as a storage type.
+* Allow the help string to be customised.
+* Use `const` for the type in the `argv` parameter, since the contents of the
+ arguments is never modified.
### Bug Fixes
* Building against GCC 4.9 was broken due to overly strict shadow warnings.
+* Fixed an ambiguous overload in the `parse_positional` function when an
+ `initializer_list` was directly passed.
+* Fixed precedence in the Boolean value regex.
## 2.0
diff --git a/thirdparty/cxxopts/CMakeLists.txt
b/thirdparty/cxxopts/CMakeLists.txt
index 192b529..892caa9 100644
--- a/thirdparty/cxxopts/CMakeLists.txt
+++ b/thirdparty/cxxopts/CMakeLists.txt
@@ -20,14 +20,22 @@
cmake_minimum_required(VERSION 3.1)
project(cxxopts)
+enable_testing()
-set(VERSION "1.2.0")
+set(VERSION "2.1.2")
option(CXXOPTS_BUILD_EXAMPLES "Set to ON to build examples" ON)
+option(CXXOPTS_BUILD_TESTS "Set to ON to build tests" OFF)
# request c++11 without gnu extension for the whole project and enable more
warnings
-set(CMAKE_CXX_STANDARD 11)
+if (CXXOPTS_CXX_STANDARD)
+ set(CMAKE_CXX_STANDARD ${CXXOPTS_CXX_STANDARD})
+else()
+ set(CMAKE_CXX_STANDARD 11)
+endif()
+
set(CMAKE_CXX_EXTENSIONS OFF)
+
if(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /W2")
elseif(CMAKE_CXX_COMPILER_ID MATCHES "[Cc]lang" OR CMAKE_CXX_COMPILER_ID
MATCHES "GNU")
diff --git a/thirdparty/cxxopts/README.md b/thirdparty/cxxopts/README.md
index c5c171b..e575e61 100644
--- a/thirdparty/cxxopts/README.md
+++ b/thirdparty/cxxopts/README.md
@@ -1,5 +1,10 @@
[](https://travis-ci.org/jarro2783/cxxopts)
+# Release versions
+
+Note that `master` is generally a work in progress, and you probably want to
use a
+tagged release version.
+
# Quick start
This is a lightweight C++ option parser library, supporting the standard GNU
@@ -52,6 +57,17 @@ exception will be thrown.
Note that the result of `options.parse` should only be used as long as the
`options` object that created it is in scope.
+## Exceptions
+
+Exceptional situations throw C++ exceptions. There are two types of
+exceptions: errors defining the options, and errors when parsing a list of
+arguments. All exceptions derive from `cxxopts::OptionException`. Errors
+defining options derive from `cxxopts::OptionSpecException` and errors
+parsing arguments derive from `cxxopts::OptionParseException`.
+
+All exceptions define a `what()` function to get a printable string
+explaining the error.
+
## Help groups
Options can be placed into groups for the purposes of displaying help messages.
@@ -116,4 +132,3 @@ expressions. For example GCC >= 4.9 or clang with libc++.
# TODO list
* Allow unrecognised options.
-* Various help strings.
diff --git a/thirdparty/cxxopts/in
[nifi-minifi-cpp] branch master updated: MINIFICPP-17: Move processors to standard-processors in preparation for future changes.
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 293b982 MINIFICPP-17: Move processors to standard-processors in preparation for future changes. 293b982 is described below commit 293b982801f6475647a67021325690bec4eb Author: Marc Parisi AuthorDate: Wed Apr 17 16:10:06 2019 -0400 MINIFICPP-17: Move processors to standard-processors in preparation for future changes. This closes #538. Approved on GH By @arpadboda. MINIFICPP-17: Fix linter error MINIFICPP-17: Update per comments Signed-off-by: Marc Parisi --- CMakeLists.txt | 3 + cmake/BuildTests.cmake | 24 --- extensions/coap/tests/CMakeLists.txt | 7 +- extensions/http-curl/tests/CMakeLists.txt | 10 +-- .../standard-processors}/CMakeLists.txt| 34 +- .../processors/AppendHostInfo.cpp | 2 +- .../processors/AppendHostInfo.h| 0 .../processors/ExecuteProcess.cpp | 2 +- .../processors/ExecuteProcess.h| 0 .../processors/ExtractText.cpp | 2 +- .../standard-processors}/processors/ExtractText.h | 0 .../processors/GenerateFlowFile.cpp| 2 +- .../processors/GenerateFlowFile.h | 0 .../standard-processors}/processors/GetFile.cpp| 2 +- .../standard-processors}/processors/GetFile.h | 2 +- .../standard-processors}/processors/GetTCP.cpp | 2 +- .../standard-processors}/processors/GetTCP.h | 0 .../processors/HashContent.cpp | 2 +- .../standard-processors}/processors/HashContent.h | 0 .../processors/ListenSyslog.cpp| 2 +- .../standard-processors}/processors/ListenSyslog.h | 0 .../processors/LogAttribute.cpp| 2 +- .../standard-processors}/processors/LogAttribute.h | 0 .../standard-processors}/processors/PutFile.cpp| 2 +- .../standard-processors}/processors/PutFile.h | 0 .../processors/RouteOnAttribute.cpp| 2 +- .../processors/RouteOnAttribute.h | 0 .../standard-processors}/processors/TailFile.cpp | 2 +- .../standard-processors}/processors/TailFile.h | 0 .../processors/UpdateAttribute.cpp | 2 +- .../processors/UpdateAttribute.h | 0 .../standard-processors/tests/CMakeLists.txt | 75 ++ .../tests}/integration/SecureSocketGetTCPTest.cpp | 6 +- .../tests}/integration/TailFileCronTest.cpp| 4 +- .../tests}/integration/TailFileTest.cpp| 4 +- .../tests}/integration/TestExecuteProcess.cpp | 4 +- .../tests}/unit/ClassLoaderTests.cpp | 2 +- .../tests}/unit/ControllerServiceTests.cpp | 10 +-- .../tests}/unit/ExtractTextTests.cpp | 8 +-- .../tests}/unit/GetFileTests.cpp | 6 +- .../tests}/unit/GetTCPTests.cpp| 8 +-- .../tests}/unit/HashContentTest.cpp| 8 +-- .../tests}/unit/ManifestTests.cpp | 6 +- .../tests}/unit/ProcessorTests.cpp | 8 +-- .../tests}/unit/PutFileTests.cpp | 4 +- .../tests}/unit/SchedulingAgentTests.cpp | 6 +- .../tests}/unit/TailFileTests.cpp | 10 +-- .../tests}/unit/UpdateAttributeTests.cpp | 8 +-- .../tests}/unit/YamlConfigurationTests.cpp | 2 +- libminifi/include/processors/ProcessorUtils.h | 55 libminifi/src/core/FlowConfiguration.cpp | 48 ++ libminifi/src/core/yaml/YamlConfiguration.cpp | 2 +- .../processors/ProcessorUtils.cpp} | 28 libminifi/test/archive-tests/CMakeLists.txt| 5 +- libminifi/test/bustache-tests/CMakeLists.txt | 5 +- libminifi/test/civetweb-tests/CMakeLists.txt | 5 +- libminifi/test/civetweb-tests/CivetwebTests.cpp| 6 +- .../test/expression-language-tests/CMakeLists.txt | 12 ++-- libminifi/test/gps-tests/CMakeLists.txt| 5 +- libminifi/test/script-tests/CMakeLists.txt | 8 ++- libminifi/test/sqlite-tests/CMakeLists.txt | 6 +- nanofi/CMakeLists.txt | 13 ++-- nanofi/src/api/nanofi.cpp | 7 +- 63 files changed, 309 insertions(+), 181 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index b4fcaf3..890ab07 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -472,6 +472,9 @@ if (WIN32 OR NOT USE_SYSTEM_ZLIB) add_dependencies(minifi zlib-external) endif(WIN32 OR NOT USE_SYSTEM_ZLIB) +createExtension(STANDARD-PROCESSORS "STANDARD PROCESSORS" "Pr
[nifi-minifi-cpp] branch master updated: MINIFICPP-799 - Fixed bison install script in MacOS bootstrap
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 2fc64e8 MINIFICPP-799 - Fixed bison install script in MacOS bootstrap
2fc64e8 is described below
commit 2fc64e874e79678184456237fd734de80a905f8b
Author: Daniel Bakai
AuthorDate: Wed Apr 24 11:27:00 2019 +0200
MINIFICPP-799 - Fixed bison install script in MacOS bootstrap
This closes #542.
Signed-off-by: Marc Parisi
---
darwin.sh | 9 +++--
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/darwin.sh b/darwin.sh
index 91a3139..4f79f69 100644
--- a/darwin.sh
+++ b/darwin.sh
@@ -50,12 +50,9 @@ install_bison() {
fi
fi
if [ "$BISON_INSTALLED" = "false" ]; then
-## ensure that the toolchain is installed
-INSTALL_BASE="sudo zypper in -y gcc gcc-c++"
-${INSTALL_BASE}
-wget https://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.xz
-tar xvf bison-3.0.4.tar.xz
-pushd bison-3.0.4
+wget https://ftp.gnu.org/gnu/bison/bison-3.0.5.tar.xz
+tar xvf bison-3.0.5.tar.xz
+pushd bison-3.0.5
./configure
make
sudo make install
[nifi-minifi-cpp] branch master updated: MINIFICPP-797 - Maximum File Count property of PutFile processor is not working properly
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new eb56f1b MINIFICPP-797 - Maximum File Count property of PutFile
processor is not working properly
eb56f1b is described below
commit eb56f1b62d1aaf2f37342f9148b0eb05c92d24d5
Author: Arpad Boda
AuthorDate: Tue Apr 2 22:34:17 2019 +0200
MINIFICPP-797 - Maximum File Count property of PutFile processor is not
working properly
MINIFICPP-797 - Added test to verify MaxDestFiles attribute
This closes #531.
Signed-off-by: Marc Parisi
---
libminifi/src/processors/PutFile.cpp | 52 +++--
libminifi/test/TestBase.cpp | 15 +
libminifi/test/TestBase.h| 16 -
libminifi/test/unit/PutFileTests.cpp | 64
4 files changed, 100 insertions(+), 47 deletions(-)
diff --git a/libminifi/src/processors/PutFile.cpp
b/libminifi/src/processors/PutFile.cpp
index b279010..72ec850 100644
--- a/libminifi/src/processors/PutFile.cpp
+++ b/libminifi/src/processors/PutFile.cpp
@@ -138,49 +138,21 @@ void PutFile::onTrigger(core::ProcessContext *context,
core::ProcessSession *ses
if (S_ISDIR(statResult.st_mode)) {
// it's a directory, count the files
int64_t ct = 0;
-#ifndef WIN32
- DIR *myDir = opendir(directory.c_str());
- if (!myDir) {
-logger_->log_warn("Could not open %s", directory);
+
+ // Callback, called for each file entry in the listed directory
+ // Return value is used to break (false) or continue (true) listing
+ auto lambda = [, this](const std::string&, const std::string&) ->
bool {
+return ++ct < max_dest_files_;
+ };
+
+ utils::file::FileUtils::list_dir(directory, lambda, logger_, false);
+
+ if (ct >= max_dest_files_) {
+logger_->log_warn("Routing to failure because the output directory %s
has at least %u files, which exceeds the "
+ "configured max number of files", directory,
max_dest_files_);
session->transfer(flowFile, Failure);
return;
}
- struct dirent* entry = nullptr;
-
- while ((entry = readdir(myDir)) != nullptr) {
-if ((strcmp(entry->d_name, ".") != 0) && (strcmp(entry->d_name, "..")
!= 0)) {
- ct++;
- if (ct >= max_dest_files_) {
-logger_->log_warn("Routing to failure because the output directory
%s has at least %u files, which exceeds the "
- "configured max number of files",
- directory, max_dest_files_);
-session->transfer(flowFile, Failure);
-closedir(myDir);
-return;
- }
-}
- }
- closedir(myDir);
-#else
- HANDLE hFind;
- WIN32_FIND_DATA FindFileData;
-
- if ((hFind = FindFirstFile(directory.c_str(), )) !=
INVALID_HANDLE_VALUE) {
-do {
- if ((strcmp(FindFileData.cFileName, ".") != 0) &&
(strcmp(FindFileData.cFileName, "..") != 0)) {
-ct++;
-if (ct >= max_dest_files_) {
- logger_->log_warn("Routing to failure because the output
directory %s has at least %u files, which exceeds the "
- "configured max number of files", directory,
max_dest_files_);
- session->transfer(flowFile, Failure);
- FindClose(hFind);
- return;
-}
- }
-}while (FindNextFile(hFind, ));
-FindClose(hFind);
- }
-#endif
}
}
diff --git a/libminifi/test/TestBase.cpp b/libminifi/test/TestBase.cpp
index 0d2b92b..43eead4 100644
--- a/libminifi/test/TestBase.cpp
+++ b/libminifi/test/TestBase.cpp
@@ -32,7 +32,7 @@ TestPlan::TestPlan(std::shared_ptr
content_repo, std::s
stream_factory =
org::apache::nifi::minifi::io::StreamFactory::getInstance(std::make_shared());
}
-std::shared_ptr TestPlan::addProcessor(const
std::shared_ptr , const std::string ,
core::Relationship relationship, bool linkToPrevious) {
+std::shared_ptr TestPlan::addProcessor(const
std::shared_ptr , const std::string , const
std::initializer_list& relationships, bool linkToPrevious) {
if (finalized) {
return nullptr;
}
@@ -50,20 +50,23 @@ std::shared_ptr
TestPlan::addProcessor(const std::shared_ptrgetUUIDStr()] = processor;
if (!linkToPrevious) {
-termination_ = relationship;
+termination_ = *(relationships.begin());
} else {
std::shared_ptr last = processor_queue_.back();
if (last == nullptr) {
last = processor;
- termination_ = relationship;
+ termination_ = *(relations
[nifi-minifi-cpp] branch master updated: MINIFICPP-726 - Enhance ExtractText to have more feature parity with the Java impl
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 8022070 MINIFICPP-726 - Enhance ExtractText to have more feature
parity with the Java impl
8022070 is described below
commit 8022070f4d030559663ba3cfca12dcfa3a909a4e
Author: Arpad Boda
AuthorDate: Mon Apr 8 12:04:48 2019 +0200
MINIFICPP-726 - Enhance ExtractText to have more feature parity with the
Java impl
MINIFICPP-726 - Add error handling for invalid regular expressions
MINIFICPP-726 - Add gcc4.8 compatible solution
This closes #533.
Signed-off-by: Marc Parisi
---
libminifi/include/processors/ExtractText.h | 18 ++-
libminifi/src/processors/ExtractText.cpp | 172 -
libminifi/test/unit/ExtractTextTests.cpp | 63 +++
3 files changed, 244 insertions(+), 9 deletions(-)
diff --git a/libminifi/include/processors/ExtractText.h
b/libminifi/include/processors/ExtractText.h
index 4caf349..5e6ff4c 100644
--- a/libminifi/include/processors/ExtractText.h
+++ b/libminifi/include/processors/ExtractText.h
@@ -50,19 +50,30 @@ public:
//! Supported Properties
static core::Property Attribute;
static core::Property SizeLimit;
+
+static core::Property RegexMode;
+static core::Property IgnoreCaptureGroupZero;
+static core::Property InsensitiveMatch;
+static core::Property MaxCaptureGroupLen;
+static core::Property EnableRepeatingCaptureGroup;
+
//! Supported Relationships
static core::Relationship Success;
//! Default maximum bytes to read into an attribute
static constexpr int DEFAULT_SIZE_LIMIT = 2 * 1024 * 1024;
//! OnTrigger method, implemented by NiFi ExtractText
-void onTrigger(core::ProcessContext *context, core::ProcessSession
*session) override;
+void onTrigger(core::ProcessContext *context, core::ProcessSession
*session);
//! Initialize, over write by NiFi ExtractText
-void initialize(void) override;
+void initialize(void);
+
+virtual bool supportsDynamicProperties() {
+ return true;
+};
class ReadCallback : public InputStreamCallback {
public:
-ReadCallback(std::shared_ptr flowFile,
core::ProcessContext *ct);
+ReadCallback(std::shared_ptr flowFile,
core::ProcessContext *ct, std::shared_ptr lgr);
~ReadCallback() {}
int64_t process(std::shared_ptr stream);
@@ -70,6 +81,7 @@ public:
std::shared_ptr flowFile_;
core::ProcessContext *ctx_;
std::vector buffer_;
+std::shared_ptr logger_;
};
protected:
diff --git a/libminifi/src/processors/ExtractText.cpp
b/libminifi/src/processors/ExtractText.cpp
index fcd9eea..d1a5c0d 100644
--- a/libminifi/src/processors/ExtractText.cpp
+++ b/libminifi/src/processors/ExtractText.cpp
@@ -21,6 +21,7 @@
#include
#include
#include
+#include
#include
#include
@@ -30,6 +31,12 @@
#include "core/ProcessSession.h"
#include "core/FlowFile.h"
+#if !defined(_WIN32)
+#if __cplusplus <= 201103L
+#include
+#endif
+#endif
+
namespace org {
namespace apache {
namespace nifi {
@@ -37,13 +44,44 @@ namespace minifi {
namespace processors {
#define MAX_BUFFER_SIZE 4096
+#define MAX_CAPTURE_GROUP_SIZE 1024
core::Property
ExtractText::Attribute(core::PropertyBuilder::createProperty("Attribute")->withDescription("Attribute
to set from content")->build());
// despite there being a size value, ExtractText was initially built with a
numeric for this property
core::Property ExtractText::SizeLimit(
-core::PropertyBuilder::createProperty("Size
Limit")->withDescription("Maximum number of bytes to read into the attribute. 0
for no limit. Default is 2MB.")->withDefaultValue(
-DEFAULT_SIZE_LIMIT)->build());
+core::PropertyBuilder::createProperty("Size Limit")
+->withDescription("Maximum number of bytes to read into the attribute. 0
for no limit. Default is 2MB.")
+->withDefaultValue(DEFAULT_SIZE_LIMIT)->build());
+
+core::Property ExtractText::RegexMode(
+core::PropertyBuilder::createProperty("Regex Mode")
+->withDescription("Set this to extract parts of flowfile content using
regular experssions in dynamic properties")
+->withDefaultValue(false)->build());
+
+core::Property ExtractText::IgnoreCaptureGroupZero(
+core::PropertyBuilder::createProperty("Include Capture Group 0")
+->withDescription("Indicates that Capture Group 0 should be included as an
attribute. "
+ "Capture Group 0 represents the entirety of the regular
expression match, is typically not used, and could have considerable length."
[nifi-minifi-cpp] branch master updated: MINIFICPP-804 - C API Python examples are broken
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 9d378e9 MINIFICPP-804 - C API Python examples are broken
9d378e9 is described below
commit 9d378e978c93a0df08ffd65be1168abdbf113b9f
Author: Arpad Boda
AuthorDate: Wed Apr 17 15:44:11 2019 +0200
MINIFICPP-804 - C API Python examples are broken
Closes #537.
Signed-off-by: Marc Parisi
---
nanofi/include/api/nanofi.h | 2 +-
python/getFile.py | 6 --
python/minifi/__init__.py | 10 ++
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/nanofi/include/api/nanofi.h b/nanofi/include/api/nanofi.h
index c014030..04e5712 100644
--- a/nanofi/include/api/nanofi.h
+++ b/nanofi/include/api/nanofi.h
@@ -262,7 +262,7 @@ flow_file_record *invoke_file(standalone_processor* proc,
const char* path);
**/
flow_file_record *invoke_chunk(standalone_processor *proc, uint8_t *buf,
uint64_t size);
-int transfer(processor_session* session, flow *flow, const char *rel);
+DEPRECATED int transfer(processor_session* session, flow *flow, const char
*rel);
/**
* Creates a flow file record based on a file
diff --git a/python/getFile.py b/python/getFile.py
index fd3e335..c8aff98 100644
--- a/python/getFile.py
+++ b/python/getFile.py
@@ -36,7 +36,9 @@ class GetFilePrinterProcessor(PyProcessor):
if not flow_file.add_attribute("python_test","value2"):
print("Cannot add the same attribute twice!")
print ("original file name: " +
flow_file.get_attribute("filename"))
-self.transfer(session, flow_file, "success")
+target_relationship = "success"
+if not self.transfer(session, flow_file, target_relationship):
+print("transfer to relationship " + target_relationship +
" failed")
return CALLBACK(onTrigger)
@@ -67,7 +69,7 @@ processor.set_property("Keep Source File", "true")
current_module = sys.modules[__name__]
-processor =
minifi.create_python_processor(current_module,"GetFilePrinterProcessor")
+processor = minifi.create_python_processor(current_module,
"GetFilePrinterProcessor")
ff = minifi.get_next_flowfile()
if ff:
diff --git a/python/minifi/__init__.py b/python/minifi/__init__.py
index b6de1d6..4632347 100644
--- a/python/minifi/__init__.py
+++ b/python/minifi/__init__.py
@@ -60,7 +60,7 @@ class Processor(object):
self._minifi = minifi
def set_property(self, name, value):
-self._minifi.set_property( self._proc, name.encode("UTF-8"),
value.encode("UTF-8"))
+return self._minifi.set_property( self._proc, name.encode("UTF-8"),
value.encode("UTF-8")) == 0
class PyProcessor(object):
def __init__(self, minifi, flow):
@@ -79,7 +79,7 @@ class PyProcessor(object):
return None
def transfer(self, session, ff, rel):
-self._minifi.transfer(session, self._flow, rel.encode("UTF-8"))
+return self._minifi.transfer_to_relationship(ff.get_instance(),
session, rel.encode("UTF-8")) == 0
@abstractmethod
def _onTriggerCallback(self):
@@ -120,8 +120,7 @@ class FlowFile(object):
def add_attribute(self, name, value):
vallen = len(value)
-ret = self._minifi.add_attribute(self._ff, name.encode("UTF-8"),
value.encode("UTF-8"), vallen)
-return True if ret == 0 else False
+return self._minifi.add_attribute(self._ff, name.encode("UTF-8"),
value.encode("UTF-8"), vallen) == 0
def update_attribute(self, name, value):
vallen = len(value)
@@ -167,6 +166,9 @@ class MiNiFi(object):
""" transfer ff """
self._minifi.transfer.argtypes = [ctypes.POINTER(CProcessSession),
ctypes.POINTER(CFlow) , ctypes.c_char_p ]
self._minifi.transfer.restype = ctypes.c_int
+""" transfer ff to relationship """
+self._minifi.transfer_to_relationship.argtypes =
[ctypes.POINTER(CFlowFile), ctypes.POINTER(CProcessSession), ctypes.c_char_p ]
+self._minifi.transfer_to_relationship.restype = ctypes.c_int
""" add attribute to ff """
self._minifi.add_attribute.argtypes = [ctypes.POINTER(CFlowFile),
ctypes.c_char_p, ctypes.c_char_p, ctypes.c_int ]
self._minifi.add_attribute.restype = ctypes.c_int
[nifi-minifi-cpp] branch master updated: MINIFICPP-811 - fix file permissions
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 365b3c6 MINIFICPP-811 - fix file permissions 365b3c6 is described below commit 365b3c6b6af4b0893b585f66e4d7a0b121f60130 Author: Dustin Rodrigues AuthorDate: Sat Apr 20 13:26:48 2019 -0400 MINIFICPP-811 - fix file permissions This closes #541. Signed-off-by: Marc Parisi --- extensions/jni/nifi-minifi-jni-assembly/pom.xml | 0 libminifi/test/unit/SiteToSiteHelper.h | 0 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/extensions/jni/nifi-minifi-jni-assembly/pom.xml b/extensions/jni/nifi-minifi-jni-assembly/pom.xml old mode 100755 new mode 100644 diff --git a/libminifi/test/unit/SiteToSiteHelper.h b/libminifi/test/unit/SiteToSiteHelper.h old mode 100755 new mode 100644
[nifi-minifi-cpp] branch master updated: MINIFICPP-808 - small doc updates
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 9751cfd MINIFICPP-808 - small doc updates
9751cfd is described below
commit 9751cfd55bed2a719029a75a585155ca7caa854a
Author: Dustin Rodrigues
AuthorDate: Fri Apr 19 09:48:22 2019 -0400
MINIFICPP-808 - small doc updates
This closes #539.
Signed-off-by: Marc Parisi
---
PROCESSORS.md | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/PROCESSORS.md b/PROCESSORS.md
index 2090632..079e055 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -123,7 +123,7 @@ default values, and whether a property supports the NiFi
Expression Language.
| Base Directory | /tmp/ | | A base directory where pcap files are stored
temporarily |
| Batch Size | 50 | | Number of captured packets written into a PCAP file |
| Capture Bluetooth | false | | Captures bluetooth interfaces if true |
-| Network Controller | .* | | Regular expression of the network controller(s)
to which packet capture will be attached|
+| Network Controller | .\* | | Regular expression of the network
controller(s) to which packet capture will be attached|
### Relationships
@@ -632,13 +632,14 @@ default values, and whether a property supports the NiFi
Expression Language.
| Name | Default Value | Allowable Values | Description |
| - | - | - | - |
-| **Base Path** | contentListener | | Base path for incoming connections |
+| Base Path | contentListener | | Base path for incoming connections |
| **Listening Port** | | | The Port to listen on for incoming connections |
| SSL Certificate | | | File containing PEM-formatted file including TLS/SSL
certificate and key |
| SSL Certificate Authority | | | File containing trusted PEM-formatted
certificates |
| SSL Verify Peer | no | yes, no | Whether or not to verify the client's
certificate |
| SSL Minimum Version | SSL2 | SSL2, SSL3, TLS1.0, TLS1.1, TLS1.2 | Minimum
TLS/SSL version allowed |
| HTTP Headers to receive as Attributes (Regex) | | | Specifies the Regular
Expression that determines the names of HTTP Headers that should be passed
along as FlowFile attributes |
+| Authorized DN Pattern | .\* | | Specifies the Regular Expression that
determines the names of HTTP Headers that should be passed along as FlowFile
attributes |
### Relationships
@@ -702,10 +703,10 @@ default values, and whether a property supports the NiFi
Expression Language.
| Name | Default Value | Allowable Values | Description |
| - | - | - | - |
-| **Directory** | | | The directory to which files should be written. You may
use expression language such as `/aa/bb/${path}`**Supports Expression
Language: true** |
-| **Conflict Resolution Strategy** | fail | replace, ignore, fail | Indicates
what should happen when a file with the same name already exists in the output
directory |
+| Directory | . | | The directory to which files should be written. You may
use expression language such as `/aa/bb/${path}`**Supports Expression
Language: true** |
+| Conflict Resolution Strategy | fail | replace, ignore, fail | Indicates what
should happen when a file with the same name already exists in the output
directory |
| **Create Missing Directories** | true | true, false | If true, then missing
destination directories will be created. If false, flowfiles are penalized and
sent to failure. |
-| **Maximum File Count** | | | Specifies the maximum number of files that can
exist in the output directory |
+| Maximum File Count | | | Specifies the maximum number of files that can
exist in the output directory |
### Relationships
[nifi-minifi-cpp] branch master updated: MINIFICPP-694 - Ensure that processor consist of proper phases
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 81d552b MINIFICPP-694 - Ensure that processor consist of proper phases
81d552b is described below
commit 81d552b0fb1ff02f2cb5e81f2b9b114561175277
Author: Arpad Boda
AuthorDate: Tue Apr 16 15:46:53 2019 +0200
MINIFICPP-694 - Ensure that processor consist of proper phases
This closes #536.
Signed-off-by: Marc Parisi
---
nanofi/include/api/nanofi.h| 22 --
nanofi/include/core/cstructs.h | 5 +++-
nanofi/include/cxx/CallbackProcessor.h | 15
nanofi/include/cxx/Plan.h | 12 ++
nanofi/src/api/nanofi.cpp | 4 ++--
nanofi/src/cxx/CallbackProcessor.cpp | 10 ++--
nanofi/src/cxx/Plan.cpp| 42 +++---
nanofi/tests/CAPITests.cpp | 12 --
8 files changed, 91 insertions(+), 31 deletions(-)
diff --git a/nanofi/include/api/nanofi.h b/nanofi/include/api/nanofi.h
index 7b2cb18..c014030 100644
--- a/nanofi/include/api/nanofi.h
+++ b/nanofi/include/api/nanofi.h
@@ -379,13 +379,31 @@ int8_t remove_attribute(flow_file_record*, const char *
key);
int transmit_flowfile(flow_file_record *, nifi_instance *);
+
+/
+ * ##
+ * API functions for user-defined processor
+ * ##
+ */
+
+typedef struct {
+ const char * name;
+ ontrigger_callback * ontr_cb;
+ onschedule_callback * onsc_cb;
+} custom_processor_args;
+
+
/**
* Adds a custom processor for later instantiation
* @param name name of the processor
- * @param logic the callback to be invoked when the processor is triggered
+ * @param in the name and the callbacks used for the processor
+ * @attention it's recommended to use this function via the variadic arg
macro: the caller doesn't need to create the
+ * parameter struct and callback arguments can be optional.
* @return 0 on success, -1 otherwise (name already in use for eg.)
**/
-int add_custom_processor(const char * name, processor_logic* logic);
+int var_add_custom_processor(custom_processor_args in);
+
+#define add_custom_processor(...)
var_add_custom_processor((custom_processor_args){__VA_ARGS__});
/**
* Removes a custom processor
diff --git a/nanofi/include/core/cstructs.h b/nanofi/include/core/cstructs.h
index c137cf8..7d258f7 100644
--- a/nanofi/include/core/cstructs.h
+++ b/nanofi/include/core/cstructs.h
@@ -126,7 +126,10 @@ typedef enum FS {
ROLLBACK
} FailureStrategy;
-typedef void (processor_logic)(processor_session*, processor_context *);
+typedef void (ontrigger_callback)(processor_session*, processor_context *);
+typedef void (onschedule_callback)(processor_context *);
+
+typedef ontrigger_callback processor_logic;
typedef struct file_buffer {
uint8_t * buffer;
diff --git a/nanofi/include/cxx/CallbackProcessor.h
b/nanofi/include/cxx/CallbackProcessor.h
index 21e8200..62eaa68 100644
--- a/nanofi/include/cxx/CallbackProcessor.h
+++ b/nanofi/include/cxx/CallbackProcessor.h
@@ -54,7 +54,8 @@ class CallbackProcessor : public core::Processor {
*/
CallbackProcessor(std::string name, utils::Identifier uuid =
utils::Identifier())
: Processor(name, uuid),
-callback_(nullptr),
+ontrigger_callback_({}),
+onschedule_callback_({}),
objref_(nullptr),
logger_(logging::LoggerFactory::getLogger()) {
}
@@ -67,12 +68,15 @@ class CallbackProcessor : public core::Processor {
public:
- void setCallback(void *obj,std::function ontrigger_callback) {
+ void setCallback(void *obj, std::function ontrigger_callback,
+ std::function
onschedule_callback = {}) {
objref_ = obj;
-callback_ = ontrigger_callback;
+ontrigger_callback_ = ontrigger_callback;
+onschedule_callback_ = onschedule_callback;
}
- // OnTrigger method, implemented by NiFi CallbackProcessor
+ virtual void onSchedule(core::ProcessContext *context,
core::ProcessSessionFactory *sessionFactory);
+ // OnTrigger method, implemented by MiNiFi CallbackProcessor
virtual void onTrigger(core::ProcessContext *context, core::ProcessSession
*session); // override;
// Initialize, over write by NiFi CallbackProcessor
virtual void initialize(); // override;
@@ -83,7 +87,8 @@ class CallbackProcessor : public core::Processor {
protected:
void *objref_;
- std::function
callback_;
+ std::function
ontrigger_callback_;
+ std::function onschedule_callback_;
private:
// Logger
std::shared_ptr logger_;
diff --git a/nanofi/include/cxx/Plan.h b/nanofi/include/cxx/Plan.h
index 87e9642..d988810 100644
--- a/nanofi/include/cxx/Plan.h
+++ b/nanofi/include/cxx
[nifi-minifi-cpp] branch master updated: MINIFICPP-794: Incorporate Flow File Expiration settings on connections from YAML definitions
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new a413dcb MINIFICPP-794: Incorporate Flow File Expiration settings on
connections from YAML definitions
a413dcb is described below
commit a413dcb6e11f31d602fb61c85d766ce56717fe68
Author: József Kett
AuthorDate: Fri Apr 12 17:51:29 2019 +0200
MINIFICPP-794: Incorporate Flow File Expiration settings on connections
from YAML definitions
This closes #535.
Signed-off-by: Marc Parisi
---
libminifi/src/core/yaml/YamlConfiguration.cpp | 10 ++
libminifi/test/unit/YamlConfigurationTests.cpp | 2 ++
2 files changed, 12 insertions(+)
diff --git a/libminifi/src/core/yaml/YamlConfiguration.cpp
b/libminifi/src/core/yaml/YamlConfiguration.cpp
index 756b5af..cc4fc3d 100644
--- a/libminifi/src/core/yaml/YamlConfiguration.cpp
+++ b/libminifi/src/core/yaml/YamlConfiguration.cpp
@@ -656,6 +656,16 @@ void YamlConfiguration::parseConnectionYaml(YAML::Node
*connectionsNode, core::P
}
connection->setDestinationUUID(destUUID);
+if (connectionNode["flowfile expiration"]) {
+ uint64_t expirationDuration = 0;
+ std::string expiration = connectionNode["flowfile
expiration"].as();
+ TimeUnit unit;
+ if (core::Property::StringToTime(expiration, expirationDuration,
unit) && core::Property::ConvertTimeUnitToMS(expirationDuration, unit,
expirationDuration)) {
+logger_->log_debug("parseConnection: flowfile expiration => [%s]",
expirationDuration);
+connection->setFlowExpirationDuration(expirationDuration);
+ }
+}
+
if (connection) {
parent->addConnection(connection);
}
diff --git a/libminifi/test/unit/YamlConfigurationTests.cpp
b/libminifi/test/unit/YamlConfigurationTests.cpp
index 12ce423..ca4c0ae 100644
--- a/libminifi/test/unit/YamlConfigurationTests.cpp
+++ b/libminifi/test/unit/YamlConfigurationTests.cpp
@@ -162,6 +162,7 @@ TEST_CASE("Test YAML Config Processing",
"[YamlConfiguration]") {
REQUIRE(!it.second->getUUIDStr().empty());
REQUIRE(it.second->getDestination());
REQUIRE(it.second->getSource());
+REQUIRE(6 == it.second->getFlowExpirationDuration());
}
}
@@ -463,6 +464,7 @@ NiFi Properties Overrides: {}
REQUIRE(!it.second->getUUIDStr().empty());
REQUIRE(it.second->getDestination());
REQUIRE(it.second->getSource());
+REQUIRE(0 == it.second->getFlowExpirationDuration());
}
}
[nifi-minifi-cpp] branch master updated: MINIFICPP-706 - RawSiteToSite: remove code duplication
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 7514caa MINIFICPP-706 - RawSiteToSite: remove code duplication
7514caa is described below
commit 7514caacbf4722f7e3c47224ec62d918d6f4804d
Author: Arpad Boda
AuthorDate: Wed Jan 9 14:15:13 2019 +0100
MINIFICPP-706 - RawSiteToSite: remove code duplication
This closes #470.
Signed-off-by: Marc Parisi
---
libminifi/include/sitetosite/RawSocketProtocol.h | 11 +--
libminifi/include/sitetosite/SiteToSiteClient.h | 8 +-
libminifi/src/sitetosite/RawSocketProtocol.cpp | 98 +---
libminifi/src/sitetosite/SiteToSiteClient.cpp| 37 -
4 files changed, 23 insertions(+), 131 deletions(-)
diff --git a/libminifi/include/sitetosite/RawSocketProtocol.h
b/libminifi/include/sitetosite/RawSocketProtocol.h
index 395b597..1624ed6 100644
--- a/libminifi/include/sitetosite/RawSocketProtocol.h
+++ b/libminifi/include/sitetosite/RawSocketProtocol.h
@@ -112,9 +112,6 @@ class RawSiteToSiteClient : public
sitetosite::SiteToSiteClient {
}
- void setPeer(std::unique_ptr peer) {
-peer_ = std::move(peer);
- }
/**
* Provides a reference to the time out
* @returns timeout
@@ -146,18 +143,14 @@ class RawSiteToSiteClient : public
sitetosite::SiteToSiteClient {
virtual int writeRequestType(RequestType type);
// read Request Type
virtual int readRequestType(RequestType );
+
// read Respond
virtual int readRespond(const std::shared_ptr ,
RespondCode , std::string );
// write respond
virtual int writeRespond(const std::shared_ptr ,
RespondCode code, std::string message);
// getRespondCodeContext
virtual RespondCodeContext *getRespondCodeContext(RespondCode code) {
-for (unsigned int i = 0; i < sizeof(SiteToSiteRequest::respondCodeContext)
/ sizeof(RespondCodeContext); i++) {
- if (SiteToSiteRequest::respondCodeContext[i].code == code) {
-return ::respondCodeContext[i];
- }
-}
-return NULL;
+return SiteToSiteClient::getRespondCodeContext(code);
}
// Creation of a new transaction, return the transaction ID if success,
diff --git a/libminifi/include/sitetosite/SiteToSiteClient.h
b/libminifi/include/sitetosite/SiteToSiteClient.h
index 6469e82..e1fffc3 100644
--- a/libminifi/include/sitetosite/SiteToSiteClient.h
+++ b/libminifi/include/sitetosite/SiteToSiteClient.h
@@ -221,12 +221,8 @@ class SiteToSiteClient : public core::Connectable {
// deleteTransaction
virtual void deleteTransaction(std::string transactionID);
- virtual void tearDown();
-
- // write Request Type
- virtual int writeRequestType(RequestType type);
- // read Request Type
- virtual int readRequestType(RequestType );
+ virtual void tearDown() = 0;
+
// read Respond
virtual int readResponse(const std::shared_ptr ,
RespondCode , std::string );
// write respond
diff --git a/libminifi/src/sitetosite/RawSocketProtocol.cpp
b/libminifi/src/sitetosite/RawSocketProtocol.cpp
index c0bf499..316397a 100644
--- a/libminifi/src/sitetosite/RawSocketProtocol.cpp
+++ b/libminifi/src/sitetosite/RawSocketProtocol.cpp
@@ -395,97 +395,37 @@ bool
RawSiteToSiteClient::getPeerList(std::vector ) {
}
}
-int RawSiteToSiteClient::writeRequestType(RequestType type) {
- if (type >= MAX_REQUEST_TYPE)
-return -1;
-
- return peer_->writeUTF(SiteToSiteRequest::RequestTypeStr[type]);
-}
-
-int RawSiteToSiteClient::readRequestType(RequestType ) {
- std::string requestTypeStr;
-
- int ret = peer_->readUTF(requestTypeStr);
-
- if (ret <= 0)
-return ret;
+ int RawSiteToSiteClient::writeRequestType(RequestType type) {
+if (type >= MAX_REQUEST_TYPE)
+ return -1;
- for (int i = NEGOTIATE_FLOWFILE_CODEC; i <= SHUTDOWN; i++) {
-if (SiteToSiteRequest::RequestTypeStr[i] == requestTypeStr) {
- type = (RequestType) i;
- return ret;
-}
+return peer_->writeUTF(SiteToSiteRequest::RequestTypeStr[type]);
}
- return -1;
-}
-
-int RawSiteToSiteClient::readRespond(const std::shared_ptr
, RespondCode , std::string ) {
- uint8_t firstByte;
-
- int ret = peer_->read(firstByte);
-
- if (ret <= 0 || firstByte != CODE_SEQUENCE_VALUE_1)
-return -1;
-
- uint8_t secondByte;
-
- ret = peer_->read(secondByte);
-
- if (ret <= 0 || secondByte != CODE_SEQUENCE_VALUE_2)
-return -1;
-
- uint8_t thirdByte;
-
- ret = peer_->read(thirdByte);
-
- if (ret <= 0)
-return ret;
+ int RawSiteToSiteClient::readRequestType(RequestType ) {
+std::string requestTypeStr;
- code = (RespondCode) thirdByte;
+int ret = peer_->readUTF(requestTypeStr);
- RespondCodeContext *resCode = getRespondCodeContext(code);
-
- if (resCode == NULL) {
-// Not a valid respond code
-return -1;
-
[nifi-minifi-cpp] branch master updated: MINIFICPP-608: Add Cron example
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 37a72af MINIFICPP-608: Add Cron example 37a72af is described below commit 37a72afb509408bd5a8aca8f544f80bfa2514e5d Author: Marc Parisi AuthorDate: Mon Mar 4 09:08:21 2019 -0500 MINIFICPP-608: Add Cron example This closes #495. Approved on GH by arpadboda Signed-off-by: Marc Parisi --- CONFIGURE.md| 5 + LICENSE | 25 - cmake/BuildTests.cmake | 4 + controller/CMakeLists.txt | 2 +- extensions/ExtensionHeader.txt | 2 +- libminifi/CMakeLists.txt| 2 +- libminifi/include/CronDrivenSchedulingAgent.h | 74 +++ libminifi/include/FlowController.h | 3 + libminifi/include/core/ProcessGroup.h | 5 +- libminifi/include/core/Processor.h | 21 libminifi/src/CronDrivenSchedulingAgent.cpp | 85 + libminifi/src/FlowController.cpp| 21 +++- libminifi/src/core/ProcessGroup.cpp | 43 ++--- libminifi/src/core/yaml/YamlConfiguration.cpp | 12 ++- libminifi/test/integration/TailFileCronTest.cpp | 108 + libminifi/test/resources/TestTailFileCron.yml | 63 main/CMakeLists.txt | 2 +- nanofi/CMakeLists.txt | 4 +- thirdparty/cron/Cron.h | 121 thirdparty/cron/LICENSE.TXT | 21 20 files changed, 593 insertions(+), 30 deletions(-) diff --git a/CONFIGURE.md b/CONFIGURE.md index 2ad8a57..5e6c31c 100644 --- a/CONFIGURE.md +++ b/CONFIGURE.md @@ -66,6 +66,11 @@ Additionally, users can utilize the MiNiFi Toolkit Converter (version 0.0.1 - sc max concurrent tasks: 1 Properties: +### Scheduling strategies +Currently Apache NiFi MiNiFi C++ supports TIMER_DRIVEN, EVENT_DRIVEN, and CRON_DRIVEN. TIMER_DRIVEN uses periods to execute your processor(s) at given intervals. +The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. CRON_DRIVEN executes at the desired intervals +based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ). + ### SiteToSite Security Configuration in minifi.properties diff --git a/LICENSE b/LICENSE index 8f2a2cd..1c05dee 100644 --- a/LICENSE +++ b/LICENSE @@ -1663,7 +1663,6 @@ ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. This project bundles `FindFLEX.cmake` and` FindBISON.cmake` from Kitware, Inc. These files are available under a 3-Clause BSD license: -= Copyright 2009 Kitware, Inc. Copyright 2006 Tristan Carel Modified 2010 by Jon Siwek, adding HEADER option @@ -1704,4 +1703,26 @@ These files are available under a 3-Clause BSD license: This software is distributed WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the License for more information. -= \ No newline at end of file + +This project bundles Borma, which is available under an MIT Licenses: +MIT License + +Copyright (c) 2017 Bosma + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/cmake/BuildTests.cmake b/cmake/BuildTests.cmake index c57c0b3..7063e65 100644 --- a/cmake/BuildTests.cmake +++ b/cmake/BuildTests.cmake @@ -37,6 +37,7 @@ endif() function(appendIncludes testName) target_include_directorie
[nifi-minifi-cpp] branch master updated: MINIFICPP-672 - Bootstrap agent through Python
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6b40502 MINIFICPP-672 - Bootstrap agent through Python
6b40502 is described below
commit 6b40502cbc819f9979b6767a2a872deb1823560b
Author: Arpad Boda
AuthorDate: Tue Mar 12 14:53:14 2019 +0100
MINIFICPP-672 - Bootstrap agent through Python
This closes #527.
Signed-off-by: Marc Parisi
---
bootstrap.py | 237 +++
1 file changed, 237 insertions(+)
diff --git a/bootstrap.py b/bootstrap.py
new file mode 100755
index 000..cfe76e8
--- /dev/null
+++ b/bootstrap.py
@@ -0,0 +1,237 @@
+#!/usr/bin/python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function
+
+MINIFI_SUBFOLDER = '/nifi/nifi-minifi-cpp/'
+APACHE_CLOSER_REPO_JSON_URL =
'https://www.apache.org/dyn/closer.cgi?as_json=1=/nifi/nifi-minifi-cpp'
+APACHE_MIRROR_LIST = "http://www.apache.org/mirrors/;
+
+import argparse
+import sys
+
+if sys.version_info[0] < 3:
+ from urllib2 import urlopen
+ input = raw_input
+else:
+ from urllib.request import urlopen
+
+import json
+import os.path
+import platform
+import tarfile
+
+
+from distutils.util import strtobool
+from ftplib import FTP
+
+def install_package(package_name):
+ try:
+import pip
+if hasattr(pip, 'main'):
+ pipcode = pip.main(['install', package])
+else:
+ pipcode = pip._internal.main(['install', package])
+return pipcode == 0
+ except:
+return False
+
+
+distro_available = False
+
+try:
+ import distro
+
+ distro_available = True
+except:
+ distro_available = install_package("distro")
+
+
+def get_distro():
+ if is_mac():
+return ["osx", "", "darwin"]
+ try:
+if distro_available:
+ return distro.linux_distribution(full_distribution_name=False)
+else:
+ return platform.linux_distribution()
+ except:
+return ["N/A", "N/A", "N/A"]
+
+
+def is_mac():
+ return platform.system() == "Darwin"
+
+
+def mapped_distro():
+ distro_info = get_distro()
+ distro = distro_info[0].lower()
+ release = distro_info[2].lower()
+ if any(d in distro for d in ["rhel", "red hat", "centos"]):
+return "rhel", release
+ else:
+return distro, release
+
+
+def find_closest_mirror():
+ try:
+url = urlopen(APACHE_CLOSER_REPO_JSON_URL)
+data = json.loads(url.read().decode())
+
+return data['ftp'][0]
+
+ except Exception as e:
+print ("Failed to find closest mirror, please specify one!")
+return ""
+
+
+def get_release_and_binaries_from_ftp(host, apache_dir, version = None):
+ ftp = FTP(host)
+ ftp.login()
+ ftp.cwd(apache_dir + MINIFI_SUBFOLDER)
+ # list files with ftplib
+ file_list = list(filter(lambda x: any(char.isdigit() for char in x),
+ ftp.nlst(""))) # to filter "." and ".." - relese
names contain number
+ file_list.sort(reverse=True)
+ if not version:
+latest_release = file_list[0]
+ else:
+if version not in file_list:
+ print("The specified version (" + version + ") doesn't exist. Please use
one of the following: " + ", ".join(file_list))
+ exit(-1)
+latest_release = version
+
+ ftp.cwd("./" + latest_release)
+ binaries = list(filter(lambda x: any(char.isdigit() for char in x),
ftp.nlst("")))
+
+ ftp.quit()
+
+ return latest_release, binaries
+
+
+def download_binary_from_ftp(host, apache_dir, release, binary):
+ successful_download = False
+
+ try:
+ftp = FTP(host)
+ftp.login()
+ftp.cwd(apache_dir + MINIFI_SUBFOLDER + release)
+
+print ("Downloading: ftp://; + host + "/" + MINIFI_SUBFOLDER + release +
"/" + binary)
+
+with open(os.path.join(os.getcwd(), binary), "wb") as targetfile:
+ ftp.retrbinary("RETR " + binary, targetfi
[nifi-minifi-cpp] branch master updated: MINIFICPP-798 - Default to batch mode for JNI Maven builds
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 81ff62d MINIFICPP-798 - Default to batch mode for JNI Maven builds
81ff62d is described below
commit 81ff62de39d31ee4a5e21b216dc9dca0673eb6eb
Author: Aldrin Piri
AuthorDate: Tue Apr 2 14:30:28 2019 -0700
MINIFICPP-798 - Default to batch mode for JNI Maven builds
This closes #532.
Signed-off-by: Marc Parisi
---
extensions/jni/CMakeLists.txt | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/extensions/jni/CMakeLists.txt b/extensions/jni/CMakeLists.txt
index d7c8348..2fcd529 100644
--- a/extensions/jni/CMakeLists.txt
+++ b/extensions/jni/CMakeLists.txt
@@ -46,7 +46,7 @@ file(COPY ${JNI_FRAMEWORK_JAR_SOURCE} DESTINATION
${JNI_FRAMEWORK_JAR_BIN})
file(COPY ${JNI_ASSEMBLY_JAR_SOURCE} DESTINATION ${JNI_ASSEMBLY_JAR_BIN})
message("Building NiFi assembly for packaging ")
-execute_process(COMMAND "mvn" "package" "-Denforcer.skip=true"
+execute_process(COMMAND "mvn" "-q" "package" "-Denforcer.skip=true"
WORKING_DIRECTORY "${JNI_ASSEMBLY_JAR_BIN}/nifi-minifi-jni-assembly"
RESULT_VARIABLE mvn_result
OUTPUT_VARIABLE mvn_output)
@@ -64,7 +64,7 @@ endif()
message("Building NiFi JNI Jar for packaging ")
-execute_process(COMMAND "mvn" "package"
+execute_process(COMMAND "mvn" "-q" "package"
WORKING_DIRECTORY "${JNI_FRAMEWORK_JAR_BIN}/nifi-framework-jni"
RESULT_VARIABLE mvn_result
OUTPUT_VARIABLE mvn_output)
[nifi-minifi-cpp] branch master updated: MINIFICPP-789: Minor updates to readme
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 7b49287 MINIFICPP-789: Minor updates to readme 7b49287 is described below commit 7b49287d4191835d84ee6b792a199431a4c73a06 Author: Marc Parisi AuthorDate: Mon Mar 25 12:31:54 2019 -0400 MINIFICPP-789: Minor updates to readme This closes #523. Approved on GH by @apiri. Signed-off-by: Marc Parisi --- Extensions.md | 38 +- 1 file changed, 5 insertions(+), 33 deletions(-) diff --git a/Extensions.md b/Extensions.md index 63753c3..8ecca59 100644 --- a/Extensions.md +++ b/Extensions.md @@ -14,7 +14,7 @@ --> # Apache MiNiFi Extensions Guide -To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select option m in the bootstrap.sh guided build process defined in the [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping) +To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping) # Extensions by example @@ -110,42 +110,14 @@ void *createHttpCurlFactory(void); } ``` -#Using your object factory function +## Using your object factory function To use your C function, you must define the sequence "Class Loader Functions" in your YAML file under FlowController. This will indicate to the code that the factory function is to be called and we will create the mappings defined, above. Note that for the case of HTTP-CURL we have made it so that if libcURL exists and it is not disabled, the createHttpCurlFactory function is automatically loaded. To do this use the function FlowConfiguration::add_static_func -- this will add your function to the list of registered resources and will do so in a thread safe way. If you take this approach you cannot disable your library with an argument within the YAML file. -#Enabling Modules +## Common extension practices -#Enabling PacketCapture +Extensions should minimize dependencies that are dynamically linked. This means that you should statically link to dependencies when possible. You should verify that your resulting binary doesn't have system dependencies via ldd (in Linux) or dumpbbin /dependents on Microsoft Windows. -Packet Capture can be enabled to support capturing pcap files from all network interfaces on the machine. To do enable this, you must type the following when building MiNiFi C++ -``` - cmake -DENABLE_PCAP=TRUE .. -``` - -The cmake process will include a notification that the third party dependency, PcapPlusPlus has created its platform specific modules. - -You will see a message such as this when building. This configuration script will contain the name of your platforma and be an indication that -the configuration was successful: - - - PcapPlusPlus Linux configuration script - - -When the PCAP extension is built, you will be able to use the PacketCapture Processor, which has two major options: 'Batch Size' and 'Batch Directory' - -Batch Size will limit the number of packets taht are combined within a given PCAP. Batch Directory will allow the user to specify the scratch directory for network data to be written to. - -Note that if Batch Directory is not specified, /tmp/ will be used. - -*Running PcapTests requires root privileges on Linux. - -#Expressions - -To evaluate dynamic expressions using the [NiFi Expression Language](https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html), -use the `bool getProperty(const std::string , std::string , const std::shared_ptr _file);` -method within processor extensions. The expression defined in the property will be compiled if it hasn't already been, and -the expression will be evaluated against the provided flow file (may be `nullptr`). The result of the evaluation is -stored into the `` parameter. +It is advised that you place your extension in bootstrap.sh to allow users a visual menu for enabling or disabling your extension. It should not be enabled by default unless discussed with the community at large. If there is reliance on specialized hardware you should perform as many compile time tests as possible to ensure your extension cannot be enabled.
[nifi-minifi-cpp] branch master updated: MINIFICPP-772: Change from /Wall to /W3, a more sensical warning level
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6576136 MINIFICPP-772: Change from /Wall to /W3, a more sensical
warning level
6576136 is described below
commit 6576136f8e7218b75dafc6edd896c27c77e38b07
Author: Marc Parisi
AuthorDate: Mon Mar 25 15:17:22 2019 -0400
MINIFICPP-772: Change from /Wall to /W3, a more sensical warning level
This closes #524.
Approvd on GH by Arpadboda
Signed-off-by: Marc Parisi
---
libminifi/CMakeLists.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libminifi/CMakeLists.txt b/libminifi/CMakeLists.txt
index 313ef98..de58efe 100644
--- a/libminifi/CMakeLists.txt
+++ b/libminifi/CMakeLists.txt
@@ -72,7 +72,7 @@ endif()
endif()
if (WIN32)
- set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /Wall")
+ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /W3")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wno-reorder")
endif()
[nifi-minifi-cpp] branch master updated: MINIFICPP-781 - Fix documentation typos found in 0.6.0
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new 18df297 MINIFICPP-781 - Fix documentation typos found in 0.6.0 18df297 is described below commit 18df297398c7755e23283d3e8b3238badd8bbf8d Author: Arpad Boda AuthorDate: Wed Mar 20 14:31:38 2019 +0100 MINIFICPP-781 - Fix documentation typos found in 0.6.0 This closes #518. Signed-off-by: Marc Parisi --- NOTICE| 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/NOTICE b/NOTICE index 04d03e2..625e9c2 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache NiFi MiNiFi -Copyright 2017 The Apache Software Foundation +Copyright 2019 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). diff --git a/README.md b/README.md index c2ed3c8..4886e23 100644 --- a/README.md +++ b/README.md @@ -470,7 +470,7 @@ provides the command to build your distro and the output file in your build dire | Debian 9 | make debian | nifi-minifi-cpp-debian-$VERSION-bin.tar.gz | Fedora 29 | make fedora | nifi-minifi-cpp-fedora-$VERSION-bin.tar.gz | Ubuntu 16 | make u16 | nifi-minifi-cpp-xenial-$VERSION-bin.tar.gz -| Ubuntu 16 | make u18 | nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz +| Ubuntu 18 | make u18 | nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz ### Cleaning
[nifi-minifi-cpp] branch master updated: MINIFICPP-786 - GetFile processor doesn't seem to work with default configuration on Win, logged details are insufficient, too
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new b821df8 MINIFICPP-786 - GetFile processor doesn't seem to work with
default configuration on Win, logged details are insufficient, too
b821df8 is described below
commit b821df856e334f488dfb6f0c8ca4c267262f8dcb
Author: Arpad Boda
AuthorDate: Sat Mar 23 19:49:17 2019 +0100
MINIFICPP-786 - GetFile processor doesn't seem to work with default
configuration on Win, logged details are insufficient, too
This closes #522.
Signed-off-by: Marc Parisi
---
libminifi/include/utils/file/FileUtils.h | 85
libminifi/src/processors/GetFile.cpp | 77 +
2 files changed, 97 insertions(+), 65 deletions(-)
diff --git a/libminifi/include/utils/file/FileUtils.h
b/libminifi/include/utils/file/FileUtils.h
index 89aa803..53bb054 100644
--- a/libminifi/include/utils/file/FileUtils.h
+++ b/libminifi/include/utils/file/FileUtils.h
@@ -80,6 +80,16 @@ class FileUtils {
* @param path current path to delete
* @param delete_files_recursively deletes recursively
*/
+
+ static char get_separator()
+ {
+#ifdef WIN32
+return '\\';
+#else
+return '/';
+#endif
+ }
+
static int64_t delete_dir(const std::string , bool
delete_files_recursively = true) {
#ifdef BOOST_VERSION
try {
@@ -301,6 +311,81 @@ class FileUtils {
#endif
}
+ /*
+ * Provides a platform-independent function to list a directory
+ * Callback is called for every file found: first argument is the path of
the directory, second is the filename
+ * Return value of the callback is used to continue (true) or stop (false)
listing
+ */
+ static void list_dir(const std::string& dir, std::function callback,
+ const std::shared_ptr , bool
recursive = true) {
+
+logger->log_debug("Performing file listing against %s", dir);
+#ifndef WIN32
+DIR *d = opendir(dir.c_str());
+if (!d) {
+ logger->log_warn("Failed to open directory: %s", dir.c_str());
+ return;
+}
+
+struct dirent *entry;
+while ((entry = readdir(d)) != NULL) {
+
+ std::string d_name = entry->d_name;
+ std::string path = dir + get_separator() + d_name;
+
+ struct stat statbuf;
+ if (stat(path.c_str(), ) != 0) {
+logger->log_warn("Failed to stat %s", path);
+continue;
+ }
+
+ if (S_ISDIR(statbuf.st_mode)) {
+// if this is a directory
+if (recursive && strcmp(d_name.c_str(), "..") != 0 &&
strcmp(d_name.c_str(), ".") != 0) {
+ list_dir(path, callback, logger, recursive);
+}
+ } else {
+if (!callback(dir, d_name)) {
+ break;
+}
+ }
+}
+closedir(d);
+#else
+HANDLE hFind;
+WIN32_FIND_DATA FindFileData;
+
+std::string pathToSearch = dir + "\\*.*";
+hFind = FindFirstFileA(pathToSearch.c_str(), );
+
+if(hFind == INVALID_HANDLE_VALUE) {
+ logger->log_warn("Failed to open directory: %s", dir.c_str());
+ return;
+}
+
+do {
+ struct stat statbuf {};
+ if (strcmp(FindFileData.cFileName, ".") != 0 &&
strcmp(FindFileData.cFileName, "..") != 0) {
+std::string path = dir + get_separator() + FindFileData.cFileName;
+if (stat(path.c_str(), ) != 0) {
+ logger->log_warn("Failed to stat %s", path);
+ continue;
+}
+if (S_ISDIR(statbuf.st_mode)) {
+ if (recursive) {
+list_dir(path, callback, logger, recursive);
+ }
+}
+else {
+ if (!callback(dir, FindFileData.cFileName)) {
+break;
+ }
+}
+ }
+} while (FindNextFileA(hFind, ));
+FindClose(hFind);
+#endif
+ }
};
} /* namespace file */
diff --git a/libminifi/src/processors/GetFile.cpp
b/libminifi/src/processors/GetFile.cpp
index 3fbd41f..8d8feeb 100644
--- a/libminifi/src/processors/GetFile.cpp
+++ b/libminifi/src/processors/GetFile.cpp
@@ -41,9 +41,6 @@
#include "core/ProcessSession.h"
#include "core/TypedValues.h"
-#ifndef S_ISDIR
-#define S_ISDIR(mode) (((mode) & S_IFMT) == S_IFDIR)
-#endif
#define R_OK4 /* Test for read permission. */
#define W_OK2 /* Test for write permission. */
#define F_OK0 /* Test for existence. */
@@ -211,6 +208,8 @@ bool GetFile::isListingEmpty() {
}
void GetFile::putListing(std::string fileName) {
+ logger_->log_trace("Adding file to queue: %s", fileName);
+
std::lock_guard lock(mutex_);
_dirList.push(fileName);
@@ -220,15 +219,14 @@ void GetFile::pollListing(std::que
[nifi-minifi-cpp] branch master updated: MINIFICPP-783: Fix bootstrap curl install on Darwin
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6cae874 MINIFICPP-783: Fix bootstrap curl install on Darwin
6cae874 is described below
commit 6cae874b73c8ac84a762a29c3aed46ba966932f9
Author: Kevin Doran
AuthorDate: Thu Mar 21 09:55:17 2019 -0400
MINIFICPP-783: Fix bootstrap curl install on Darwin
This closes #519.
Signed-off-by: Marc Parisi
---
darwin.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/darwin.sh b/darwin.sh
index 086a105..91a3139 100644
--- a/darwin.sh
+++ b/darwin.sh
@@ -83,7 +83,7 @@ build_deps(){
if [ "$KEY" = "$option" ]; then
FOUND_VALUE="$VALUE"
if [ "$FOUND_VALUE" = "libcurl" ]; then
-brew install curl --with-openssl
+brew install curl
elif [ "$FOUND_VALUE" = "libpcap" ]; then
INSTALLED+=("libpcap")
elif [ "$FOUND_VALUE" = "openssl" ]; then
[nifi-minifi-cpp] annotated tag rel/minifi-cpp-0.6.0 created (now 139b837)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to annotated tag rel/minifi-cpp-0.6.0 in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. at 139b837 (tag) tagging 28ade6cf75e8a0ccce78699f147f4ad9baaf70c2 (commit) replaces rel/minifi-cpp-0.5.0 by Marc Parisi on Fri Mar 22 13:51:53 2019 -0400 - Log - MINIFICPP-770 signed tag approved for release of NiFi MiNiFi C++ 0.6.0 -BEGIN PGP SIGNATURE- iQJIBAABCgAyFiEEt7FS1WuRFP1esSgcL0gcNblC53wFAlyVILkUHHBocm9ja2Vy QGFwYWNoZS5vcmcACgkQL0gcNblC53xBoxAAsPdyfL6/rPhpeS9OZ3mrX9bMCJWK LfkmmepgTm6KxFixXQ+MOYStRuvqDoGV/Zb6+PQns964ivXmE1AAk6dwZkcTFnm+ QXe3flx5CVXieQLdfAQVq12aUOA62dz7X8cC6Mmhoaa8qDbkqaL9kdOOaZr+NEW2 VVuEHkQkHGrwMZgTeaoEaDZRbbmmBFKqxzFjsMDUDgngho8p3Pnt+c0OioIQQT02 ywiFiVacJ3HPyhxAhjYhUZWw0kJeom9mz7kg2gIA3CIBBVJbnGWTrisyMmd/WjXo dUC1JwfnP2Md3S4xPNu8kG6VljhZQluLAopRS5SlA9D4ZhEbkdGmUVEogIcdYysr NJUWKmsCGTCHVqFuCKbfjxWotpctC2GirtfIXzvSq4AF0WqxoXHsrrYdFONV3tzK ro212TcEKSAi+7sxqHSDeZmcodfdB3Bgn6tfwyp6iukKPMPK1G4EkZU5Dne8HmFE KtuyIAkSey0MVn7i4V0cO1SYbZYR+pb1rEtdaoSfSWO6fvDyQ5KdueGcXl06ifIp ubKEFEWNHk8rsainxI70L7R/iEDtDGzoJcQIcuIo8+kCnpDptmcBHC5q9d+U5XqT EzaY0dJiVJNqvnlgqFddnXSPtScT07cNouZuAFhdxxif2va+uJKhvPGqFSCK1XNs 4uxh7ySvsmCe1Vc= =MiEM -END PGP SIGNATURE- --- No new revisions were added by this update.
svn commit: r1856071 - /nifi/site/trunk/minifi/download.html
Author: phrocker Date: Fri Mar 22 17:48:54 2019 New Revision: 1856071 URL: http://svn.apache.org/viewvc?rev=1856071=rev Log: MINIFICPP-770: Add 0.6.0 downloads Modified: nifi/site/trunk/minifi/download.html Modified: nifi/site/trunk/minifi/download.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/minifi/download.html?rev=1856071=1856070=1856071=diff == --- nifi/site/trunk/minifi/download.html (original) +++ nifi/site/trunk/minifi/download.html Fri Mar 22 17:48:54 2019 @@ -186,6 +186,122 @@ MiNiFi C++ + cpp-0.6.0 + + + Sources: + + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz;>nifi-minifi-cpp-0.6.0-source.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.sha512;>sha512 ) + + + + + Binaries + + Centos 7 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz;>nifi-minifi-cpp-centos-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha512;>sha512 ) + + Debian 9 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz;>nifi-minifi-cpp-debian-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha512;>sha512 ) + + Fedora 29 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz;>nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha512;>sha512 ) + + MSFT Windows - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi;>nifi-minifi-cpp-0.6.0.msi +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.sha512;>sha512 ) + + OS X - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz;>nifi-minifi-cpp-osx-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.sha512;>sha512 ) + + + Ubuntu 16 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6
[nifi-site] branch master updated: MINIFICPP-770: 0.6.0 release
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/master by this push: new 191c6b8 MINIFICPP-770: 0.6.0 release 191c6b8 is described below commit 191c6b8d28fb61c0b154503d2d314eb6da44c4bd Author: Marc Parisi AuthorDate: Fri Mar 22 13:45:25 2019 -0400 MINIFICPP-770: 0.6.0 release --- src/pages/html/minifi/download.hbs | 116 + 1 file changed, 116 insertions(+) diff --git a/src/pages/html/minifi/download.hbs b/src/pages/html/minifi/download.hbs index 5550be7..c08f903 100644 --- a/src/pages/html/minifi/download.hbs +++ b/src/pages/html/minifi/download.hbs @@ -87,6 +87,122 @@ title: Apache NiFi - MiNiFi Downloads MiNiFi C++ + cpp-0.6.0 + + + Sources: + + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz;>nifi-minifi-cpp-0.6.0-source.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0-source.tar.gz.sha512;>sha512 ) + + + + + Binaries + + Centos 7 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz;>nifi-minifi-cpp-centos-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha512;>sha512 ) + + Debian 9 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz;>nifi-minifi-cpp-debian-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha512;>sha512 ) + + Fedora 29 - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz;>nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha512;>sha512 ) + + MSFT Windows - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi;>nifi-minifi-cpp-0.6.0.msi +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-0.6.0.msi.sha512;>sha512 ) + + OS X - x86_64 + https://www.apache.org/dyn/closer.lua?path=/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz;>nifi-minifi-cpp-osx-0.6.0-bin.tar.gz +( https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.asc;>asc, + https://www.apache.org/dist/nifi/nifi-minifi-cpp/0.6.0/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.sha256;>sha256, + https://www.apache.org/dist/nifi/nifi
svn commit: r33142 - /dev/nifi/nifi-minifi-cpp/0.6.0/
Author: phrocker Date: Fri Mar 22 17:06:43 2019 New Revision: 33142 Log: “MINIFI-770” Removed: dev/nifi/nifi-minifi-cpp/0.6.0/
svn commit: r33141 - /dev/nifi/nifi-minifi-cpp/0.6.0-rc2/ /release/nifi/nifi-minifi-cpp/0.6.0/
Author: phrocker Date: Fri Mar 22 17:06:00 2019 New Revision: 33141 Log: “MINIFI-770” Added: release/nifi/nifi-minifi-cpp/0.6.0/ - copied from r33140, dev/nifi/nifi-minifi-cpp/0.6.0-rc2/ Removed: dev/nifi/nifi-minifi-cpp/0.6.0-rc2/
svn commit: r33074 - /dev/nifi/nifi-minifi-cpp/0.6.0-rc2/
Author: phrocker Date: Tue Mar 19 16:44:25 2019 New Revision: 33074 Log: MINIFICPP-770: Commit 0.6.0-rc2 Added: dev/nifi/nifi-minifi-cpp/0.6.0-rc2/ dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0-source.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0-source.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0-source.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0-source.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0.msi (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0.msi.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0.msi.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-0.6.0.msi.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-jni-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-jni-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-jni-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-bionic-jni-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-jni-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-jni-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-jni-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-centos-jni-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-jni-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-jni-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-jni-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-debian-jni-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-jni-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-jni-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-jni-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-fedora-jni-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-jni-0.6.0.msi (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-jni-0.6.0.msi.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-jni-0.6.0.msi.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-jni-0.6.0.msi.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-jni-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-jni-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-jni-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-osx-jni-0.6.0-bin.tar.gz.sha512 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-xenial-0.6.0-bin.tar.gz (with props) dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-xenial-0.6.0-bin.tar.gz.asc dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi-minifi-cpp-xenial-0.6.0-bin.tar.gz.sha256 dev/nifi/nifi-minifi-cpp/0.6.0-rc2/nifi
[nifi-minifi-cpp] branch MINIFICPP-770-RCX created (now 28ade6c)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch MINIFICPP-770-RCX in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. at 28ade6c MINIFICPP-779: Change debian and apt repos to use libpython3-dev, with knowledge that they'll be 3.5+. If a user does an update they will either need to install the older version or already have it installed to run the convenience binaries based on how libpython is linked No new revisions were added by this update.
