[nifi] branch main updated: NIFI-7607: Fixed bug that caused the wrong Controller Service to be de-referenced when a component that used to reference a Controller Service is changed to reference a dif
This is an automated email from the ASF dual-hosted git repository.
bbende pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new e17db80 NIFI-7607: Fixed bug that caused the wrong Controller Service
to be de-referenced when a component that used to reference a Controller
Service is changed to reference a different Controller Service (#4391)
e17db80 is described below
commit e17db80514870077ac379186b73762833dcacff1
Author: markap14
AuthorDate: Tue Jul 7 16:38:42 2020 -0400
NIFI-7607: Fixed bug that caused the wrong Controller Service to be
de-referenced when a component that used to reference a Controller Service is
changed to reference a different Controller Service (#4391)
---
.../main/java/org/apache/nifi/controller/AbstractComponentNode.java| 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
index a1f5af1..a7db9ec 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
@@ -347,7 +347,8 @@ public abstract class AbstractComponentNode implements
ComponentNode {
if (!propertyConfiguration.equals(oldConfiguration)) {
if (descriptor.getControllerServiceDefinition() != null) {
if (oldConfiguration != null) {
-final ControllerServiceNode oldNode =
serviceProvider.getControllerServiceNode(effectiveValue);
+final String oldEffectiveValue =
oldConfiguration.getEffectiveValue(getParameterContext());
+final ControllerServiceNode oldNode =
serviceProvider.getControllerServiceNode(oldEffectiveValue);
if (oldNode != null) {
oldNode.removeReference(this, descriptor);
}
[nifi] branch main updated: NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available
This is an automated email from the ASF dual-hosted git repository. kdoran pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 43fb57e NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims. Added new StandardOidcIdentityProviderGroovyTest file. Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test. Added comments in getAvailableClaims() method. Fixed typos in NiFi Docs Admin Guide. Added [...] 43fb57e is described below commit 43fb57e7bb4b3a3a7714578816ab57ece63349c8 Author: mtien AuthorDate: Wed May 20 18:14:57 2020 -0700 NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims. Added new StandardOidcIdentityProviderGroovyTest file. Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test. Added comments in getAvailableClaims() method. Fixed typos in NiFi Docs Admin Guide. Added license to Groovy test. Fixed a checkstyle error. Refactor exchangeAuthorizationCode method. Added unit tests. Verified all unit tests added so far are passing. Refactored code. Added unit tests. Refactored OIDC provider to decouple constructor & network-dependent initialization. Added unit tests. Added unit tests. Refactored OIDC provider to separately authorize the client. Added unit tests. Added unit tests. NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT. Signed-off-by: Nathan Gough This closes #4344. --- .../src/main/asciidoc/administration-guide.adoc| 5 +- .../web/security/oidc/OidcIdentityProvider.java| 6 +- .../apache/nifi/web/security/oidc/OidcService.java | 17 +- .../oidc/StandardOidcIdentityProvider.java | 391 -- .../StandardOidcIdentityProviderGroovyTest.groovy | 584 + 5 files changed, 847 insertions(+), 156 deletions(-) diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index b2653d2..90daf3a 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -367,10 +367,9 @@ To enable authentication via OpenId Connect the following properties must be con |`nifi.security.user.oidc.read.timeout` | Read timeout when communicating with the OpenId Connect Provider. |`nifi.security.user.oidc.client.id` | The client id for NiFi after registration with the OpenId Connect Provider. |`nifi.security.user.oidc.client.secret` | The client secret for NiFi after registration with the OpenId Connect Provider. -|`nifi.security.user.oidc.preferred.jwsalgorithm` | The preferred algorithm for for validating identity tokens. If this value is blank, it will default to `RS256` which is required to be supported +|`nifi.security.user.oidc.preferred.jwsalgorithm` | The preferred algorithm for validating identity tokens. If this value is blank, it will default to `RS256` which is required to be supported |`nifi.security.user.oidc.additional.scopes` | Comma separated scopes that are sent to OpenId Connect Provider in addition to `openid` and `email`. -|`nifi.security.user.oidc.claim.identifying.user` | Claim that identifies the user to be logged in; default is `email`. May need to be requested via the `nifi.security.user.oidc.additional.scopes` before usage. -by the OpenId Connect Provider according to the specification. If this value is `HS256`, `HS384`, or `HS512`, NiFi will attempt to validate HMAC protected tokens using the specified client secret. +|`nifi.security.user.oidc.claim.identifying.user` | Claim that identifies the user to be logged in; default is `email`. May need to be requested via the `nifi.security.user.oidc.additional.scopes` before usage by the OpenId Connect Provider according to the specification. If this value is `HS256`, `HS384`, or `HS512`, NiFi will attempt to validate HMAC protected tokens using the specified client secret. If this value is `none`, NiFi will attempt to validate unsecured/plain tokens. Other values for this algorithm will attempt to parse as an RSA or EC algorithm to be used in conjunction with the JSON Web Key (JWK) provided through the jwks_uri in the metadata found at the discovery URL. |== diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi
[nifi] branch master updated (c396927 -> aa741cc)
This is an automated email from the ASF dual-hosted git repository. thenatog pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from c396927 NIFI-7594 In HandleHttpRequest deleting multipart file resources after processing. add aa741cc NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims. Added new StandardOidcIdentityProviderGroovyTest file. Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test. Added comments in getAvailableClaims() method. Fixed typos in NiFi Docs Admin Guide. Added [...] No new revisions were added by this update. Summary of changes: .../src/main/asciidoc/administration-guide.adoc| 5 +- .../web/security/oidc/OidcIdentityProvider.java| 6 +- .../apache/nifi/web/security/oidc/OidcService.java | 17 +- .../oidc/StandardOidcIdentityProvider.java | 391 -- .../StandardOidcIdentityProviderGroovyTest.groovy | 584 + 5 files changed, 847 insertions(+), 156 deletions(-) create mode 100644 nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/test/groovy/org/apache/nifi/web/security/oidc/StandardOidcIdentityProviderGroovyTest.groovy
[nifi] branch main updated: NIFI-5213: Allow AvroReader to process files w embedded schema even when the access strategy is explicit schema NIFI-5213: Incorporated review comments
This is an automated email from the ASF dual-hosted git repository.
mthomsen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new f040c6a NIFI-5213: Allow AvroReader to process files w embedded
schema even when the access strategy is explicit schema NIFI-5213: Incorporated
review comments
f040c6a is described below
commit f040c6aadbedb0fb5e778e7e13069d66cc97af45
Author: Matthew Burgess
AuthorDate: Fri May 18 00:29:52 2018 -0400
NIFI-5213: Allow AvroReader to process files w embedded schema even when
the access strategy is explicit schema
NIFI-5213: Incorporated review comments
This closes #2718
Signed-off-by: Mike Thomsen
---
.../nifi-record-serialization-services/pom.xml | 2 +
.../nifi/avro/AvroReaderWithExplicitSchema.java| 48 --
.../avro/TestAvroReaderWithExplicitSchema.java | 99 +
.../src/test/resources/avro/avro_embed_schema.avro | Bin 0 -> 216 bytes
.../src/test/resources/avro/avro_schemaless.avro | Bin 0 -> 18 bytes
.../src/test/resources/avro/avro_schemaless.avsc | 25 ++
6 files changed, 167 insertions(+), 7 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml
b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml
index f17fe3c..4c4ba4c 100755
---
a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml
+++
b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml
@@ -132,6 +132,8 @@
apache-rat-plugin
+
src/test/resources/avro/avro_embed_schema.avro
+
src/test/resources/avro/avro_schemaless.avro
src/test/resources/avro/datatypes.avsc
src/test/resources/avro/decimals.avsc
src/test/resources/avro/logical-types.avsc
diff --git
a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java
b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java
index 9197e47..9c0dbf5 100644
---
a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java
+++
b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReaderWithExplicitSchema.java
@@ -17,30 +17,59 @@
package org.apache.nifi.avro;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.SequenceInputStream;
+
import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileStream;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DecoderFactory;
+import org.apache.commons.io.input.TeeInputStream;
import org.apache.nifi.serialization.record.RecordSchema;
-import java.io.EOFException;
-import java.io.IOException;
-import java.io.InputStream;
-
public class AvroReaderWithExplicitSchema extends AvroRecordReader {
private final InputStream in;
private final RecordSchema recordSchema;
private final DatumReader datumReader;
-private final BinaryDecoder decoder;
+private BinaryDecoder decoder;
private GenericRecord genericRecord;
+private DataFileStream dataFileStream;
-public AvroReaderWithExplicitSchema(final InputStream in, final
RecordSchema recordSchema, final Schema avroSchema) {
+public AvroReaderWithExplicitSchema(final InputStream in, final
RecordSchema recordSchema, final Schema avroSchema) throws IOException {
this.in = in;
this.recordSchema = recordSchema;
datumReader = new NonCachingDatumReader<>(avroSchema);
-decoder = DecoderFactory.get().binaryDecoder(in, null);
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+TeeInputStream teeInputStream = new TeeInputStream(in, baos);
+// Try to parse as a DataFileStream, if it works, glue the streams
back together and delegate calls to the DataFileStream
+try {
+dataFileStream = new DataFileStream<>(teeInputStream, new
NonCachingDatumReader<>());
+} catch
[nifi-minifi-cpp] branch master updated (189b1bd -> 589db71)
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git.
from 189b1bd MINIFICPP-1280 - StringUtils::replaceAll hangs in case
from_string is empty
new 1e5e3ae MINIFICPP-1274 - Commit delete operation before shutdown
new ff785ce MINIFICPP-1273 - Drain connections on flow shutdown
new 589db71 MINIFICPP-1276 - CAPITests should clean up the temporary
files it creates
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CMakeLists.txt | 2 +
extensions/rocksdb-repos/FlowFileRepository.cpp| 1 +
libminifi/include/Connection.h | 2 +-
libminifi/include/core/ProcessGroup.h | 2 +
libminifi/include/core/Repository.h| 2 +-
libminifi/src/Connection.cpp | 10 +-
libminifi/src/FlowController.cpp | 5 +-
libminifi/src/core/ProcessGroup.cpp| 19 +++-
.../test/flow-tests}/CMakeLists.txt| 19 ++--
libminifi/test/flow-tests/FlowControllerTests.cpp | 125 +
libminifi/test/rocksdb-tests/RepoTests.cpp | 81 +
nanofi/include/api/nanofi.h| 6 +-
nanofi/src/api/nanofi.cpp | 4 +-
nanofi/tests/CAPITests.cpp | 14 +--
14 files changed, 254 insertions(+), 38 deletions(-)
copy {extensions/librdkafka/tests => libminifi/test/flow-tests}/CMakeLists.txt
(63%)
create mode 100644 libminifi/test/flow-tests/FlowControllerTests.cpp
[nifi-minifi-cpp] 02/03: MINIFICPP-1273 - Drain connections on flow shutdown
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit ff785cea511c4266525a4f9b6f5eccdd90817aa4
Author: Adam Debreceni
AuthorDate: Tue Jun 30 11:27:09 2020 +0200
MINIFICPP-1273 - Drain connections on flow shutdown
Signed-off-by: Arpad Boda
This closes #827
---
CMakeLists.txt| 2 +
libminifi/include/Connection.h| 2 +-
libminifi/include/core/ProcessGroup.h | 2 +
libminifi/src/Connection.cpp | 10 +-
libminifi/src/FlowController.cpp | 5 +-
libminifi/src/core/ProcessGroup.cpp | 19 +++-
libminifi/test/flow-tests/CMakeLists.txt | 31 ++
libminifi/test/flow-tests/FlowControllerTests.cpp | 125 ++
8 files changed, 185 insertions(+), 11 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 700b95f..3479070 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -766,6 +766,8 @@ endif()
## Add KeyValueStorageService tests
registerTest("${TEST_DIR}/keyvalue-tests")
+registerTest("${TEST_DIR}/flow-tests")
+
include(BuildDocs)
include(DockerConfig)
diff --git a/libminifi/include/Connection.h b/libminifi/include/Connection.h
index 64a805e..5a40d87 100644
--- a/libminifi/include/Connection.h
+++ b/libminifi/include/Connection.h
@@ -168,7 +168,7 @@ class Connection : public core::Connectable, public
std::enable_shared_from_this
// Poll the flow file from queue, the expired flow file record also being
returned
std::shared_ptr
poll(std::set> &expiredFlowRecords);
// Drain the flow records
- void drain();
+ void drain(bool delete_permanently);
void yield() override {}
diff --git a/libminifi/include/core/ProcessGroup.h
b/libminifi/include/core/ProcessGroup.h
index fcbb78a..9944658 100644
--- a/libminifi/include/core/ProcessGroup.h
+++ b/libminifi/include/core/ProcessGroup.h
@@ -228,6 +228,8 @@ class ProcessGroup {
void getConnections(std::map>
&connectionMap);
+ void drainConnections();
+
protected:
void startProcessingProcessors(const
std::shared_ptr timeScheduler, const
std::shared_ptr &eventScheduler, const
std::shared_ptr &cronScheduler); // NOLINT
diff --git a/libminifi/src/Connection.cpp b/libminifi/src/Connection.cpp
index 9038e91..06cfcce 100644
--- a/libminifi/src/Connection.cpp
+++ b/libminifi/src/Connection.cpp
@@ -262,15 +262,17 @@ std::shared_ptr
Connection::poll(std::set lock(mutex_);
while (!queue_.empty()) {
std::shared_ptr item = queue_.front();
queue_.pop();
-logger_->log_debug("Delete flow file UUID %s from connection %s, because
it expired", item->getUUIDStr(), name_);
-if (flow_repository_->Delete(item->getUUIDStr())) {
- item->setStoredToRepository(false);
+logger_->log_debug("Delete flow file UUID %s from connection %s",
item->getUUIDStr(), name_);
+if (delete_permanently) {
+ if (flow_repository_->Delete(item->getUUIDStr())) {
+item->setStoredToRepository(false);
+ }
}
}
queued_data_size_ = 0;
diff --git a/libminifi/src/FlowController.cpp b/libminifi/src/FlowController.cpp
index 3976936..669285b 100644
--- a/libminifi/src/FlowController.cpp
+++ b/libminifi/src/FlowController.cpp
@@ -250,6 +250,9 @@ int16_t FlowController::stop(bool force, uint64_t
timeToWait) {
* -Stopping the schedulers doesn't actually quit the onTrigger functions
of processors
* -They only guarantee that the processors are not scheduled any more
* -After the threadpool is stopped we can make sure that processors don't
need repos and controllers anymore */
+if (this->root_) {
+ this->root_->drainConnections();
+}
this->flow_file_repo_->stop();
this->provenance_repo_->stop();
// stop the ControllerServices
@@ -923,7 +926,7 @@ int16_t FlowController::clearConnection(const std::string
&connection) {
auto conn = connections.find(connection);
if (conn != connections.end()) {
logger_->log_info("Clearing connection %s", connection);
- conn->second->drain();
+ conn->second->drain(true);
}
}
return -1;
diff --git a/libminifi/src/core/ProcessGroup.cpp
b/libminifi/src/core/ProcessGroup.cpp
index 9eec44c..ee88d79 100644
--- a/libminifi/src/core/ProcessGroup.cpp
+++ b/libminifi/src/core/ProcessGroup.cpp
@@ -92,13 +92,12 @@ ProcessGroup::~ProcessGroup() {
onScheduleTimer_->stop();
}
- for (auto &&connection : connections_) {
-connection->drain();
+ for (auto&& connection : connections_) {
+connection->drain(false);
}
- for (std::set::iterator it = child_process_groups_.begin();
it != child_process_groups_.end(); ++it) {
-ProcessGroup *processGroup(*it);
-delete processGroup;
+ for (ProcessGroup* childGroup : child_process_groups_) {
+delete childGroup;
}
}
@@ -403,6 +402,16
[nifi-minifi-cpp] 01/03: MINIFICPP-1274 - Commit delete operation before shutdown
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 1e5e3ae8e006e64dff81096c4cf558b452fd3d1b
Author: Adam Debreceni
AuthorDate: Mon Jun 29 14:51:08 2020 +0200
MINIFICPP-1274 - Commit delete operation before shutdown
Signed-off-by: Arpad Boda
This closes #826
---
extensions/rocksdb-repos/FlowFileRepository.cpp | 1 +
libminifi/include/core/Repository.h | 2 +-
libminifi/test/rocksdb-tests/RepoTests.cpp | 81 +
3 files changed, 83 insertions(+), 1 deletion(-)
diff --git a/extensions/rocksdb-repos/FlowFileRepository.cpp
b/extensions/rocksdb-repos/FlowFileRepository.cpp
index 428db97..127ce9b 100644
--- a/extensions/rocksdb-repos/FlowFileRepository.cpp
+++ b/extensions/rocksdb-repos/FlowFileRepository.cpp
@@ -119,6 +119,7 @@ void FlowFileRepository::run() {
last = now;
}
}
+ flush();
}
void FlowFileRepository::prune_stored_flowfiles() {
diff --git a/libminifi/include/core/Repository.h
b/libminifi/include/core/Repository.h
index 3de1a54..e44c479 100644
--- a/libminifi/include/core/Repository.h
+++ b/libminifi/include/core/Repository.h
@@ -242,7 +242,7 @@ class Repository : public virtual
core::SerializableComponent, public core::Trac
// thread
std::thread thread_;
// whether the monitoring thread is running for the repo while it was enabled
- bool running_;
+ std::atomic running_;
// whether stop accepting provenace event
std::atomic repo_full_;
// repoSize
diff --git a/libminifi/test/rocksdb-tests/RepoTests.cpp
b/libminifi/test/rocksdb-tests/RepoTests.cpp
index e970c40..7521d6f 100644
--- a/libminifi/test/rocksdb-tests/RepoTests.cpp
+++ b/libminifi/test/rocksdb-tests/RepoTests.cpp
@@ -326,3 +326,84 @@ TEST_CASE("Test FlowFile Restore", "[TestFFR6]") {
LogTestController::getInstance().reset();
}
+
+TEST_CASE("Flush deleted flowfiles before shutdown", "[TestFFR7]") {
+ class TestFlowFileRepository: public core::repository::FlowFileRepository{
+ public:
+explicit TestFlowFileRepository(const std::string& name)
+: core::SerializableComponent(name),
+ FlowFileRepository(name, FLOWFILE_REPOSITORY_DIRECTORY,
MAX_FLOWFILE_REPOSITORY_ENTRY_LIFE_TIME,
+ MAX_FLOWFILE_REPOSITORY_STORAGE_SIZE, 1) {}
+void flush() override {
+ FlowFileRepository::flush();
+ if (onFlush_) {
+onFlush_();
+ }
+}
+std::function onFlush_;
+ };
+
+ TestController testController;
+ char format[] = "/var/tmp/testRepo.XX";
+ auto dir = testController.createTempDirectory(format);
+
+ auto config = std::make_shared();
+ config->set(minifi::Configure::nifi_flowfile_repository_directory_default,
utils::file::FileUtils::concat_path(dir, "flowfile_repository"));
+
+ auto content_repo =
std::make_shared();
+
+ auto connection = std::make_shared(nullptr, nullptr,
"Connection");
+ std::map>
connectionMap{{connection->getUUIDStr(), connection}};
+ // initialize repository
+ {
+std::shared_ptr ff_repository =
std::make_shared("flowFileRepository");
+
+std::atomic flush_counter{0};
+
+std::atomic stop{false};
+std::thread shutdown{[&] {
+ while (!stop.load()) {
+std::this_thread::sleep_for(std::chrono::milliseconds{10});
+ }
+ ff_repository->stop();
+}};
+
+ff_repository->onFlush_ = [&] {
+ if (++flush_counter != 1) {
+return;
+ }
+
+ for (int keyIdx = 0; keyIdx < 100; ++keyIdx) {
+auto file = std::make_shared(ff_repository,
nullptr);
+file->setUuidConnection(connection->getUUIDStr());
+// Serialize is sync
+REQUIRE(file->Serialize());
+if (keyIdx % 2 == 0) {
+ // delete every second flowFile
+ REQUIRE(ff_repository->Delete(file->getUUIDStr()));
+}
+ }
+ stop = true;
+ // wait for the shutdown thread to start waiting for the worker thread
+ std::this_thread::sleep_for(std::chrono::milliseconds{100});
+};
+
+ff_repository->setConnectionMap(connectionMap);
+REQUIRE(ff_repository->initialize(config));
+ff_repository->loadComponent(content_repo);
+ff_repository->start();
+
+shutdown.join();
+ }
+
+ // check if the deleted flowfiles are indeed deleted
+ {
+std::shared_ptr ff_repository =
std::make_shared("flowFileRepository");
+ff_repository->setConnectionMap(connectionMap);
+REQUIRE(ff_repository->initialize(config));
+ff_repository->loadComponent(content_repo);
+ff_repository->start();
+std::this_thread::sleep_for(std::chrono::milliseconds{100});
+REQUIRE(connection->getQueueSize() == 50);
+ }
+}
[nifi-minifi-cpp] 03/03: MINIFICPP-1276 - CAPITests should clean up the temporary files it creates
This is an automated email from the ASF dual-hosted git repository.
aboda pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 589db71624b7f90f27e80fe3aa40e056a22c6374
Author: Adam Markovics
AuthorDate: Mon Jul 6 11:08:34 2020 +0200
MINIFICPP-1276 - CAPITests should clean up the temporary files it creates
Signed-off-by: Arpad Boda
This closes #830
---
nanofi/include/api/nanofi.h | 6 +-
nanofi/src/api/nanofi.cpp | 4 ++--
nanofi/tests/CAPITests.cpp | 14 +++---
3 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/nanofi/include/api/nanofi.h b/nanofi/include/api/nanofi.h
index f401073..bec0f83 100644
--- a/nanofi/include/api/nanofi.h
+++ b/nanofi/include/api/nanofi.h
@@ -68,13 +68,17 @@ void set_terminate_callback(void (*terminate_callback)());
* ##
*/
+nifi_instance *create_instance_repo(const char *url, nifi_port *port, const
char* const repo_type);
+
/**
* Creates a new MiNiFi instance
* @param url remote URL the instance connects to
* @param port remote port the instance connects to
* @return pointer to the new instance
**/
-nifi_instance *create_instance(const char *url, nifi_port *port);
+static nifi_instance *create_instance(const char *url, nifi_port *port){
+ return create_instance_repo(url, port, "filesystemrepository");
+}
/**
* Initialize remote connection of instance for transfers
diff --git a/nanofi/src/api/nanofi.cpp b/nanofi/src/api/nanofi.cpp
index 29944bb..616398c 100644
--- a/nanofi/src/api/nanofi.cpp
+++ b/nanofi/src/api/nanofi.cpp
@@ -108,7 +108,7 @@ class DirectoryConfiguration {
* @Deprecated for API version 0.2 in favor of the following prototype
* nifi_instance *create_instance(nifi_port const *port) {
*/
-nifi_instance *create_instance(const char *url, nifi_port *port) {
+nifi_instance *create_instance_repo(const char *url, nifi_port *port, const
char* const repo_type) {
// make sure that we have a thread safe way of initializing the content
directory
DirectoryConfiguration::initialize();
@@ -118,7 +118,7 @@ nifi_instance *create_instance(const char *url, nifi_port
*port) {
* This API will gradually move away from C++, hence malloc is used for
nifi_instance
* Since minifi::Instance is currently being used, then we need to use new
in that case.
*/
- instance->instance_ptr = new minifi::Instance(url, port->port_id,
"filesystemrepository");
+ instance->instance_ptr = new minifi::Instance(url, port->port_id, repo_type);
NULL_CHECK(nullptr, instance->instance_ptr);
diff --git a/nanofi/tests/CAPITests.cpp b/nanofi/tests/CAPITests.cpp
index 4112155..b130c16 100644
--- a/nanofi/tests/CAPITests.cpp
+++ b/nanofi/tests/CAPITests.cpp
@@ -15,29 +15,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include
#include
-#include
#include
-#include
-#include
#include
-
#include "utils/file/FileUtils.h"
#include "TestBase.h"
-
-#include
-#include
#include "api/nanofi.h"
-std::string test_file_content = "C API raNdOMcaSe test d4t4 th1s is!";
-std::string test_file_name = "tstFile.ext";
+const std::string test_file_content = "C API raNdOMcaSe test d4t4 th1s is!";
+const std::string test_file_name = "tstFile.ext";
static nifi_instance *create_instance_obj(const char *name =
"random_instance") {
nifi_port port;
char port_str[] = "12345";
port.port_id = port_str;
- return create_instance("random_instance", &port);
+ return create_instance_repo("random_instance", &port, "volatilerepository");
}
static int failure_count = 0;
[nifi] branch main updated: NIFI-7542 Upgrade additional jackson-databind versions.
This is an automated email from the ASF dual-hosted git repository.
mthomsen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new f43dc80 NIFI-7542 Upgrade additional jackson-databind versions.
f43dc80 is described below
commit f43dc8063f4f7c3ddd7efe700c9c9be4da4179fb
Author: mtien
AuthorDate: Thu Jun 25 13:40:23 2020 -0700
NIFI-7542 Upgrade additional jackson-databind versions.
This close #4362
Signed-off-by: Mike Thomsen
---
nifi-nar-bundles/nifi-easyrules-bundle/nifi-easyrules-service/pom.xml | 2 +-
.../nifi-elasticsearch-client-service-api/pom.xml | 2 +-
nifi-nar-bundles/nifi-graph-bundle/nifi-graph-processors/pom.xml| 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-easyrules-bundle/nifi-easyrules-service/pom.xml
b/nifi-nar-bundles/nifi-easyrules-bundle/nifi-easyrules-service/pom.xml
index 6620a33..481956a 100644
--- a/nifi-nar-bundles/nifi-easyrules-bundle/nifi-easyrules-service/pom.xml
+++ b/nifi-nar-bundles/nifi-easyrules-bundle/nifi-easyrules-service/pom.xml
@@ -62,7 +62,7 @@
com.fasterxml.jackson.core
jackson-databind
-2.9.10.4
+${jackson-databind.version}
diff --git
a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api/pom.xml
b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api/pom.xml
index 257ceaa..cd87e6d 100644
---
a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api/pom.xml
+++
b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api/pom.xml
@@ -46,7 +46,7 @@
com.fasterxml.jackson.core
jackson-databind
-2.9.10.4
+${jackson-databind.version}
compile
diff --git a/nifi-nar-bundles/nifi-graph-bundle/nifi-graph-processors/pom.xml
b/nifi-nar-bundles/nifi-graph-bundle/nifi-graph-processors/pom.xml
index c5ca9b5..2c46c30 100644
--- a/nifi-nar-bundles/nifi-graph-bundle/nifi-graph-processors/pom.xml
+++ b/nifi-nar-bundles/nifi-graph-bundle/nifi-graph-processors/pom.xml
@@ -52,7 +52,7 @@
com.fasterxml.jackson.core
jackson-databind
-2.9.10.4
+${jackson-databind.version}
org.apache.nifi
[nifi] branch main updated: NIFI-7602 added verify phase so contrib-check can actually execute
This is an automated email from the ASF dual-hosted git repository. mthomsen pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new f8e5be3 NIFI-7602 added verify phase so contrib-check can actually execute f8e5be3 is described below commit f8e5be3c09de7cfc4d040743d739ddc1fc823aab Author: Joe Witt AuthorDate: Mon Jul 6 11:49:11 2020 -0700 NIFI-7602 added verify phase so contrib-check can actually execute This closes #4388 Signed-off-by: Mike Thomsen --- .github/workflows/ci-workflow.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci-workflow.yml b/.github/workflows/ci-workflow.yml index 5de3f68..f794489 100644 --- a/.github/workflows/ci-workflow.yml +++ b/.github/workflows/ci-workflow.yml @@ -79,7 +79,7 @@ jobs: env: MAVEN_OPTS: -Xmx2g -XX:ReservedCodeCacheSize=1g -XX:+UseG1GC -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN -Dmaven.surefire.arguments="-Duser.language=fr -Duser.region=FR -Duser.timezone=Europe/Paris" run: | - mvn -V -T 0.8C package -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu + mvn -V -T 0.8C package verify -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu - name: Disk After Build run: | df -h @@ -161,7 +161,7 @@ jobs: env: MAVEN_OPTS: -Xmx2g -XX:ReservedCodeCacheSize=1g -XX:+UseG1GC -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN -Dmaven.surefire.arguments="-Duser.language=en -Duser.region=AU -Duser.timezone=Australia/Melbourne" run: | - mvn -V -T 0.8C package -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu + mvn -V -T 0.8C package verify -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu - name: Disk After Build run: | df -h @@ -238,7 +238,7 @@ jobs: env: MAVEN_OPTS: -Xmx2g -XX:ReservedCodeCacheSize=1g -XX:+UseG1GC -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN -Dmaven.surefire.arguments="-Duser.language=ja -Duser.region=JP -Duser.timezone=Asia/Tokyo" run: | - mvn -V -T 0.8C package -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu + mvn -V -T 0.8C package verify -B -Pcontrib-check,include-grpc -Ddir-only -ntp -ff -pl -nifi-assembly,-nifi-toolkit/nifi-toolkit-assembly,-nifi-system-tests -nsu - name: Disk After Build run: | df -h
[nifi] branch main updated: NIFI-7603 fix doc
This is an automated email from the ASF dual-hosted git repository. mthomsen pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new ea02788 NIFI-7603 fix doc ea02788 is described below commit ea02788de63c44434eee01dc3776ed85a9b3fea4 Author: KovalevIV AuthorDate: Mon Jul 6 19:49:28 2020 +0300 NIFI-7603 fix doc This closes #4387 Signed-off-by: Mike Thomsen --- .../org.apache.nifi.lookup.RestLookupService/additionalDetails.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/resources/docs/org.apache.nifi.lookup.RestLookupService/additionalDetails.html b/nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/resources/docs/org.apache.nifi.lookup.RestLookupService/additionalDetails.html index b38b919..4f0d4f9 100644 --- a/nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/resources/docs/org.apache.nifi.lookup.RestLookupService/additionalDetails.html +++ b/nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/resources/docs/org.apache.nifi.lookup.RestLookupService/additionalDetails.html @@ -31,7 +31,7 @@ put -body; contains a string representing JSON, XML, etc. to be sent with any +request.body; contains a string representing JSON, XML, etc. to be sent with any of those methods except for "get". mime.type; specifies media type of the request body, required when 'body' is passed. *; any other keys can be configured to pass variables to resolve target URLs. See 'Dynamic URLs' section below.
[nifi] branch main updated: NIFI-7595 Upgrade log4j-core dependency.
This is an automated email from the ASF dual-hosted git repository. mthomsen pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 74b4d5a NIFI-7595 Upgrade log4j-core dependency. 74b4d5a is described below commit 74b4d5ac5ff359103886d1bd575d15bcb4c2530f Author: mtien AuthorDate: Wed Jul 1 14:22:15 2020 -0700 NIFI-7595 Upgrade log4j-core dependency. This closes #4382 Signed-off-by: Mike Thomsen --- .../nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/pom.xml | 2 +- .../nifi-elasticsearch-restapi-processors/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/pom.xml b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/pom.xml index 032d6c5..7c416c5 100644 --- a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/pom.xml +++ b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/pom.xml @@ -75,7 +75,7 @@ language governing permissions and limitations under the License. --> org.apache.logging.log4j log4j-core -2.8.2 +2.13.3 org.apache.nifi diff --git a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml index 642f406..5c95a88 100644 --- a/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml +++ b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml @@ -74,7 +74,7 @@ language governing permissions and limitations under the License. --> org.apache.logging.log4j log4j-core -2.8.2 +2.13.3 org.apache.nifi
