[GitHub] [nifi] s9514171 opened a new pull request #4406: NIFI-7623 Take unnecessary behavior out when using registry to update…
s9514171 opened a new pull request #4406:
URL: https://github.com/apache/nifi/pull/4406
… the flow
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFIREG-403) Nifi-Registry showing anonymous version from loged user on NiFi
[ https://issues.apache.org/jira/browse/NIFIREG-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157068#comment-17157068 ] Alejandro Fiel Martínez commented on NIFIREG-403: - Sorry, I misunderstood the feature, can close this. Will wait for openid to have full log. Thanks! > Nifi-Registry showing anonymous version from loged user on NiFi > --- > > Key: NIFIREG-403 > URL: https://issues.apache.org/jira/browse/NIFIREG-403 > Project: NiFi Registry > Issue Type: Bug > Environment: NiFi Registry 0.5.0 on Amazon Linux AMI 2 >Reporter: Alejandro Fiel Martínez >Priority: Major > > Hi, I have 2 NiFi 1.11.4 installed along NiFi Registry. > I log in on NiFi with openId, no login on Registry (Nginx). > When I generate a new version fron NiFi to Registry, the user that generated > that version it showed as "anonymous" on Registry when it was loged in on > NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFIREG-403) Nifi-Registry showing anonymous version from loged user on NiFi
[ https://issues.apache.org/jira/browse/NIFIREG-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157062#comment-17157062 ] Bryan Bende commented on NIFIREG-403: - That is not how it works... NiFi Registry is an application with a REST API, the UI and NiFi are both clients of the REST API. From the perspective of saving a version of a flow, it is just a call to the REST API, it could be from NiFI or from curl or from NiFi CLI, or any tool. If registry is unsecured then we have not authenticated whoever is making that call and therefore we have no way to know what user identity to store as the committer. In the case when NiFi is saving to NiFI Registry, NiFi does send over the identity of the user, but it still requires that NiFi itself authenticates to registry and registry verifies that NiFi is an authorized proxy. > Nifi-Registry showing anonymous version from loged user on NiFi > --- > > Key: NIFIREG-403 > URL: https://issues.apache.org/jira/browse/NIFIREG-403 > Project: NiFi Registry > Issue Type: Bug > Environment: NiFi Registry 0.5.0 on Amazon Linux AMI 2 >Reporter: Alejandro Fiel Martínez >Priority: Major > > Hi, I have 2 NiFi 1.11.4 installed along NiFi Registry. > I log in on NiFi with openId, no login on Registry (Nginx). > When I generate a new version fron NiFi to Registry, the user that generated > that version it showed as "anonymous" on Registry when it was loged in on > NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFIREG-403) Nifi-Registry showing anonymous version from loged user on NiFi
[ https://issues.apache.org/jira/browse/NIFIREG-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156957#comment-17156957 ] Alejandro Fiel Martínez edited comment on NIFIREG-403 at 7/13/20, 9:36 PM: --- NiFi Registry is not yet secured (waiting for openid support) I push the changes from a logged account in Apache NiFi. I think NiFi Registry should take the name of the "commiting" user on the NiFi instance, the login to Registry should allow me to create buckets and view the changes commited by each user on NiFi. Or...not? Thanks was (Author: alejandrofm): NiFi Registry is not yet secured (waiting for openid support) I push the changes from a logged account in Apache NiFi. I think NiFi Registry should take the name of the "commiting" user on the NiFi instance, the login to Registry should allow me to create buckets and view the changes commited by each user. Or...not? Thanks > Nifi-Registry showing anonymous version from loged user on NiFi > --- > > Key: NIFIREG-403 > URL: https://issues.apache.org/jira/browse/NIFIREG-403 > Project: NiFi Registry > Issue Type: Bug > Environment: NiFi Registry 0.5.0 on Amazon Linux AMI 2 >Reporter: Alejandro Fiel Martínez >Priority: Major > > Hi, I have 2 NiFi 1.11.4 installed along NiFi Registry. > I log in on NiFi with openId, no login on Registry (Nginx). > When I generate a new version fron NiFi to Registry, the user that generated > that version it showed as "anonymous" on Registry when it was loged in on > NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFIREG-403) Nifi-Registry showing anonymous version from loged user on NiFi
[ https://issues.apache.org/jira/browse/NIFIREG-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156957#comment-17156957 ] Alejandro Fiel Martínez commented on NIFIREG-403: - NiFi Registry is not yet secured (waiting for openid support) I push the changes from a logged account in Apache NiFi. I think NiFi Registry should take the name of the "commiting" user on the NiFi instance, the login to Registry should allow me to create buckets and view the changes commited by each user. Or...not? Thanks > Nifi-Registry showing anonymous version from loged user on NiFi > --- > > Key: NIFIREG-403 > URL: https://issues.apache.org/jira/browse/NIFIREG-403 > Project: NiFi Registry > Issue Type: Bug > Environment: NiFi Registry 0.5.0 on Amazon Linux AMI 2 >Reporter: Alejandro Fiel Martínez >Priority: Major > > Hi, I have 2 NiFi 1.11.4 installed along NiFi Registry. > I log in on NiFi with openId, no login on Registry (Nginx). > When I generate a new version fron NiFi to Registry, the user that generated > that version it showed as "anonymous" on Registry when it was loged in on > NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
arpadboda commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453885919
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
+ConsumeWindowsEventLog::ProcessOldEvents
+ };
+ for (const core::Property& property :
properties_required_or_with_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + " value ")) {
+ FAIL("Property did not get queried: " << property.getName());
+}
+ }
+
+ auto properties_optional_without_default_value = {
+ConsumeWindowsEventLog::EventHeaderDelimiter
+ };
+ for (const core::Property& property :
properties_optional_without_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + ", empty value")) {
+ FAIL("Optional property did not get queried: " << property.getName());
+}
+ }
+
+ REQUIRE(LogTestController::getInstance().contains("Successfully configured
CWEL"));
+}
+
+TEST_CASE("ConsumeWindowsEventLog onSchedule throws if it cannot create the
bookmark", "[create][bookmark]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor =
[GitHub] [nifi] pvillard31 commented on pull request #4405: NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allo…
pvillard31 commented on pull request #4405: URL: https://github.com/apache/nifi/pull/4405#issuecomment-657642155 Will review and play with it during the week... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7633) Provide Process Groups with a FlowFile Concurrency level for transferring a batch of FlowFiles at once
[ https://issues.apache.org/jira/browse/NIFI-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-7633: - Fix Version/s: 1.12.0 Status: Patch Available (was: Open) > Provide Process Groups with a FlowFile Concurrency level for transferring a > batch of FlowFiles at once > -- > > Key: NIFI-7633 > URL: https://issues.apache.org/jira/browse/NIFI-7633 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.12.0 > > > NIFI-7476 provides the ability for a Process Group to bring in a single > FlowFile at a time instead of continually streaming data. It also provides > the ability to output data as a batch. If a Process Group outputs the data in > a batch, another Process Group should have the ability to pull in all of that > data in a single batch as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 opened a new pull request #4405: NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allo…
markap14 opened a new pull request #4405:
URL: https://github.com/apache/nifi/pull/4405
…w data to be pulled into a ProcessGroup as a batch
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453732134
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
+ConsumeWindowsEventLog::ProcessOldEvents
+ };
+ for (const core::Property& property :
properties_required_or_with_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + " value ")) {
+ FAIL("Property did not get queried: " << property.getName());
+}
+ }
+
+ auto properties_optional_without_default_value = {
+ConsumeWindowsEventLog::EventHeaderDelimiter
+ };
+ for (const core::Property& property :
properties_optional_without_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + ", empty value")) {
+ FAIL("Optional property did not get queried: " << property.getName());
+}
+ }
+
+ REQUIRE(LogTestController::getInstance().contains("Successfully configured
CWEL"));
+}
+
+TEST_CASE("ConsumeWindowsEventLog onSchedule throws if it cannot create the
bookmark", "[create][bookmark]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor =
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453731872
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
Review comment:
I've added a comment about deprecation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453731600
##
File path: extensions/windows-event-log/tests/BookmarkTests.cpp
##
@@ -0,0 +1,245 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Bookmark.h"
+
+#include
+
+#include "TestBase.h"
+#include "utils/gsl.h"
+#include "wel/UniqueEvtHandle.h"
+
+using Bookmark = org::apache::nifi::minifi::processors::Bookmark;
+using unique_evt_handle = org::apache::nifi::minifi::wel::unique_evt_handle;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+const std::wstring APPLICATION_CHANNEL = L"Application";
+
+constexpr DWORD BOOKMARK_TESTS_OPCODE = 10368; // random opcode hopefully
won't clash with something important
+
+std::unique_ptr createBookmark(TestPlan _plan,
+ const std::wstring ,
+ const std::string =
IdGenerator::getIdGenerator()->generate().to_string()) {
+ const auto state_manager =
test_plan.getStateManagerProvider()->getCoreComponentStateManager(uuid);
+ const auto logger = test_plan.getLogger();
+ return utils::make_unique(channel, L"*", "", uuid, false,
state_manager, logger);
+}
+
+void reportEvent(const std::wstring& channel, const char* message) {
+ auto event_source = RegisterEventSourceW(nullptr, channel.c_str());
+ auto event_source_deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, EVENTLOG_INFORMATION_TYPE, 0,
+ BOOKMARK_TESTS_OPCODE, nullptr, 1, 0, , nullptr);
+}
+
+std::wstring bookmarkHandleAsXml(EVT_HANDLE event) {
+ REQUIRE(event);
+ constexpr std::size_t BUFFER_SIZE = 1024;
+ std::array buffer = {};
+ DWORD buffer_used;
+ DWORD property_count;
+ if (!EvtRender(nullptr, event, EvtRenderBookmark, buffer.size(),
buffer.data(), _used, _count)) {
+FAIL("Could not render event; error code: " << GetLastError());
+ }
+ return std::wstring{buffer.data()};
+}
+
+std::wstring bookmarkAsXml(const std::unique_ptr& bookmark) {
+ REQUIRE(bookmark);
+ REQUIRE(*bookmark);
+ return bookmarkHandleAsXml(bookmark->getBookmarkHandleFromXML());
+}
+
+unique_evt_handle queryEvents() {
+ std::wstring query = L"Event/System/EventID=" +
std::to_wstring(BOOKMARK_TESTS_OPCODE);
+ unique_evt_handle results{EvtQuery(NULL, APPLICATION_CHANNEL.c_str(),
query.c_str(), EvtQueryChannelPath | EvtQueryReverseDirection)};
+ if (!results) {
+FAIL("EvtQuery() failed; error code: " << GetLastError());
+ }
+ return results;
+}
+
+unique_evt_handle getFirstEventFromResults(const unique_evt_handle& results) {
+ REQUIRE(results);
+ EVT_HANDLE event_raw_handle = 0;
+ DWORD timeout_ms = 100;
Review comment:
done
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453731504
##
File path: extensions/windows-event-log/tests/BookmarkTests.cpp
##
@@ -0,0 +1,245 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Bookmark.h"
+
+#include
+
+#include "TestBase.h"
+#include "utils/gsl.h"
+#include "wel/UniqueEvtHandle.h"
+
+using Bookmark = org::apache::nifi::minifi::processors::Bookmark;
+using unique_evt_handle = org::apache::nifi::minifi::wel::unique_evt_handle;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+const std::wstring APPLICATION_CHANNEL = L"Application";
+
+constexpr DWORD BOOKMARK_TESTS_OPCODE = 10368; // random opcode hopefully
won't clash with something important
+
+std::unique_ptr createBookmark(TestPlan _plan,
+ const std::wstring ,
+ const std::string =
IdGenerator::getIdGenerator()->generate().to_string()) {
+ const auto state_manager =
test_plan.getStateManagerProvider()->getCoreComponentStateManager(uuid);
+ const auto logger = test_plan.getLogger();
+ return utils::make_unique(channel, L"*", "", uuid, false,
state_manager, logger);
+}
+
+void reportEvent(const std::wstring& channel, const char* message) {
+ auto event_source = RegisterEventSourceW(nullptr, channel.c_str());
+ auto event_source_deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, EVENTLOG_INFORMATION_TYPE, 0,
+ BOOKMARK_TESTS_OPCODE, nullptr, 1, 0, , nullptr);
+}
+
+std::wstring bookmarkHandleAsXml(EVT_HANDLE event) {
+ REQUIRE(event);
+ constexpr std::size_t BUFFER_SIZE = 1024;
+ std::array buffer = {};
+ DWORD buffer_used;
+ DWORD property_count;
+ if (!EvtRender(nullptr, event, EvtRenderBookmark, buffer.size(),
buffer.data(), _used, _count)) {
+FAIL("Could not render event; error code: " << GetLastError());
+ }
+ return std::wstring{buffer.data()};
+}
+
+std::wstring bookmarkAsXml(const std::unique_ptr& bookmark) {
+ REQUIRE(bookmark);
+ REQUIRE(*bookmark);
+ return bookmarkHandleAsXml(bookmark->getBookmarkHandleFromXML());
+}
+
+unique_evt_handle queryEvents() {
+ std::wstring query = L"Event/System/EventID=" +
std::to_wstring(BOOKMARK_TESTS_OPCODE);
+ unique_evt_handle results{EvtQuery(NULL, APPLICATION_CHANNEL.c_str(),
query.c_str(), EvtQueryChannelPath | EvtQueryReverseDirection)};
+ if (!results) {
+FAIL("EvtQuery() failed; error code: " << GetLastError());
+ }
+ return results;
+}
+
+unique_evt_handle getFirstEventFromResults(const unique_evt_handle& results) {
+ REQUIRE(results);
+ EVT_HANDLE event_raw_handle = 0;
+ DWORD timeout_ms = 100;
+ DWORD num_results_found = 0;
+ bool result = EvtNext(results.get(), 1, _raw_handle, timeout_ms, 0,
_results_found);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4377: NIFI-7568 - Fixing Kerberos mappings
thenatog commented on pull request #4377: URL: https://github.com/apache/nifi/pull/4377#issuecomment-657599212 Created a separate issue to add/test OIDC identity mappings: https://issues.apache.org/jira/browse/NIFI-7634. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-7634) Add OIDC identity mappings
Nathan Gough created NIFI-7634: -- Summary: Add OIDC identity mappings Key: NIFI-7634 URL: https://issues.apache.org/jira/browse/NIFI-7634 Project: Apache NiFi Issue Type: New Feature Components: Security Affects Versions: 1.11.4 Reporter: Nathan Gough Assignee: Nathan Gough Fix For: 1.12.0 Add identity mappings for OIDC identities, such that the user database and JWT and NiFi will use the mapped identity if configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #835: MINIFICPP-1281 - Improve test performance by using event polling instead of sleep by sync
fgerlits commented on a change in pull request #835:
URL: https://github.com/apache/nifi-minifi-cpp/pull/835#discussion_r453682665
##
File path: libminifi/include/utils/IntegrationTestUtils.h
##
@@ -0,0 +1,66 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "../../../libminifi/test/TestBase.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+template
+bool verifyEventHappenedInPollTime(const std::chrono::duration&
wait_duration, Fun&& check) {
+ std::chrono::system_clock::time_point wait_end =
std::chrono::system_clock::now() + wait_duration;
+ do {
+if (std::forward(check)()) {
+ return true;
+}
+std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ } while (std::chrono::system_clock::now() < wait_end);
+ return false;
+}
+
+template
+bool verifyLogLinePresenceInPollTime(const std::chrono::duration&
wait_duration, String... patterns) {
+ // This helper is to be removed once we upgrade to support gcc 4.9+ only
Review comment:
what is the gcc issue here? https://godbolt.org/z/hbMvss compiles fine
on gcc 4.8.5 (and even on 4.7.4), at least in this sandbox
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1121:
Description:
Or version of spdlog is 2+ years old. The new spdlog version uses a new version
of the cppformat (back then)/fmt (now) formatting library.
We should consider directly depending on {{fmt}} since we already have it as a
transitive dependency and it would be useful for e.g. formatting
exception/error messages, etc.
*Update (hunyadi):*
Seems like we have to skip version 1.0 with upgrading. There are quite a lot of
non-documented breaking changes, for example this commit:
[https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
in spdlog are not documented and the codebase is kept up-to-date with commits
sent directly to the central repository.
was:
Or version of spdlog is 2+ years old. The new spdlog version uses a new version
of the cppformat (back then)/fmt (now) formatting library.
We should consider directly depending on {{fmt}} since we already have it as a
transitive dependency and it would be useful for e.g. formatting
exception/error messages, etc.
*Update (hunyadi):*
Seems like we have to skip version 1.0 with upgrading. There are quite a lot of
non-documented breaking changes, for example this commit:
[https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
breaks ::instance() on stdout/stderr sinks.
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
>
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks. Unfortunately, changes like this
> in spdlog are not documented and the codebase is kept up-to-date with commits
> sent directly to the central repository.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi updated MINIFICPP-1121:
Description:
Or version of spdlog is 2+ years old. The new spdlog version uses a new version
of the cppformat (back then)/fmt (now) formatting library.
We should consider directly depending on {{fmt}} since we already have it as a
transitive dependency and it would be useful for e.g. formatting
exception/error messages, etc.
*Update (hunyadi):*
Seems like we have to skip version 1.0 with upgrading. There are quite a lot of
non-documented breaking changes, for example this commit:
[https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
breaks ::instance() on stdout/stderr sinks.
was:
Or version of spdlog is 2+ years old. The new spdlog version uses a new version
of the cppformat (back then)/fmt (now) formatting library.
We should consider directly depending on {{fmt}} since we already have it as a
transitive dependency and it would be useful for e.g. formatting
exception/error messages, etc.
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
>
> *Update (hunyadi):*
> Seems like we have to skip version 1.0 with upgrading. There are quite a lot
> of non-documented breaking changes, for example this commit:
> [https://github.com/gabime/spdlog/commit/6f4cd8d397a443f095c1dce5c025f55684c70eac#diff-9458442ae281c51018015fd2773dc688]
> breaks ::instance() on stdout/stderr sinks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
arpadboda commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453665360
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -119,7 +107,125 @@ TEST_CASE("Flow shutdown drains connections",
"[TestFlow1]") {
controller->stop(true);
+ REQUIRE(sinkProc->trigger_count == 0);
+
for (auto& it : connectionMap) {
REQUIRE(it.second->isEmpty());
}
}
+
+TEST_CASE("Flow shutdown waits for a while", "[TestFlow2]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ // before we could initiate the shutdown
+ sinkProc->yield(100);
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 0);
+
+ controller->stop(true);
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 3);
+}
+
+TEST_CASE("Flow stopped after grace period", "[TestFlow3]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+
testController.configuration_->set(minifi::Configure::nifi_flowcontroller_drain_timeout,
"1000 ms");
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ sinkProc->yield(100);
+
+ sinkProc->onTriggerCb_ = [&]{
+static std::atomic first_onTrigger{true};
+bool isFirst = true;
+// sleep only on the first trigger
+if (first_onTrigger.compare_exchange_strong(isFirst, false)) {
+ std::this_thread::sleep_for(std::chrono::milliseconds{1500});
+}
+ };
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
Review comment:
Just checked the code: there are functions to increment and decrement
active tasks of a processor. The decrement happens after the session is
commited, which a piece of good news for us:
```
bool Processor::isRunning() {
return (state_ == RUNNING && active_tasks_ > 0);
}
```
This should properly be able to tell us whether the execution (including
committing and destroying the session) is already completed or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
arpadboda commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453665360
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -119,7 +107,125 @@ TEST_CASE("Flow shutdown drains connections",
"[TestFlow1]") {
controller->stop(true);
+ REQUIRE(sinkProc->trigger_count == 0);
+
for (auto& it : connectionMap) {
REQUIRE(it.second->isEmpty());
}
}
+
+TEST_CASE("Flow shutdown waits for a while", "[TestFlow2]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ // before we could initiate the shutdown
+ sinkProc->yield(100);
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 0);
+
+ controller->stop(true);
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 3);
+}
+
+TEST_CASE("Flow stopped after grace period", "[TestFlow3]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+
testController.configuration_->set(minifi::Configure::nifi_flowcontroller_drain_timeout,
"1000 ms");
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ sinkProc->yield(100);
+
+ sinkProc->onTriggerCb_ = [&]{
+static std::atomic first_onTrigger{true};
+bool isFirst = true;
+// sleep only on the first trigger
+if (first_onTrigger.compare_exchange_strong(isFirst, false)) {
+ std::this_thread::sleep_for(std::chrono::milliseconds{1500});
+}
+ };
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
Review comment:
Just checked the code: there are functions to increment and decrement
active tasks of a processor. The decrement happens after the session is
commited, which a piece of good news for us:
```
bool Processor::isRunning() {
return (state_ == RUNNING && active_tasks_ > 0);
}
```
This should properly be able to tell us whether the execution (including
committing and destroying the session) already finished or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
arpadboda commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453665360
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -119,7 +107,125 @@ TEST_CASE("Flow shutdown drains connections",
"[TestFlow1]") {
controller->stop(true);
+ REQUIRE(sinkProc->trigger_count == 0);
+
for (auto& it : connectionMap) {
REQUIRE(it.second->isEmpty());
}
}
+
+TEST_CASE("Flow shutdown waits for a while", "[TestFlow2]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ // before we could initiate the shutdown
+ sinkProc->yield(100);
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 0);
+
+ controller->stop(true);
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 3);
+}
+
+TEST_CASE("Flow stopped after grace period", "[TestFlow3]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+
testController.configuration_->set(minifi::Configure::nifi_flowcontroller_drain_timeout,
"1000 ms");
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ sinkProc->yield(100);
+
+ sinkProc->onTriggerCb_ = [&]{
+static std::atomic first_onTrigger{true};
+bool isFirst = true;
+// sleep only on the first trigger
+if (first_onTrigger.compare_exchange_strong(isFirst, false)) {
+ std::this_thread::sleep_for(std::chrono::milliseconds{1500});
+}
+ };
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
Review comment:
Just checked the code: there are functions to increment and decrement
active tasks of a processor. The decrement happens after the session is
commited, which a piece of good news for us:
```
bool Processor::isRunning() {
return (state_ == RUNNING && active_tasks_ > 0);
}
```
This should properly be able to tell us whether the execution (including
committing and destroying the session) is already finished or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-7497) AWS Credentials for Assume Role need to be able to configure STS Endpoint
[
https://issues.apache.org/jira/browse/NIFI-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende resolved NIFI-7497.
---
Fix Version/s: 1.12.0
Resolution: Fixed
> AWS Credentials for Assume Role need to be able to configure STS Endpoint
> -
>
> Key: NIFI-7497
> URL: https://issues.apache.org/jira/browse/NIFI-7497
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Neptune Salt
>Priority: Minor
> Fix For: 1.12.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As a user of NiFi, when I want to enable cross account access in certain
> environments, I want to be able to override the STS endpoint for the security
> token service.
> This arises from the limitations here:
> [https://github.com/aws/aws-sdk-java/blob/b1b1a21fa46f8948fcf39e8b3a76f6ebe00e14b9/aws-java-sdk-sts/src/main/java/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.java#L291]
> The relevant comment being:
>
> {code:java}
> /**
> * Sets the AWS Security Token Service (STS) endpoint where session
> credentials are retrieved
> * from. The default AWS Security Token Service (STS) endpoint
> ("sts.amazonaws.com")
> * works for all accounts that are not for China (Beijing) region or
> GovCloud. You only need to
> * change the endpoint to "sts.cn-north-1.amazonaws.com.cn" when you are
> requesting session
> * credentials for services in China(Beijing) region or
> "sts.us-gov-west-1.amazonaws.com" for
> * GovCloud. Setting this invalidates existing session
> credentials.
> *
> * @deprecated This method may be removed in a future major version.
> Create multiple providers
> * if you need to work with multiple STS endpoints.
> */
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] bbende merged pull request #4402: NIFI-7497 Removed a few style check bugs that crept up in the last co…
bbende merged pull request #4402: URL: https://github.com/apache/nifi/pull/4402 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7497) AWS Credentials for Assume Role need to be able to configure STS Endpoint
[
https://issues.apache.org/jira/browse/NIFI-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156714#comment-17156714
]
ASF subversion and git services commented on NIFI-7497:
---
Commit bb5e9fb00b21c6cbf782c9ce4ddf818789b92084 in nifi's branch
refs/heads/main from Mike Thomsen
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bb5e9fb ]
NIFI-7497 Removed a few style check bugs that crept up in the last commit.
(#4402)
> AWS Credentials for Assume Role need to be able to configure STS Endpoint
> -
>
> Key: NIFI-7497
> URL: https://issues.apache.org/jira/browse/NIFI-7497
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Neptune Salt
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As a user of NiFi, when I want to enable cross account access in certain
> environments, I want to be able to override the STS endpoint for the security
> token service.
> This arises from the limitations here:
> [https://github.com/aws/aws-sdk-java/blob/b1b1a21fa46f8948fcf39e8b3a76f6ebe00e14b9/aws-java-sdk-sts/src/main/java/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.java#L291]
> The relevant comment being:
>
> {code:java}
> /**
> * Sets the AWS Security Token Service (STS) endpoint where session
> credentials are retrieved
> * from. The default AWS Security Token Service (STS) endpoint
> ("sts.amazonaws.com")
> * works for all accounts that are not for China (Beijing) region or
> GovCloud. You only need to
> * change the endpoint to "sts.cn-north-1.amazonaws.com.cn" when you are
> requesting session
> * credentials for services in China(Beijing) region or
> "sts.us-gov-west-1.amazonaws.com" for
> * GovCloud. Setting this invalidates existing session
> credentials.
> *
> * @deprecated This method may be removed in a future major version.
> Create multiple providers
> * if you need to work with multiple STS endpoints.
> */
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-7633) Provide Process Groups with a FlowFile Concurrency level for transferring a batch of FlowFiles at once
Mark Payne created NIFI-7633: Summary: Provide Process Groups with a FlowFile Concurrency level for transferring a batch of FlowFiles at once Key: NIFI-7633 URL: https://issues.apache.org/jira/browse/NIFI-7633 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne NIFI-7476 provides the ability for a Process Group to bring in a single FlowFile at a time instead of continually streaming data. It also provides the ability to output data as a batch. If a Process Group outputs the data in a batch, another Process Group should have the ability to pull in all of that data in a single batch as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #821: MINIFICPP-1251 - Implement and test RetryFlowFile processor
arpadboda commented on a change in pull request #821:
URL: https://github.com/apache/nifi-minifi-cpp/pull/821#discussion_r453631636
##
File path: extensions/standard-processors/processors/RetryFlowFile.cpp
##
@@ -0,0 +1,183 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RetryFlowFile.h"
+
+#include "core/PropertyValidation.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property
RetryFlowFile::RetryAttribute(core::PropertyBuilder::createProperty("Retry
Attribute")
+->withDescription(
+"The name of the attribute that contains the current retry count for
the FlowFile."
+"WARNING: If the name matches an attribute already on the FlowFile
that does not contain a numerical value, "
+"the processor will either overwrite that attribute with '1' or fail
based on configuration.")
+->withDefaultValue("flowfile.retries")
Review comment:
Looking at NiFi code is always a good source of hints and what we see
there worth consideration, but it's not an unerring guidance or revelation.
In case we think that our implementation definitely requires a valid value
for the given property to work, I think we should make it required.
In case of properties that define attribute names I would also consider
something like this:
https://www.javadoc.io/doc/org.apache.nifi/nifi-processor-utils/0.7.0/org/apache/nifi/processor/util/StandardValidators.html#NON_BLANK_VALIDATOR
As we request a valid string here that can name an attribute.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7629) Consume & Publish PubSub processors do not evaluate EL for property PROJECT_ID despite its setting
[ https://issues.apache.org/jira/browse/NIFI-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-7629: - Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Consume & Publish PubSub processors do not evaluate EL for property > PROJECT_ID despite its setting > -- > > Key: NIFI-7629 > URL: https://issues.apache.org/jira/browse/NIFI-7629 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.11.1, 1.11.4 > Environment: Nifi package for version 1.11.1 run on macOS Catalina > 10.15.5 >Reporter: Cuong Nguyen >Assignee: Cuong Nguyen >Priority: Major > Labels: beginner > Fix For: 1.12.0 > > Attachments: image-2020-07-11-11-52-52-828.png, > image-2020-07-11-11-55-10-841.png, image-2020-07-11-12-02-10-374.png > > Time Spent: 20m > Remaining Estimate: 0h > > When trying to set property Project ID (processor ConsumeGCPubSub, Bundle > org.apache.nifi - nifi-gcp-nar) with an EL value > !image-2020-07-11-11-52-52-828.png|width=800,height=550! > !image-2020-07-11-12-02-10-374.png|width=800,height=550! > Error message occurs as below > !image-2020-07-11-11-55-10-841.png|width=813,height=210! > The problem does not occur if using plain text value for the same property. > Investigating this processor's source code, I found the cause for this issue > (lastest source code at main branch): > [https://github.com/apache/nifi/blob/8d53f5d0c91a4e759ea7033bb4d0143ba96d2c21/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/ConsumeGCPubSub.java#L182] > The same problem happens with processor PublishGCPubSub: > > [https://github.com/apache/nifi/blob/8d53f5d0c91a4e759ea7033bb4d0143ba96d2c21/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L212] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFIREG-394) Extension's onConfigured should be called with the same ClassLoader
[ https://issues.apache.org/jira/browse/NIFIREG-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFIREG-394. - Fix Version/s: 0.7.0 Resolution: Fixed > Extension's onConfigured should be called with the same ClassLoader > --- > > Key: NIFIREG-394 > URL: https://issues.apache.org/jira/browse/NIFIREG-394 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Christian Englert >Priority: Major > Fix For: 0.7.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Extensions trying to create instances during the onConfigured call can have > classloader exceptions. > AuthorizerFactory for example is creating an extension classloader to create > the instance of the extension, then using another classloader to call > .onConfigured > In case the extension needs the ConfigurationContext in order to be able to > create instances of Classes where the dependencies are not included in the > classloader context used to call .onConfirued() this will lead to problems. > Thus, I'd suggest .onConfigured should be called with the same classloader -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 merged pull request #4403: NIFI-7629: Make property PROJECT_ID evaluate EL on processors Consume…
markap14 merged pull request #4403: URL: https://github.com/apache/nifi/pull/4403 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7629) Consume & Publish PubSub processors do not evaluate EL for property PROJECT_ID despite its setting
[ https://issues.apache.org/jira/browse/NIFI-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156690#comment-17156690 ] ASF subversion and git services commented on NIFI-7629: --- Commit c7a235f85478d39bb11eb44994a8719fd1f4294c in nifi's branch refs/heads/main from Cuong Nguyen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c7a235f ] NIFI-7629: Make property PROJECT_ID evaluate EL on processors ConsumeGCPubSub & PublishGCPubSub > Consume & Publish PubSub processors do not evaluate EL for property > PROJECT_ID despite its setting > -- > > Key: NIFI-7629 > URL: https://issues.apache.org/jira/browse/NIFI-7629 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.11.1, 1.11.4 > Environment: Nifi package for version 1.11.1 run on macOS Catalina > 10.15.5 >Reporter: Cuong Nguyen >Assignee: Cuong Nguyen >Priority: Major > Labels: beginner > Attachments: image-2020-07-11-11-52-52-828.png, > image-2020-07-11-11-55-10-841.png, image-2020-07-11-12-02-10-374.png > > Time Spent: 20m > Remaining Estimate: 0h > > When trying to set property Project ID (processor ConsumeGCPubSub, Bundle > org.apache.nifi - nifi-gcp-nar) with an EL value > !image-2020-07-11-11-52-52-828.png|width=800,height=550! > !image-2020-07-11-12-02-10-374.png|width=800,height=550! > Error message occurs as below > !image-2020-07-11-11-55-10-841.png|width=813,height=210! > The problem does not occur if using plain text value for the same property. > Investigating this processor's source code, I found the cause for this issue > (lastest source code at main branch): > [https://github.com/apache/nifi/blob/8d53f5d0c91a4e759ea7033bb4d0143ba96d2c21/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/ConsumeGCPubSub.java#L182] > The same problem happens with processor PublishGCPubSub: > > [https://github.com/apache/nifi/blob/8d53f5d0c91a4e759ea7033bb4d0143ba96d2c21/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L212] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #4403: NIFI-7629: Make property PROJECT_ID evaluate EL on processors Consume…
markap14 commented on pull request #4403: URL: https://github.com/apache/nifi/pull/4403#issuecomment-657543201 Thanks @ndc629, the changes look good to me. +1 merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-registry] asfgit closed pull request #281: NIFIREG-394: call onConfigured with the same classloader
asfgit closed pull request #281: URL: https://github.com/apache/nifi-registry/pull/281 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFIREG-401) Use String instead of enum for FlowFileOutboundPolicy and FlowFileConcurrency
[ https://issues.apache.org/jira/browse/NIFIREG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFIREG-401. - Fix Version/s: 0.7.0 Assignee: Bryan Bende (was: Mark Payne) Resolution: Fixed This was resolved as part of NIFIREG-398. > Use String instead of enum for FlowFileOutboundPolicy and FlowFileConcurrency > - > > Key: NIFIREG-401 > URL: https://issues.apache.org/jira/browse/NIFIREG-401 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Mark Payne >Assignee: Bryan Bende >Priority: Major > Fix For: 0.7.0 > > > The VersionedProcessGroup has an enum for the FlowfileOutboundPolicy and for > the FlowFileConcurrency. These are likely to to change some in future > releases of nifi (possibly even before the next release of nifi). The > registry doesn't really care what these values are. So it makes sense to use > a String for these values instead of the more restrictive enum for the > registry side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFIREG-403) Nifi-Registry showing anonymous version from loged user on NiFi
[ https://issues.apache.org/jira/browse/NIFIREG-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156686#comment-17156686 ] Bryan Bende commented on NIFIREG-403: - Are you saying NiFi Registry is not secured? It has to be secured in order to authenticate the incoming requests which then produces a user identity, otherwise all requests are made by the anonymous user. > Nifi-Registry showing anonymous version from loged user on NiFi > --- > > Key: NIFIREG-403 > URL: https://issues.apache.org/jira/browse/NIFIREG-403 > Project: NiFi Registry > Issue Type: Bug > Environment: NiFi Registry 0.5.0 on Amazon Linux AMI 2 >Reporter: Alejandro Fiel Martínez >Priority: Major > > Hi, I have 2 NiFi 1.11.4 installed along NiFi Registry. > I log in on NiFi with openId, no login on Registry (Nginx). > When I generate a new version fron NiFi to Registry, the user that generated > that version it showed as "anonymous" on Registry when it was loged in on > NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453609434
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
+ConsumeWindowsEventLog::ProcessOldEvents
+ };
+ for (const core::Property& property :
properties_required_or_with_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + " value ")) {
+ FAIL("Property did not get queried: " << property.getName());
+}
+ }
+
+ auto properties_optional_without_default_value = {
+ConsumeWindowsEventLog::EventHeaderDelimiter
+ };
+ for (const core::Property& property :
properties_optional_without_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + ", empty value")) {
+ FAIL("Optional property did not get queried: " << property.getName());
+}
+ }
+
+ REQUIRE(LogTestController::getInstance().contains("Successfully configured
CWEL"));
+}
+
+TEST_CASE("ConsumeWindowsEventLog onSchedule throws if it cannot create the
bookmark", "[create][bookmark]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor =
[GitHub] [nifi] simonbence closed pull request #4349: NIFI-7549 Adding Hazelcast based DistributedMapCacheClient support
simonbence closed pull request #4349: URL: https://github.com/apache/nifi/pull/4349 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1121) Upgrade spdlog
[
https://issues.apache.org/jira/browse/MINIFICPP-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Hunyadi reassigned MINIFICPP-1121:
---
Assignee: Adam Hunyadi
> Upgrade spdlog
> --
>
> Key: MINIFICPP-1121
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1121
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Marton Szasz
>Assignee: Adam Hunyadi
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
>
> Or version of spdlog is 2+ years old. The new spdlog version uses a new
> version of the cppformat (back then)/fmt (now) formatting library.
> We should consider directly depending on {{fmt}} since we already have it as
> a transitive dependency and it would be useful for e.g. formatting
> exception/error messages, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
arpadboda commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453578121
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
+ConsumeWindowsEventLog::ProcessOldEvents
+ };
+ for (const core::Property& property :
properties_required_or_with_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + " value ")) {
+ FAIL("Property did not get queried: " << property.getName());
+}
+ }
+
+ auto properties_optional_without_default_value = {
+ConsumeWindowsEventLog::EventHeaderDelimiter
+ };
+ for (const core::Property& property :
properties_optional_without_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + ", empty value")) {
+ FAIL("Optional property did not get queried: " << property.getName());
+}
+ }
+
+ REQUIRE(LogTestController::getInstance().contains("Successfully configured
CWEL"));
+}
+
+TEST_CASE("ConsumeWindowsEventLog onSchedule throws if it cannot create the
bookmark", "[create][bookmark]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor =
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
arpadboda commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453573778
##
File path: extensions/windows-event-log/tests/BookmarkTests.cpp
##
@@ -0,0 +1,245 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Bookmark.h"
+
+#include
+
+#include "TestBase.h"
+#include "utils/gsl.h"
+#include "wel/UniqueEvtHandle.h"
+
+using Bookmark = org::apache::nifi::minifi::processors::Bookmark;
+using unique_evt_handle = org::apache::nifi::minifi::wel::unique_evt_handle;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+const std::wstring APPLICATION_CHANNEL = L"Application";
+
+constexpr DWORD BOOKMARK_TESTS_OPCODE = 10368; // random opcode hopefully
won't clash with something important
+
+std::unique_ptr createBookmark(TestPlan _plan,
+ const std::wstring ,
+ const std::string =
IdGenerator::getIdGenerator()->generate().to_string()) {
+ const auto state_manager =
test_plan.getStateManagerProvider()->getCoreComponentStateManager(uuid);
+ const auto logger = test_plan.getLogger();
+ return utils::make_unique(channel, L"*", "", uuid, false,
state_manager, logger);
+}
+
+void reportEvent(const std::wstring& channel, const char* message) {
+ auto event_source = RegisterEventSourceW(nullptr, channel.c_str());
+ auto event_source_deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, EVENTLOG_INFORMATION_TYPE, 0,
+ BOOKMARK_TESTS_OPCODE, nullptr, 1, 0, , nullptr);
+}
+
+std::wstring bookmarkHandleAsXml(EVT_HANDLE event) {
+ REQUIRE(event);
+ constexpr std::size_t BUFFER_SIZE = 1024;
+ std::array buffer = {};
+ DWORD buffer_used;
+ DWORD property_count;
+ if (!EvtRender(nullptr, event, EvtRenderBookmark, buffer.size(),
buffer.data(), _used, _count)) {
+FAIL("Could not render event; error code: " << GetLastError());
+ }
+ return std::wstring{buffer.data()};
+}
+
+std::wstring bookmarkAsXml(const std::unique_ptr& bookmark) {
+ REQUIRE(bookmark);
+ REQUIRE(*bookmark);
+ return bookmarkHandleAsXml(bookmark->getBookmarkHandleFromXML());
+}
+
+unique_evt_handle queryEvents() {
+ std::wstring query = L"Event/System/EventID=" +
std::to_wstring(BOOKMARK_TESTS_OPCODE);
+ unique_evt_handle results{EvtQuery(NULL, APPLICATION_CHANNEL.c_str(),
query.c_str(), EvtQueryChannelPath | EvtQueryReverseDirection)};
+ if (!results) {
+FAIL("EvtQuery() failed; error code: " << GetLastError());
+ }
+ return results;
+}
+
+unique_evt_handle getFirstEventFromResults(const unique_evt_handle& results) {
+ REQUIRE(results);
+ EVT_HANDLE event_raw_handle = 0;
+ DWORD timeout_ms = 100;
+ DWORD num_results_found = 0;
+ bool result = EvtNext(results.get(), 1, _raw_handle, timeout_ms, 0,
_results_found);
Review comment:
I just meant embedding in the if statement, but I can live with the
current code as well, not a big deal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
arpadboda commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453573490
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
Review comment:
Nope, I was just interested in the added value. This covers another
path, it's fine, thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453570853
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
+ ); // NOLINT
+
+ REQUIRE_NOTHROW(
+auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ ); // NOLINT
+}
+
+TEST_CASE("ConsumeWindowsEventLog properties work with default values",
"[create][properties]") {
+ TestController test_controller;
+ LogTestController::getInstance().setDebug();
+ LogTestController::getInstance().setTrace();
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor = test_plan->addProcessor("ConsumeWindowsEventLog", "cwel");
+ test_controller.runSession(test_plan);
+
+ auto properties_required_or_with_default_value = {
+ConsumeWindowsEventLog::Channel,
+ConsumeWindowsEventLog::Query,
+// ConsumeWindowsEventLog::RenderFormatXML, // FIXME(fgerlits): not
defined, does not exist in NiFi either; should be removed
+ConsumeWindowsEventLog::MaxBufferSize,
+// ConsumeWindowsEventLog::InactiveDurationToReconnect, //
FIXME(fgerlits): obsolete, see definition; should be removed
+ConsumeWindowsEventLog::IdentifierMatcher,
+ConsumeWindowsEventLog::IdentifierFunction,
+ConsumeWindowsEventLog::ResolveAsAttributes,
+ConsumeWindowsEventLog::EventHeader,
+ConsumeWindowsEventLog::OutputFormat,
+ConsumeWindowsEventLog::BatchCommitSize,
+ConsumeWindowsEventLog::BookmarkRootDirectory,
+ConsumeWindowsEventLog::ProcessOldEvents
+ };
+ for (const core::Property& property :
properties_required_or_with_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + " value ")) {
+ FAIL("Property did not get queried: " << property.getName());
+}
+ }
+
+ auto properties_optional_without_default_value = {
+ConsumeWindowsEventLog::EventHeaderDelimiter
+ };
+ for (const core::Property& property :
properties_optional_without_default_value) {
+if (!LogTestController::getInstance().contains("property name " +
property.getName() + ", empty value")) {
+ FAIL("Optional property did not get queried: " << property.getName());
+}
+ }
+
+ REQUIRE(LogTestController::getInstance().contains("Successfully configured
CWEL"));
+}
+
+TEST_CASE("ConsumeWindowsEventLog onSchedule throws if it cannot create the
bookmark", "[create][bookmark]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ auto processor =
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453559053
##
File path: extensions/windows-event-log/tests/ConsumeWindowsEventLogTests.cpp
##
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConsumeWindowsEventLog.h"
+
+#include "core/ConfigurableComponent.h"
+#include "../../../../extensions/standard-processors/processors/LogAttribute.h"
+#include "TestBase.h"
+
+using ConsumeWindowsEventLog =
org::apache::nifi::minifi::processors::ConsumeWindowsEventLog;
+using LogAttribute = org::apache::nifi::minifi::processors::LogAttribute;
+using ConfigurableComponent =
org::apache::nifi::minifi::core::ConfigurableComponent;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+core::Relationship Success{"success", "Everything is fine"};
+
+const std::string APPLICATION_CHANNEL = "Application";
+
+constexpr DWORD CWEL_TESTS_OPCODE = 14985; // random opcode hopefully won't
clash with something important
+
+void reportEvent(const std::string& channel, const char* message, WORD
log_level = EVENTLOG_INFORMATION_TYPE) {
+ auto event_source = RegisterEventSourceA(nullptr, channel.c_str());
+ auto deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, log_level, 0, CWEL_TESTS_OPCODE, nullptr, 1, 0,
, nullptr);
+}
+
+} // namespace
+
+TEST_CASE("ConsumeWindowsEventLog constructor works", "[create]") {
+ TestController test_controller;
+ std::shared_ptr test_plan = test_controller.createPlan();
+
+ REQUIRE_NOTHROW(ConsumeWindowsEventLog processor_one("one"));
+
+ REQUIRE_NOTHROW(
+utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
+ConsumeWindowsEventLog processor_two("two", uuid);
Review comment:
It is slightly different, because the default value of `uuid` is
`utils::Identifier()`, which is blank/all-0s. I wanted to exercise both code
paths, but it's probably not really necessary. Do you want me to remove it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #836: MINIFICPP-1248 Create unit tests for the ConsumeWindowsEventLog processor
fgerlits commented on a change in pull request #836:
URL: https://github.com/apache/nifi-minifi-cpp/pull/836#discussion_r453548537
##
File path: extensions/windows-event-log/tests/BookmarkTests.cpp
##
@@ -0,0 +1,245 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Bookmark.h"
+
+#include
+
+#include "TestBase.h"
+#include "utils/gsl.h"
+#include "wel/UniqueEvtHandle.h"
+
+using Bookmark = org::apache::nifi::minifi::processors::Bookmark;
+using unique_evt_handle = org::apache::nifi::minifi::wel::unique_evt_handle;
+using IdGenerator = org::apache::nifi::minifi::utils::IdGenerator;
+
+namespace {
+
+const std::wstring APPLICATION_CHANNEL = L"Application";
+
+constexpr DWORD BOOKMARK_TESTS_OPCODE = 10368; // random opcode hopefully
won't clash with something important
+
+std::unique_ptr createBookmark(TestPlan _plan,
+ const std::wstring ,
+ const std::string =
IdGenerator::getIdGenerator()->generate().to_string()) {
+ const auto state_manager =
test_plan.getStateManagerProvider()->getCoreComponentStateManager(uuid);
+ const auto logger = test_plan.getLogger();
+ return utils::make_unique(channel, L"*", "", uuid, false,
state_manager, logger);
+}
+
+void reportEvent(const std::wstring& channel, const char* message) {
+ auto event_source = RegisterEventSourceW(nullptr, channel.c_str());
+ auto event_source_deleter = gsl::finally([_source](){
DeregisterEventSource(event_source); });
+ ReportEventA(event_source, EVENTLOG_INFORMATION_TYPE, 0,
+ BOOKMARK_TESTS_OPCODE, nullptr, 1, 0, , nullptr);
+}
+
+std::wstring bookmarkHandleAsXml(EVT_HANDLE event) {
+ REQUIRE(event);
+ constexpr std::size_t BUFFER_SIZE = 1024;
+ std::array buffer = {};
+ DWORD buffer_used;
+ DWORD property_count;
+ if (!EvtRender(nullptr, event, EvtRenderBookmark, buffer.size(),
buffer.data(), _used, _count)) {
+FAIL("Could not render event; error code: " << GetLastError());
+ }
+ return std::wstring{buffer.data()};
+}
+
+std::wstring bookmarkAsXml(const std::unique_ptr& bookmark) {
+ REQUIRE(bookmark);
+ REQUIRE(*bookmark);
+ return bookmarkHandleAsXml(bookmark->getBookmarkHandleFromXML());
+}
+
+unique_evt_handle queryEvents() {
+ std::wstring query = L"Event/System/EventID=" +
std::to_wstring(BOOKMARK_TESTS_OPCODE);
+ unique_evt_handle results{EvtQuery(NULL, APPLICATION_CHANNEL.c_str(),
query.c_str(), EvtQueryChannelPath | EvtQueryReverseDirection)};
+ if (!results) {
+FAIL("EvtQuery() failed; error code: " << GetLastError());
+ }
+ return results;
+}
+
+unique_evt_handle getFirstEventFromResults(const unique_evt_handle& results) {
+ REQUIRE(results);
+ EVT_HANDLE event_raw_handle = 0;
+ DWORD timeout_ms = 100;
+ DWORD num_results_found = 0;
+ bool result = EvtNext(results.get(), 1, _raw_handle, timeout_ms, 0,
_results_found);
Review comment:
we check it in the next line; would you prefer to inline the `EvtNext`
call into `if (result)`? (or maybe rename `result` to `success`?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Wastack commented on a change in pull request #4370: NIFI-6128 UnpackContent: Store unpacked file data
Wastack commented on a change in pull request #4370:
URL: https://github.com/apache/nifi/pull/4370#discussion_r453512358
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
##
@@ -321,6 +337,13 @@ public void process(final InputStream in) throws
IOException {
attributes.put(CoreAttributes.ABSOLUTE_PATH.key(), absPathString);
attributes.put(CoreAttributes.MIME_TYPE.key(),
OCTET_STREAM);
+attributes.put(FILE_PERMISSIONS_ATTRIBUTE,
FileInfo.permissionToString(tarEntry.getMode()));
+attributes.put(FILE_OWNER_ATTRIBUTE,
String.valueOf(tarEntry.getUserName()));
+attributes.put(FILE_GROUP_ATTRIBUTE,
String.valueOf(tarEntry.getGroupName()));
+
+final String timeAsString =
DATE_TIME_FORMATTER.format(tarEntry.getModTime().toInstant());
+
attributes.put(FILE_LAST_MODIFIED_TIME_ATTRIBUTE, timeAsString);
Review comment:
Let's modify file.creationTime to the same as lastModifiedTime in order
to avoid (minimize) confusion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
adamdebreceni commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453479898
##
File path: libminifi/test/flow-tests/FlowControllerTests.cpp
##
@@ -119,7 +107,125 @@ TEST_CASE("Flow shutdown drains connections",
"[TestFlow1]") {
controller->stop(true);
+ REQUIRE(sinkProc->trigger_count == 0);
+
for (auto& it : connectionMap) {
REQUIRE(it.second->isEmpty());
}
}
+
+TEST_CASE("Flow shutdown waits for a while", "[TestFlow2]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ // before we could initiate the shutdown
+ sinkProc->yield(100);
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 0);
+
+ controller->stop(true);
+
+ REQUIRE(sourceProc->trigger_count.load() == 1);
+ REQUIRE(sinkProc->trigger_count.load() == 3);
+}
+
+TEST_CASE("Flow stopped after grace period", "[TestFlow3]") {
+ TestControllerWithFlow testController(yamlConfig);
+ auto controller = testController.controller_;
+ auto root = testController.root_;
+
+
testController.configuration_->set(minifi::Configure::nifi_flowcontroller_drain_timeout,
"1000 ms");
+
+ auto sourceProc =
std::static_pointer_cast(root->findProcessor("Generator"));
+ auto sinkProc =
std::static_pointer_cast(root->findProcessor("TestProcessor"));
+
+ // prevent the initial trigger
+ // in case the source got triggered
+ // and the scheduler triggers the sink
+ sinkProc->yield(100);
+
+ sinkProc->onTriggerCb_ = [&]{
+static std::atomic first_onTrigger{true};
+bool isFirst = true;
+// sleep only on the first trigger
+if (first_onTrigger.compare_exchange_strong(isFirst, false)) {
+ std::this_thread::sleep_for(std::chrono::milliseconds{1500});
+}
+ };
+
+ testController.startFlow();
+
+ // wait for the source processor to enqueue its flowFiles
+ std::this_thread::sleep_for(std::chrono::milliseconds{50});
Review comment:
we are not waiting for the trigger but for the FlowFiles to be enqueued
into the Connection, during shutdown we cannot check if a source processor
stopped, thus we can only check the connections' contents, should we expose the
fact that there are active sessions for a processor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
adamdebreceni commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453476744
##
File path: libminifi/src/FlowController.cpp
##
@@ -239,8 +239,35 @@ int16_t FlowController::stop(bool force, uint64_t
timeToWait) {
if (running_) {
// immediately indicate that we are not running
logger_->log_info("Stop Flow Controller");
-if (this->root_)
+if (this->root_) {
+ // stop source processors first
+ this->root_->stopProcessing(timer_scheduler_, event_scheduler_,
cron_scheduler_, [] (const std::shared_ptr& proc) -> bool {
+return !proc->hasIncomingConnections();
+ });
+ std::chrono::milliseconds shutdown_timer{0};
+ // we enable C2 to progressively increase the timeout
+ // in case it sees that waiting for a little longer could
+ // allow the FlowFiles to be processed
+ auto shutdown_timeout = [&]() -> std::chrono::milliseconds {
+if (timeToWait != 0) {
+ return std::chrono::milliseconds{timeToWait};
+}
+static const core::TimePeriodValue default_timeout{"10 sec"};
Review comment:
done
##
File path: libminifi/src/core/ProcessGroup.cpp
##
@@ -412,6 +414,18 @@ void ProcessGroup::drainConnections() {
}
}
+std::size_t ProcessGroup::getTotalFlowFileCount() const {
+ std::size_t sum = 0;
+ for (auto& conn : connections_) {
+sum += conn->getQueueSize();
+ }
+
+ for (ProcessGroup* childGroup : child_process_groups_) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #833: MINIFICPP-1272 - Graceful shutdown on flow update
adamdebreceni commented on a change in pull request #833:
URL: https://github.com/apache/nifi-minifi-cpp/pull/833#discussion_r453467576
##
File path: libminifi/src/core/ProcessGroup.cpp
##
@@ -412,6 +414,18 @@ void ProcessGroup::drainConnections() {
}
}
+std::size_t ProcessGroup::getTotalFlowFileCount() const {
+ std::size_t sum = 0;
+ for (auto& conn : connections_) {
Review comment:
that would make me think, that the `Connection` is `const` but only the
`std::shared_ptr<...>` would be
if it is clear for everybody else I can make the change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
