[GitHub] [nifi-minifi-cpp] am-c-p-p edited a comment on pull request #757: MINIFICPP-1177 Implementation based on Nifi.
am-c-p-p edited a comment on pull request #757: URL: https://github.com/apache/nifi-minifi-cpp/pull/757#issuecomment-614127977 Implementation is rewritten from scratch is closely based on Nifi Java TailFile https://github.com/apache/nifi/blob/63379c35201cb58d0a5c5a4d55bb72504551973c/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java so would be useful to compare this C++ implementation with the Java one. To do: 1. State is saved in memory for now and will use RocksDB controller service when it is merged to apache. 2. Needs to be added the same test cases as in https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestTailFile.java. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-7399) Change Controller service scope
DEOM Damien created NIFI-7399: - Summary: Change Controller service scope Key: NIFI-7399 URL: https://issues.apache.org/jira/browse/NIFI-7399 Project: Apache NiFi Issue Type: Improvement Reporter: DEOM Damien It would be very convenient to add an option to change a controller service scope -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] bakaid commented on pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on pull request #605: URL: https://github.com/apache/nifi-minifi-cpp/pull/605#issuecomment-619571850 @arpadboda @szaszm Rebased to latest master (including C2 refactor), added test for C2 DESCRIBE corecomponentstate, addressed review comments. I have tested state handling on shutdown and C2 configuration reload with TailFile, it seems to work fine. I am going to continue with final verification of functionality/state migration of stateful processors on different platforms, if those work fine I think this should be ready to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605:
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r415336955
##
File path: libminifi/src/c2/C2Agent.cpp
##
@@ -435,6 +437,27 @@ void C2Agent::handle_c2_server_response(const
C2ContentResponse ) {
update_sink_->drainRepositories();
C2Payload response(Operation::ACKNOWLEDGE, resp.ident, false, true);
enqueue_c2_response(std::move(response));
+ } else if (resp.name == "corecomponentstate") {
Review comment:
This is unfortunately still very much non-trivial to do (unlike testing
DESCRIBE, which I do now), so I've added a TODO for it as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605:
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r415336687
##
File path: extensions/sql/processors/QueryDatabaseTable.cpp
##
@@ -308,19 +240,43 @@ void QueryDatabaseTable::processOnSchedule(const
core::ProcessContext )
context.getProperty(s_sqlQuery.getName(), sqlQuery_);
context.getProperty(s_maxRowsPerFlowFile.getName(), maxRowsPerFlowFile_);
- std::string stateDir;
- context.getProperty(s_stateDirectory.getName(), stateDir);
- if (stateDir.empty()) {
-logger_->log_error("State Directory is empty");
-return;
+ mapState_.clear();
+
+ state_manager_ = context.getStateManager();
+ if (state_manager_ == nullptr) {
+throw Exception(PROCESSOR_EXCEPTION, "Failed to get StateManager");
}
- pState_ = std::make_unique(tableName_, stateDir, getUUIDStr(),
logger_);
- if (!*pState_) {
-return;
+ std::unordered_map state_map;
+ if (state_manager_->get(state_map)) {
+if (state_map[TABLENAME_KEY] != tableName_) {
+ state_manager_->clear();
+} else {
+ for (auto&& elem : state_map) {
+if (elem.first.find(MAXVALUE_KEY_PREFIX) == 0) {
+ mapState_.emplace(elem.first.substr(MAXVALUE_KEY_PREFIX.length()),
std::move(elem.second));
+}
+ }
+}
+ } else {
+// Try to migrate legacy state file
+std::string stateDir;
+context.getProperty(s_stateDirectory.getName(), stateDir);
+if (!stateDir.empty()) {
+ LegacyState legacyState(tableName_, stateDir, getUUIDStr(), logger_);
+ if (legacyState) {
+mapState_ = legacyState.getStateMap();
+if (saveState() && state_manager_->persist()) {
+ logger_->log_info("State migration successful");
+ legacyState.moveStateFileToMigrated();
+} else {
+ logger_->log_warn("Failed to persists migrated state");
+}
+ } else {
+logger_->log_warn("Could not migrate state from specified State
Directory %s", stateDir);
+ }
+}
Review comment:
It is short and very much unlikely to be reused, so I don't see much
benefit in moving to a function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605:
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r415333419
##
File path: libminifi/src/c2/C2Agent.cpp
##
@@ -596,6 +619,29 @@ void C2Agent::handle_describe(const C2ContentResponse
) {
}
enqueue_c2_response(std::move(response));
}
+ } else if (resp.name == "corecomponentstate") {
Review comment:
Added `C2DescribeCoreComponentStateTest` for this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] msharee9 opened a new pull request #771: MINIFICPP-1183 Cleanup C2 tests
msharee9 opened a new pull request #771: URL: https://github.com/apache/nifi-minifi-cpp/pull/771 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
