[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #791: MINIFICPP-1177 Improvements to the TailFile processor
arpadboda commented on a change in pull request #791:
URL: https://github.com/apache/nifi-minifi-cpp/pull/791#discussion_r438830323
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -40,60 +35,283 @@
#include
#include
+#include "io/CRCStream.h"
#include "utils/file/FileUtils.h"
#include "utils/file/PathUtils.h"
#include "utils/TimeUtil.h"
#include "utils/StringUtils.h"
#include "utils/RegexUtils.h"
-#ifdef HAVE_REGEX_CPP
-#include
-#else
-#include
-#endif
#include "TailFile.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
-#ifndef S_ISDIR
-#define S_ISDIR(mode) (((mode) & S_IFMT) == S_IFDIR)
-#endif
-
-#if defined(__clang__)
-#pragma clang diagnostic push
-#pragma clang diagnostic ignored "-Wsign-compare"
-#elif defined(__GNUC__) || defined(__GNUG__)
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wsign-compare"
-#endif
-
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace processors {
-core::Property TailFile::FileName("File to Tail", "Fully-qualified filename of
the file that should be tailed when using single file mode, or a file regex
when using multifile mode", "");
-core::Property TailFile::StateFile("State File", "DEPRECATED. Only use it for
state migration from the legacy state file.",
- "TailFileState");
-core::Property TailFile::Delimiter("Input Delimiter", "Specifies the character
that should be used for delimiting the data being tailed"
- "from the incoming file."
- "If none is specified, data will be
ingested as it becomes available.",
- "");
+core::Property TailFile::FileName(
+core::PropertyBuilder::createProperty("File to Tail")
+->withDescription("Fully-qualified filename of the file that should be
tailed when using single file mode, or a file regex when using multifile mode")
+->isRequired(true)
+->build());
+
+core::Property TailFile::StateFile(
+core::PropertyBuilder::createProperty("State File")
+->withDescription("DEPRECATED. Only use it for state migration from
the legacy state file.")
+->isRequired(false)
+->withDefaultValue("TailFileState")
+->build());
+
+core::Property TailFile::Delimiter(
+core::PropertyBuilder::createProperty("Input Delimiter")
+->withDescription("Specifies the character that should be used for
delimiting the data being tailed"
+ "from the incoming file. If none is specified, data will be ingested
as it becomes available.")
+->isRequired(false)
+->withDefaultValue("\\n")
+->build());
core::Property TailFile::TailMode(
-core::PropertyBuilder::createProperty("tail-mode", "Tailing
Mode")->withDescription(
-"Specifies the tail file mode. In 'Single file' mode only a single
file will be watched. "
+core::PropertyBuilder::createProperty("tail-mode", "Tailing Mode")
+->withDescription("Specifies the tail file mode. In 'Single file' mode
only a single file will be watched. "
"In 'Multiple file' mode a regex may be used. Note that in multiple
file mode we will still continue to watch for rollover on the initial set of
watched files. "
"The Regex used to locate multiple files will be run during the
schedule phrase. Note that if rotated files are matched by the regex, those
files will be tailed.")->isRequired(true)
-->withAllowableValue("Single
file")->withAllowableValue("Multiple file")->withDefaultValue("Single
file")->build());
-
-core::Property
TailFile::BaseDirectory(core::PropertyBuilder::createProperty("tail-base-directory",
"Base Directory")->isRequired(false)->build());
+->withAllowableValue("Single
file")->withAllowableValue("Multiple file")->withDefaultValue("Single file")
+->build());
+
+core::Property TailFile::BaseDirectory(
+core::PropertyBuilder::createProperty("tail-base-directory", "Base
Directory")
+->withDescription("Base directory used to look for files to tail. This
property is required when using Multiple file mode.")
+->isRequired(false)
+->build());
+
+core::Property TailFile::RecursiveLookup(
+core::PropertyBuilder::createProperty("Recursive lookup")
+->withDescription("When using Multiple file mode, this property
determines whether files are tailed in "
+"child directories of the Base Directory or not.")
+->isRequired(false)
+->withDefaultValue(false)
+->build());
+
+core::Property TailFile::LookupFrequency(
+core::PropertyBuilder::createProperty("Lookup frequency")
+->withDescription("When using Multiple file mode, this property
specifies the minimum duration "
+"the processor will wait between looking for new files to tail in the
Base Directory.")
+->isRequired(false)
+
[jira] [Commented] (NIFI-7524) Add retry relationship to ExecuteSQL
[ https://issues.apache.org/jira/browse/NIFI-7524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133733#comment-17133733 ] Mark Payne commented on NIFI-7524: -- I would recommend a name like "comms.failure" or something of that nature, that describes the failure condition. A name such as "retry" is too prescriptive - it leaves the decision of what to do in such a case in the hands of a flow designer but at the same time names the relationship in such a way that presumes it knows how the condition should be handled. > Add retry relationship to ExecuteSQL > > > Key: NIFI-7524 > URL: https://issues.apache.org/jira/browse/NIFI-7524 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Priority: Major > > Currently ExecuteSQL has success and failure relationships, where all > errors/failures are routed to failure. However for transient exceptions such > as "Connection refused", it would be better to have a retry relationship to > indicate that the flow file could be processed successfully in the future, > rather than invalid SQL errors, e.g. The retry relationship exists in other > RDBMS processors such as PutSQL and PutDatabaseRecord, and would be helpful > for ExecuteSQL as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] wanghongqi commented on pull request #4326: NIFI-7519:Add a ExecuteSQL property on DBCPConnectionPool
wanghongqi commented on pull request #4326: URL: https://github.com/apache/nifi/pull/4326#issuecomment-642940050 I retry the unit tests and not error! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] bbende commented on a change in pull request #4329: Nifi 7442 Add CLI commands to the registry for handling users, groups and policies
bbende commented on a change in pull request #4329: URL: https://github.com/apache/nifi/pull/4329#discussion_r439063970 ## File path: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ## @@ -106,6 +106,12 @@ The following are available commands: registry import-flow-version registry sync-flow-versions registry transfer-flow-version + registry create-user + registry update-user + registry create-user-group + registry update-user-group + registry create-policy Review comment: Lets update this to be 'get-policy' since we got rid of create and folded into update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-7524) Add retry relationship to ExecuteSQL
Matt Burgess created NIFI-7524: -- Summary: Add retry relationship to ExecuteSQL Key: NIFI-7524 URL: https://issues.apache.org/jira/browse/NIFI-7524 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Currently ExecuteSQL has success and failure relationships, where all errors/failures are routed to failure. However for transient exceptions such as "Connection refused", it would be better to have a retry relationship to indicate that the flow file could be processed successfully in the future, rather than invalid SQL errors, e.g. The retry relationship exists in other RDBMS processors such as PutSQL and PutDatabaseRecord, and would be helpful for ExecuteSQL as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-3792) A processor to facilitate retrying FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-3792: --- Status: Patch Available (was: Open) > A processor to facilitate retrying FlowFiles > > > Key: NIFI-3792 > URL: https://issues.apache.org/jira/browse/NIFI-3792 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Joe Percivall >Assignee: Joe Percivall >Priority: Major > > When dealing with processor failures in production, potentially related to > network issues, the current best practice is to retry the flowfile a couple > times before declaring it a failure[1]. This currently requires multiple > processors and penalizing the flowfile isn't possible. Also if the flow is > not fast enough, back-pressure can cause a livelocked state which requires > manual intervention. > A new processor should be added to not only encourage best practices but also > offer configuration options to deal with un-optimal situations. > [1] > https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #791: MINIFICPP-1177 Improvements to the TailFile processor
szaszm commented on a change in pull request #791:
URL: https://github.com/apache/nifi-minifi-cpp/pull/791#discussion_r439020454
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -40,60 +35,283 @@
#include
#include
+#include "io/CRCStream.h"
#include "utils/file/FileUtils.h"
#include "utils/file/PathUtils.h"
#include "utils/TimeUtil.h"
#include "utils/StringUtils.h"
#include "utils/RegexUtils.h"
-#ifdef HAVE_REGEX_CPP
-#include
-#else
-#include
-#endif
#include "TailFile.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
-#ifndef S_ISDIR
-#define S_ISDIR(mode) (((mode) & S_IFMT) == S_IFDIR)
-#endif
-
-#if defined(__clang__)
-#pragma clang diagnostic push
-#pragma clang diagnostic ignored "-Wsign-compare"
-#elif defined(__GNUC__) || defined(__GNUG__)
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wsign-compare"
-#endif
-
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace processors {
-core::Property TailFile::FileName("File to Tail", "Fully-qualified filename of
the file that should be tailed when using single file mode, or a file regex
when using multifile mode", "");
-core::Property TailFile::StateFile("State File", "DEPRECATED. Only use it for
state migration from the legacy state file.",
- "TailFileState");
-core::Property TailFile::Delimiter("Input Delimiter", "Specifies the character
that should be used for delimiting the data being tailed"
- "from the incoming file."
- "If none is specified, data will be
ingested as it becomes available.",
- "");
+core::Property TailFile::FileName(
+core::PropertyBuilder::createProperty("File to Tail")
+->withDescription("Fully-qualified filename of the file that should be
tailed when using single file mode, or a file regex when using multifile mode")
+->isRequired(true)
+->build());
+
+core::Property TailFile::StateFile(
+core::PropertyBuilder::createProperty("State File")
+->withDescription("DEPRECATED. Only use it for state migration from
the legacy state file.")
+->isRequired(false)
+->withDefaultValue("TailFileState")
+->build());
+
+core::Property TailFile::Delimiter(
+core::PropertyBuilder::createProperty("Input Delimiter")
+->withDescription("Specifies the character that should be used for
delimiting the data being tailed"
+ "from the incoming file. If none is specified, data will be ingested
as it becomes available.")
+->isRequired(false)
+->withDefaultValue("\\n")
+->build());
core::Property TailFile::TailMode(
-core::PropertyBuilder::createProperty("tail-mode", "Tailing
Mode")->withDescription(
-"Specifies the tail file mode. In 'Single file' mode only a single
file will be watched. "
+core::PropertyBuilder::createProperty("tail-mode", "Tailing Mode")
+->withDescription("Specifies the tail file mode. In 'Single file' mode
only a single file will be watched. "
"In 'Multiple file' mode a regex may be used. Note that in multiple
file mode we will still continue to watch for rollover on the initial set of
watched files. "
"The Regex used to locate multiple files will be run during the
schedule phrase. Note that if rotated files are matched by the regex, those
files will be tailed.")->isRequired(true)
-->withAllowableValue("Single
file")->withAllowableValue("Multiple file")->withDefaultValue("Single
file")->build());
-
-core::Property
TailFile::BaseDirectory(core::PropertyBuilder::createProperty("tail-base-directory",
"Base Directory")->isRequired(false)->build());
+->withAllowableValue("Single
file")->withAllowableValue("Multiple file")->withDefaultValue("Single file")
+->build());
+
+core::Property TailFile::BaseDirectory(
+core::PropertyBuilder::createProperty("tail-base-directory", "Base
Directory")
+->withDescription("Base directory used to look for files to tail. This
property is required when using Multiple file mode.")
+->isRequired(false)
+->build());
+
+core::Property TailFile::RecursiveLookup(
+core::PropertyBuilder::createProperty("Recursive lookup")
+->withDescription("When using Multiple file mode, this property
determines whether files are tailed in "
+"child directories of the Base Directory or not.")
+->isRequired(false)
+->withDefaultValue(false)
+->build());
+
+core::Property TailFile::LookupFrequency(
+core::PropertyBuilder::createProperty("Lookup frequency")
+->withDescription("When using Multiple file mode, this property
specifies the minimum duration "
+"the processor will wait between looking for new files to tail in the
Base Directory.")
+->isRequired(false)
+
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #791: MINIFICPP-1177 Improvements to the TailFile processor
szaszm commented on a change in pull request #791:
URL: https://github.com/apache/nifi-minifi-cpp/pull/791#discussion_r438997857
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -204,52 +418,78 @@ void TailFile::parseStateFileLine(char *buf) {
}
std::string value = equal;
- key = trimRight(key);
- value = trimRight(value);
+ key = utils::StringUtils::trimRight(key);
+ value = utils::StringUtils::trimRight(value);
if (key == "FILENAME") {
std::string fileLocation, fileName;
if (utils::file::PathUtils::getFileNameAndPath(value, fileLocation,
fileName)) {
logger_->log_debug("State migration received path %s, file %s",
fileLocation, fileName);
- tail_states_.insert(std::make_pair(fileName, TailState { fileLocation,
fileName, 0, 0 }));
+ state.emplace(fileName, TailState{fileLocation, fileName});
} else {
- tail_states_.insert(std::make_pair(value, TailState { fileLocation,
value, 0, 0 }));
+ state.emplace(value, TailState{fileLocation, value});
}
}
if (key == "POSITION") {
// for backwards compatibility
-if (tail_states_.size() != 1) {
+if (tail_states_.size() != std::size_t{1}) {
throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION,
"Incompatible state file types");
}
const auto position = std::stoull(value);
logger_->log_debug("Received position %d", position);
Review comment:
Could you change fix this logging line to use the `llu` as the format
specifier, since `position` is `unsigned long long` here?
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -260,72 +500,60 @@ bool TailFile::recoverState(const
std::shared_ptr& context
try {
const std::string& current = state_map.at("file." + std::to_string(i)
+ ".current");
uint64_t position = std::stoull(state_map.at("file." +
std::to_string(i) + ".position"));
+uint64_t checksum = readOptionalUint64(state_map, "file." +
std::to_string(i) + ".checksum");
std::string fileLocation, fileName;
if (utils::file::PathUtils::getFileNameAndPath(current, fileLocation,
fileName)) {
logger_->log_debug("Received path %s, file %s", fileLocation,
fileName);
- new_tail_states.emplace(fileName, TailState { fileLocation,
fileName, position, 0 });
+ new_tail_states.emplace(current, TailState{fileLocation, fileName,
position, std::chrono::system_clock::time_point{}, checksum});
} else {
- new_tail_states.emplace(current, TailState { fileLocation, current,
position, 0 });
+ new_tail_states.emplace(current, TailState{fileLocation, current,
position, std::chrono::system_clock::time_point{}, checksum});
}
} catch (...) {
continue;
}
}
-state_load_success = true;
-tail_states_ = std::move(new_tail_states);
for (const auto& s : tail_states_) {
- logger_->log_debug("TailState %s: %s, %s, %llu, %llu", s.first,
s.second.path_, s.second.current_file_name_, s.second.currentTailFilePosition_,
s.second.currentTailFileModificationTime_);
+ logger_->log_debug("TailState %s: %s, %s, %llu, %llu",
Review comment:
Could you fix the format string to use `PRIu64` as the format specifier
for `uint64_t`? It's not guaranteed to be the same as `unsigned long long`,
it's actually `unsigned long` on my machine. `PRIu64` is declared in
``.
##
File path: extensions/standard-processors/tests/unit/TailFileTests.cpp
##
@@ -652,46 +872,606 @@ TEST_CASE("TailFileWithMultileRolledOverFiles",
"[tailfiletest2]") {
char format[] = "/tmp/gt.XX";
auto dir = testController.createTempDirectory(format);
- // Define test input file
- std::string in_file(dir);
- in_file.append("fruits.txt");
+ std::string test_file = dir + utils::file::FileUtils::get_separator() +
"fruits.log";
- for (int i = 2; 0 <= i; --i) {
-if (i < 2) {
- std::this_thread::sleep_for(std::chrono::milliseconds(1000)); // make
sure the new file gets newer modification time
-}
-std::ofstream in_file_stream(in_file + (i > 0 ? std::to_string(i) : ""));
-for (int j = 0; j <= i; j++) {
- in_file_stream << "Apple" << DELIM;
-}
-in_file_stream.close();
- }
+ std::ofstream test_file_stream_0(test_file, std::ios::binary);
+ test_file_stream_0 << "Apple" << DELIM << "Orange" << DELIM;
+ test_file_stream_0.flush();
// Build MiNiFi processing graph
auto tail_file = plan->addProcessor("TailFile", "Tail");
plan->setProperty(tail_file, processors::TailFile::Delimiter.getName(),
std::string(1, DELIM));
- plan->setProperty(tail_file, processors::TailFile::FileName.getName(),
in_file);
+ plan->setProperty(tail_file, processors::TailFile::FileName.getName(),
test_file);
auto log_attr = plan->addProcessor("LogAttribute", "Log",
core::Relationship("success", "description"), true);
[jira] [Assigned] (NIFI-4760) TestListenSMTP tests fail on OSX
[ https://issues.apache.org/jira/browse/NIFI-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee reassigned NIFI-4760: - Assignee: Joey Frazee > TestListenSMTP tests fail on OSX > > > Key: NIFI-4760 > URL: https://issues.apache.org/jira/browse/NIFI-4760 > Project: Apache NiFi > Issue Type: Test >Affects Versions: 1.5.0 >Reporter: Jeffrey T. Zemerick >Assignee: Joey Frazee >Priority: Minor > > When building 1.5.0 on OSX High Sierra with the following command: > mvn clean install -Pcontrib-check > (At first I thought it was because I was doing -T 1C but it fails without > parallel, too.) > Some tests in TestListenSMTP reliably fail: > Results : > Failed tests: > TestListenSMTP.validateSuccessfulInteraction:88 > TestListenSMTP.validateSuccessfulInteractionWithTls:150 > TestListenSMTP.validateTooLargeMessage:192 > The relevant test output: > validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP) > Time elapsed: 15.069 sec <<< FAILURE! > java.lang.AssertionError > at > org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteraction(TestListenSMTP.java:88) > validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP) > Time elapsed: 16.514 sec <<< FAILURE! > java.lang.AssertionError > at > org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteractionWithTls(TestListenSMTP.java:150) > validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP) > Time elapsed: 21.516 sec <<< FAILURE! > java.lang.AssertionError > at > org.apache.nifi.processors.email.TestListenSMTP.validateTooLargeMessage(TestListenSMTP.java:192) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on pull request #791: MINIFICPP-1177 Improvements to the TailFile processor
arpadboda commented on pull request #791: URL: https://github.com/apache/nifi-minifi-cpp/pull/791#issuecomment-642724988 @fgerlits : one of the tests failed in CI: https://travis-ci.org/github/apache/nifi-minifi-cpp/jobs/697214111 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #791: MINIFICPP-1177 Improvements to the TailFile processor
arpadboda commented on a change in pull request #791:
URL: https://github.com/apache/nifi-minifi-cpp/pull/791#discussion_r438834296
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -204,52 +418,78 @@ void TailFile::parseStateFileLine(char *buf) {
}
std::string value = equal;
- key = trimRight(key);
- value = trimRight(value);
+ key = utils::StringUtils::trimRight(key);
+ value = utils::StringUtils::trimRight(value);
if (key == "FILENAME") {
std::string fileLocation, fileName;
if (utils::file::PathUtils::getFileNameAndPath(value, fileLocation,
fileName)) {
logger_->log_debug("State migration received path %s, file %s",
fileLocation, fileName);
- tail_states_.insert(std::make_pair(fileName, TailState { fileLocation,
fileName, 0, 0 }));
+ state.emplace(fileName, TailState{fileLocation, fileName});
} else {
- tail_states_.insert(std::make_pair(value, TailState { fileLocation,
value, 0, 0 }));
+ state.emplace(value, TailState{fileLocation, value});
}
}
if (key == "POSITION") {
// for backwards compatibility
-if (tail_states_.size() != 1) {
+if (tail_states_.size() != std::size_t{1}) {
throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION,
"Incompatible state file types");
}
const auto position = std::stoull(value);
logger_->log_debug("Received position %d", position);
-tail_states_.begin()->second.currentTailFilePosition_ = position;
+state.begin()->second.position_ = position;
}
if (key.find(CURRENT_STR) == 0) {
const auto file = key.substr(strlen(CURRENT_STR));
std::string fileLocation, fileName;
if (utils::file::PathUtils::getFileNameAndPath(value, fileLocation,
fileName)) {
- tail_states_[file].path_ = fileLocation;
- tail_states_[file].current_file_name_ = fileName;
+ state[file].path_ = fileLocation;
+ state[file].file_name_ = fileName;
} else {
throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION, "State file
contains an invalid file name");
}
}
if (key.find(POSITION_STR) == 0) {
const auto file = key.substr(strlen(POSITION_STR));
-tail_states_[file].currentTailFilePosition_ = std::stoull(value);
+state[file].position_ = std::stoull(value);
}
}
+bool TailFile::recoverState(const std::shared_ptr&
context) {
+ std::map new_tail_states;
+ bool state_load_success = getStateFromStateManager(new_tail_states) ||
+getStateFromLegacyStateFile(context,
new_tail_states);
+ if (!state_load_success) {
+return false;
+ }
+ logger_->log_debug("load state succeeded");
-bool TailFile::recoverState(const std::shared_ptr&
context) {
- bool state_load_success = false;
+ if (tail_mode_ == Mode::SINGLE) {
+if (tail_states_.size() == 1) {
+ auto state_it = tail_states_.begin();
+ const auto it = new_tail_states.find(state_it->first);
+ if (it != new_tail_states.end()) {
+state_it->second = it->second;
+ }
+} else {
+ throw minifi::Exception(ExceptionType::PROCESSOR_EXCEPTION, "This should
never happen: "
Review comment:
Well, shouldn't but I think it could.
Not sure if you handle this anywhere, but we should be prepared for config
changes.
In case any of the significant properties of the processor changes, we
should simply ignore the previously saved state. I mean mode, path, rotation
pattern, delimiter, etc.
This means that we should save the current value of these properties to
state and compare with the current config when loading the state.
If you already handle it at some point in the code, please ignore my
overelaborated comment.
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -17,19 +17,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include
#include
-#include
-#include
-
-#include
#ifndef WIN32
#include
Review comment:
Do we still need this ifdef here?
Guess these logics are moved to fileutils now.
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -40,60 +35,283 @@
#include
#include
+#include "io/CRCStream.h"
#include "utils/file/FileUtils.h"
#include "utils/file/PathUtils.h"
#include "utils/TimeUtil.h"
#include "utils/StringUtils.h"
#include "utils/RegexUtils.h"
-#ifdef HAVE_REGEX_CPP
-#include
-#else
-#include
-#endif
#include "TailFile.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
-#ifndef S_ISDIR
-#define S_ISDIR(mode) (((mode) & S_IFMT) == S_IFDIR)
-#endif
-
-#if defined(__clang__)
-#pragma clang diagnostic push
-#pragma clang diagnostic ignored "-Wsign-compare"
-#elif defined(__GNUC__) || defined(__GNUG__)
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wsign-compare"
-#endif
-
namespace org {
[jira] [Created] (NIFI-7523) Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task
Peter Turcsanyi created NIFI-7523: - Summary: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task Key: NIFI-7523 URL: https://issues.apache.org/jira/browse/NIFI-7523 Project: Apache NiFi Issue Type: Improvement Reporter: Peter Turcsanyi Assignee: Peter Turcsanyi SSL Context Service property is currently used only for the Kafka connection. Use the truststore from SSL Context Service when connecting to Atlas via HTTPS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] patricker commented on pull request #3511: NIFI-6175 Spark Livy - Improving Livy
patricker commented on pull request #3511: URL: https://github.com/apache/nifi/pull/3511#issuecomment-642694758 @naru014 Yes, it can. It also includes Batch processing controller service/processor. The one that comes with NiFi currently has severe limitations. I keep forgetting to get this one merged. I just need to update it for the latest version of NiFI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy edited a comment on pull request #4314: NIFI-7442 Add CLI commands to the registry in order to support automa…
tpalfy edited a comment on pull request #4314: URL: https://github.com/apache/nifi/pull/4314#issuecomment-642671399 I was reviewing this PR and had some additional updates. I will submit a new PR with Bence’s work + mine (Bence's commit is preserved in the new PR). https://github.com/apache/nifi/pull/4329 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy opened a new pull request #4329: Nifi 7442 Add CLI commands to the registry for handling users, groups and policies
tpalfy opened a new pull request #4329: URL: https://github.com/apache/nifi/pull/4329 https://issues.apache.org/jira/browse/NIFI-7442 New commands were added to the CLI to be able to create/update users, user groups and access policies in NiFi Registry. As NiFi depends on released Registry, some code temporarily takes place in NiFi which should be moved to the Registry in the long term. Also, this comes with some code duplication as well (which will be removed in a later point too). ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on pull request #4314: NIFI-7442 Add CLI commands to the registry in order to support automa…
tpalfy commented on pull request #4314: URL: https://github.com/apache/nifi/pull/4314#issuecomment-642671399 I was reviewing this PR and had some additional updates. I will submit a new PR with Bence’s work + mine (Bence's commit is preserved in the new PR). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFIREG-398) Add support for FlowFile Concurrency and Outbound Policy to VersionedProcessGroup
[ https://issues.apache.org/jira/browse/NIFIREG-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFIREG-398. - Fix Version/s: 1.0.0 Resolution: Fixed > Add support for FlowFile Concurrency and Outbound Policy to > VersionedProcessGroup > - > > Key: NIFIREG-398 > URL: https://issues.apache.org/jira/browse/NIFIREG-398 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > NiFi's Process Groups can now be configured with a FlowFile Concurrency and > an Outbound Policy setting, since NIFI-7476. VersionedProcessGroup needs to > be updated to allow for these settings also so that it can be properly > persisted to the registry. > The Flow Comparator also needs to take these into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFICPP-1153) Nanofi CTailFile tests use the same input file
[ https://issues.apache.org/jira/browse/MINIFICPP-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Gerlits reassigned MINIFICPP-1153: - Assignee: Ferenc Gerlits (was: Arpad Boda) > Nanofi CTailFile tests use the same input file > -- > > Key: MINIFICPP-1153 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1153 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.7.0 >Reporter: Arpad Boda >Assignee: Ferenc Gerlits >Priority: Major > Fix For: 0.8.0 > > Time Spent: 40m > Remaining Estimate: 0h > > All these testcases use a file named "e.txt" in cwd, which often results in > failures. > Should use properly generated tempfiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-6163) Reporting task can be set to RUNNING from REST API when invalid
[
https://issues.apache.org/jira/browse/NIFI-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Makarov Vasiliy Nicolaevich reassigned NIFI-6163:
-
Assignee: Makarov Vasiliy Nicolaevich
> Reporting task can be set to RUNNING from REST API when invalid
> ---
>
> Key: NIFI-6163
> URL: https://issues.apache.org/jira/browse/NIFI-6163
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.9.1
>Reporter: Bryan Bende
>Assignee: Makarov Vasiliy Nicolaevich
>Priority: Minor
>
> Using the REST API you can submit a request to start a reporting task that
> has validation errors using a curl command like this:
> {code:java}
> curl -X PUT -H Content-Type:application/json
> "http://localhost:8080/nifi-api/reporting-tasks/ca0808a7-0169-1000-9546-62009238a9da/run-status;
> -d
> '{"revision":{"clientId":"ca05c584-0169-1000-497d-4c35dd4f2a58","version":7},"disconnectedNodeAcknowledged":false,"state":"RUNNING"}'{code}
> This call succeeds and if you refresh the UI it will show as running, even
> though it should never have been allowed to be running since it was invalid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7057) Fix Script Permissions For Docker Integration Test
[ https://issues.apache.org/jira/browse/NIFI-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133128#comment-17133128 ] Makarov Vasiliy Nicolaevich commented on NIFI-7057: --- I have checked these files out - they all are with correct 775 permission status. So I think issue is already resolved. [~Absolutesantaja] One question is should we move these permissions in Dockerfile like in NIFI-7080 ? > Fix Script Permissions For Docker Integration Test > -- > > Key: NIFI-7057 > URL: https://issues.apache.org/jira/browse/NIFI-7057 > Project: Apache NiFi > Issue Type: Bug >Reporter: Shawn Weeks >Assignee: Makarov Vasiliy Nicolaevich >Priority: Trivial > > The following files need to have their permissions updated for the Docker > profile to run it's tests. I'll add more files if I find them. > chmod 755 ./nifi-toolkit-assembly/docker/tests/exit-codes.sh > chmod 755 ./nifi-toolkit/nifi-toolkit-assembly/docker/tests/tls-toolkit.sh > chmod 755 ./nifi-docker/dockermaven/integration-test.sh > chmod 755 ./nifi-docker/dockermaven/sh/*.sh > chmod 755 ./nifi-docker/dockerhub/sh/*.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] feitgraph commented on pull request #4328: NIFI-7035 The first curator connection issue is logged as ERROR until…
feitgraph commented on pull request #4328: URL: https://github.com/apache/nifi/pull/4328#issuecomment-642532453 To repeat the issue I managed to change nifi.properties so that nifi.zookeeper.connect.string contained one or more hosts that are not up. After which nifi-app.log was spammed each 1ms with these sorts of exceptions and stack traces (which are not that helpful). So the curator lib has out of the box the switch to log only the first problem with the zookeeper connection, and other logs will be in DEBUG level. See [https://jira.apache.org/jira/browse/NIFI-7035](url) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] feitgraph commented on pull request #4328: NIFI-7035 The first curator connection issue is logged as ERROR until…
feitgraph commented on pull request #4328: URL: https://github.com/apache/nifi/pull/4328#issuecomment-642532773 Problem with the mac workflow it seems not related to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #805: MINIFICPP-1252 Add gsl-lite
arpadboda closed pull request #805: URL: https://github.com/apache/nifi-minifi-cpp/pull/805 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7522) Processor Kafka 2.5 / Static Membership
[ https://issues.apache.org/jira/browse/NIFI-7522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Firenz updated NIFI-7522: - Description: Kafka 2.4 introduced the static membership [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. I don't know the versionning strategy for Kafka Processors. We have 2 choices : * Thanks to dynamic properties, upgrading the 2.0 processor with 2.5 libs (confusing but lazy update) * Create a new 2.5 process set So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, test ok. But i need to learn the full process to create the branch/pull/merge request etc. For people in the urge or just curious, my last build is attached. was: Kafka 2.4 introduced the static membership [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. I don't know the versionning strategy for Kafka Processors. There is no difference between the 2.0 and 2.5 processor (no new fields, the group.instance.id is added with dynamic properties). So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, test ok. But i need to learn the full process to create the branch/pull/merge request etc. For people in the urge or just curious, my last build is attached. > Processor Kafka 2.5 / Static Membership > --- > > Key: NIFI-7522 > URL: https://issues.apache.org/jira/browse/NIFI-7522 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Firenz >Priority: Minor > Attachments: nifi-kafka-2-5-nar-1.11.5-SNAPSHOT.nar > > > Kafka 2.4 introduced the static membership > [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] > > So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. > > I don't know the versionning strategy for Kafka Processors. We have 2 choices > : > * Thanks to dynamic properties, upgrading the 2.0 processor with 2.5 libs > (confusing but lazy update) > * Create a new 2.5 process set > > So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, > test ok. But i need to learn the full process to create the branch/pull/merge > request etc. For people in the urge or just curious, my last build is > attached. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7522) Processor Kafka 2.5 / Static Membership
[ https://issues.apache.org/jira/browse/NIFI-7522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Firenz updated NIFI-7522: - Description: Kafka 2.4 introduced the static membership [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. I don't know the versionning strategy for Kafka Processors. We have 2 choices : * Thanks to dynamic properties, upgrading the 2.0 processors set with 2.5 kafka libs (confusing but lazy update) * Create a new 2.5 processors set So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, test ok. But i need to learn the full process to create the branch/pull/merge request etc. For people in the urge or just curious, my last build is attached. was: Kafka 2.4 introduced the static membership [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. I don't know the versionning strategy for Kafka Processors. We have 2 choices : * Thanks to dynamic properties, upgrading the 2.0 processor with 2.5 libs (confusing but lazy update) * Create a new 2.5 process set So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, test ok. But i need to learn the full process to create the branch/pull/merge request etc. For people in the urge or just curious, my last build is attached. > Processor Kafka 2.5 / Static Membership > --- > > Key: NIFI-7522 > URL: https://issues.apache.org/jira/browse/NIFI-7522 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Firenz >Priority: Minor > Attachments: nifi-kafka-2-5-nar-1.11.5-SNAPSHOT.nar > > > Kafka 2.4 introduced the static membership > [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] > > So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. > > I don't know the versionning strategy for Kafka Processors. We have 2 choices > : > * Thanks to dynamic properties, upgrading the 2.0 processors set with 2.5 > kafka libs (confusing but lazy update) > * Create a new 2.5 processors set > > So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, > test ok. But i need to learn the full process to create the branch/pull/merge > request etc. For people in the urge or just curious, my last build is > attached. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7522) Processor Kafka 2.5 / Static Membership
Firenz created NIFI-7522: Summary: Processor Kafka 2.5 / Static Membership Key: NIFI-7522 URL: https://issues.apache.org/jira/browse/NIFI-7522 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Firenz Attachments: nifi-kafka-2-5-nar-1.11.5-SNAPSHOT.nar Kafka 2.4 introduced the static membership [[KIP-345|https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]] So i propose to add a Publish/Consume 2_5 with the last Kafka Client 2.5. I don't know the versionning strategy for Kafka Processors. There is no difference between the 2.0 and 2.5 processor (no new fields, the group.instance.id is added with dynamic properties). So i created a new set of 2.5 processors from 1.11.5-SNAPSHOT. It builds, test ok. But i need to learn the full process to create the branch/pull/merge request etc. For people in the urge or just curious, my last build is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] feitgraph opened a new pull request #4328: NIFI-7035 The first curator connection issue is logged as ERROR until…
feitgraph opened a new pull request #4328:
URL: https://github.com/apache/nifi/pull/4328
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Helps not to spam error curator logs on the case of lost node, NIFI-7035.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
