[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj no worries, I'll get this merged. Thanks ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @phrocker That was the last one, I promise. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj We also haven't explored disabling optimizations to speed up catch builds or using CATCH_CONFIG_FAST_COMPILE ( which is supposed to help by 20% ) for travis builds. On my machine with some port changes across configs test runs take about 60seconds versus a few minutes. For my latest branch ( https://github.com/apache/nifi-minifi-cpp/compare/master...phrocker:MINIFICPP-60?expand=1 ) it helped tremendously. I'll get your PR merged in a bit. You are free to squash or fixup...otherwise I'll simply squash the commits into a single one and merge. Thanks! ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj It's not archive that is the problem. It's the usage of uint64_t over long. * ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 The build fails both with or without the `struct` keyword; I'll try a different approach next. There are also some other things I forgot to add when moving to the extension. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @phrocker I already have a travis-ci.org account, but I've granted access to @NifiLocal and synced the account. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj I emailed Travis their response: "" We've been experiencing some issues with our abuse system creating false positives lately, and I suspect that's what's happening here. I took a look in our system, and it looks like the organization NiFiLocal, where the pull request is coming from, doesn't exist within Travis, which is causing our system to flag the pull request as suspicious. If you can have the creator of the pull request log into travis-ci.org and sync their Github account, I should be able to go into our system and flag the NiFiLocal as not abusive, which will allow the pull request to build normally. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @phrocker Cool. I can squash into one commit and force push, if needed. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj thanks for the fix. Something in the way this PR is submitted is causing travis not to run as as result I"ll go through re-verification on all platforms and merge once I have an opportunity to do that. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj Ignore the boost removal, that was unrelated. This processor will be moved to an EXCLUDE_BOOST so that those who wish to exclude boost will not see this and other libarchive processors. So you can keep that header. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 I'm changing the types for size, mtime and mtime_nsec in `ArchiveEntryMetadata`'s declaration instead of casting. Why'd you take out the boost header include? ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue:
https://github.com/apache/nifi-minifi-cpp/pull/148
diff --git a/libminifi/src/processors/FocusArchiveEntry.cpp
b/libminifi/src/processors/FocusArchiveEntry.cpp
index c262217..42962a6 100644
--- a/libminifi/src/processors/FocusArchiveEntry.cpp
+++ b/libminifi/src/processors/FocusArchiveEntry.cpp
@@ -24,8 +24,6 @@
#include
-#include
-
#include
#include
@@ -154,11 +152,11 @@ void
FocusArchiveEntry::onTrigger(core::ProcessContext *context,
entryVal["entry_name"] = Json::Value(entryMetadata.entryName);
entryVal["entry_type"] = Json::Value(entryMetadata.entryType);
entryVal["entry_perm"] = Json::Value(entryMetadata.entryPerm);
- entryVal["entry_size"] = Json::Value(entryMetadata.entrySize);
+ entryVal["entry_size"] =
Json::Value((uint64_t)entryMetadata.entrySize);
entryVal["entry_uid"] = Json::Value(entryMetadata.entryUID);
entryVal["entry_gid"] = Json::Value(entryMetadata.entryGID);
- entryVal["entry_mtime"] = Json::Value(entryMetadata.entryMTime);
- entryVal["entry_mtime_nsec"] =
Json::Value(entryMetadata.entryMTimeNsec);
+ entryVal["entry_mtime"] =
Json::Value((uint64_t)entryMetadata.entryMTime);
+ entryVal["entry_mtime_nsec"] =
Json::Value((int64_t)entryMetadata.entryMTimeNsec);
if (entryMetadata.entryType == AE_IFREG) {
entryVal["stash_key"] = Json::Value(entryMetadata.stashKey);
@@ -221,7 +219,7 @@ int64_t
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
archive_read_support_filter_all(inputArchive);
// Read callback which reads from ifstream
- auto read = [] (archive * a, void *d, const void **buf) -> int64_t {
+ auto read = [] (struct archive * a, void *d, const void **buf) -> long {
auto data = static_cast(d);
*buf = data->buf;
int64_t read = 0;
@@ -241,7 +239,7 @@ int64_t
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
};
// Close callback for libarchive
- auto close = [] (archive *, void *) -> int {
+ auto close = [] (struct archive *, void *) -> int {
// Because we do not need to close the stream, do nothing & return
success
return 0;
};
@@ -249,7 +247,7 @@ int64_t
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
// Read each item in the archive
int res;
- if ((res = archive_read_open(inputArchive, , NULL, read, close))) {
+ if ((res = archive_read_open(inputArchive, , nullptr, read,
close))) {
logger_->log_error(
"FocusArchiveEntry can't open due to archive error: %s",
archive_error_string(inputArchive));
---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj These failures are isolated to OSX. I noticed that travis isn't building your PRs to show you the build failure. I'll take a look to see why you aren't getting travis runs. My guess is that you don't have a link to travis through your GH account...but I'll try to verify that. Additionally, there are linter errors. Sorry I didn't notice that before. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @phrocker For the first error, I'm pretty sure the only type it can convert to is int64. The JsonCPP docs don't have any ambiguities there, it's just signed and unsigned int and int64. For the second, I don't see how it's even possible since archive.h is included at the top of that file. Can you check to see if it's using the thirdparty/ headers, or could it be pulling them from somewhere else on your system? ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj I see two build failures. FocusArchiveEntry.cpp:161:38: error: ambiguous conversion for functional-style cast from 'const long' to 'Json::Value' FocusArchiveEntry.cpp:252:14: error: no matching function for call to 'archive_read_open' ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 Thank @achristianson for the original implementation, way back in March/April against 070d8758fded4836f17816c1b1a34a48aa1bba20. I just brought it up to date. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @phrocker With that header addition I believe we should be good to go. Thanks! ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj One minor change RE the license header for ProcessSessionReadCallback. Assuming @apiri is good, we can merge. I'm going to make a follow on to break these archive classes into an extension, but I'm not satisfied with what I'm "calling them," so I won't burden you with any additional rebases. Thanks for this! ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 Rebased. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj pls clarify with @phrocker if the stream-based export is truly necessary at this point. The greater concern may be ensuring that the stash resource claims are properly tracked by the flow file repo (rocksdb). I *believe* that would require adding a bit of code to FlowFileRecord's Serialize/Deserialize methods, but would like @phrocker to confirm since he's more knowledgeable on the core repos than I am. ---
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 Things left to look into: * [ ] Move things which depend on libarchive to an extension * [ ] Use streams for exportContent * [ ] Move exportContent ReadCallback to another file ---
