subject:"\[jira\] \[Commented\] \(MINIFICPP\-39\) Create FocusArchive processor"

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-11-03 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237561#comment-16237561
 ] 

Caleb Johnson commented on MINIFICPP-39:


I think this should be marked as resolved?

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220780#comment-16220780
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj It's not archive that is the problem. It's the usage of uint64_t 
over long. *


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220731#comment-16220731
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
The build fails both with or without the `struct` keyword; I'll try a 
different approach next. There are also some other things I forgot to add when 
moving to the extension.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219283#comment-16219283
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@phrocker I already have a travis-ci.org account, but I've granted access 
to @NifiLocal and synced the account.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219255#comment-16219255
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj I emailed Travis
their response:
""
We've been experiencing some issues with our abuse system creating false 
positives lately, and I suspect that's what's happening here. I took a look in 
our system, and it looks like the organization NiFiLocal, where the pull 
request is coming from, doesn't exist within Travis, which is causing our 
system to flag the pull request as suspicious. If you can have the creator of 
the pull request log into travis-ci.org and sync their Github account, I should 
be able to go into our system and flag the NiFiLocal as not abusive, which will 
allow the pull request to build normally.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218664#comment-16218664
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@phrocker Cool. I can squash into one commit and force push, if needed.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218657#comment-16218657
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj thanks for the fix. Something in the way this PR is submitted is 
causing travis not to run as as result I"ll go through re-verification on all 
platforms and merge once I have an opportunity to do that. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218655#comment-16218655
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj Ignore the boost removal, that was unrelated. This processor will 
be moved to an EXCLUDE_BOOST so that those who wish to exclude boost will not 
see this and other libarchive processors. So you can keep that header. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218607#comment-16218607
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
I'm changing the types for size, mtime and mtime_nsec in 
`ArchiveEntryMetadata`'s declaration instead of casting. Why'd you take out the 
boost header include?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218551#comment-16218551
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
diff --git a/libminifi/src/processors/FocusArchiveEntry.cpp 
b/libminifi/src/processors/FocusArchiveEntry.cpp
index c262217..42962a6 100644
--- a/libminifi/src/processors/FocusArchiveEntry.cpp
+++ b/libminifi/src/processors/FocusArchiveEntry.cpp
@@ -24,8 +24,6 @@
 
 #include 
 
-#include 
-
 #include 
 #include 
 
@@ -154,11 +152,11 @@ void 
FocusArchiveEntry::onTrigger(core::ProcessContext *context,
   entryVal["entry_name"] = Json::Value(entryMetadata.entryName);
   entryVal["entry_type"] = Json::Value(entryMetadata.entryType);
   entryVal["entry_perm"] = Json::Value(entryMetadata.entryPerm);
-  entryVal["entry_size"] = Json::Value(entryMetadata.entrySize);
+  entryVal["entry_size"] = 
Json::Value((uint64_t)entryMetadata.entrySize);
   entryVal["entry_uid"] = Json::Value(entryMetadata.entryUID);
   entryVal["entry_gid"] = Json::Value(entryMetadata.entryGID);
-  entryVal["entry_mtime"] = Json::Value(entryMetadata.entryMTime);
-  entryVal["entry_mtime_nsec"] = 
Json::Value(entryMetadata.entryMTimeNsec);
+  entryVal["entry_mtime"] = 
Json::Value((uint64_t)entryMetadata.entryMTime);
+  entryVal["entry_mtime_nsec"] = 
Json::Value((int64_t)entryMetadata.entryMTimeNsec);
 
   if (entryMetadata.entryType == AE_IFREG) {
   entryVal["stash_key"] = Json::Value(entryMetadata.stashKey);
@@ -221,7 +219,7 @@ int64_t 
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
   archive_read_support_filter_all(inputArchive);
 
   // Read callback which reads from ifstream
-  auto read = [] (archive * a, void *d, const void **buf) -> int64_t {
+  auto read = [] (struct archive * a, void *d, const void **buf) -> long {
 auto data = static_cast(d);
 *buf = data->buf;
 int64_t read = 0;
@@ -241,7 +239,7 @@ int64_t 
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
   };
 
   // Close callback for libarchive
-  auto close = [] (archive *, void *) -> int {
+  auto close = [] (struct archive *, void *) -> int {
 // Because we do not need to close the stream, do nothing & return 
success
 return 0;
   };
@@ -249,7 +247,7 @@ int64_t 
FocusArchiveEntry::ReadCallback::process(std::shared_ptr
   // Read each item in the archive
   int res;
 
-  if ((res = archive_read_open(inputArchive, , NULL, read, close))) {
+  if ((res = archive_read_open(inputArchive, , nullptr, read, 
close))) {
   logger_->log_error(
   "FocusArchiveEntry can't open due to archive error: %s",
   archive_error_string(inputArchive));


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218527#comment-16218527
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj These failures are isolated to OSX. I noticed that travis isn't 
building your PRs to show you the build failure. I'll take a look to see why 
you aren't getting travis runs. My guess is that you don't have a link to 
travis through your GH account...but I'll try to verify that.
Additionally, there are linter errors. Sorry I didn't notice that before. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217290#comment-16217290
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@phrocker For the first error, I'm pretty sure the only type it can convert 
to is int64. The JsonCPP docs don't have any ambiguities there, it's just 
signed and unsigned int and int64.

For the second, I don't see how it's even possible since archive.h is 
included at the top of that file.

Can you check to see if it's using the thirdparty/ headers, or could it be 
pulling them from somewhere else on your system?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216845#comment-16216845
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj I see two build failures.

FocusArchiveEntry.cpp:161:38: error: ambiguous conversion for 
functional-style cast from 'const long' to 'Json::Value'

FocusArchiveEntry.cpp:252:14: error: no matching function for call to 
'archive_read_open'


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215277#comment-16215277
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
Thank @achristianson for the original implementation, way back in 
March/April against 070d8758fded4836f17816c1b1a34a48aa1bba20. I just brought it 
up to date.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215154#comment-16215154
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@phrocker With that header addition I believe we should be good to go.  
Thanks!


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215117#comment-16215117
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj One minor change RE the license header for 
ProcessSessionReadCallback. Assuming @apiri is good, we can merge. I'm going to 
make a follow on to break these archive classes into an extension, but I'm not 
satisfied with what I'm "calling them," so I won't burden you with any 
additional rebases. Thanks for this!


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215114#comment-16215114
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r146259050
  
--- Diff: libminifi/include/core/ProcessSessionReadCallback.h ---
@@ -0,0 +1,33 @@
+#ifndef __PROCESS_SESSION_READ_CALLBACK_H__
--- End diff --

Since you broke this out into a different file ( thanks! ) it will need a 
license header. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214969#comment-16214969
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
Rebased.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214169#comment-16214169
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user minifirocks commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r146120896
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
   bool keepSource,
   uint64_t offset, char inputDelimiter);
 
+  /**
+   * Exports the data stream to a file
+   * @param string file to export stream to
+   * @param flow flow file
+   * @param bool whether or not to keep the content in the flow file
+   */
+  bool exportContent(const std::string ,
--- End diff --

the mergeContent.h
// Archive Class
class ArchiveMerge {
public: 
do not reply on persistent storage, it use archive_write_open(arch, this, 
NULL, archive_write, NULL); to write the process content in RAM into flowfile.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211295#comment-16211295
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
@calebj pls clarify with @phrocker if the stream-based export is truly 
necessary at this point.

The greater concern may be ensuring that the stash resource claims are 
properly tracked by the flow file repo (rocksdb). I *believe* that would 
require adding a bit of code to FlowFileRecord's Serialize/Deserialize methods, 
but would like @phrocker to confirm since he's more knowledgeable on the core 
repos than I am.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211285#comment-16211285
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145748583
  
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public 
io::Serializable {
 return content_full_fath_;
   }
 
+  /**
+   * Cleanly relinquish a resource claim
+   */
--- End diff --

The flowfile is releasing its own claim to the resource. There are a few 
cases where the flow file has a claim to a resource that it should release to 
any other components that might have a claim on the resource (which might 
currently be a hypothetical-only case).


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211283#comment-16211283
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747665
  
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public 
io::Serializable {
 return content_full_fath_;
   }
 
+  /**
+   * Cleanly relinquish a resource claim
+   */
--- End diff --

Clarified in the latest commit


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211281#comment-16211281
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747588
  
--- Diff: libminifi/include/core/FlowConfiguration.h ---
@@ -35,6 +35,8 @@
 #include "processors/ExecuteProcess.h"
 #include "processors/AppendHostInfo.h"
 #include "processors/MergeContent.h"
+#include "processors/FocusArchiveEntry.h"
--- End diff --

Clarified in the latest commit


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211270#comment-16211270
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/148
  
Things left to look into:
* [ ] Move things which depend on libarchive to an extension
* [ ] Use streams for exportContent
* [ ] Move exportContent ReadCallback to another file



> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211268#comment-16211268
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145745217
  
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include 
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211247#comment-16211247
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743845
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211260#comment-16211260
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744366
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -19,6 +19,7 @@
 #define __PROCESS_SESSION_H__
 
 #include 
+#include 
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211259#comment-16211259
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744343
  
--- Diff: libminifi/include/core/FlowFile.h ---
@@ -50,6 +50,32 @@ class FlowFile : public core::Connectable {
   void clearResourceClaim();
 
   /**
+   * Returns a pointer to this flow file record's
+   * claim at the given stash key
+   */
+  std::shared_ptr getStashClaim(const std::string );
+
+  /**
+   * Sets the given stash key to the inbound claim argument
+   */
+  void setStashClaim(const std::string , 
std::shared_ptr );
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211251#comment-16211251
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743902
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
>

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211238#comment-16211238
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145742734
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

Neither. It's the 3-clause UC Regents copyright, which is in the three 
aforementioned source files. The originals also skip the third list item.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211261#comment-16211261
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145734644
  
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include 
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --

I would opt to just not use the using statement here. Not worth breaking 
the convention for the one class.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211262#comment-16211262
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743112
  
--- Diff: libminifi/src/core/ProcessSession.cpp ---
@@ -799,6 +799,152 @@ void ProcessSession::import(std::string source, 
std::shared_ptr
   }
 }
 
+bool ProcessSession::exportContent(
+const std::string ,
+const std::string ,
+std::shared_ptr ,
+bool keepContent) {
+  logger_->log_info(
+  "Exporting content of %s to %s",
+  flow->getUUIDStr().c_str(),
+  destination.c_str());
+
+  ReadCallback cb(tmpFile, destination, logger_);
+  read(flow, );
+
+  logger_->log_info("Committing %s", destination.c_str());
+  bool commit_ok = cb.commit();
+
+  if (commit_ok) {
+logger_->log_info("Commit OK.");
+  } else {
+logger_->log_error(
+  "Commit of %s to %s failed!",
+  flow->getUUIDStr().c_str(),
+  destination.c_str());
+  }
+  return commit_ok;
+}
+
+bool ProcessSession::exportContent(
+const std::string ,
+std::shared_ptr ,
+bool keepContent) {
+  std::string tmpFileName = boost::filesystem::unique_path().native();
+  return exportContent(destination, tmpFileName, flow, keepContent);
+}
+
+ProcessSession::ReadCallback::ReadCallback(const std::string ,
+   const std::string ,
+   
std::shared_ptr logger)
+: _tmpFile(tmpFile),
+  _tmpFileOs(tmpFile, std::ios::binary),
+  _destFile(destFile),
+  logger_(logger) {
+}
+
+// Copy the entire file contents to the temporary file
+int64_t 
ProcessSession::ReadCallback::process(std::shared_ptr stream) {
+  // Copy file contents into tmp file
+  _writeSucceeded = false;
+  size_t size = 0;
+  uint8_t buffer[8192];
+  do {
+int read = stream->read(buffer, 8192);
+if (read < 0) {
+  return -1;
+}
+if (read == 0) {
+  break;
+}
+_tmpFileOs.write(reinterpret_cast(buffer), read);
+size += read;
+  } while (size < stream->getSize());
+  _writeSucceeded = true;
+  return size;
+}
+
+// Renames tmp file to final destination
+// Returns true if commit succeeded
+bool ProcessSession::ReadCallback::commit() {
+  bool success = false;
+
+  logger_->log_info("committing export operation to %s", 
_destFile.c_str());
+
+  if (_writeSucceeded) {
+_tmpFileOs.close();
+
+if (rename(_tmpFile.c_str(), _destFile.c_str())) {
+  logger_->log_info("commit export operation to %s failed because 
rename() call failed", _destFile.c_str());
+} else {
+  success = true;
+  logger_->log_info("commit export operation to %s succeeded", 
_destFile.c_str());
+}
+  } else {
+logger_->log_error("commit export operation to %s failed because write 
failed", _destFile.c_str());
+  }
+  return success;
+}
+
+// Clean up resources
+ProcessSession::ReadCallback::~ReadCallback() {
+  // Close tmp file
+  _tmpFileOs.close();
+
+  // Clean up tmp file, if necessary
+  unlink(_tmpFile.c_str());
+}
+
+
+void ProcessSession::stash(const std::string , 
std::shared_ptr flow) {
--- End diff --

@phrocker by 'tmp file,' are you referring to the stash claims?

We would want those to operate the same as the primary content claim. It 
looks like this may not be the case currently. Would adding the stash claims to 
the data stored/retrieved in FlowFileRecord's Serialize/Deserialize cover all 
the bases, or would other changes be required as well?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
>

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211255#comment-16211255
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744191
  
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include 
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
+
+//! FocusArchiveEntry Class
+class FocusArchiveEntry : public core::Processor {
+ public:
+  //! Constructor
+  /*!
+   * Create a new processor
+   */
+  explicit FocusArchiveEntry(std::string name, uuid_t uuid = NULL)
+  : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+  }
+  //! Destructor
+  virtual ~FocusArchiveEntry()   {
+  }
+  //! Processor Name
+  static constexpr char const* ProcessorName = "FocusArchiveEntry";
+  //! Supported Properties
+  static core::Property Path;
+  //! Supported Relationships
+  static core::Relationship Success;
+
+  bool set_del_or_update_attr(std::shared_ptr, const 
std::string, std::string*) const;
--- End diff --

Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211250#comment-16211250
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743879
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
+  uuid_generate(stashKeyUuid);
+  uuid_unparse_lower(stashKeyUuid, stashKey);
+  logger_->log_debug(
+  "FocusArchiveEntry generated stash key %s for entry %s",
+  stashKey,
+  entryMetadata.entryName.c_str());
+  entryMetadata.stashKey.assign(stashKey);
+
+  if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+  }
+
+  // Stash the content
+  session->stash(entryMetadata.stashKey, flowFile);
+}
+  }
+
+  // Restore target archive entry
+  if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+  } else {
+logger_->log_warn(
+  "FocusArchiveEntry failed to locate target entry: %s",
+  targetEntry.c_str());
+  }
+
+  // Set new/updated lens

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211248#comment-16211248
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743863
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
+  uuid_generate(stashKeyUuid);
+  uuid_unparse_lower(stashKeyUuid, stashKey);
+  logger_->log_debug(
+  "FocusArchiveEntry generated stash key %s for entry %s",
+  stashKey,
+  entryMetadata.entryName.c_str());
+  entryMetadata.stashKey.assign(stashKey);
+
+  if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+  }
+
+  // Stash the content
+  session->stash(entryMetadata.stashKey, flowFile);
+}
+  }
+
+  // Restore target archive entry
+  if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+  } else {
+logger_->log_warn(
+  "FocusArchiveEntry failed to locate target entry: %s",
+  targetEntry.c_str());
+  }
+
+  // Set new/updated lens

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211226#comment-16211226
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145740328
  
--- Diff: CMakeLists.txt ---
@@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL 
"Disable dynamic SSL libr
 set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
 add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3)
 add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL)
+add_subdirectory(thirdparty/libarchive-3.3.2)
--- End diff --

@calebj I think @achristianson point is valid that it shouldn't be done 
here. I will submit a PR to the other PR and then when that is merged ( should 
be the next day ), you will get that for free via the rebase. So feel free to 
ignore any comment about "extensions" since that is on me. thanks!


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211221#comment-16211221
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user calebj commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145738276
  
--- Diff: CMakeLists.txt ---
@@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL 
"Disable dynamic SSL libr
 set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
 add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3)
 add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL)
+add_subdirectory(thirdparty/libarchive-3.3.2)
--- End diff --

I'm not sure how to do that, are there any examples?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211184#comment-16211184
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145731149
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
   bool keepSource,
   uint64_t offset, char inputDelimiter);
 
+  /**
+   * Exports the data stream to a file
+   * @param string file to export stream to
+   * @param flow flow file
+   * @param bool whether or not to keep the content in the flow file
+   */
+  bool exportContent(const std::string ,
--- End diff --

Export is simply meant to be the inverse of import.

Digging into the code, export is used in this change set in 
UnfocusArchiveEntry in order to export the flow file content into a 
scratch/working location so that it can be re-assembled back into an archive.

We're ultimately calling archive_write_data (line 286 of 
UnfocusArchiveEntry.cpp), which takes data from a byte buffer. Therefore, we 
don't necessarily require a persistent filesystem for this change, as this 
scratch/working area could be in RAM or some other medium as long as there's 
enough space.

As it stands, these new archive processors do depend on persistent 
filesystem storage, but this addition of exportContent does not result in 
ProcessSession or any core component depending on any storage implementation 
where it previously did not. We're simply adding an mirror capability to import 
which is optional to use, and where the caller is responsible for environmental 
considerations/requirements such as persistent storage.

Assuming we move these new archive processors into an extension, a current 
prerequisite of using that extension will be persistent filesystem storage. 
This should be documented. This leaves open the door to future 
implementations/improvements which use RAM or some other medium to reconstitute 
the archive, but I think that level of functionality is not required 
immediately.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211163#comment-16211163
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145724061
  
--- Diff: CMakeLists.txt ---
@@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL 
"Disable dynamic SSL libr
 set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
 add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3)
 add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL)
+add_subdirectory(thirdparty/libarchive-3.3.2)
--- End diff --

Great! We'll work on getting the other PR in an extension and this can be 
rebased. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211128#comment-16211128
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user achristianson commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145717915
  
--- Diff: CMakeLists.txt ---
@@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL 
"Disable dynamic SSL libr
 set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
 add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3)
 add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL)
+add_subdirectory(thirdparty/libarchive-3.3.2)
--- End diff --

Both this and MINIFICPP-72 use libarchive. Not opposed to this being moved 
to an extension, but it needs to be coordinated between the two.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211108#comment-16211108
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145714089
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

I was assuming that was the answer and the fourth clause was cherry-picked.

I assume the the author then simply meant "which under a 3-clause UC 
Regents [License] as below" since the copyright is a subset of the license.

Thanks.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211107#comment-16211107
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145714071
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

I now see this was commit was taken from the other, outstanding PR, so will 
make sure we get that taken care of there and we can cherry pick the work from 
here onto that.  I believe a number of these changes have already been covered 
there


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211101#comment-16211101
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145713014
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
--- End diff --

archive_getdate.c which is still bundled with this commit is listed as 
public domain and we should have appropriate attribution in our license


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211090#comment-16211090
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145711436
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

Oh, interesting.  I now see there is a 3-clause below the header that the 
maintainer (I believe) of libarchive has as a 2 clause


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211088#comment-16211088
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145711080
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

Certainly is listed as a 3-clause.

archive_entry.c, archive_read_support_filter_compress.c, and 
archive_write_add_filter_compress.c all have 2 clause headers


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread marco polo (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211006#comment-16211006
 ] 

marco polo commented on MINIFICPP-39:
-

[~calebj] I'll try to pull down your PR.

I've built master on U14 a few days ago, so I will try with that PR. We may be 
linking to the wrong object in the test. 

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210999#comment-16210999
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145691644
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
--- End diff --

Is there any reason you were unable to use the ID generator? 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
>

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211000#comment-16211000
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145690546
  
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include 
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --

I'm not necessarily against a using statement for loggers, but this would 
be the only one. I would like to see us to continue following convention 
outside of the extensions directory. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210996#comment-16210996
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688494
  
--- Diff: libminifi/include/core/FlowConfiguration.h ---
@@ -35,6 +35,8 @@
 #include "processors/ExecuteProcess.h"
 #include "processors/AppendHostInfo.h"
 #include "processors/MergeContent.h"
+#include "processors/FocusArchiveEntry.h"
--- End diff --

This isn't necessary if you use an extension. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210992#comment-16210992
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688253
  
--- Diff: LICENSE ---
@@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
OUT OF OR IN
 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+This projects includes libarchive bundle (https://www.libarchive.org)
+which is available under a BSD License by Tim Kientzle and others
+
+Copyright (c) 2003-2009 Tim Kientzle and other authors 
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer
+   in this position and unchanged.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (END LICENSE TEXT)
+
+All libarchive C sources (including .c and .h files)
+and documentation files are subject to the copyright notice reproduced
+above.
+
+This libarchive includes below files 
+libarchive/archive_entry.c
+libarchive/archive_read_support_filter_compress.c
+libarchive/archive_write_add_filter_compress.c 
+which under a 3-clause UC Regents copyright as below
+/*-
+ * Copyright (c) 1993
+ *  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 4. Neither the name of the University nor the names of its contributors
--- End diff --

Is this a BSD-3 clause or a BSD-4 clause? 

BSD-4-clause may not be included as I understand it. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210990#comment-16210990
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688553
  
--- Diff: libminifi/include/core/FlowFile.h ---
@@ -50,6 +50,32 @@ class FlowFile : public core::Connectable {
   void clearResourceClaim();
 
   /**
+   * Returns a pointer to this flow file record's
+   * claim at the given stash key
+   */
+  std::shared_ptr getStashClaim(const std::string );
+
+  /**
+   * Sets the given stash key to the inbound claim argument
+   */
+  void setStashClaim(const std::string , 
std::shared_ptr );
--- End diff --

please use const on all shared ptrs 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211003#comment-16211003
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688790
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -19,6 +19,7 @@
 #define __PROCESS_SESSION_H__
 
 #include 
+#include 
--- End diff --

Please don't use boost in core components. We should find another way to do 
this and extract this functionality to a boost supporting extension. 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210995#comment-16210995
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145690194
  
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include 
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
+
+//! FocusArchiveEntry Class
+class FocusArchiveEntry : public core::Processor {
+ public:
+  //! Constructor
+  /*!
+   * Create a new processor
+   */
+  explicit FocusArchiveEntry(std::string name, uuid_t uuid = NULL)
+  : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+  }
+  //! Destructor
+  virtual ~FocusArchiveEntry()   {
+  }
+  //! Processor Name
+  static constexpr char const* ProcessorName = "FocusArchiveEntry";
+  //! Supported Properties
+  static core::Property Path;
+  //! Supported Relationships
+  static core::Relationship Success;
+
+  bool set_del_or_update_attr(std::shared_ptr, const 
std::string, std::string*) const;
--- End diff --

why std::string * over const std::string &?  The semantics imply you will 
be changing the third argument. 





> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211002#comment-16211002
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145692040
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
+  uuid_generate(stashKeyUuid);
+  uuid_unparse_lower(stashKeyUuid, stashKey);
+  logger_->log_debug(
+  "FocusArchiveEntry generated stash key %s for entry %s",
+  stashKey,
+  entryMetadata.entryName.c_str());
+  entryMetadata.stashKey.assign(stashKey);
+
+  if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+  }
+
+  // Stash the content
+  session->stash(entryMetadata.stashKey, flowFile);
+}
+  }
+
+  // Restore target archive entry
+  if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+  } else {
+logger_->log_warn(
+  "FocusArchiveEntry failed to locate target entry: %s",
+  targetEntry.c_str());
+  }
+
+  // Set new/updated lens

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210998#comment-16210998
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145689143
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
   bool keepSource,
   uint64_t offset, char inputDelimiter);
 
+  /**
+   * Exports the data stream to a file
+   * @param string file to export stream to
+   * @param flow flow file
+   * @param bool whether or not to keep the content in the flow file
+   */
+  bool exportContent(const std::string ,
--- End diff --

Does export content imply we will always have persistent storage? If not we 
could use std::shared_ptr stream for input and output and apply 
a file stream to them? 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211001#comment-16211001
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145692128
  
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include 
+#include 
+
+#include 
+
+#include 
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total 
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool 
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr 
flowFile, const std::string key, std::string* value) const {
+  if (value == nullptr)
+return flowFile->removeAttribute(key);
+  else if (flowFile->updateAttribute(key, *value))
+return true;
+  else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(Path);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+  core::ProcessSession *session) {
+  auto flowFile = session->get();
+  std::shared_ptr flowFileRecord = 
std::static_pointer_cast(flowFile);
+
+  if (!flowFile) {
+return;
+  }
+
+  std::string targetEntry;
+  context->getProperty(Path.getName(), targetEntry);
+
+  // Extract archive contents
+  ArchiveMetadata archiveMetadata;
+  archiveMetadata.focusedEntry = targetEntry;
+  ReadCallback cb();
+  session->read(flowFile, );
+
+  // For each extracted entry, import & stash to key
+  std::string targetEntryStashKey;
+
+  for (auto  : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+  logger_->log_info("FocusArchiveEntry importing %s from %s",
+  entryMetadata.entryName.c_str(),
+  entryMetadata.tmpFileName.c_str());
+  session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+  char stashKey[37];
+  uuid_t stashKeyUuid;
+  uuid_generate(stashKeyUuid);
+  uuid_unparse_lower(stashKeyUuid, stashKey);
+  logger_->log_debug(
+  "FocusArchiveEntry generated stash key %s for entry %s",
+  stashKey,
+  entryMetadata.entryName.c_str());
+  entryMetadata.stashKey.assign(stashKey);
+
+  if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+  }
+
+  // Stash the content
+  session->stash(entryMetadata.stashKey, flowFile);
+}
+  }
+
+  // Restore target archive entry
+  if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+  } else {
+logger_->log_warn(
+  "FocusArchiveEntry failed to locate target entry: %s",
+  targetEntry.c_str());
+  }
+
+  // Set new/updated lens

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210997#comment-16210997
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145691328
  
--- Diff: libminifi/src/core/ProcessSession.cpp ---
@@ -799,6 +799,152 @@ void ProcessSession::import(std::string source, 
std::shared_ptr
   }
 }
 
+bool ProcessSession::exportContent(
+const std::string ,
+const std::string ,
+std::shared_ptr ,
+bool keepContent) {
+  logger_->log_info(
+  "Exporting content of %s to %s",
+  flow->getUUIDStr().c_str(),
+  destination.c_str());
+
+  ReadCallback cb(tmpFile, destination, logger_);
+  read(flow, );
+
+  logger_->log_info("Committing %s", destination.c_str());
+  bool commit_ok = cb.commit();
+
+  if (commit_ok) {
+logger_->log_info("Commit OK.");
+  } else {
+logger_->log_error(
+  "Commit of %s to %s failed!",
+  flow->getUUIDStr().c_str(),
+  destination.c_str());
+  }
+  return commit_ok;
+}
+
+bool ProcessSession::exportContent(
+const std::string ,
+std::shared_ptr ,
+bool keepContent) {
+  std::string tmpFileName = boost::filesystem::unique_path().native();
+  return exportContent(destination, tmpFileName, flow, keepContent);
+}
+
+ProcessSession::ReadCallback::ReadCallback(const std::string ,
+   const std::string ,
+   
std::shared_ptr logger)
+: _tmpFile(tmpFile),
+  _tmpFileOs(tmpFile, std::ios::binary),
+  _destFile(destFile),
+  logger_(logger) {
+}
+
+// Copy the entire file contents to the temporary file
+int64_t 
ProcessSession::ReadCallback::process(std::shared_ptr stream) {
+  // Copy file contents into tmp file
+  _writeSucceeded = false;
+  size_t size = 0;
+  uint8_t buffer[8192];
+  do {
+int read = stream->read(buffer, 8192);
+if (read < 0) {
+  return -1;
+}
+if (read == 0) {
+  break;
+}
+_tmpFileOs.write(reinterpret_cast(buffer), read);
+size += read;
+  } while (size < stream->getSize());
+  _writeSucceeded = true;
+  return size;
+}
+
+// Renames tmp file to final destination
+// Returns true if commit succeeded
+bool ProcessSession::ReadCallback::commit() {
+  bool success = false;
+
+  logger_->log_info("committing export operation to %s", 
_destFile.c_str());
+
+  if (_writeSucceeded) {
+_tmpFileOs.close();
+
+if (rename(_tmpFile.c_str(), _destFile.c_str())) {
+  logger_->log_info("commit export operation to %s failed because 
rename() call failed", _destFile.c_str());
+} else {
+  success = true;
+  logger_->log_info("commit export operation to %s succeeded", 
_destFile.c_str());
+}
+  } else {
+logger_->log_error("commit export operation to %s failed because write 
failed", _destFile.c_str());
+  }
+  return success;
+}
+
+// Clean up resources
+ProcessSession::ReadCallback::~ReadCallback() {
+  // Close tmp file
+  _tmpFileOs.close();
+
+  // Clean up tmp file, if necessary
+  unlink(_tmpFile.c_str());
+}
+
+
+void ProcessSession::stash(const std::string , 
std::shared_ptr flow) {
--- End diff --

What are the ramifications if power is lost and the rocksdb repos WAL 
causes us to repeat this flow? Would that previous tmp file be left around? 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210993#comment-16210993
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688465
  
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public 
io::Serializable {
 return content_full_fath_;
   }
 
+  /**
+   * Cleanly relinquish a resource claim
+   */
--- End diff --

What does relinquish a claim mean in this context? From what/where?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210991#comment-16210991
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145687725
  
--- Diff: CMakeLists.txt ---
@@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL 
"Disable dynamic SSL libr
 set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
 add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3)
 add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL)
+add_subdirectory(thirdparty/libarchive-3.3.2)
--- End diff --

Can we move this into an extension and have this be excluded based on 
either an inclusion or exclusion?


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210994#comment-16210994
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145688909
  
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
   bool keepSource,
   uint64_t offset, char inputDelimiter);
 
+  /**
+   * Exports the data stream to a file
+   * @param string file to export stream to
+   * @param flow flow file
+   * @param bool whether or not to keep the content in the flow file
+   */
+  bool exportContent(const std::string ,
+ std::shared_ptr ,
+ bool keepContent);
+
+  bool exportContent(const std::string ,
+ const std::string ,
+ std::shared_ptr ,
+ bool keepContent);
+
+  // Stash the content to a key
+  void stash(const std::string , std::shared_ptr flow);
+   // Restore content previously stashed to a key
+  void restore(const std::string , std::shared_ptr 
flow);
+
 // Prevent default copy constructor and assignment operation
 // Only support pass by reference or pointer
   ProcessSession(const ProcessSession ) = delete;
   ProcessSession =(const ProcessSession ) = delete;
 
+  class ReadCallback : public InputStreamCallback {
--- End diff --

Can we move this elsewhere? 


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210938#comment-16210938
 ] 

Caleb Johnson commented on MINIFICPP-39:


It is based on Ubuntu 14.04, and there is no package that mentions rocksdb in 
any configured repo. The specific issue I'm having is this:

{noformat}
[ 78%] Linking CXX shared library librocksdb.so
/usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o: relocation 
R_X86_64_32 against `.data' can not be used when making a shared object; 
recompile with -fPIC
CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols: 
Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
make[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
make: *** [all] Error 2
{noformat}

However, running make VERBOSE=1 reveals that the compiler arguments _do_ begin 
with -fPIC.

I've isolated the libarchive additions of MINIFICPP-72 into a separate commit 
and rebased this changeset onto it in the MINIFI-244-rc2 branch, which is what 
I'll be using in the PR. This should make things a bit smoother for both.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210940#comment-16210940
 ] 

ASF GitHub Bot commented on MINIFICPP-39:
-

GitHub user calebj opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/148

 MINIFI-244 Un/FocusArchive processors

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

Split into two commits to share a common base with #146.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NiFiLocal/nifi-minifi-cpp MINIFI-244-rc2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/148.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #148


commit 5cf2ae943e9eb2ab8c289fc25da451b96da38a04
Author: Caleb Johnson 
Date:   2017-10-18T18:21:29Z

Pull in MINIFICPP-72's libarchive

commit 3a32355c8f81e7216fca5ba8a83e9781aaeedb04
Author: Caleb Johnson 
Date:   2017-10-17T15:45:53Z

MINIFI-244 Un/FocusArchive processors




> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-18 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209461#comment-16209461
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

If your cloud9 environment is based on ubuntu, you should be able to apt-get 
install rocksdb & the development headers (check the nifi dev mailing list for 
a recent post by [~phrocker] on this.

I looked at the code, and see your function to verify the contents of the file 
in the unit test. Nice work!

I can't think of anything else off-hand that would preclude a PR where the 
development community can formally review everything. Lucky for you, another 
developer recently brought in libarchive (MINIFICPP-72), and I think they 
worked out the somewhat complex LICENSING documentation. Watch out for some 
minor conflicts since that PR isn't merged yet (both that branch and yours will 
bring in libarchive). Things should be compatible, but I imagine there will be 
a small amount of conflicting/redundant code in the CMakeLists.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-17 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208156#comment-16208156
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

Thanks. Will check it out and get back to you.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-17 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208033#comment-16208033
 ] 

Caleb Johnson commented on MINIFICPP-39:


[~achristianson], check the MINIFI-244-rc branch for what I think is ready to 
PR. It has full tests via PutFile, and has been rebased, squashed, and linted 
(is that what you call a clean linter pass?).

I can't get rocksdb to build on cloud9 for some reason, but the Un/FocusArchive 
tests build and run. Unfortunately, cloud9 doesn't have support for Docker.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-13 Thread marco polo (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204354#comment-16204354
 ] 

marco polo commented on MINIFICPP-39:
-

[~calebj] An alternative is to use log attributes to do validation through the 
log output. Not a great way for validation, but we'll get to the 
current_flowfile_ eventually. When we have a chance and would always welcome 
contributions there. 

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-13 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203718#comment-16203718
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

[~calebj], it looks like we currently don't have that facility in the cpp unit 
test framework. So for now, the main options are to use the putfile, and verify 
the output after it's written to disk, or to use the docker system integration 
test framework. The docs on the docker system integration test framework are at 
https://github.com/apache/nifi-minifi-cpp/blob/master/docker/test/integration/README.md.
 To do this, you'd have to add support for these processors (simple python 
classes), then create a custom OutputValidator (look at the 
SingleFileOutputValidator for an example).

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-12 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202421#comment-16202421
 ] 

Caleb Johnson commented on MINIFICPP-39:


[~achristianson] Thanks for the feedback. I've added the attribute update and 
removed rapidjson in the latest commits. I wanted to ask about the test flow 
for Un/FocusArchive next.

As I commented near the end of the unit test, the "current flowfile" is always 
nullptr after runNextProcessor(). I'd like to be able to access it to ensure 
the attributes and contents are correct, preferably without going through 
PutFile. Is there a way to do that without going through temp files using the 
TestBase machinery?

I'm also working on ManipulateArchive. It's been rebased and compiles without 
any issues, so I'll be working on tests for that as well. Then, ApplyTemplate 
and ExtractText.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-12 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202138#comment-16202138
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

[~calebj] nice job on the rebase and setting up tests. A couple of notes:

1) I noticed this bit seems to be unlimplemented in UnfocusArchive.onTrigger:

{{
  // Set new/updated lens stack to attribute; height(stack') = height(stack) - 1

  // if (!flowFile->updateAttribute("lens.archive.stack", stackStr))
  // {
  //flowFile->addAttribute("lens.archive.stack", stackStr);
  // }
}}

The lens stack is already "popped" with this line: 
lensStack.resize(lensStack.size() - 1);, so I think we just need to serialize 
the json after that and set it back into the lens.archive.stack attribute.

2) The unit tests do cover onTrigger of both Focus & Unfocus, which is good, 
but it would be nice to have some additional verification of the actual output 
file. I.e. verify that it is a valid tar/what have you, and that the content is 
valid. Along similar lines, it would be nice to verify that the flow file 
content is correct after FocusArchive.

Looking good over all. I think this is close to ready for a PR.


> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-12 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201979#comment-16201979
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

Sure thing, will take a look.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-11 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200729#comment-16200729
 ] 

Caleb Johnson commented on MINIFICPP-39:


[~achristianson],

Can you give [this 
branch|https://github.com/NiFiLocal/nifi-minifi-cpp/tree/MINIFI-244-rebase2] a 
look? If it looks fine, the next step for me is to switch everything that uses 
rapidjson over to jsoncpp. But for now, it's rebased to the latest master and I 
_think _it'll build.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-10-04 Thread Andrew Christianson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191353#comment-16191353
 ] 

Andrew Christianson commented on MINIFICPP-39:
--

[~calebj],

Sorry for the delay --I've been out with a brand new baby.

I do not have any WIP unit tests for this, but I would be happy to answer any 
questions about the code. There is a workable unit test framework in minifi-cpp 
now. I would take a look at PutFileTests for an example processor unit test.

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor

2017-09-29 Thread Caleb Johnson (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186205#comment-16186205
 ] 

Caleb Johnson commented on MINIFICPP-39:


[~achristianson],

I've rebased this and started working on tests in a branch on NifiLocal. If you 
have any WIPs for tests, could you send them to me?

> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive 
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on” 
> some particular piece/path of a complex data object such that you can more 
> precisely target particular operations without losing the context or 
> structure of the overall data you’re working with." 
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" 
> an entry in the archive, perform processing *in-context* of that entry, then 
> re-focus on the overall archive. This allows for transformation or other 
> processing of an entry in the archive without losing the overall context of 
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

73 matches

Mail list logo