[jira] [Commented] (NIFIREG-36) Add Jackson JsonInclude Annotations to NiFi Registry Data Model classes
[ https://issues.apache.org/jira/browse/NIFIREG-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212143#comment-16212143 ] ASF GitHub Bot commented on NIFIREG-36: --- Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/23 This is now rebased against the branch that adds integration tests. We should wait to merge this until that branch is finalized and merged to master so this can be rebased again if needed, but let me know what you think of the changes in the last commit on this branch. I added a couple other changes to this as well, including: - a custom object mapper in the client - mapper providers are now loaded via package scanning in NiFiRegistryResourceConfig (possible by also adding Spring @Component annotations) > Add Jackson JsonInclude Annotations to NiFi Registry Data Model classes > --- > > Key: NIFIREG-36 > URL: https://issues.apache.org/jira/browse/NIFIREG-36 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Kevin Doran >Assignee: Kevin Doran > Fix For: 0.0.1 > > > Currently, NiFi Registry responses include null values for optional fields in > the serialized Json. This ticket is to add Jackson annotations that prevent > explicit "null" from being serialized for optional fields, instead just > omitting the optional fields, as not all clients/frameworks can interpret > null correctly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry issue #23: NIFIREG-36 Add Json Mappers to server and client
Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/23 This is now rebased against the branch that adds integration tests. We should wait to merge this until that branch is finalized and merged to master so this can be rebased again if needed, but let me know what you think of the changes in the last commit on this branch. I added a couple other changes to this as well, including: - a custom object mapper in the client - mapper providers are now loaded via package scanning in NiFiRegistryResourceConfig (possible by also adding Spring @Component annotations) ---
[jira] [Commented] (NIFI-1625) ExtractText - Description of Capture Group is not clear
[ https://issues.apache.org/jira/browse/NIFI-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212122#comment-16212122 ] Randy Bovay commented on NIFI-1625: --- Rekha, I would agree, let's close this out > ExtractText - Description of Capture Group is not clear > --- > > Key: NIFI-1625 > URL: https://issues.apache.org/jira/browse/NIFI-1625 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Randy Bovay >Priority: Trivial > > ExtractText ONLY captures the first 1024 (default) characters. > The help text says this applies to the capture group values. It's not clear > that this is on the 'input', but leads one to believe it's on the actual new > properties that are being captured. > > Better wording should be > "Specifies the Maximum length of the input record that will be evaluated for > the capture. The input record will only be evaluated Up TO this length, and > the rest will be ignored" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-3348) G1 Values Not staying in correct column on refresh. In Cluster UI JVM Tab.
[ https://issues.apache.org/jira/browse/NIFI-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212110#comment-16212110 ] Randy Bovay commented on NIFI-3348: --- Pierre, Yes, that looks fine now. Can close this out as fixed. > G1 Values Not staying in correct column on refresh. In Cluster UI JVM Tab. > -- > > Key: NIFI-3348 > URL: https://issues.apache.org/jira/browse/NIFI-3348 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Randy Bovay >Priority: Minor > > The Values in the G1 Old Generation and G1 Young Generation will flip back > and forth in each column for the node as you hit refresh from the UI. > These are in the Cluster UI, JVM Tab. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212020#comment-16212020
]
Caleb Johnson commented on MINIFICPP-262:
-
I think that updating civetweb would be for the best, but its tests need to be
excluded from the `make tests` list and/or moved to its own civet-tests target.
auto_ptr is only present in jsoncpp, yaml-cpp and catch. There aren't any in
the MiNiFi codebase, so those issues for the third-party maintainers to
address. Those headers are just included in so many places that it floods my
build log.
>From this point on, I might just use an independently bootstrapped debian or
>ubuntu chroot to do dev builds. It's the second best thing to Docker, which
>Cloud9 doesn't support.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-1706) Extend QueryDatabaseTable to support arbitrary queries
[ https://issues.apache.org/jira/browse/NIFI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211993#comment-16211993 ] ASF GitHub Bot commented on NIFI-1706: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2162 Yes, sorry for the delay, got swamped > Extend QueryDatabaseTable to support arbitrary queries > -- > > Key: NIFI-1706 > URL: https://issues.apache.org/jira/browse/NIFI-1706 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.4.0 >Reporter: Paul Bormans >Assignee: Peter Wicks > Labels: features > > The QueryDatabaseTable is able to observe a configured database table for new > rows and yield these into the flowfile. The model of an rdbms however is > often (if not always) normalized so you would need to join various tables in > order to "flatten" the data into useful events for a processing pipeline as > can be build with nifi or various tools within the hadoop ecosystem. > The request is to extend the processor to specify an arbitrary sql query > instead of specifying the table name + columns. > In addition (this may be another issue?) it is desired to limit the number of > rows returned per run. Not just because of bandwidth issue's from the nifi > pipeline onwards but mainly because huge databases may not be able to return > so many records within a reasonable time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2162: NIFI-1706 Extend QueryDatabaseTable to support arbitrary q...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2162 Yes, sorry for the delay, got swamped ---
[jira] [Commented] (NIFI-1706) Extend QueryDatabaseTable to support arbitrary queries
[ https://issues.apache.org/jira/browse/NIFI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211979#comment-16211979 ] ASF GitHub Bot commented on NIFI-1706: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2162 @mattyb149 Can you take another look? I think I've addressed all of your concerns. > Extend QueryDatabaseTable to support arbitrary queries > -- > > Key: NIFI-1706 > URL: https://issues.apache.org/jira/browse/NIFI-1706 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.4.0 >Reporter: Paul Bormans >Assignee: Peter Wicks > Labels: features > > The QueryDatabaseTable is able to observe a configured database table for new > rows and yield these into the flowfile. The model of an rdbms however is > often (if not always) normalized so you would need to join various tables in > order to "flatten" the data into useful events for a processing pipeline as > can be build with nifi or various tools within the hadoop ecosystem. > The request is to extend the processor to specify an arbitrary sql query > instead of specifying the table name + columns. > In addition (this may be another issue?) it is desired to limit the number of > rows returned per run. Not just because of bandwidth issue's from the nifi > pipeline onwards but mainly because huge databases may not be able to return > so many records within a reasonable time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2162: NIFI-1706 Extend QueryDatabaseTable to support arbitrary q...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/2162 @mattyb149 Can you take another look? I think I've addressed all of your concerns. ---
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211965#comment-16211965
]
marco polo commented on MINIFICPP-262:
--
[~calebj] Thanks for running that.
That's disconcerting to hear about Arch. Do you think we could we could just
replace our third party directories with civetweb 1.10v? Did you run anything
to validate it? I'm not terribly worried about c++17 compatibility, but if the
fix is as easy as updating third party dependencies, and it causes no
compatibility issues, then I'm not sure there is any reason to not update.
I thought json_cpp used auto_ptr as well, so I'm surprised that didn't cause
issues either. At any rate, would you be willing to open a ticket RE the
dependencies potentially needing to be updated? I'll open one if you prefer,
but wanted to make sure you get credit for finding that.
"I never had to merge PR150 to get a successful build on Trusty. Did you want
me to test that, or the opposite?" I don't want to ask you to do that. I
noticed some commits were missing from a PR merge and I think 150 needs to get
in. It fixes a broken travis build too, so I'll double check tomorrow with
Trusty and Centos 7 just to be certain I don't further break things.
Thanks again.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211947#comment-16211947
]
Caleb Johnson commented on MINIFICPP-262:
-
I can't reproduce the issue either on a fresh Trusty VM. There's just something
screwy with Cloud9's setup.
On my personal Arch laptop, I wasn't able to get civetweb-1.9.1 or rocksdb to
build, respectively due to C++17 and openssl 1.1.0 incompatibility (I think),
but updating to civetweb v1.10 and rocksdb v5.7.2 (installed to system from
AUR) allows it to build and test correctly. The civetweb-1.10 tests aren't part
of the build, though it does try to run them. As a side note, it looks like
std::auto_ptr is deprecated now.
I never had to merge PR150 to get a successful build on Trusty. Did you want me
to test that, or the opposite?
The error I got for rocksdb is this:
{noformat}
Scanning dependencies of target build_version
[ 0%] Building CXX object
thirdparty/rocksdb/CMakeFiles/build_version.dir/__/__/build_version.cc.o
[ 0%] Built target build_version
Scanning dependencies of target rocksdb
[ 0%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/cache/lru_cache.cc.o
[ 0%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/cache/sharded_cache.cc.o
[ 0%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/cache/clock_cache.cc.o
[ 4%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/db/builder.cc.o
[ 4%] Building CXX object thirdparty/rocksdb/CMakeFiles/rocksdb.dir/db/c.cc.o
[ 4%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/db/column_family.cc.o
[ 4%] Building CXX object
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/db/compacted_db_impl.cc.o
/home/caleb/.backup/main/Projects/Coding/NiFi/nifi-minifi-cpp/thirdparty/rocksdb/cache/lru_cache.cc:
In constructor ‘rocksdb::LRUCache::LRUCache(size_t, int, bool, double)’:
/home/caleb/.backup/main/Projects/Coding/NiFi/nifi-minifi-cpp/thirdparty/rocksdb/cache/lru_cache.cc:464:42:
error: ‘new’ of type ‘rocksdb::LRUCacheShard’ with extended alignment 64
[-Werror=aligned-new=]
shards_ = new LRUCacheShard[num_shards_];
^
/home/caleb/.backup/main/Projects/Coding/NiFi/nifi-minifi-cpp/thirdparty/rocksdb/cache/lru_cache.cc:464:42:
note: uses ‘void* operator new [](std::size_t)’, which does not have an
alignment parameter
/home/caleb/.backup/main/Projects/Coding/NiFi/nifi-minifi-cpp/thirdparty/rocksdb/cache/lru_cache.cc:464:42:
note: use ‘-faligned-new’ to enable C++17 over-aligned new support
cc1plus: all warnings being treated as errors
make[3]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb.dir/build.make:87:
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/cache/lru_cache.cc.o] Error 1
make[3]: *** Waiting for unfinished jobs
make[2]: *** [CMakeFiles/Makefile2:3974:
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:3986:
thirdparty/rocksdb/CMakeFiles/rocksdb.dir/rule] Error 2
make: *** [Makefile:1484: rocksdb] Error 2
{noformat}
And for civetweb:
{noformat}
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c: In function
‘ssl_get_client_cert_info’:
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c:11848:9: error:
implicit declaration of function ‘i2c_ASN1_INTEGER’; did you mean
‘i2a_ASN1_INTEGER’? [-Wimplicit-function-declaration]
len = i2c_ASN1_INTEGER(serial, NULL);
^~~~
i2a_ASN1_INTEGER
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c: In function
‘set_ssl_option’:
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c:12129:36: warning:
conversion to ‘long unsigned int’ from ‘long int’ may change the sign of the
result [-Wsign-conversion]
SSL_CTX_set_options(ctx->ssl_ctx, ssl_get_protocol(protocol_ver));
^~~~
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c: In function
‘uninitialize_ssl’:
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c:12250:3: warning:
‘ERR_remove_state’ is deprecated [-Wdeprecated-declarations]
ERR_remove_state(0);
^~~~
In file included from /usr/include/openssl/ct.h:13:0,
from /usr/include/openssl/ssl.h:61,
from
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c:1245:
/usr/include/openssl/err.h:248:1: note: declared here
DEPRECATEDIN_1_0_0(void ERR_remove_state(unsigned long pid))
^
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c: In function
‘close_connection’:
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c:12505:3: warning:
‘ERR_remove_state’ is deprecated [-Wdeprecated-declarations]
ERR_remove_state(0);
^~~~
In file included from /usr/include/openssl/ct.h:13:0,
from /usr/include/openssl/ssl.h:61,
from
nifi-minifi-cpp/thirdparty/civetweb-1.9.1/src/civetweb.c
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211891#comment-16211891
]
marco polo commented on MINIFICPP-262:
--
[~achristianson] Can you provide some details for your centos 7 instance? cmake
version, installed rocksdb globally or within third party, etc?
I just spun one up in ec2 and can give you a key for it...it builds without an
issue using the branch that hosts PR150. I can merge that directly since it was
part of a formerly approved PR, but I'd like to verify via a consensus that
it's resolved. I suspect though that this does nothing to address the issue you
are encountering.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211673#comment-16211673
]
marco polo commented on MINIFICPP-262:
--
Cool. You may and should hit the issue in PR 150. We didn't merge all commits
from the initial PR, so I'm applying them individually. The lineage is a little
screwed up there, but I can't reproduce your issue. I can believe that it is
occurring though. I just can't replicate to test a fix. It's clearly trying to
load the static lib, which isn't compiled with fpic. I'm going to look at it
further and post a branch.
It seems that Cloud9 needed a CC, so it was a non starter for me.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (NIFI-4508) AMQP Processor that uses basicConsume
Randy Bovay created NIFI-4508: - Summary: AMQP Processor that uses basicConsume Key: NIFI-4508 URL: https://issues.apache.org/jira/browse/NIFI-4508 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.3.0 Reporter: Randy Bovay Due to poor performance of the AMQP Processor, we need to be able to have a basicConsume based interface to RabbitMQ. https://community.hortonworks.com/questions/66799/consumeamqp-performance-issue-less-than-50-msgs-se.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4507) consumeAMQP Expression Language support
Randy Bovay created NIFI-4507: - Summary: consumeAMQP Expression Language support Key: NIFI-4507 URL: https://issues.apache.org/jira/browse/NIFI-4507 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.3.0 Reporter: Randy Bovay The ConsumeAMQP Processors do not have ExpressionLanguage support. Would like to add this so that I can consume from multiple Queues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211615#comment-16211615
]
Caleb Johnson commented on MINIFICPP-262:
-
Sure, I'll spin up a vanilla Trusty VM and get back to you. I'm not sure about
the distro, but if you sign into Cloud9 you should be able to investigate and
repro.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFIREG-35) Implement a client for the REST API
[ https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211608#comment-16211608 ] ASF GitHub Bot commented on NIFIREG-35: --- Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/22 > Implement a client for the REST API > --- > > Key: NIFIREG-35 > URL: https://issues.apache.org/jira/browse/NIFIREG-35 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.0.1 > > > It would be helpful to offer a basic client for interacting with the REST API > and save everyone the work of setting up the plumbing for Jersey client or > some other library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (NIFIREG-35) Implement a client for the REST API
[ https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFIREG-35. Resolution: Fixed > Implement a client for the REST API > --- > > Key: NIFIREG-35 > URL: https://issues.apache.org/jira/browse/NIFIREG-35 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.0.1 > > > It would be helpful to offer a basic client for interacting with the REST API > and save everyone the work of setting up the plumbing for Jersey client or > some other library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry pull request #22: NIFIREG-35 Initial NiFi Registry Client
Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/22 ---
[jira] [Commented] (NIFI-4499) Cannot view content of flow file when NiFi is being accessed via a proxy
[ https://issues.apache.org/jira/browse/NIFI-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211603#comment-16211603 ] ASF GitHub Bot commented on NIFI-4499: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2217 Thanks @jtstorck! This has been merged to master. > Cannot view content of flow file when NiFi is being accessed via a proxy > > > Key: NIFI-4499 > URL: https://issues.apache.org/jira/browse/NIFI-4499 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.4.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > > When accessing the content viewer from the NiFi UI, while NiFi is being > proxied, the configured URL of the content viewer (in nifi.properties) will > result in a 404 when trying to view content. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (NIFI-4499) Cannot view content of flow file when NiFi is being accessed via a proxy
[ https://issues.apache.org/jira/browse/NIFI-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFI-4499. --- Resolution: Fixed Fix Version/s: 1.5.0 > Cannot view content of flow file when NiFi is being accessed via a proxy > > > Key: NIFI-4499 > URL: https://issues.apache.org/jira/browse/NIFI-4499 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.4.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.5.0 > > > When accessing the content viewer from the NiFi UI, while NiFi is being > proxied, the configured URL of the content viewer (in nifi.properties) will > result in a 404 when trying to view content. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4499) Cannot view content of flow file when NiFi is being accessed via a proxy
[ https://issues.apache.org/jira/browse/NIFI-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211598#comment-16211598 ] ASF GitHub Bot commented on NIFI-4499: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2217 Reviewing... > Cannot view content of flow file when NiFi is being accessed via a proxy > > > Key: NIFI-4499 > URL: https://issues.apache.org/jira/browse/NIFI-4499 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.4.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > > When accessing the content viewer from the NiFi UI, while NiFi is being > proxied, the configured URL of the content viewer (in nifi.properties) will > result in a 404 when trying to view content. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211601#comment-16211601
]
ASF GitHub Bot commented on NIFI-3248:
--
Github user JohannesDaniel commented on the issue:
https://github.com/apache/nifi/pull/2199
I dont think that the timezone and the commit issues are still important.
GetSolr now takes the timestamp directly from the results. Commit delays wont
be a problem as the state is only updated when new documents are retrieved, no
matter at which time the query was executed. The same applies to the timezone
issue.
> GetSolr can miss recently updated documents
> ---
>
> Key: NIFI-3248
> URL: https://issues.apache.org/jira/browse/NIFI-3248
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1,
> 1.0.1
>Reporter: Koji Kawamura
>Assignee: Johannes Peter
> Attachments: nifi-flow.png, query-result-with-curly-bracket.png,
> query-result-with-square-bracket.png
>
>
> GetSolr holds the last query timestamp so that it only fetches documents
> those have been added or updated since the last query.
> However, GetSolr misses some of those updated documents, and once the
> documents date field value becomes older than last query timestamp, the
> document won't be able to be queried by GetSolr any more.
> This JIRA is for tracking the process of investigating this behavior, and
> discussion on them.
> Here are things that can be a cause of this behavior:
> |#|Short description|Should we address it?|
> |1|Timestamp range filter, curly or square bracket?|No|
> |2|Timezone difference between update and query|Additional docs might be
> helpful|
> |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least,
> add 'commit lag-time'?|
> h2. 1. Timestamp range filter, curly or square bracket?
> At the first glance, using curly and square bracket in mix looked strange
> ([source
> code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]).
> But these difference has a meaning.
> The square bracket on the range query is inclusive and the curly bracket is
> exclusive. If we use inclusive on both sides and a document has a time stamp
> exactly on the boundary then it could be returned in two consecutive
> executions, and we only want it in one.
> This is intentional, and it should be as it is.
> h2. 2. Timezone difference between update and query
> Solr treats date fields as [UTC
> representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|].
> If date field String value of an updated document represents time without
> timezone, and NiFi is running on an environment using timezone other than
> UTC, GetSolr can't perform date range query as users expect.
> Let's say NiFi is running with JST(UTC+9). A process added a document to Solr
> at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it
> as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any
> documents updated from 15:00 to 15:10 JST. GetSolr formatted dates using UTC,
> i.e. 6:00 to 6:10 UTC. The updated document won't be matched with the date
> range filter.
> To avoid this, updated documents must have proper timezone in date field
> string representation.
> If one uses NiFi expression language to set current timestamp to that date
> field, following NiFi expression can be used:
> {code}
> ${now():format("-MM-dd'T'HH:mm:ss.SSSZ")}
> {code}
> It will produce a result like:
> {code}
> 2016-12-27T15:30:04.895+0900
> {code}
> Then it will be indexed in Solr with UTC and will be queried by GetSolr as
> expected.
> h2. 3. Lag comes from NearRealTIme nature of Solr
> Solr provides Near Real Time search capability, that means, the recently
> updated documents can be queried in Near Real Time, but it's not real time.
> This latency can be controlled by either on client side which requests the
> update operation by specifying "commitWithin" parameter, or on the Solr
> server side, "autoCommit" and "autoSoftCommit" in
> [solrconfig.xml|https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-Commits].
> Since commit and updating index can be costly, it's recommended to set this
> interval long enough up to the maximum tolerable latency.
> However, this can be problematic with GetSolr. For instance, as shown in the
> simple NiFi flow below, GetSolr can miss updated documents:
> {code}
> t1: GetSolr queried
> t2: GenerateFlowFile set date = t2
> t3: PutSolrContentStream stored new doc
> t4: GetSolr queried again, from t1 to t4, but
[jira] [Commented] (NIFI-4499) Cannot view content of flow file when NiFi is being accessed via a proxy
[ https://issues.apache.org/jira/browse/NIFI-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211604#comment-16211604 ] ASF GitHub Bot commented on NIFI-4499: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2217 > Cannot view content of flow file when NiFi is being accessed via a proxy > > > Key: NIFI-4499 > URL: https://issues.apache.org/jira/browse/NIFI-4499 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.4.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > > When accessing the content viewer from the NiFi UI, while NiFi is being > proxied, the configured URL of the content viewer (in nifi.properties) will > result in a 404 when trying to view content. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4499) Cannot view content of flow file when NiFi is being accessed via a proxy
[ https://issues.apache.org/jira/browse/NIFI-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211599#comment-16211599 ] ASF subversion and git services commented on NIFI-4499: --- Commit fb839fff5617a91b9d6c6fd4f43d78d1e12c560a in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=fb839ff ] NIFI-4499 Updated default content-viewer URL property to be relative rather than assuming it is deployed at the root context. This closes #2217 > Cannot view content of flow file when NiFi is being accessed via a proxy > > > Key: NIFI-4499 > URL: https://issues.apache.org/jira/browse/NIFI-4499 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.4.0 >Reporter: Jeff Storck >Assignee: Jeff Storck > > When accessing the content viewer from the NiFi UI, while NiFi is being > proxied, the configured URL of the content viewer (in nifi.properties) will > result in a 404 when trying to view content. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi pull request #2217: NIFI-4499 Updated default content-viewer URL proper...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2217 ---
[GitHub] nifi issue #2199: NIFI-3248: Improvement of GetSolr Processor
Github user JohannesDaniel commented on the issue: https://github.com/apache/nifi/pull/2199 I dont think that the timezone and the commit issues are still important. GetSolr now takes the timestamp directly from the results. Commit delays wont be a problem as the state is only updated when new documents are retrieved, no matter at which time the query was executed. The same applies to the timezone issue. ---
[GitHub] nifi issue #2217: NIFI-4499 Updated default content-viewer URL property to b...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2217 Thanks @jtstorck! This has been merged to master. ---
[GitHub] nifi issue #2217: NIFI-4499 Updated default content-viewer URL property to b...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2217 Reviewing... ---
[jira] [Commented] (NIFIREG-35) Implement a client for the REST API
[ https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211596#comment-16211596 ] ASF GitHub Bot commented on NIFIREG-35: --- Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/22 +1, I verified functionality after rebase. I'm good with that, thanks @bbende! > Implement a client for the REST API > --- > > Key: NIFIREG-35 > URL: https://issues.apache.org/jira/browse/NIFIREG-35 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.0.1 > > > It would be helpful to offer a basic client for interacting with the REST API > and save everyone the work of setting up the plumbing for Jersey client or > some other library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry issue #22: NIFIREG-35 Initial NiFi Registry Client
Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/22 +1, I verified functionality after rebase. I'm good with that, thanks @bbende! ---
[GitHub] nifi issue #2219: NiFi-4436: Add UI controls for starting/stopping/reverting...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2219 I am going to update this PR to indicate that it is a work in progress as merging will break the build because it depends on Registry artifacts. Will hold off on merging until we either release the Registry or update the documentation to clearly describe the build the dependencies. ---
[jira] [Commented] (MINIFICPP-262) Rocksdb fails to link
[
https://issues.apache.org/jira/browse/MINIFICPP-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211586#comment-16211586
]
marco polo commented on MINIFICPP-262:
--
Can you provide more information [~calebj] on your u14 distro? Have you seen
this on U14.04 vanilla? I am unable to reproduce there. There is a vestigial
issue from missed commits on the original ticket, so I've made another PR (
https://github.com/apache/nifi-minifi-cpp/pull/150 ), but that's not your
issue. I will try Cent7 now.
> Rocksdb fails to link
> -
>
> Key: MINIFICPP-262
> URL: https://issues.apache.org/jira/browse/MINIFICPP-262
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Andrew Christianson
>
> Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be
> having the same issue on Ubuntu 14.04 as part of MINIFI-244.
> {code}
> [ 38%] Linking CXX static library librocksdb.a
> [ 38%] Linking CXX shared library librocksdb.so
> [ 60%] Built target rocksdb
> [ 61%] Building CXX object
> CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
> [ 61%] Building CXX object
> CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
> /usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o:
> relocation R_X86_64_32 against `.data' can not be used when making a shared
> object; recompile with -fPIC
> CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
> Bad value
> collect2: error: ld returned 1 exit status
> gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
> gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
> gmake[1]: *** Waiting for unfinished jobs
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-113) Move from LevelDB to Rocks DB for all repositories.
[ https://issues.apache.org/jira/browse/MINIFICPP-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211581#comment-16211581 ] ASF GitHub Bot commented on MINIFICPP-113: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/150 MINIFI-372: Resolve issues with missed commits Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFI-372-FIX Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit d83aa8cd1d6087d3f908de5dcd3f19a16ad40dc2 Author: Marc Parisi Date: 2017-10-19T19:20:57Z MINIFI-372: Resolve issues with missed commits > Move from LevelDB to Rocks DB for all repositories. > > > Key: MINIFICPP-113 > URL: https://issues.apache.org/jira/browse/MINIFICPP-113 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: marco polo >Assignee: marco polo >Priority: Minor > > Can also be used as a file system repo where we want to minimize the number > of inodes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #150: MINIFI-372: Resolve issues with missed co...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/150 MINIFI-372: Resolve issues with missed commits Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFI-372-FIX Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit d83aa8cd1d6087d3f908de5dcd3f19a16ad40dc2 Author: Marc Parisi Date: 2017-10-19T19:20:57Z MINIFI-372: Resolve issues with missed commits ---
[jira] [Created] (NIFI-4506) Add date functions to Record Path
Bryan Bende created NIFI-4506:
-
Summary: Add date functions to Record Path
Key: NIFI-4506
URL: https://issues.apache.org/jira/browse/NIFI-4506
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.4.0, 1.3.0, 1.2.0
Reporter: Bryan Bende
Priority: Minor
We should support some date related functions in record path. At a minimum I
think having a format function like:
{code}
format( /someField, '-MM-dd', defaultValue)
{code}
The main use case for this is using PartitionRecord to partition by month, day,
or hour on a date field.
Currently you have treat the date as a string and use a sub-string operation to
get the part you are interested in, which also assumes the date is in a string
form in the first place.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFIREG-35) Implement a client for the REST API
[ https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211550#comment-16211550 ] ASF GitHub Bot commented on NIFIREG-35: --- Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/22 @kevdoran i rebased the code against master to resolve conflicts, i think for now we can leave the fields stuff and the interfaces alone and merge this in, and then maybe we can address either of them if needed, are you good with that? > Implement a client for the REST API > --- > > Key: NIFIREG-35 > URL: https://issues.apache.org/jira/browse/NIFIREG-35 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.0.1 > > > It would be helpful to offer a basic client for interacting with the REST API > and save everyone the work of setting up the plumbing for Jersey client or > some other library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry issue #22: NIFIREG-35 Initial NiFi Registry Client
Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/22 @kevdoran i rebased the code against master to resolve conflicts, i think for now we can leave the fields stuff and the interfaces alone and merge this in, and then maybe we can address either of them if needed, are you good with that? ---
[jira] [Created] (MINIFICPP-262) Rocksdb fails to link
Andrew Christianson created MINIFICPP-262:
-
Summary: Rocksdb fails to link
Key: MINIFICPP-262
URL: https://issues.apache.org/jira/browse/MINIFICPP-262
Project: NiFi MiNiFi C++
Issue Type: Bug
Reporter: Andrew Christianson
Rocksdb fails to link when building on CentOS 7.4. [~calebj] seems to be having
the same issue on Ubuntu 14.04 as part of MINIFI-244.
{code}
[ 38%] Linking CXX static library librocksdb.a
[ 38%] Linking CXX shared library librocksdb.so
[ 60%] Built target rocksdb
[ 61%] Building CXX object
CMakeFiles/Tests.dir/thirdparty/spdlog-20170710/include/spdlog/dummy.cpp.o
[ 61%] Building CXX object
CMakeFiles/ControllerServiceIntegrationTests.dir/libminifi/test/TestBase.cpp.o
/usr/bin/ld: CMakeFiles/build_version.dir/__/__/build_version.cc.o: relocation
R_X86_64_32 against `.data' can not be used when making a shared object;
recompile with -fPIC
CMakeFiles/build_version.dir/__/__/build_version.cc.o: error adding symbols:
Bad value
collect2: error: ld returned 1 exit status
gmake[2]: *** [thirdparty/rocksdb/librocksdb.so.5.7.0] Error 1
gmake[1]: *** [thirdparty/rocksdb/CMakeFiles/rocksdb-shared.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (NIFI-4505) MapCache/SimpleMaCache/PersistentMapCache: Add keyset method
Jon Kessler created NIFI-4505: - Summary: MapCache/SimpleMaCache/PersistentMapCache: Add keyset method Key: NIFI-4505 URL: https://issues.apache.org/jira/browse/NIFI-4505 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.4.0 Reporter: Jon Kessler Priority: Minor Suggest adding a keyset method to the MapCache and implementations as well as to any client/interface that make use of a MapCache. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4504) SimpleMapCache/PersistentMapCache: Add removeAndGet and removeByPatternAndGet
Jon Kessler created NIFI-4504: - Summary: SimpleMapCache/PersistentMapCache: Add removeAndGet and removeByPatternAndGet Key: NIFI-4504 URL: https://issues.apache.org/jira/browse/NIFI-4504 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.4.0 Reporter: Jon Kessler Priority: Minor Typical map implementations return the value that was removed when performing a remove. Because you couldn't update the existing remove methods without it being a breaking change I suggest adding new versions of the remove and removeByPattern methods that return the removed value(s). These changes should also be applied up the chain to any class that makes use of these classes such as the MapCacheServer and AtomicDistributedMapCacheClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFIREG-35) Implement a client for the REST API
[ https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211363#comment-16211363 ] ASF GitHub Bot commented on NIFIREG-35: --- Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/22 Thanks for the review! I am working on resolving the conflicts now. > Implement a client for the REST API > --- > > Key: NIFIREG-35 > URL: https://issues.apache.org/jira/browse/NIFIREG-35 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.0.1 > > > It would be helpful to offer a basic client for interacting with the REST API > and save everyone the work of setting up the plumbing for Jersey client or > some other library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry issue #22: NIFIREG-35 Initial NiFi Registry Client
Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/22 Thanks for the review! I am working on resolving the conflicts now. ---
[jira] [Commented] (NIFIREG-35) Implement a client for the REST API
[
https://issues.apache.org/jira/browse/NIFIREG-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211358#comment-16211358
]
ASF GitHub Bot commented on NIFIREG-35:
---
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/22#discussion_r145762243
--- Diff:
nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/FlowResource.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import org.apache.nifi.registry.authorization.Authorizer;
+import org.apache.nifi.registry.field.Fields;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.Set;
+
+@Component
+@Path("/flows")
+@Api(
+value = "/flows",
+description = "Gets metadata about flows."
+)
+public class FlowResource extends AuthorizableApplicationResource {
+
+private final RegistryService registryService;
+
+@Autowired
+public FlowResource(
+final RegistryService registryService,
+final AuthorizationService authorizationService,
+final Authorizer authorizer) {
+super(authorizer, authorizationService);
+this.registryService = registryService;
+}
+
+@GET
+@Path("fields")
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Retrieves the available field names that can be used
for searching or sorting on flows.",
+response = Fields.class
+)
+public Response getAvailableFlowFields() {
+final Set flowFields = registryService.getFlowFields();
+final Fields fields = new Fields(flowFields);
+return Response.status(Response.Status.OK).entity(fields).build();
+}
+
+}
--- End diff --
I like the idea of possibly combining them all into one call under
/fields... the way it was currently done was simply to align with the resource
type being returned by the REST end-point, but maybe that is not the easiest to
maintain.
The reason I wanted to make it dynamic was because I envisioned someone
later modifying the DB entities and having no idea that there were these
end-points that returned field names, and now either fields are missing, or
fields are present that were removed, etc.
A different option I considered was not having any of the /fields
end-points and relying on the names in the entities returned from the REST API,
but since the DB and REST layer are using different objects we would then need
to maintain a mapping of field names between the two, which also seems error
prone.
I'm kind of torn because it feels awkward right now to return entities with
certain fields, and then say oh sorry these are the actual names you can
search/sort on. Let me keep thinking about it.
> Implement a client for the REST API
> ---
>
> Key: NIFIREG-35
> URL: https://issues.apache.org/jira/browse/NIFIREG-35
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 0.0.1
>
>
> It would be helpful to offer a basic client for interacting with the REST API
> and save everyone the work of setting up the plumbing for Jersey client or
> some other library.
[jira] [Commented] (MINIFICPP-261) Extensions not added to docker/minificppsource, causing make docker to fail
[ https://issues.apache.org/jira/browse/MINIFICPP-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211357#comment-16211357 ] ASF GitHub Bot commented on MINIFICPP-261: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/149 > Extensions not added to docker/minificppsource, causing make docker to fail > --- > > Key: MINIFICPP-261 > URL: https://issues.apache.org/jira/browse/MINIFICPP-261 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Andrew Christianson >Assignee: Andrew Christianson > > We're not adding the extensions/ dir to docker/minificppsource/, which > CMakeLists.txt references, so docker builds are failing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-registry pull request #22: NIFIREG-35 Initial NiFi Registry Client
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/22#discussion_r145762243
--- Diff:
nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/FlowResource.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import org.apache.nifi.registry.authorization.Authorizer;
+import org.apache.nifi.registry.field.Fields;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.Set;
+
+@Component
+@Path("/flows")
+@Api(
+value = "/flows",
+description = "Gets metadata about flows."
+)
+public class FlowResource extends AuthorizableApplicationResource {
+
+private final RegistryService registryService;
+
+@Autowired
+public FlowResource(
+final RegistryService registryService,
+final AuthorizationService authorizationService,
+final Authorizer authorizer) {
+super(authorizer, authorizationService);
+this.registryService = registryService;
+}
+
+@GET
+@Path("fields")
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Retrieves the available field names that can be used
for searching or sorting on flows.",
+response = Fields.class
+)
+public Response getAvailableFlowFields() {
+final Set flowFields = registryService.getFlowFields();
+final Fields fields = new Fields(flowFields);
+return Response.status(Response.Status.OK).entity(fields).build();
+}
+
+}
--- End diff --
I like the idea of possibly combining them all into one call under
/fields... the way it was currently done was simply to align with the resource
type being returned by the REST end-point, but maybe that is not the easiest to
maintain.
The reason I wanted to make it dynamic was because I envisioned someone
later modifying the DB entities and having no idea that there were these
end-points that returned field names, and now either fields are missing, or
fields are present that were removed, etc.
A different option I considered was not having any of the /fields
end-points and relying on the names in the entities returned from the REST API,
but since the DB and REST layer are using different objects we would then need
to maintain a mapping of field names between the two, which also seems error
prone.
I'm kind of torn because it feels awkward right now to return entities with
certain fields, and then say oh sorry these are the actual names you can
search/sort on. Let me keep thinking about it.
---
[GitHub] nifi-minifi-cpp pull request #149: MINIFICPP-261 Copy extensions when copyin...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/149 ---
[jira] [Commented] (MINIFICPP-261) Extensions not added to docker/minificppsource, causing make docker to fail
[ https://issues.apache.org/jira/browse/MINIFICPP-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211318#comment-16211318 ] ASF GitHub Bot commented on MINIFICPP-261: -- GitHub user achristianson opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/149 MINIFICPP-261 Copy extensions when copying minificppsource for docker… … build Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-261 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 commit ce2d9580c4ee18e88f1d7525fcbc0519f4cfb1de Author: Andrew I. Christianson Date: 2017-10-19T16:33:42Z MINIFICPP-261 Copy extensions when copying minificppsource for docker build > Extensions not added to docker/minificppsource, causing make docker to fail > --- > > Key: MINIFICPP-261 > URL: https://issues.apache.org/jira/browse/MINIFICPP-261 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Andrew Christianson >Assignee: Andrew Christianson > > We're not adding the extensions/ dir to docker/minificppsource/, which > CMakeLists.txt references, so docker builds are failing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #149: MINIFICPP-261 Copy extensions when copyin...
GitHub user achristianson opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/149 MINIFICPP-261 Copy extensions when copying minificppsource for docker⦠⦠build Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-261 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 commit ce2d9580c4ee18e88f1d7525fcbc0519f4cfb1de Author: Andrew I. Christianson Date: 2017-10-19T16:33:42Z MINIFICPP-261 Copy extensions when copying minificppsource for docker build ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211295#comment-16211295 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj pls clarify with @phrocker if the stream-based export is truly necessary at this point. The greater concern may be ensuring that the stash resource claims are properly tracked by the flow file repo (rocksdb). I *believe* that would require adding a bit of code to FlowFileRecord's Serialize/Deserialize methods, but would like @phrocker to confirm since he's more knowledgeable on the core repos than I am. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 @calebj pls clarify with @phrocker if the stream-based export is truly necessary at this point. The greater concern may be ensuring that the stash resource claims are properly tracked by the flow file repo (rocksdb). I *believe* that would require adding a bit of code to FlowFileRecord's Serialize/Deserialize methods, but would like @phrocker to confirm since he's more knowledgeable on the core repos than I am. ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211285#comment-16211285
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145748583
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public
io::Serializable {
return content_full_fath_;
}
+ /**
+ * Cleanly relinquish a resource claim
+ */
--- End diff --
The flowfile is releasing its own claim to the resource. There are a few
cases where the flow file has a claim to a resource that it should release to
any other components that might have a claim on the resource (which might
currently be a hypothetical-only case).
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145748583
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public
io::Serializable {
return content_full_fath_;
}
+ /**
+ * Cleanly relinquish a resource claim
+ */
--- End diff --
The flowfile is releasing its own claim to the resource. There are a few
cases where the flow file has a claim to a resource that it should release to
any other components that might have a claim on the resource (which might
currently be a hypothetical-only case).
---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211283#comment-16211283
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747665
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public
io::Serializable {
return content_full_fath_;
}
+ /**
+ * Cleanly relinquish a resource claim
+ */
--- End diff --
Clarified in the latest commit
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211281#comment-16211281 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747588 --- Diff: libminifi/include/core/FlowConfiguration.h --- @@ -35,6 +35,8 @@ #include "processors/ExecuteProcess.h" #include "processors/AppendHostInfo.h" #include "processors/MergeContent.h" +#include "processors/FocusArchiveEntry.h" --- End diff -- Clarified in the latest commit > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747588 --- Diff: libminifi/include/core/FlowConfiguration.h --- @@ -35,6 +35,8 @@ #include "processors/ExecuteProcess.h" #include "processors/AppendHostInfo.h" #include "processors/MergeContent.h" +#include "processors/FocusArchiveEntry.h" --- End diff -- Clarified in the latest commit ---
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145747665
--- Diff: libminifi/include/FlowFileRecord.h ---
@@ -164,6 +164,11 @@ class FlowFileRecord : public core::FlowFile, public
io::Serializable {
return content_full_fath_;
}
+ /**
+ * Cleanly relinquish a resource claim
+ */
--- End diff --
Clarified in the latest commit
---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211270#comment-16211270 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 Things left to look into: * [ ] Move things which depend on libarchive to an extension * [ ] Use streams for exportContent * [ ] Move exportContent ReadCallback to another file > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211268#comment-16211268
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145745217
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743863
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
+ uuid_generate(stashKeyUuid);
+ uuid_unparse_lower(stashKeyUuid, stashKey);
+ logger_->log_debug(
+ "FocusArchiveEntry generated stash key %s for entry %s",
+ stashKey,
+ entryMetadata.entryName.c_str());
+ entryMetadata.stashKey.assign(stashKey);
+
+ if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+ }
+
+ // Stash the content
+ session->stash(entryMetadata.stashKey, flowFile);
+}
+ }
+
+ // Restore target archive entry
+ if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+ } else {
+logger_->log_warn(
+ "FocusArchiveEntry failed to locate target entry: %s",
+ targetEntry.c_str());
+ }
+
+ // Set new/updated lens stack to attribute
+ {
+Json::Value lensStack;
+Json::Reader reader;
+
+std::string existingLensStack;
+
+if (flowFile->getAttribute("lens.archive.stack", existingLensStack
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211247#comment-16211247 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743845 --- Diff: LICENSE --- @@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +This projects includes libarchive bundle (https://www.libarchive.org) +which is available under a BSD License by Tim Kientzle and others + --- End diff -- Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211260#comment-16211260 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744366 --- Diff: libminifi/include/core/ProcessSession.h --- @@ -19,6 +19,7 @@ #define __PROCESS_SESSION_H__ #include +#include --- End diff -- Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744191
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
+
+//! FocusArchiveEntry Class
+class FocusArchiveEntry : public core::Processor {
+ public:
+ //! Constructor
+ /*!
+ * Create a new processor
+ */
+ explicit FocusArchiveEntry(std::string name, uuid_t uuid = NULL)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+ }
+ //! Destructor
+ virtual ~FocusArchiveEntry() {
+ }
+ //! Processor Name
+ static constexpr char const* ProcessorName = "FocusArchiveEntry";
+ //! Supported Properties
+ static core::Property Path;
+ //! Supported Relationships
+ static core::Relationship Success;
+
+ bool set_del_or_update_attr(std::shared_ptr, const
std::string, std::string*) const;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
---
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743112
--- Diff: libminifi/src/core/ProcessSession.cpp ---
@@ -799,6 +799,152 @@ void ProcessSession::import(std::string source,
std::shared_ptr
}
}
+bool ProcessSession::exportContent(
+const std::string &destination,
+const std::string &tmpFile,
+std::shared_ptr &flow,
+bool keepContent) {
+ logger_->log_info(
+ "Exporting content of %s to %s",
+ flow->getUUIDStr().c_str(),
+ destination.c_str());
+
+ ReadCallback cb(tmpFile, destination, logger_);
+ read(flow, &cb);
+
+ logger_->log_info("Committing %s", destination.c_str());
+ bool commit_ok = cb.commit();
+
+ if (commit_ok) {
+logger_->log_info("Commit OK.");
+ } else {
+logger_->log_error(
+ "Commit of %s to %s failed!",
+ flow->getUUIDStr().c_str(),
+ destination.c_str());
+ }
+ return commit_ok;
+}
+
+bool ProcessSession::exportContent(
+const std::string &destination,
+std::shared_ptr &flow,
+bool keepContent) {
+ std::string tmpFileName = boost::filesystem::unique_path().native();
+ return exportContent(destination, tmpFileName, flow, keepContent);
+}
+
+ProcessSession::ReadCallback::ReadCallback(const std::string &tmpFile,
+ const std::string &destFile,
+
std::shared_ptr logger)
+: _tmpFile(tmpFile),
+ _tmpFileOs(tmpFile, std::ios::binary),
+ _destFile(destFile),
+ logger_(logger) {
+}
+
+// Copy the entire file contents to the temporary file
+int64_t
ProcessSession::ReadCallback::process(std::shared_ptr stream) {
+ // Copy file contents into tmp file
+ _writeSucceeded = false;
+ size_t size = 0;
+ uint8_t buffer[8192];
+ do {
+int read = stream->read(buffer, 8192);
+if (read < 0) {
+ return -1;
+}
+if (read == 0) {
+ break;
+}
+_tmpFileOs.write(reinterpret_cast(buffer), read);
+size += read;
+ } while (size < stream->getSize());
+ _writeSucceeded = true;
+ return size;
+}
+
+// Renames tmp file to final destination
+// Returns true if commit succeeded
+bool ProcessSession::ReadCallback::commit() {
+ bool success = false;
+
+ logger_->log_info("committing export operation to %s",
_destFile.c_str());
+
+ if (_writeSucceeded) {
+_tmpFileOs.close();
+
+if (rename(_tmpFile.c_str(), _destFile.c_str())) {
+ logger_->log_info("commit export operation to %s failed because
rename() call failed", _destFile.c_str());
+} else {
+ success = true;
+ logger_->log_info("commit export operation to %s succeeded",
_destFile.c_str());
+}
+ } else {
+logger_->log_error("commit export operation to %s failed because write
failed", _destFile.c_str());
+ }
+ return success;
+}
+
+// Clean up resources
+ProcessSession::ReadCallback::~ReadCallback() {
+ // Close tmp file
+ _tmpFileOs.close();
+
+ // Clean up tmp file, if necessary
+ unlink(_tmpFile.c_str());
+}
+
+
+void ProcessSession::stash(const std::string &key,
std::shared_ptr flow) {
--- End diff --
@phrocker by 'tmp file,' are you referring to the stash claims?
We would want those to operate the same as the primary content claim. It
looks like this may not be the case currently. Would adding the stash claims to
the data stored/retrieved in FlowFileRecord's Serialize/Deserialize cover all
the bases, or would other changes be required as well?
---
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743845 --- Diff: LICENSE --- @@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +This projects includes libarchive bundle (https://www.libarchive.org) +which is available under a BSD License by Tim Kientzle and others + --- End diff -- Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211259#comment-16211259
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744343
--- Diff: libminifi/include/core/FlowFile.h ---
@@ -50,6 +50,32 @@ class FlowFile : public core::Connectable {
void clearResourceClaim();
/**
+ * Returns a pointer to this flow file record's
+ * claim at the given stash key
+ */
+ std::shared_ptr getStashClaim(const std::string &key);
+
+ /**
+ * Sets the given stash key to the inbound claim argument
+ */
+ void setStashClaim(const std::string &key,
std::shared_ptr &claim);
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211251#comment-16211251
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743902
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working wi
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744366 --- Diff: libminifi/include/core/ProcessSession.h --- @@ -19,6 +19,7 @@ #define __PROCESS_SESSION_H__ #include +#include --- End diff -- Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211238#comment-16211238 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145742734 --- Diff: LICENSE --- @@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +This projects includes libarchive bundle (https://www.libarchive.org) +which is available under a BSD License by Tim Kientzle and others + +Copyright (c) 2003-2009 Tim Kientzle and other authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer + in this position and unchanged. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR +IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. (END LICENSE TEXT) + +All libarchive C sources (including .c and .h files) +and documentation files are subject to the copyright notice reproduced +above. + +This libarchive includes below files +libarchive/archive_entry.c +libarchive/archive_read_support_filter_compress.c +libarchive/archive_write_add_filter_compress.c +which under a 3-clause UC Regents copyright as below +/*- + * Copyright (c) 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + *notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + *notice, this list of conditions and the following disclaimer in the + *documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors --- End diff -- Neither. It's the 3-clause UC Regents copyright, which is in the three aforementioned source files. The originals also skip the third list item. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message w
[GitHub] nifi-minifi-cpp issue #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/148 Things left to look into: * [ ] Move things which depend on libarchive to an extension * [ ] Use streams for exportContent * [ ] Move exportContent ReadCallback to another file ---
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145745217
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211261#comment-16211261
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145734644
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --
I would opt to just not use the using statement here. Not worth breaking
the convention for the one class.
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211262#comment-16211262
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743112
--- Diff: libminifi/src/core/ProcessSession.cpp ---
@@ -799,6 +799,152 @@ void ProcessSession::import(std::string source,
std::shared_ptr
}
}
+bool ProcessSession::exportContent(
+const std::string &destination,
+const std::string &tmpFile,
+std::shared_ptr &flow,
+bool keepContent) {
+ logger_->log_info(
+ "Exporting content of %s to %s",
+ flow->getUUIDStr().c_str(),
+ destination.c_str());
+
+ ReadCallback cb(tmpFile, destination, logger_);
+ read(flow, &cb);
+
+ logger_->log_info("Committing %s", destination.c_str());
+ bool commit_ok = cb.commit();
+
+ if (commit_ok) {
+logger_->log_info("Commit OK.");
+ } else {
+logger_->log_error(
+ "Commit of %s to %s failed!",
+ flow->getUUIDStr().c_str(),
+ destination.c_str());
+ }
+ return commit_ok;
+}
+
+bool ProcessSession::exportContent(
+const std::string &destination,
+std::shared_ptr &flow,
+bool keepContent) {
+ std::string tmpFileName = boost::filesystem::unique_path().native();
+ return exportContent(destination, tmpFileName, flow, keepContent);
+}
+
+ProcessSession::ReadCallback::ReadCallback(const std::string &tmpFile,
+ const std::string &destFile,
+
std::shared_ptr logger)
+: _tmpFile(tmpFile),
+ _tmpFileOs(tmpFile, std::ios::binary),
+ _destFile(destFile),
+ logger_(logger) {
+}
+
+// Copy the entire file contents to the temporary file
+int64_t
ProcessSession::ReadCallback::process(std::shared_ptr stream) {
+ // Copy file contents into tmp file
+ _writeSucceeded = false;
+ size_t size = 0;
+ uint8_t buffer[8192];
+ do {
+int read = stream->read(buffer, 8192);
+if (read < 0) {
+ return -1;
+}
+if (read == 0) {
+ break;
+}
+_tmpFileOs.write(reinterpret_cast(buffer), read);
+size += read;
+ } while (size < stream->getSize());
+ _writeSucceeded = true;
+ return size;
+}
+
+// Renames tmp file to final destination
+// Returns true if commit succeeded
+bool ProcessSession::ReadCallback::commit() {
+ bool success = false;
+
+ logger_->log_info("committing export operation to %s",
_destFile.c_str());
+
+ if (_writeSucceeded) {
+_tmpFileOs.close();
+
+if (rename(_tmpFile.c_str(), _destFile.c_str())) {
+ logger_->log_info("commit export operation to %s failed because
rename() call failed", _destFile.c_str());
+} else {
+ success = true;
+ logger_->log_info("commit export operation to %s succeeded",
_destFile.c_str());
+}
+ } else {
+logger_->log_error("commit export operation to %s failed because write
failed", _destFile.c_str());
+ }
+ return success;
+}
+
+// Clean up resources
+ProcessSession::ReadCallback::~ReadCallback() {
+ // Close tmp file
+ _tmpFileOs.close();
+
+ // Clean up tmp file, if necessary
+ unlink(_tmpFile.c_str());
+}
+
+
+void ProcessSession::stash(const std::string &key,
std::shared_ptr flow) {
--- End diff --
@phrocker by 'tmp file,' are you referring to the stash claims?
We would want those to operate the same as the primary content claim. It
looks like this may not be the case currently. Would adding the stash claims to
the data stored/retrieved in FlowFileRecord's Serialize/Deserialize cover all
the bases, or would other changes be required as well?
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re w
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145734644
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
--- End diff --
I would opt to just not use the using statement here. Not worth breaking
the convention for the one class.
---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211255#comment-16211255
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744191
--- Diff: libminifi/include/processors/FocusArchiveEntry.h ---
@@ -0,0 +1,115 @@
+/**
+ * @file FocusArchiveEntry.h
+ * FocusArchiveEntry class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+#define LIBMINIFI_INCLUDE_PROCESSORS_FOCUSARCHIVEENTRY_H_
+
+#include
+#include
+#include
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Core.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+using logging::LoggerFactory;
+
+//! FocusArchiveEntry Class
+class FocusArchiveEntry : public core::Processor {
+ public:
+ //! Constructor
+ /*!
+ * Create a new processor
+ */
+ explicit FocusArchiveEntry(std::string name, uuid_t uuid = NULL)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+ }
+ //! Destructor
+ virtual ~FocusArchiveEntry() {
+ }
+ //! Processor Name
+ static constexpr char const* ProcessorName = "FocusArchiveEntry";
+ //! Supported Properties
+ static core::Property Path;
+ //! Supported Relationships
+ static core::Relationship Success;
+
+ bool set_del_or_update_attr(std::shared_ptr, const
std::string, std::string*) const;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145744343
--- Diff: libminifi/include/core/FlowFile.h ---
@@ -50,6 +50,32 @@ class FlowFile : public core::Connectable {
void clearResourceClaim();
/**
+ * Returns a pointer to this flow file record's
+ * claim at the given stash key
+ */
+ std::shared_ptr getStashClaim(const std::string &key);
+
+ /**
+ * Sets the given stash key to the inbound claim argument
+ */
+ void setStashClaim(const std::string &key,
std::shared_ptr &claim);
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211250#comment-16211250
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743879
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
+ uuid_generate(stashKeyUuid);
+ uuid_unparse_lower(stashKeyUuid, stashKey);
+ logger_->log_debug(
+ "FocusArchiveEntry generated stash key %s for entry %s",
+ stashKey,
+ entryMetadata.entryName.c_str());
+ entryMetadata.stashKey.assign(stashKey);
+
+ if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+ }
+
+ // Stash the content
+ session->stash(entryMetadata.stashKey, flowFile);
+}
+ }
+
+ // Restore target archive entry
+ if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+ } else {
+logger_->log_warn(
+ "FocusArchiveEntry failed to locate target entry: %s",
+ targetEntry.c_str());
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211248#comment-16211248
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743863
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
+ uuid_generate(stashKeyUuid);
+ uuid_unparse_lower(stashKeyUuid, stashKey);
+ logger_->log_debug(
+ "FocusArchiveEntry generated stash key %s for entry %s",
+ stashKey,
+ entryMetadata.entryName.c_str());
+ entryMetadata.stashKey.assign(stashKey);
+
+ if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+ }
+
+ // Stash the content
+ session->stash(entryMetadata.stashKey, flowFile);
+}
+ }
+
+ // Restore target archive entry
+ if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+ } else {
+logger_->log_warn(
+ "FocusArchiveEntry failed to locate target entry: %s",
+ targetEntry.c_str());
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743879
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
+ uuid_generate(stashKeyUuid);
+ uuid_unparse_lower(stashKeyUuid, stashKey);
+ logger_->log_debug(
+ "FocusArchiveEntry generated stash key %s for entry %s",
+ stashKey,
+ entryMetadata.entryName.c_str());
+ entryMetadata.stashKey.assign(stashKey);
+
+ if (entryMetadata.entryName == targetEntry) {
+targetEntryStashKey = entryMetadata.stashKey;
+ }
+
+ // Stash the content
+ session->stash(entryMetadata.stashKey, flowFile);
+}
+ }
+
+ // Restore target archive entry
+ if (targetEntryStashKey != "") {
+session->restore(targetEntryStashKey, flowFile);
+ } else {
+logger_->log_warn(
+ "FocusArchiveEntry failed to locate target entry: %s",
+ targetEntry.c_str());
+ }
+
+ // Set new/updated lens stack to attribute
+ {
+Json::Value lensStack;
+Json::Reader reader;
+
+std::string existingLensStack;
+
+if (flowFile->getAttribute("lens.archive.stack", existingLensStack
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145743902
--- Diff: libminifi/src/processors/FocusArchiveEntry.cpp ---
@@ -0,0 +1,340 @@
+/**
+ * @file FocusArchiveEntry.cpp
+ * FocusArchiveEntry class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "processors/FocusArchiveEntry.h"
+
+#include
+#include
+
+#include
+
+#include
+
+#include
+#include
+
+#include
+#include
+#include
+
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+
+#include "json/json.h"
+#include "json/writer.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property FocusArchiveEntry::Path(
+"Path",
+"The path within the archive to focus (\"/\" to focus the total
archive)",
+"");
+core::Relationship FocusArchiveEntry::Success(
+"success",
+"success operational on the flow record");
+
+bool
FocusArchiveEntry::set_del_or_update_attr(std::shared_ptr
flowFile, const std::string key, std::string* value) const {
+ if (value == nullptr)
+return flowFile->removeAttribute(key);
+ else if (flowFile->updateAttribute(key, *value))
+return true;
+ else
+return flowFile->addAttribute(key, *value);
+}
+
+void FocusArchiveEntry::initialize() {
+ //! Set the supported properties
+ std::set properties;
+ properties.insert(Path);
+ setSupportedProperties(properties);
+ //! Set the supported relationships
+ std::set relationships;
+ relationships.insert(Success);
+ setSupportedRelationships(relationships);
+}
+
+void FocusArchiveEntry::onTrigger(core::ProcessContext *context,
+ core::ProcessSession *session) {
+ auto flowFile = session->get();
+ std::shared_ptr flowFileRecord =
std::static_pointer_cast(flowFile);
+
+ if (!flowFile) {
+return;
+ }
+
+ std::string targetEntry;
+ context->getProperty(Path.getName(), targetEntry);
+
+ // Extract archive contents
+ ArchiveMetadata archiveMetadata;
+ archiveMetadata.focusedEntry = targetEntry;
+ ReadCallback cb(&archiveMetadata);
+ session->read(flowFile, &cb);
+
+ // For each extracted entry, import & stash to key
+ std::string targetEntryStashKey;
+
+ for (auto &entryMetadata : archiveMetadata.entryMetadata) {
+if (entryMetadata.entryType == AE_IFREG) {
+ logger_->log_info("FocusArchiveEntry importing %s from %s",
+ entryMetadata.entryName.c_str(),
+ entryMetadata.tmpFileName.c_str());
+ session->import(entryMetadata.tmpFileName, flowFile, false, 0);
+ char stashKey[37];
+ uuid_t stashKeyUuid;
--- End diff --
Fixed in d2e7e34ab8b331ac484b9b16bd51455799a1502b
---
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145742734 --- Diff: LICENSE --- @@ -534,4 +534,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +This projects includes libarchive bundle (https://www.libarchive.org) +which is available under a BSD License by Tim Kientzle and others + +Copyright (c) 2003-2009 Tim Kientzle and other authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer + in this position and unchanged. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR +IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. (END LICENSE TEXT) + +All libarchive C sources (including .c and .h files) +and documentation files are subject to the copyright notice reproduced +above. + +This libarchive includes below files +libarchive/archive_entry.c +libarchive/archive_read_support_filter_compress.c +libarchive/archive_write_add_filter_compress.c +which under a 3-clause UC Regents copyright as below +/*- + * Copyright (c) 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + *notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + *notice, this list of conditions and the following disclaimer in the + *documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors --- End diff -- Neither. It's the 3-clause UC Regents copyright, which is in the three aforementioned source files. The originals also skip the third list item. ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211226#comment-16211226 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145740328 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- @calebj I think @achristianson point is valid that it shouldn't be done here. I will submit a PR to the other PR and then when that is merged ( should be the next day ), you will get that for free via the rebase. So feel free to ignore any comment about "extensions" since that is on me. thanks! > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145740328 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- @calebj I think @achristianson point is valid that it shouldn't be done here. I will submit a PR to the other PR and then when that is merged ( should be the next day ), you will get that for free via the rebase. So feel free to ignore any comment about "extensions" since that is on me. thanks! ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211221#comment-16211221 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145738276 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- I'm not sure how to do that, are there any examples? > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user calebj commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145738276 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- I'm not sure how to do that, are there any examples? ---
[jira] [Commented] (NIFI-3950) Separate AWS ControllerService API
[
https://issues.apache.org/jira/browse/NIFI-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211206#comment-16211206
]
ASF GitHub Bot commented on NIFI-3950:
--
Github user christophercurrie commented on the issue:
https://github.com/apache/nifi/pull/2140
Ok, I've added those files. Let me know if they look OK and if there's
anything else that needs doing.
> Separate AWS ControllerService API
> --
>
> Key: NIFI-3950
> URL: https://issues.apache.org/jira/browse/NIFI-3950
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: James Wing
>Priority: Minor
>
> The nifi-aws-bundle currently contains the interface for the
> AWSCredentialsProviderService as well as the service implementation, and
> dependent abstract classes and processor classes.
> This results in the following warning logged as NiFi loads:
> {quote}
> org.apache.nifi.nar.ExtensionManager Component
> org.apache.nifi.processors.aws.s3.PutS3Object is bundled with its referenced
> Controller Service APIs
> org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService.
> The service APIs should not be bundled with component implementations that
> reference it.
> {quote}
> Some [discussion of this issue and potential solutions occurred on the dev
> list|http://apache-nifi.1125220.n5.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-td17038.html].
> We also need a migration plan in addition to the new structure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi issue #2140: NIFI-3950 Refactor AWS bundle
Github user christophercurrie commented on the issue: https://github.com/apache/nifi/pull/2140 Ok, I've added those files. Let me know if they look OK and if there's anything else that needs doing. ---
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211184#comment-16211184
]
ASF GitHub Bot commented on MINIFICPP-39:
-
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145731149
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
bool keepSource,
uint64_t offset, char inputDelimiter);
+ /**
+ * Exports the data stream to a file
+ * @param string file to export stream to
+ * @param flow flow file
+ * @param bool whether or not to keep the content in the flow file
+ */
+ bool exportContent(const std::string &destination,
--- End diff --
Export is simply meant to be the inverse of import.
Digging into the code, export is used in this change set in
UnfocusArchiveEntry in order to export the flow file content into a
scratch/working location so that it can be re-assembled back into an archive.
We're ultimately calling archive_write_data (line 286 of
UnfocusArchiveEntry.cpp), which takes data from a byte buffer. Therefore, we
don't necessarily require a persistent filesystem for this change, as this
scratch/working area could be in RAM or some other medium as long as there's
enough space.
As it stands, these new archive processors do depend on persistent
filesystem storage, but this addition of exportContent does not result in
ProcessSession or any core component depending on any storage implementation
where it previously did not. We're simply adding an mirror capability to import
which is optional to use, and where the caller is responsible for environmental
considerations/requirements such as persistent storage.
Assuming we move these new archive processors into an extension, a current
prerequisite of using that extension will be persistent filesystem storage.
This should be documented. This leaves open the door to future
implementations/improvements which use RAM or some other medium to reconstitute
the archive, but I think that level of functionality is not required
immediately.
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user achristianson commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145731149
--- Diff: libminifi/include/core/ProcessSession.h ---
@@ -151,11 +152,47 @@ class ProcessSession {
bool keepSource,
uint64_t offset, char inputDelimiter);
+ /**
+ * Exports the data stream to a file
+ * @param string file to export stream to
+ * @param flow flow file
+ * @param bool whether or not to keep the content in the flow file
+ */
+ bool exportContent(const std::string &destination,
--- End diff --
Export is simply meant to be the inverse of import.
Digging into the code, export is used in this change set in
UnfocusArchiveEntry in order to export the flow file content into a
scratch/working location so that it can be re-assembled back into an archive.
We're ultimately calling archive_write_data (line 286 of
UnfocusArchiveEntry.cpp), which takes data from a byte buffer. Therefore, we
don't necessarily require a persistent filesystem for this change, as this
scratch/working area could be in RAM or some other medium as long as there's
enough space.
As it stands, these new archive processors do depend on persistent
filesystem storage, but this addition of exportContent does not result in
ProcessSession or any core component depending on any storage implementation
where it previously did not. We're simply adding an mirror capability to import
which is optional to use, and where the caller is responsible for environmental
considerations/requirements such as persistent storage.
Assuming we move these new archive processors into an extension, a current
prerequisite of using that extension will be persistent filesystem storage.
This should be documented. This leaves open the door to future
implementations/improvements which use RAM or some other medium to reconstitute
the archive, but I think that level of functionality is not required
immediately.
---
[jira] [Commented] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211171#comment-16211171
]
ASF GitHub Bot commented on NIFI-3248:
--
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145727845
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -170,159 +210,213 @@ protected void init(final
ProcessorInitializationContext context) {
return this.descriptors;
}
+final static Set propertyNamesForActivatingClearState = new
HashSet();
+static {
+propertyNamesForActivatingClearState.add(SOLR_TYPE.getName());
+propertyNamesForActivatingClearState.add(SOLR_LOCATION.getName());
+propertyNamesForActivatingClearState.add(COLLECTION.getName());
+propertyNamesForActivatingClearState.add(SOLR_QUERY.getName());
+propertyNamesForActivatingClearState.add(DATE_FIELD.getName());
+propertyNamesForActivatingClearState.add(RETURN_FIELDS.getName());
+}
+
@Override
public void onPropertyModified(PropertyDescriptor descriptor, String
oldValue, String newValue) {
-lastEndDatedRef.set(UNINITIALIZED_LAST_END_DATE_VALUE);
+if
(propertyNamesForActivatingClearState.contains(descriptor.getName()))
+clearState.set(true);
}
-@OnStopped
-public void onStopped() {
-writeLastEndDate();
-}
+@OnScheduled
+public void clearState(final ProcessContext context) throws
IOException {
+if (clearState.getAndSet(false)) {
+context.getStateManager().clear(Scope.CLUSTER);
+final Map newStateMap = new
HashMap();
-@OnRemoved
-public void onRemoved() {
-final File lastEndDateCache = new File(FILE_PREFIX +
getIdentifier());
-if (lastEndDateCache.exists()) {
-lastEndDateCache.delete();
-}
-}
+newStateMap.put(STATE_MANAGER_CURSOR_MARK, "*");
-@Override
-public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
-final ComponentLog logger = getLogger();
-readLastEndDate();
-
-final SimpleDateFormat sdf = new
SimpleDateFormat(LAST_END_DATE_PATTERN, Locale.US);
-sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
-final String currDate = sdf.format(new Date());
-
-final boolean initialized =
!UNINITIALIZED_LAST_END_DATE_VALUE.equals(lastEndDatedRef.get());
-
-final String query = context.getProperty(SOLR_QUERY).getValue();
-final SolrQuery solrQuery = new SolrQuery(query);
-solrQuery.setRows(context.getProperty(BATCH_SIZE).asInteger());
-
-// if initialized then apply a filter to restrict results from the
last end time til now
-if (initialized) {
-StringBuilder filterQuery = new StringBuilder();
-filterQuery.append(context.getProperty(DATE_FIELD).getValue())
-.append(":{").append(lastEndDatedRef.get()).append("
TO ")
-.append(currDate).append("]");
-solrQuery.addFilterQuery(filterQuery.toString());
-logger.info("Applying filter query {}", new
Object[]{filterQuery.toString()});
-}
+final String initialDate =
context.getProperty(DATE_FILTER).getValue();
+if (StringUtils.isBlank(initialDate))
+newStateMap.put(STATE_MANAGER_FILTER, "*");
+else
+newStateMap.put(STATE_MANAGER_FILTER, initialDate);
-final String returnFields =
context.getProperty(RETURN_FIELDS).getValue();
-if (returnFields != null && !returnFields.trim().isEmpty()) {
-for (String returnField : returnFields.trim().split("[,]")) {
-solrQuery.addField(returnField.trim());
-}
+context.getStateManager().setState(newStateMap, Scope.CLUSTER);
+
+id_field = null;
}
+}
-final String fullSortClause =
context.getProperty(SORT_CLAUSE).getValue();
-if (fullSortClause != null && !fullSortClause.trim().isEmpty()) {
-for (String sortClause : fullSortClause.split("[,]")) {
-String[] sortParts = sortClause.trim().split("[ ]");
-solrQuery.addSort(sortParts[0],
SolrQuery.ORDER.valueOf(sortParts[1]));
-}
+@Override
+protected final Collection
additionalCustomValidation(ValidationC
[GitHub] nifi pull request #2199: NIFI-3248: Improvement of GetSolr Processor
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145727845
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -170,159 +210,213 @@ protected void init(final
ProcessorInitializationContext context) {
return this.descriptors;
}
+final static Set propertyNamesForActivatingClearState = new
HashSet();
+static {
+propertyNamesForActivatingClearState.add(SOLR_TYPE.getName());
+propertyNamesForActivatingClearState.add(SOLR_LOCATION.getName());
+propertyNamesForActivatingClearState.add(COLLECTION.getName());
+propertyNamesForActivatingClearState.add(SOLR_QUERY.getName());
+propertyNamesForActivatingClearState.add(DATE_FIELD.getName());
+propertyNamesForActivatingClearState.add(RETURN_FIELDS.getName());
+}
+
@Override
public void onPropertyModified(PropertyDescriptor descriptor, String
oldValue, String newValue) {
-lastEndDatedRef.set(UNINITIALIZED_LAST_END_DATE_VALUE);
+if
(propertyNamesForActivatingClearState.contains(descriptor.getName()))
+clearState.set(true);
}
-@OnStopped
-public void onStopped() {
-writeLastEndDate();
-}
+@OnScheduled
+public void clearState(final ProcessContext context) throws
IOException {
+if (clearState.getAndSet(false)) {
+context.getStateManager().clear(Scope.CLUSTER);
+final Map newStateMap = new
HashMap();
-@OnRemoved
-public void onRemoved() {
-final File lastEndDateCache = new File(FILE_PREFIX +
getIdentifier());
-if (lastEndDateCache.exists()) {
-lastEndDateCache.delete();
-}
-}
+newStateMap.put(STATE_MANAGER_CURSOR_MARK, "*");
-@Override
-public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
-final ComponentLog logger = getLogger();
-readLastEndDate();
-
-final SimpleDateFormat sdf = new
SimpleDateFormat(LAST_END_DATE_PATTERN, Locale.US);
-sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
-final String currDate = sdf.format(new Date());
-
-final boolean initialized =
!UNINITIALIZED_LAST_END_DATE_VALUE.equals(lastEndDatedRef.get());
-
-final String query = context.getProperty(SOLR_QUERY).getValue();
-final SolrQuery solrQuery = new SolrQuery(query);
-solrQuery.setRows(context.getProperty(BATCH_SIZE).asInteger());
-
-// if initialized then apply a filter to restrict results from the
last end time til now
-if (initialized) {
-StringBuilder filterQuery = new StringBuilder();
-filterQuery.append(context.getProperty(DATE_FIELD).getValue())
-.append(":{").append(lastEndDatedRef.get()).append("
TO ")
-.append(currDate).append("]");
-solrQuery.addFilterQuery(filterQuery.toString());
-logger.info("Applying filter query {}", new
Object[]{filterQuery.toString()});
-}
+final String initialDate =
context.getProperty(DATE_FILTER).getValue();
+if (StringUtils.isBlank(initialDate))
+newStateMap.put(STATE_MANAGER_FILTER, "*");
+else
+newStateMap.put(STATE_MANAGER_FILTER, initialDate);
-final String returnFields =
context.getProperty(RETURN_FIELDS).getValue();
-if (returnFields != null && !returnFields.trim().isEmpty()) {
-for (String returnField : returnFields.trim().split("[,]")) {
-solrQuery.addField(returnField.trim());
-}
+context.getStateManager().setState(newStateMap, Scope.CLUSTER);
+
+id_field = null;
}
+}
-final String fullSortClause =
context.getProperty(SORT_CLAUSE).getValue();
-if (fullSortClause != null && !fullSortClause.trim().isEmpty()) {
-for (String sortClause : fullSortClause.split("[,]")) {
-String[] sortParts = sortClause.trim().split("[ ]");
-solrQuery.addSort(sortParts[0],
SolrQuery.ORDER.valueOf(sortParts[1]));
-}
+@Override
+protected final Collection
additionalCustomValidation(ValidationContext context) {
+final Collection problems = new ArrayList<>();
+
+if
(context.getProperty(RETURN_TYPE).evaluateAttributeExpressions().getValue().equals(MODE_REC.getValue())
+&& !context.getProperty(REC
[jira] [Commented] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211163#comment-16211163 ] ASF GitHub Bot commented on MINIFICPP-39: - Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145724061 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- Great! We'll work on getting the other PR in an extension and this can be rebased. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi-minifi-cpp pull request #148: MINIFI-244 Un/FocusArchive processors
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/148#discussion_r145724061 --- Diff: CMakeLists.txt --- @@ -101,6 +101,7 @@ set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL libr set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library") add_subdirectory(thirdparty/yaml-cpp-yaml-cpp-0.5.3) add_subdirectory(thirdparty/civetweb-1.9.1 EXCLUDE_FROM_ALL) +add_subdirectory(thirdparty/libarchive-3.3.2) --- End diff -- Great! We'll work on getting the other PR in an extension and this can be rebased. ---
[jira] [Created] (NIFIREG-40) Create UI coasters for notifications
Scott Aslan created NIFIREG-40: -- Summary: Create UI coasters for notifications Key: NIFIREG-40 URL: https://issues.apache.org/jira/browse/NIFIREG-40 Project: NiFi Registry Issue Type: Sub-task Reporter: Scott Aslan -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (NIFIREG-40) Create UI coasters for notifications
[ https://issues.apache.org/jira/browse/NIFIREG-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan reassigned NIFIREG-40: -- Assignee: Scott Aslan > Create UI coasters for notifications > > > Key: NIFIREG-40 > URL: https://issues.apache.org/jira/browse/NIFIREG-40 > Project: NiFi Registry > Issue Type: Sub-task >Reporter: Scott Aslan >Assignee: Scott Aslan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211151#comment-16211151
]
ASF GitHub Bot commented on NIFI-3248:
--
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145721674
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -66,42 +79,72 @@
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CursorMarkParams;
-@Tags({"Apache", "Solr", "Get", "Pull"})
+@Tags({"Apache", "Solr", "Get", "Pull", "Records"})
@InputRequirement(Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("Queries Solr and outputs the results as a
FlowFile")
+@CapabilityDescription("Queries Solr and outputs the results as a FlowFile
in the format of XML or using a Record Writer")
+@Stateful(scopes = {Scope.CLUSTER}, description = "Stores latest date of
Date Field so that the same data will not be fetched multiple times.")
public class GetSolr extends SolrProcessor {
-public static final PropertyDescriptor SOLR_QUERY = new
PropertyDescriptor
-.Builder().name("Solr Query")
-.description("A query to execute against Solr")
+public static final String STATE_MANAGER_FILTER =
"stateManager_filter";
+public static final String STATE_MANAGER_CURSOR_MARK =
"stateManager_cursorMark";
+public static final AllowableValue MODE_XML = new
AllowableValue("XML");
+public static final AllowableValue MODE_REC = new
AllowableValue("Records");
+
+public static final PropertyDescriptor RETURN_TYPE = new
PropertyDescriptor
+.Builder().name("Return Type")
+.displayName("Return Type")
--- End diff --
The most properties were already available in the prior GetSol processor. I
expected this to be critical for backwards compatibility. For the new
properties I chose the same naming pattern.
> GetSolr can miss recently updated documents
> ---
>
> Key: NIFI-3248
> URL: https://issues.apache.org/jira/browse/NIFI-3248
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1,
> 1.0.1
>Reporter: Koji Kawamura
>Assignee: Johannes Peter
> Attachments: nifi-flow.png, query-result-with-curly-bracket.png,
> query-result-with-square-bracket.png
>
>
> GetSolr holds the last query timestamp so that it only fetches documents
> those have been added or updated since the last query.
> However, GetSolr misses some of those updated documents, and once the
> documents date field value becomes older than last query timestamp, the
> document won't be able to be queried by GetSolr any more.
> This JIRA is for tracking the process of investigating this behavior, and
> discussion on them.
> Here are things that can be a cause of this behavior:
> |#|Short description|Should we address it?|
> |1|Timestamp range filter, curly or square bracket?|No|
> |2|Timezone difference between update and query|Additional docs might be
> helpful|
> |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least,
> add 'commit lag-time'?|
> h2. 1. Timestamp range filter, curly or square bracket?
> At the first glance, using curly and square bracket in mix looked strange
> ([source
> code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]).
> But these difference has a meaning.
> The square bracket on the range query is inclusive and the curly bracket is
> exclusive. If we use inclusive on both sides and a document has a time stamp
> exactly on the boundary then it could be returned in two consecutive
> executions, and we only want it in one.
> This is intentional, and it should be as it is.
> h2. 2. Timezone difference between update and query
> Solr treats date fields as [UTC
> representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|].
> If date field String value of an updated document represents time without
> timezone, and NiFi is running on an environment using timezone other than
> UTC, GetSolr can't perform date range query as users expect.
> Let's say NiFi is running with JST(UTC+9). A process added a document to Solr
> at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it
> as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any
> docume
[GitHub] nifi pull request #2199: NIFI-3248: Improvement of GetSolr Processor
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145721674
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -66,42 +79,72 @@
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CursorMarkParams;
-@Tags({"Apache", "Solr", "Get", "Pull"})
+@Tags({"Apache", "Solr", "Get", "Pull", "Records"})
@InputRequirement(Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("Queries Solr and outputs the results as a
FlowFile")
+@CapabilityDescription("Queries Solr and outputs the results as a FlowFile
in the format of XML or using a Record Writer")
+@Stateful(scopes = {Scope.CLUSTER}, description = "Stores latest date of
Date Field so that the same data will not be fetched multiple times.")
public class GetSolr extends SolrProcessor {
-public static final PropertyDescriptor SOLR_QUERY = new
PropertyDescriptor
-.Builder().name("Solr Query")
-.description("A query to execute against Solr")
+public static final String STATE_MANAGER_FILTER =
"stateManager_filter";
+public static final String STATE_MANAGER_CURSOR_MARK =
"stateManager_cursorMark";
+public static final AllowableValue MODE_XML = new
AllowableValue("XML");
+public static final AllowableValue MODE_REC = new
AllowableValue("Records");
+
+public static final PropertyDescriptor RETURN_TYPE = new
PropertyDescriptor
+.Builder().name("Return Type")
+.displayName("Return Type")
--- End diff --
The most properties were already available in the prior GetSol processor. I
expected this to be critical for backwards compatibility. For the new
properties I chose the same naming pattern.
---
[jira] [Commented] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211144#comment-16211144
]
ASF GitHub Bot commented on NIFI-3248:
--
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145720938
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -66,42 +79,72 @@
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CursorMarkParams;
-@Tags({"Apache", "Solr", "Get", "Pull"})
+@Tags({"Apache", "Solr", "Get", "Pull", "Records"})
@InputRequirement(Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("Queries Solr and outputs the results as a
FlowFile")
+@CapabilityDescription("Queries Solr and outputs the results as a FlowFile
in the format of XML or using a Record Writer")
+@Stateful(scopes = {Scope.CLUSTER}, description = "Stores latest date of
Date Field so that the same data will not be fetched multiple times.")
public class GetSolr extends SolrProcessor {
-public static final PropertyDescriptor SOLR_QUERY = new
PropertyDescriptor
-.Builder().name("Solr Query")
-.description("A query to execute against Solr")
+public static final String STATE_MANAGER_FILTER =
"stateManager_filter";
+public static final String STATE_MANAGER_CURSOR_MARK =
"stateManager_cursorMark";
+public static final AllowableValue MODE_XML = new
AllowableValue("XML");
+public static final AllowableValue MODE_REC = new
AllowableValue("Records");
--- End diff --
Additionally, this requires parsing of response json, as the response
parsing of Schema API is not really realized in SolrJ
> GetSolr can miss recently updated documents
> ---
>
> Key: NIFI-3248
> URL: https://issues.apache.org/jira/browse/NIFI-3248
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1,
> 1.0.1
>Reporter: Koji Kawamura
>Assignee: Johannes Peter
> Attachments: nifi-flow.png, query-result-with-curly-bracket.png,
> query-result-with-square-bracket.png
>
>
> GetSolr holds the last query timestamp so that it only fetches documents
> those have been added or updated since the last query.
> However, GetSolr misses some of those updated documents, and once the
> documents date field value becomes older than last query timestamp, the
> document won't be able to be queried by GetSolr any more.
> This JIRA is for tracking the process of investigating this behavior, and
> discussion on them.
> Here are things that can be a cause of this behavior:
> |#|Short description|Should we address it?|
> |1|Timestamp range filter, curly or square bracket?|No|
> |2|Timezone difference between update and query|Additional docs might be
> helpful|
> |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least,
> add 'commit lag-time'?|
> h2. 1. Timestamp range filter, curly or square bracket?
> At the first glance, using curly and square bracket in mix looked strange
> ([source
> code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]).
> But these difference has a meaning.
> The square bracket on the range query is inclusive and the curly bracket is
> exclusive. If we use inclusive on both sides and a document has a time stamp
> exactly on the boundary then it could be returned in two consecutive
> executions, and we only want it in one.
> This is intentional, and it should be as it is.
> h2. 2. Timezone difference between update and query
> Solr treats date fields as [UTC
> representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|].
> If date field String value of an updated document represents time without
> timezone, and NiFi is running on an environment using timezone other than
> UTC, GetSolr can't perform date range query as users expect.
> Let's say NiFi is running with JST(UTC+9). A process added a document to Solr
> at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it
> as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any
> documents updated from 15:00 to 15:10 JST. GetSolr formatted dates using UTC,
> i.e. 6:00 to 6:10 UTC. The updated document won't be matched with the date
> range filter.
> To avoid this, updated documents must have proper timezone in date field
> string
[GitHub] nifi pull request #2199: NIFI-3248: Improvement of GetSolr Processor
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2199#discussion_r145720938
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java
---
@@ -66,42 +79,72 @@
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CursorMarkParams;
-@Tags({"Apache", "Solr", "Get", "Pull"})
+@Tags({"Apache", "Solr", "Get", "Pull", "Records"})
@InputRequirement(Requirement.INPUT_FORBIDDEN)
-@CapabilityDescription("Queries Solr and outputs the results as a
FlowFile")
+@CapabilityDescription("Queries Solr and outputs the results as a FlowFile
in the format of XML or using a Record Writer")
+@Stateful(scopes = {Scope.CLUSTER}, description = "Stores latest date of
Date Field so that the same data will not be fetched multiple times.")
public class GetSolr extends SolrProcessor {
-public static final PropertyDescriptor SOLR_QUERY = new
PropertyDescriptor
-.Builder().name("Solr Query")
-.description("A query to execute against Solr")
+public static final String STATE_MANAGER_FILTER =
"stateManager_filter";
+public static final String STATE_MANAGER_CURSOR_MARK =
"stateManager_cursorMark";
+public static final AllowableValue MODE_XML = new
AllowableValue("XML");
+public static final AllowableValue MODE_REC = new
AllowableValue("Records");
--- End diff --
Additionally, this requires parsing of response json, as the response
parsing of Schema API is not really realized in SolrJ
---
[jira] [Created] (NIFI-4503) Connection to support load-balancing strategies
Haimo Liu created NIFI-4503: --- Summary: Connection to support load-balancing strategies Key: NIFI-4503 URL: https://issues.apache.org/jira/browse/NIFI-4503 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Haimo Liu As an operator, I want to be able to create new list/fetch flows encapsulated within a single process group, that automatically distribute the fetch operations across the nodes in my NiFi cluster, so that I can manage each flow independently from one another and without the need for orchestration with remote process groups. It would be great to add the ability for on any given connection have a user be able to select ‘auto balance across cluster’ and it will automatically take care of distributing the objects across the cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
