[GitHub] [nifi] ottobackwards commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable
ottobackwards commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable URL: https://github.com/apache/nifi/pull/4148#discussion_r394745002 ## File path: nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/JsonPathEvaluator.java ## @@ -46,7 +47,14 @@ public JsonPathEvaluator(final Evaluator subject, final Evaluator
[GitHub] [nifi] mkazia opened a new pull request #4152: NIFI-7269 Upgrade solrj version to 7 in nifi-solr-processors
mkazia opened a new pull request #4152: NIFI-7269 Upgrade solrj version to 7 in
nifi-solr-processors
URL: https://github.com/apache/nifi/pull/4152
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Upgrade solrj version to 7 in nifi-solr-processors; fixes bug NIFI-7269._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [x] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable
MikeThomsen commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable URL: https://github.com/apache/nifi/pull/4148#discussion_r394726108 ## File path: nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/JsonPathEvaluator.java ## @@ -46,7 +47,14 @@ public JsonPathEvaluator(final Evaluator subject, final Evaluator
[jira] [Created] (NIFI-7269) Solrj in nifi-solr-nar needs upgrade
Mubashir Kazia created NIFI-7269:
Summary: Solrj in nifi-solr-nar needs upgrade
Key: NIFI-7269
URL: https://issues.apache.org/jira/browse/NIFI-7269
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.11.3
Environment: Solr 7.4 configured with Kerberos and LDAP(Basic)
Authentication
Reporter: Mubashir Kazia
If Solr is configured with Kerberos (SPNEGO) and LDAP (Basic) Authentication
then using Kerberos Authentication on any of the Solr processors like QuerySolr
or PutSolrContent fails with error
{{java.lang.IllegalStateException: Unsupported authentication scheme: Basic}}
This can be attributed to the solrj version 6.6.6 used in the nifi-solr-nar.
If the solrj version is upgraded to 7.7.2, this error no longer appears.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-6293) MongoDB Aggregation does not parse dates
[
https://issues.apache.org/jira/browse/NIFI-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen updated NIFI-6293:
---
Fix Version/s: 1.12.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> MongoDB Aggregation does not parse dates
>
>
> Key: NIFI-6293
> URL: https://issues.apache.org/jira/browse/NIFI-6293
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.2
> Environment: Ubuntu 16.04, MongoDB 4.0.3, TLS to MongoDB.
>Reporter: Emil Sarkissian
>Assignee: Eduardo Mota Fontes
>Priority: Major
> Labels: easyfix, newbie, pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> *Repro Steps*:
> # Create a "Generate Flow File" Processor to kick off things
> # Connect that to a Nifi MongoDB Aggregation Processor
> # Set the query to a $match with date criteria, for example:
> {code:java}
> [ { "$match": { "someDate": { "$gt": "2019-01-01T00:00:00Z" } } } ]{code}
>
> *Expected Result*: Query returns correct documents per the date criteria.
> *Actual Result*: Condition always evaluates to false and no documents are
> returned.
> *Notes:*
> # I have tried every possible way to make this work, including \{ "$date":
> "2019-01-01T00:00:00Z" } and also using "$expr" with Mongo, as well as
> "$dateFromString", and "$convert".
> # Using an "$expr" seems to work functionally but refuses to use any indexes
> - probably a MongoDB issue (4.0.3 is what I run) and introduces unnecessary
> complexity.
> I have fixed this for our fork, as we need to go to Production soon, but I am
> new to the contribution process and didn't want to just put a feature branch
> without explaining the issue.
> The fix is to provide a Jackson custom deserializer to handle dates based on
> what Date Format is specified on the processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-6293) MongoDB Aggregation does not parse dates
[
https://issues.apache.org/jira/browse/NIFI-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062100#comment-17062100
]
ASF subversion and git services commented on NIFI-6293:
---
Commit 943de310ad5b5fa6c128882da84c88621abd69b3 in nifi's branch
refs/heads/master from Eduardo Fontes
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=943de31 ]
NIFI-6293
Add support to Mongo Extended JSON v2
Add org.json lib
Replace evil json
Replace evil json for alternative
Include testExtendedJsonSupport
Style adjustment
Remove unecessary new JSON parser
Fix query in testExtendedJsonSupport
Parse with Jackson and BSON
Back to default MONGO_URI
This closes #4068
Signed-off-by: Mike Thomsen
> MongoDB Aggregation does not parse dates
>
>
> Key: NIFI-6293
> URL: https://issues.apache.org/jira/browse/NIFI-6293
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.2
> Environment: Ubuntu 16.04, MongoDB 4.0.3, TLS to MongoDB.
>Reporter: Emil Sarkissian
>Assignee: Eduardo Mota Fontes
>Priority: Major
> Labels: easyfix, newbie, pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> *Repro Steps*:
> # Create a "Generate Flow File" Processor to kick off things
> # Connect that to a Nifi MongoDB Aggregation Processor
> # Set the query to a $match with date criteria, for example:
> {code:java}
> [ { "$match": { "someDate": { "$gt": "2019-01-01T00:00:00Z" } } } ]{code}
>
> *Expected Result*: Query returns correct documents per the date criteria.
> *Actual Result*: Condition always evaluates to false and no documents are
> returned.
> *Notes:*
> # I have tried every possible way to make this work, including \{ "$date":
> "2019-01-01T00:00:00Z" } and also using "$expr" with Mongo, as well as
> "$dateFromString", and "$convert".
> # Using an "$expr" seems to work functionally but refuses to use any indexes
> - probably a MongoDB issue (4.0.3 is what I run) and introduces unnecessary
> complexity.
> I have fixed this for our fork, as we need to go to Production soon, but I am
> new to the contribution process and didn't want to just put a feature branch
> without explaining the issue.
> The fix is to provide a Jackson custom deserializer to handle dates based on
> what Date Format is specified on the processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4068: NIFI-6293 - Add support to Mongo Extended JSON to RunMongoAggregation Query
asfgit closed pull request #4068: NIFI-6293 - Add support to Mongo Extended JSON to RunMongoAggregation Query URL: https://github.com/apache/nifi/pull/4068 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[
https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062056#comment-17062056
]
Manuel Loayza commented on NIFI-7266:
-
[~joewitt]
Thanks a lot for checking on that.
I see a lot of these exceptions:
{code:java}
2020-03-18 21:09:20,729 WARN [Process Cluster Protocol Request-20]
o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from
ip-207-50-126.dqa.capitalone.com due to
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:221)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:133)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:314)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:269)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:219)
... 5 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:299)
... 7 common frames omitted{code}
And a post in stackoverflow says something that we already have
[https://stackoverflow.com/questions/43151807/secured-nifi-cluster-setup]
{{}}
{{}}
{code:java}
nifi.security.needClientAuth=true
nifi.cluster.protocol.is.secure=true {code}
That is curious, because the cluster seems to recover the 1st time after 15
minutes, and the 2nd time it recovered after 45 minutes.
> NIFI 1.4.0 gets unresponsive after heavy load
> -
>
> Key: NIFI-7266
> URL: https://issues.apache.org/jira/browse/NIFI-7266
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.2.0, 1.3.0, 1.4.0
>Reporter: Manuel Loayza
>Priority: Trivial
> Attachments: NIFI_PERF_LOG_1.log, Screen Shot 2020-03-17 at 3.18.27
> PM.png, image-2020-03-18-16-03-49-351.png, image-2020-03-18-16-35-49-452.png
>
>
> We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121
> + Linux CentOS
> The traffic get divided between those 2 clusters:
> 1. TPS: 2700 - EAST cluster
> 2. TPS: 980. - WEST cluster
> We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster
> with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it
> web console got unresponsive.
> I've tried many things to fix this thing, but only thing I got was to
> increase the time from 4 to 6 hours before it fails
> Our current instances are running on AWS and each EC2 instances has 8 cpus
> (c5.2xlarge), and 16GB RAM.
> I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the
> same outcome.
> I don't have a clue to figure it out what the issue is. Also I have a
> datadog dashboard to track some java head metrics but everything looks normal.
> What should I do to find why those new better instances are failing? is it
> memory or disk space or threads got stuck? Why an old NIFI cluster conf
> works better than a new NIFI?
> Hope you can help me with this.
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[
https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062056#comment-17062056
]
Manuel Loayza edited comment on NIFI-7266 at 3/18/20, 9:18 PM:
---
[~joewitt]
Thanks a lot for checking on that.
I see a lot of these exceptions:
{code:java}
2020-03-18 21:09:20,729 WARN [Process Cluster Protocol Request-20]
o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from
ip-207-50-126.dqa.capitalone.com due to
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:221)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:133)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:314)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:269)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:219)
... 5 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:299)
... 7 common frames omitted{code}
And a post in stackoverflow says something that we already have
[https://stackoverflow.com/questions/43151807/secured-nifi-cluster-setup]
{code:java}
nifi.security.needClientAuth=true
nifi.cluster.protocol.is.secure=true {code}
That is curious, because the cluster seems to recover the 1st time after 15
minutes, and the 2nd time it recovered after 45 minutes.
was (Author: mloayzag):
[~joewitt]
Thanks a lot for checking on that.
I see a lot of these exceptions:
{code:java}
2020-03-18 21:09:20,729 WARN [Process Cluster Protocol Request-20]
o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from
ip-207-50-126.dqa.capitalone.com due to
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
org.apache.nifi.cluster.protocol.ProtocolException:
java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:221)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:133)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:314)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:269)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:219)
... 5 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
at
org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:299)
... 7 common frames omitted{code}
And a post in stackoverflow says something that we already have
[https://stackoverflow.com/questions/43151807/secured-nifi-cluster-setup]
{{}}
{{}}
{code:java}
nifi.security.needClientAuth=true
nifi.cluster.protocol.is.secure=true {code}
That is curious, because the cluster seems to recover the 1st time after 15
minutes, and the 2nd time it recovered after 45 minutes.
> NIFI 1.4.0 gets unresponsive after heavy load
> -
>
> Key:
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062050#comment-17062050 ] Manuel Loayza commented on NIFI-7266: - Here some logs that I got when nifi cluster started failing [^NIFI_PERF_LOG_1.log] > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: NIFI_PERF_LOG_1.log, Screen Shot 2020-03-17 at 3.18.27 > PM.png, image-2020-03-18-16-03-49-351.png, image-2020-03-18-16-35-49-452.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r394634612
##
File path:
extensions/rocksdb-repos/controllers/RocksDbPersistableKeyValueStoreService.cpp
##
@@ -0,0 +1,200 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RocksDbPersistableKeyValueStoreService.h"
+
+#include "utils/StringUtils.h"
+
+#include
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace controllers {
+
+core::Property RocksDbPersistableKeyValueStoreService::Directory(
+core::PropertyBuilder::createProperty("Directory")->withDescription("Path
to a directory for the database")
+->isRequired(true)->build());
+
+RocksDbPersistableKeyValueStoreService::RocksDbPersistableKeyValueStoreService(const
std::string& name, const std::string& id)
+: controllers::KeyValueStoreService(name, id)
+, controllers::AbstractAutoPersistingKeyValueStoreService(name, id)
+, db_valid_(false)
+,
logger_(logging::LoggerFactory::getLogger())
{
+}
+
+RocksDbPersistableKeyValueStoreService::RocksDbPersistableKeyValueStoreService(const
std::string& name, utils::Identifier uuid /*= utils::Identifier()*/)
+: controllers::KeyValueStoreService(name, uuid)
+, controllers::AbstractAutoPersistingKeyValueStoreService(name, uuid)
+, db_valid_(false)
+,
logger_(logging::LoggerFactory::getLogger())
{
+}
+
+RocksDbPersistableKeyValueStoreService::~RocksDbPersistableKeyValueStoreService()
{
+ if (db_valid_) {
+delete db_;
+ }
+}
+
+void RocksDbPersistableKeyValueStoreService::initialize() {
+ AbstractAutoPersistingKeyValueStoreService::initialize();
+ std::set supportedProperties;
+ supportedProperties.insert(Directory);
+ updateSupportedProperties(supportedProperties);
+}
+
+void RocksDbPersistableKeyValueStoreService::onEnable() {
+ if (configuration_ == nullptr) {
+logger_->log_debug("Cannot enable RocksDbPersistableKeyValueStoreService");
+return;
+ }
+
+ AbstractAutoPersistingKeyValueStoreService::onEnable();
+
+ if (!getProperty(Directory.getName(), directory_)) {
+logger_->log_error("Invalid or missing property: Directory");
+return;
+ }
+
+ if (db_valid_) {
+db_valid_ = false;
+delete db_;
+ }
+ rocksdb::Options options;
+ options.create_if_missing = true;
+ if (!always_persist_) {
+options.manual_wal_flush = true;
+ }
+ rocksdb::Status status = rocksdb::DB::Open(options, directory_, _);
+ if (status.ok()) {
+db_valid_ = true;
+logger_->log_trace("Successfully opened RocksDB database at %s",
directory_.c_str());
+ } else {
+logger_->log_error("Failed to open RocksDB database at %s, error: %s",
directory_.c_str(), status.getState());
+return;
+ }
+
+ if (always_persist_) {
+default_write_options.sync = true;
+ }
+
+ logger_->log_trace("Enabled RocksDbPersistableKeyValueStoreService");
+}
+
+void RocksDbPersistableKeyValueStoreService::notifyStop() {
+ AbstractAutoPersistingKeyValueStoreService::notifyStop();
+
+ if (db_valid_) {
+delete db_;
+db_valid_= false;
+ }
+}
+
+bool RocksDbPersistableKeyValueStoreService::set(const std::string& key, const
std::string& value) {
+ if (!db_valid_) {
Review comment:
Replaced with unique_ptr, so no longer relevant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r394634269
##
File path:
extensions/rocksdb-repos/controllers/RocksDbPersistableKeyValueStoreService.h
##
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef
LIBMINIFI_INCLUDE_CONTROLLERS_KEYVALUE_RocksDbPersistableKeyValueStoreService_H_
+#define
LIBMINIFI_INCLUDE_CONTROLLERS_KEYVALUE_RocksDbPersistableKeyValueStoreService_H_
+
+#include "controllers/keyvalue/AbstractAutoPersistingKeyValueStoreService.h"
+#include "core/Core.h"
+#include "properties/Configure.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+
+#include "rocksdb/db.h"
+#include "rocksdb/options.h"
+#include "rocksdb/slice.h"
+
+#include
+#include
+#include
+#include
+#include
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace controllers {
+
+class RocksDbPersistableKeyValueStoreService : public
AbstractAutoPersistingKeyValueStoreService {
+ public:
+ explicit RocksDbPersistableKeyValueStoreService(const std::string& name,
const std::string& id);
+ explicit RocksDbPersistableKeyValueStoreService(const std::string& name,
utils::Identifier uuid = utils::Identifier());
+
+ virtual ~RocksDbPersistableKeyValueStoreService();
+
+ static core::Property Directory;
+
+ virtual void initialize() override;
+ virtual void onEnable() override;
+ virtual void notifyStop() override;
+
+ virtual bool set(const std::string& key, const std::string& value) override;
+
+ virtual bool get(const std::string& key, std::string& value) override;
+
+ virtual bool get(std::unordered_map& kvs) override;
+
+ virtual bool remove(const std::string& key) override;
+
+ virtual bool clear() override;
+
+ virtual bool update(const std::string& key, const std::function& update_func) override;
+
+ virtual bool persist() override;
+
+ protected:
+ std::string directory_;
+
+ rocksdb::DB* db_;
+ rocksdb::WriteOptions default_write_options;
+ bool db_valid_;
Review comment:
It was a remnant from the UnorderedMap version (which was implemented before
this). Replaced along with using a unique_ptr.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r394634441
##
File path:
extensions/rocksdb-repos/controllers/RocksDbPersistableKeyValueStoreService.cpp
##
@@ -0,0 +1,200 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RocksDbPersistableKeyValueStoreService.h"
+
+#include "utils/StringUtils.h"
+
+#include
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace controllers {
+
+core::Property RocksDbPersistableKeyValueStoreService::Directory(
+core::PropertyBuilder::createProperty("Directory")->withDescription("Path
to a directory for the database")
+->isRequired(true)->build());
+
+RocksDbPersistableKeyValueStoreService::RocksDbPersistableKeyValueStoreService(const
std::string& name, const std::string& id)
+: controllers::KeyValueStoreService(name, id)
+, controllers::AbstractAutoPersistingKeyValueStoreService(name, id)
+, db_valid_(false)
+,
logger_(logging::LoggerFactory::getLogger())
{
+}
+
+RocksDbPersistableKeyValueStoreService::RocksDbPersistableKeyValueStoreService(const
std::string& name, utils::Identifier uuid /*= utils::Identifier()*/)
+: controllers::KeyValueStoreService(name, uuid)
+, controllers::AbstractAutoPersistingKeyValueStoreService(name, uuid)
+, db_valid_(false)
+,
logger_(logging::LoggerFactory::getLogger())
{
+}
+
+RocksDbPersistableKeyValueStoreService::~RocksDbPersistableKeyValueStoreService()
{
+ if (db_valid_) {
+delete db_;
+ }
+}
+
+void RocksDbPersistableKeyValueStoreService::initialize() {
+ AbstractAutoPersistingKeyValueStoreService::initialize();
+ std::set supportedProperties;
+ supportedProperties.insert(Directory);
+ updateSupportedProperties(supportedProperties);
+}
+
+void RocksDbPersistableKeyValueStoreService::onEnable() {
+ if (configuration_ == nullptr) {
+logger_->log_debug("Cannot enable RocksDbPersistableKeyValueStoreService");
+return;
+ }
+
+ AbstractAutoPersistingKeyValueStoreService::onEnable();
+
+ if (!getProperty(Directory.getName(), directory_)) {
+logger_->log_error("Invalid or missing property: Directory");
+return;
+ }
+
+ if (db_valid_) {
+db_valid_ = false;
+delete db_;
+ }
+ rocksdb::Options options;
+ options.create_if_missing = true;
Review comment:
Good point, this was written last summer, we didn't know this back then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r394633928
##
File path:
extensions/rocksdb-repos/controllers/RocksDbPersistableKeyValueStoreService.h
##
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef
LIBMINIFI_INCLUDE_CONTROLLERS_KEYVALUE_RocksDbPersistableKeyValueStoreService_H_
+#define
LIBMINIFI_INCLUDE_CONTROLLERS_KEYVALUE_RocksDbPersistableKeyValueStoreService_H_
+
+#include "controllers/keyvalue/AbstractAutoPersistingKeyValueStoreService.h"
+#include "core/Core.h"
+#include "properties/Configure.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+
+#include "rocksdb/db.h"
+#include "rocksdb/options.h"
+#include "rocksdb/slice.h"
+
+#include
+#include
+#include
+#include
+#include
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace controllers {
+
+class RocksDbPersistableKeyValueStoreService : public
AbstractAutoPersistingKeyValueStoreService {
+ public:
+ explicit RocksDbPersistableKeyValueStoreService(const std::string& name,
const std::string& id);
+ explicit RocksDbPersistableKeyValueStoreService(const std::string& name,
utils::Identifier uuid = utils::Identifier());
+
+ virtual ~RocksDbPersistableKeyValueStoreService();
+
+ static core::Property Directory;
+
+ virtual void initialize() override;
+ virtual void onEnable() override;
+ virtual void notifyStop() override;
+
+ virtual bool set(const std::string& key, const std::string& value) override;
+
+ virtual bool get(const std::string& key, std::string& value) override;
+
+ virtual bool get(std::unordered_map& kvs) override;
+
+ virtual bool remove(const std::string& key) override;
+
+ virtual bool clear() override;
+
+ virtual bool update(const std::string& key, const std::function& update_func) override;
+
+ virtual bool persist() override;
+
+ protected:
+ std::string directory_;
+
+ rocksdb::DB* db_;
Review comment:
None of our rocksdb stuff uses a unique_ptr, but it can, rewritten.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Loayza updated NIFI-7266: Attachment: NIFI_PERF_LOG_1.log > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: NIFI_PERF_LOG_1.log, Screen Shot 2020-03-17 at 3.18.27 > PM.png, image-2020-03-18-16-03-49-351.png, image-2020-03-18-16-35-49-452.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Loayza updated NIFI-7266: Attachment: image-2020-03-18-16-35-49-452.png > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png, > image-2020-03-18-16-03-49-351.png, image-2020-03-18-16-35-49-452.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062024#comment-17062024 ] Manuel Loayza commented on NIFI-7266: - I'm getting this error in the web console once the cluster fails !image-2020-03-18-16-03-49-351.png|width=706,height=302! > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png, > image-2020-03-18-16-03-49-351.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1182) Implement Libarchive extension on Windows
Alex Marmer created MINIFICPP-1182: -- Summary: Implement Libarchive extension on Windows Key: MINIFICPP-1182 URL: https://issues.apache.org/jira/browse/MINIFICPP-1182 Project: Apache NiFi MiNiFi C++ Issue Type: Task Reporter: Alex Marmer Implement Libarchive extension on Windows. According to https://www.libarchive.org, Windows is supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Loayza updated NIFI-7266: Attachment: image-2020-03-18-16-03-49-351.png > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png, > image-2020-03-18-16-03-49-351.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] MuazmaZ commented on a change in pull request #4126: NIFI-7103 Adding PutAzureDataLakeStorage Processor to provide native support for Azure Data Lake Storage Gen 2
MuazmaZ commented on a change in pull request #4126: NIFI-7103 Adding
PutAzureDataLakeStorage Processor to provide native support for Azure Data Lake
Storage Gen 2
URL: https://github.com/apache/nifi/pull/4126#discussion_r394596420
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.Locale;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.commons.lang3.StringUtils;
+
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor;
+import
com.azure.storage.file.datalake.implementation.models.StorageErrorException;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "adlsgen2", "datalake" })
+@CapabilityDescription("Puts content into an Azure Data Lake Storage Gen 2")
+@SeeAlso({})
+@WritesAttributes({ @WritesAttribute(attribute = "azure.filesystem",
description = "The name of the Azure File System"),
+@WritesAttribute(attribute = "azure.filename", description = "The name
of the Azure File Name"),
+@WritesAttribute(attribute = "azure.primaryUri", description =
"Primary location for file content"),
+@WritesAttribute(attribute = "azure.length", description = "Length of
the file")})
+@InputRequirement(Requirement.INPUT_REQUIRED)
+
+public class PutAzureDataLakeStorage extends
AbstractAzureDataLakeStorageProcessor {
+
+ @Override
+public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+final long startNanos = System.nanoTime();
+
+final String fileSystem =
context.getProperty(FILESYSTEM).evaluateAttributeExpressions(flowFile).getValue();
+final String directory =
context.getProperty(DIRECTORY).evaluateAttributeExpressions(flowFile).getValue();
+final String fileName =
context.getProperty(FILE).evaluateAttributeExpressions(flowFile).getValue();
+final String accountName =
context.getProperty(ACCOUNT_NAME).evaluateAttributeExpressions(flowFile).getValue();
+final String accountKey =
context.getProperty(ACCOUNT_KEY).evaluateAttributeExpressions(flowFile).getValue();
+final String sasToken =
context.getProperty(PROP_SAS_TOKEN).evaluateAttributeExpressions(flowFile).getValue();
+final String endpoint = String.format(Locale.ROOT,
"https://%s.dfs.core.windows.net;, accountName);
+DataLakeServiceClient storageClient;
+if (StringUtils.isNotBlank(accountKey)) {
+final StorageSharedKeyCredential credential = new
StorageSharedKeyCredential(accountName,
+accountKey);
+storageClient = new
DataLakeServiceClientBuilder().endpoint(endpoint).credential(credential)
+
[jira] [Updated] (NIFI-7256) Unit test failures for MonitorActivity processor suggest it doesn't work as expected
[ https://issues.apache.org/jira/browse/NIFI-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7256: --- Issue Type: Test (was: Task) > Unit test failures for MonitorActivity processor suggest it doesn't work as > expected > > > Key: NIFI-7256 > URL: https://issues.apache.org/jira/browse/NIFI-7256 > Project: Apache NiFi > Issue Type: Test >Reporter: Joe Witt >Priority: Minor > Fix For: 1.12.0, 1.11.4 > > Time Spent: 20m > Remaining Estimate: 0h > > Sometimes in slower build environments this test fails. Looking at the unit > test logic suggests this test might actually be revealing a bug. > [ERROR] Failures: > [ERROR] > TestMonitorActivity.testClusterMonitorActivityRestoredByOtherNodeOnPrimary:704 > expected:<1> but was:<0> > [ERROR] Tests run: 1500, Failures: 1, Errors: 0, Skipped: 23 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7185) TestLuceneEventIndex is an unreliable test
[ https://issues.apache.org/jira/browse/NIFI-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7185: --- Issue Type: Test (was: Bug) > TestLuceneEventIndex is an unreliable test > --- > > Key: NIFI-7185 > URL: https://issues.apache.org/jira/browse/NIFI-7185 > Project: Apache NiFi > Issue Type: Test > Components: Extensions, Tools and Build > Environment: Github Actions CI - Ubuntu Java 11 - 2 cores > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > Maven home: /usr/share/apache-maven-3.6.3 > Java version: 11.0.6, vendor: Azul Systems, Inc., runtime: > /opt/hostedtoolcache/jdk/11.0.6/x64 > Default locale: en, platform encoding: UTF-8 > OS name: "linux", version: "5.0.0-1031-azure", arch: "amd64", family: "unix" >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.12.0, 1.11.4 > > Time Spent: 20m > Remaining Estimate: 0h > > [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.363 > s <<< FAILURE! - in > org.apache.nifi.provenance.index.lucene.TestLuceneEventIndex > [ERROR] > testUnauthorizedEventsGetPlaceholdersForExpandChildren(org.apache.nifi.provenance.index.lucene.TestLuceneEventIndex) > Time elapsed: 0.097 s <<< FAILURE! > java.lang.AssertionError: expected:<5> but was:<8> > at > org.apache.nifi.provenance.index.lucene.TestLuceneEventIndex.testUnauthorizedEventsGetPlaceholdersForExpandChildren(TestLuceneEventIndex.java:234) > [ERROR] Failures: > [ERROR] > TestLuceneEventIndex.testUnauthorizedEventsGetPlaceholdersForExpandChildren:234 > expected:<5> but was:<8> > [ERROR] Tests run: 85, Failures: 1, Errors: 0, Skipped: 9 > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test (default-test) on > project nifi-persistent-provenance-repository: There are test failures. > [ERROR] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 commented on a change in pull request #4147: NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
mattyb149 commented on a change in pull request #4147: NIFI-7260: Fix error
handling and re-evaluate Module Directory property on changed for scripted
controller services
URL: https://github.com/apache/nifi/pull/4147#discussion_r394473190
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/AbstractScriptedRecordFactory.java
##
@@ -42,9 +42,13 @@ public void setup() {
if (scriptNeedsReload.get() || recordFactory.get() == null) {
if
(ScriptingComponentHelper.isFile(scriptingComponentHelper.getScriptPath())) {
-reloadScriptFile(scriptingComponentHelper.getScriptPath());
+if
(!reloadScriptFile(scriptingComponentHelper.getScriptPath())) {
Review comment:
I think we need the exception to indicate to the framework that the CS was
not enabled properly, even though the processor is still allowed to run and the
validation errors are already logged (hopefully only once now)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] mattyb149 commented on issue #4147: NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
mattyb149 commented on issue #4147: NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services URL: https://github.com/apache/nifi/pull/4147#issuecomment-600725807 I talked to @markap14 , it is the current design that a CS that fails while enabling still allows the processor to run, and it doesn't disable the CS on error, so I think this PR might be good to go otherwise? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061859#comment-17061859 ] ASF subversion and git services commented on NIFI-7238: --- Commit 4d940bb151eb8d250b0319318b96d23c4a9819ae in nifi's branch refs/heads/NIFI-7238-RC1 from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4d940bb ] NIFI-7238-RC1 prepare release nifi-1.11.4-RC1 > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061860#comment-17061860 ] ASF subversion and git services commented on NIFI-7238: --- Commit ece29c38631a8817b47f03d40cb971b9d9655581 in nifi's branch refs/heads/NIFI-7238-RC1 from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ece29c3 ] NIFI-7238-RC1 prepare for next development iteration > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1181) Define and improve C2 commands and protocol for NanoFi
Murtuza Shareef created MINIFICPP-1181: -- Summary: Define and improve C2 commands and protocol for NanoFi Key: MINIFICPP-1181 URL: https://issues.apache.org/jira/browse/MINIFICPP-1181 Project: Apache NiFi MiNiFi C++ Issue Type: Story Reporter: Murtuza Shareef Assignee: Murtuza Shareef MINIFICPP-1007 currently has C2 integration with ECU over CoAP. It implements C2 START/STOP/RESTART/UPDATE commands to interact with ECU. We need to make sure that the C2 protocol here is consistent with the protocol behavior in MiNiFi C++. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1180) Investigate C2 protocol integration into NanoFi
Murtuza Shareef created MINIFICPP-1180: -- Summary: Investigate C2 protocol integration into NanoFi Key: MINIFICPP-1180 URL: https://issues.apache.org/jira/browse/MINIFICPP-1180 Project: Apache NiFi MiNiFi C++ Issue Type: Epic Reporter: Murtuza Shareef Assignee: Murtuza Shareef This epic is created to track work needed to unify C2 protocol across NanoFi and MiNiFi and to come up with an architecture to make C2 as a reusable component for both NaNoFi and MiNiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1179) Employ reliable buffering in ECU.
[ https://issues.apache.org/jira/browse/MINIFICPP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murtuza Shareef updated MINIFICPP-1179: --- Description: The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size and no data persistence to file system, making the ecu unreliable and memory inefficient. This ticket solves the above problems by # Setting a max buffer size per ECU and incrementally allocating memory when needed. # Persist data to the file system for reliability. # Flush any persisted data upon startup to the destination. was: The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size and no data persistence to file system, making the ecu unreliable and memory inefficient. This ticket solves the above problems by # Setting a max buffer size per ECU and incrementally allocating memory when needed. # Persisting data to the file system for reliability. > Employ reliable buffering in ECU. > -- > > Key: MINIFICPP-1179 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1179 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Murtuza Shareef >Assignee: Murtuza Shareef >Priority: Major > > The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size > and no data persistence to file system, making the ecu unreliable and memory > inefficient. > This ticket solves the above problems by > # Setting a max buffer size per ECU and incrementally allocating memory when > needed. > # Persist data to the file system for reliability. > # Flush any persisted data upon startup to the destination. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1179) Employ reliable buffering in ECU.
[ https://issues.apache.org/jira/browse/MINIFICPP-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murtuza Shareef updated MINIFICPP-1179: --- Description: The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size and no data persistence to file system, making the ecu unreliable and memory inefficient. This ticket solves the above problems by # Setting a max buffer size per ECU and incrementally allocating memory when needed. # Persisting data to the file system for reliability. was: The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size and no data persistence to file system, making the ecu unreliable and memory inefficient. This ticket solves the above problems by # Setting a max buffer size per ECU and incrementally allocating memory when needed. # Persisting data to the file system while in the process of flushing to the output plugin. > Employ reliable buffering in ECU. > -- > > Key: MINIFICPP-1179 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1179 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Murtuza Shareef >Assignee: Murtuza Shareef >Priority: Major > > The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size > and no data persistence to file system, making the ecu unreliable and memory > inefficient. > This ticket solves the above problems by > # Setting a max buffer size per ECU and incrementally allocating memory when > needed. > # Persisting data to the file system for reliability. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFICPP-1179) Employ reliable buffering in ECU.
Murtuza Shareef created MINIFICPP-1179: -- Summary: Employ reliable buffering in ECU. Key: MINIFICPP-1179 URL: https://issues.apache.org/jira/browse/MINIFICPP-1179 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Murtuza Shareef Assignee: Murtuza Shareef The work done as part of MINIFICPP-1007, uses a ring buffer with fixed size and no data persistence to file system, making the ecu unreliable and memory inefficient. This ticket solves the above problems by # Setting a max buffer size per ECU and incrementally allocating memory when needed. # Persisting data to the file system while in the process of flushing to the output plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1008) Implement ECU C API SDK and integrate with C2
[ https://issues.apache.org/jira/browse/MINIFICPP-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murtuza Shareef updated MINIFICPP-1008: --- Description: A C API SDK allows users to create ECUs and run them. The C2 API and the CoAP communication implementation will be implemented as part of another story. MINIFICPP-1007 The goal of this story is to develop a minimal C API as an SDK to build out an ECU and then integrate it with the communication layer (C2/CoAP layer) to be fully integrated with CoAP server. was: A C API SDK allows users to create ECUs and run them. The C2 API and the CoAP communication implementation will be implemented as part of another story. MINIFICPP-1007 The goal of this story is to develop a minimal C API as an SDK to build out an ECU and then integrate it with the communication layer (C2/CoAP layer) to be fully integrated with CoAP server (EFM). > Implement ECU C API SDK and integrate with C2 > - > > Key: MINIFICPP-1008 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1008 > Project: Apache NiFi MiNiFi C++ > Issue Type: Story >Reporter: Murtuza Shareef >Assignee: Murtuza Shareef >Priority: Major > Labels: CAPI, ECU, nanofi > > A C API SDK allows users to create ECUs and run them. The C2 API and the CoAP > communication implementation will be implemented as part of another story. > MINIFICPP-1007 > The goal of this story is to develop a minimal C API as an SDK to build out > an ECU and then integrate it with the communication layer (C2/CoAP layer) to > be fully integrated with CoAP server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061786#comment-17061786 ] ASF subversion and git services commented on NIFI-7238: --- Commit 40497d10ba9d5cb2878c925d646c4cf7ffbc in nifi's branch refs/heads/support/nifi-1.11.x from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=40497d1 ] NIFI-7238 > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
[ https://issues.apache.org/jira/browse/NIFI-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061780#comment-17061780 ] ASF subversion and git services commented on NIFI-7267: --- Commit e9cb6c1e3609bb928df0eab452d4ba14d4720d65 in nifi's branch refs/heads/support/nifi-1.11.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e9cb6c1 ] NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150) Signed-off-by: Andy LoPresto > Upgrade spring-data-redis in Redis bundle > - > > Key: NIFI-7267 > URL: https://issues.apache.org/jira/browse/NIFI-7267 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.12.0, 1.11.4 > > Time Spent: 40m > Remaining Estimate: 0h > > The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to > 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7258) Overflow in PutAzureEventHub when not configured correctly
[
https://issues.apache.org/jira/browse/NIFI-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061779#comment-17061779
]
ASF subversion and git services commented on NIFI-7258:
---
Commit 53ca44d9625dc7681d717d332c893bb50517224a in nifi's branch
refs/heads/support/nifi-1.11.x from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=53ca44d ]
NIFI-7258 - fix overflow in PutAzureEventHub when not configured correctly
Signed-off-by: Pierre Villard
This closes #4146.
> Overflow in PutAzureEventHub when not configured correctly
> --
>
> Key: NIFI-7258
> URL: https://issues.apache.org/jira/browse/NIFI-7258
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Priority: Major
> Fix For: 1.12.0, 1.11.4
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When PutAzureEventHub is not correctly configured, the processor will keep
> retrying without yielding causing resources consumption on the NiFi node. The
> log message is also not correctly formatted:
> {code:java}
> 2020-03-16 15:53:25,613 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to create
> EventHubClient due to com.microsoft.azure.eventhubs.CommunicationException: A
> communication error has occurred. This may be due to an incorrect host name
> in your connection string or a problem with your network connection.: {}
> 2020-03-16 15:53:25,613 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> com.microsoft.azure.eventhubs.CommunicationException: A communication error
> has occurred. This may be due to an incorrect host name in your connection
> string or a problem with your network connection.:
> org.apache.nifi.processor.exception.ProcessException:
> com.microsoft.azure.eventhubs.CommunicationException: A communication error
> has occurred. This may be due to an incorrect host name in your connection
> string or a problem with your network connection.
> 2020-03-16 15:53:25,617 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to create
> EventHubClient due to com.microsoft.azure.eventhubs.CommunicationException: A
> communication error has occurred. This may be due to an incorrect host name
> in your connection string or a problem with your network connection.: {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
[ https://issues.apache.org/jira/browse/NIFI-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7267: --- Fix Version/s: 1.11.4 > Upgrade spring-data-redis in Redis bundle > - > > Key: NIFI-7267 > URL: https://issues.apache.org/jira/browse/NIFI-7267 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.12.0, 1.11.4 > > Time Spent: 40m > Remaining Estimate: 0h > > The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to > 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7258) Overflow in PutAzureEventHub when not configured correctly
[
https://issues.apache.org/jira/browse/NIFI-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-7258:
---
Fix Version/s: 1.11.4
> Overflow in PutAzureEventHub when not configured correctly
> --
>
> Key: NIFI-7258
> URL: https://issues.apache.org/jira/browse/NIFI-7258
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Priority: Major
> Fix For: 1.12.0, 1.11.4
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When PutAzureEventHub is not correctly configured, the processor will keep
> retrying without yielding causing resources consumption on the NiFi node. The
> log message is also not correctly formatted:
> {code:java}
> 2020-03-16 15:53:25,613 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to create
> EventHubClient due to com.microsoft.azure.eventhubs.CommunicationException: A
> communication error has occurred. This may be due to an incorrect host name
> in your connection string or a problem with your network connection.: {}
> 2020-03-16 15:53:25,613 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> com.microsoft.azure.eventhubs.CommunicationException: A communication error
> has occurred. This may be due to an incorrect host name in your connection
> string or a problem with your network connection.:
> org.apache.nifi.processor.exception.ProcessException:
> com.microsoft.azure.eventhubs.CommunicationException: A communication error
> has occurred. This may be due to an incorrect host name in your connection
> string or a problem with your network connection.
> 2020-03-16 15:53:25,617 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.azure.eventhub.PutAzureEventHub
> PutAzureEventHub[id=e346dfcc-0170-1000--ad338113] Failed to create
> EventHubClient due to com.microsoft.azure.eventhubs.CommunicationException: A
> communication error has occurred. This may be due to an incorrect host name
> in your connection string or a problem with your network connection.: {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] joewitt commented on issue #3917: Adding GetSmbFile and PutSmbFile processors
joewitt commented on issue #3917: Adding GetSmbFile and PutSmbFile processors URL: https://github.com/apache/nifi/pull/3917#issuecomment-600645888 ..disregard the close - hit the wrong button! reopened. And just learned that reopening means the new builds are running - wow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] joewitt closed pull request #3917: Adding GetSmbFile and PutSmbFile processors
joewitt closed pull request #3917: Adding GetSmbFile and PutSmbFile processors URL: https://github.com/apache/nifi/pull/3917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] si-sun opened a new pull request #3917: Adding GetSmbFile and PutSmbFile processors
si-sun opened a new pull request #3917: Adding GetSmbFile and PutSmbFile
processors
URL: https://github.com/apache/nifi/pull/3917
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Added functionality to read and write from samba shares with GetSmbFile and
PutSmbFile processors. The advantage is to not have to manage cifs mounts on
the server/nodes. Also the processors supports windows style share locks on the
files during read and write.
It's using [SMBJ client library](https://github.com/hierynomus/smbj) for the
samba connection.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder? (only for
:nifi-smb-bundle)
- [X] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [X] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [X] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #3917: Adding GetSmbFile and PutSmbFile processors
joewitt commented on issue #3917: Adding GetSmbFile and PutSmbFile processors URL: https://github.com/apache/nifi/pull/3917#issuecomment-600645546 this ideally needs to be rebased against latest. The updates to NOTICE/LiCENSE look really well done. The code itself looks quite thoughtful. Not sure how well tests will work on different platforms but a rebase/update to latest master will help highlight that. In general though this looks like a solid contrib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (MINIFICPP-1171) Fix "%ll" format strings in log lines
[
https://issues.apache.org/jira/browse/MINIFICPP-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri resolved MINIFICPP-1171.
Resolution: Fixed
> Fix "%ll" format strings in log lines
> -
>
> Key: MINIFICPP-1171
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1171
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.8.0
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Malformed log strings don't print the value.
> As an example, this line of code:
> {code}./extensions/civetweb/processors/ListenHTTP.cpp:287:
> logger_->log_debug("ListenHTTP handling POST request of length %ll",
> req_info->content_length);{code}
> Results in the following log message:
> {code}[2020-03-03 17:00:29.710]
> [org::apache::nifi::minifi::processors::ListenHTTP::Handler] [debug]
> ListenHTTP handling POST request of length{code}
> (Compiled with g++8.3 on Debian Buster)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] JensMKofoed commented on issue #3917: Adding GetSmbFile and PutSmbFile processors
JensMKofoed commented on issue #3917: Adding GetSmbFile and PutSmbFile processors URL: https://github.com/apache/nifi/pull/3917#issuecomment-600638599 Are there any progress in this great new features? This is something I have waited for for a very long time. If you need help testing, I would be glad to help This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061726#comment-17061726 ] Joe Witt commented on NIFI-7266: Most large scale production users are on latest JDKs at all times with latest NiFi. That JDK and OS is likely not the issue here. In such a situation you'll need to have a good way of collecting logs/configuration details/etc.. and sharing that with someone that can look into that with you. > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] am-c-p-p commented on issue #739: MINIFICPP-1163 Implemented.
am-c-p-p commented on issue #739: MINIFICPP-1163 Implemented. URL: https://github.com/apache/nifi-minifi-cpp/pull/739#issuecomment-600556379 Added parameter /M (for installer with merge modules) to win_build_vs.bat. Created new WixWinLocalRedist.wsi for installer with local redistributables Dlls, ideally it would be better to have only one WixWin.wsi, but couldn't figure out simple way to do it. Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-7265. -- Resolution: Not A Problem > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > Labels: Linux, kubernetes > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at >
[jira] [Commented] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061506#comment-17061506 ] Ganesh Banda commented on NIFI-7265: [~markap14], I figured this out. There is client timeouts "minSessionTimeout and maxSessionTimeout" were very less at Zookeeper, where in which KeeperError I was getting. After increasing it as well as "nifi.zookeeper.connect.timeout and nifi.zookeeper.session.timeout" in Nifi, started working fine. Nifi is reconnecting back once after Zookeeper comes back online. > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > Labels: Linux, kubernetes > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up >
[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061468#comment-17061468 ] Pierre Villard commented on NIFI-7238: -- The pull request is opened and does not have active reviewers. Besides 1.11.4 is about including fixes, no new features. If you're in a position to help reviewing/testing the pull request, please do and it could be included in 1.12.0. > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
[ https://issues.apache.org/jira/browse/NIFI-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-7267: - Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade spring-data-redis in Redis bundle > - > > Key: NIFI-7267 > URL: https://issues.apache.org/jira/browse/NIFI-7267 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.12.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to > 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
