[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061415#comment-17061415 ] Jens M Kofoed commented on NIFI-7238: - Hi [~pvillard] - Thanks for the news it sounds great. I looked at the github that someone is working on a GetSMB/PutSMB ([https://github.com/apache/nifi/pull/3917)] Will that be a part of the next version? > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7255) nifi.properties configuration change can result in duplicate nodes listed in Cluster UI
[ https://issues.apache.org/jira/browse/NIFI-7255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061412#comment-17061412 ] Jens M Kofoed commented on NIFI-7255: - I've also tried to change a cluster from non-secure to secure and learned the hard way that nodes in the cluster are identified with hostname:port and not just the hostname. I had to completely delete my flow and running/working states to get the cluster back. Not good. And yes if you change the port of a node, it will show up as a new node. All changes in the UI are sendt to all nodes in the cluster and since the old node with the old port no loger exist but is still showen as a member og the cluster you are not able to make any changes. My work-around was to disconnect the node from the cluster and delete it. Change port in configuration with-out changing it to secure yet. Bring it back on with the new port and join the cluster. After all nodes was changed I could change the cluster to secure > nifi.properties configuration change can result in duplicate nodes listed in > Cluster UI > --- > > Key: NIFI-7255 > URL: https://issues.apache.org/jira/browse/NIFI-7255 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.7.0, 1.8.0, 1.9.0, 1.10.0 >Reporter: Matthew Clarke >Priority: Major > > The elements that go in to coming up with the "Node identifier" which is then > store in local state on a NiFi node include: > /** > the unique identifier for the node > */ > private final String id; /** > the IP or hostname to use for sending requests to the node's external > interface > */ > private final String apiAddress; /** > the port to use use for sending requests to the node's external interface, > this can be HTTP API port or HTTPS API port depending on whether //TODO: . > */ > private final int apiPort; /** > the IP or hostname to use for sending requests to the node's internal > interface > */ > private final String socketAddress; /** > the port to use for sending requests to the node's internal interface > */ > private final int socketPort; /** > The IP or hostname to use for sending FlowFiles when load balancing a > connection > */ > private final String loadBalanceAddress; /** > the port to use for sending FlowFiles when load balancing a connection > */ > private final int loadBalancePort; /** > the IP or hostname that external clients should use to communicate with this > node via Site-to-Site > */ > private final String siteToSiteAddress; /** > the port that external clients should use to communicate with this node via > Site-to-Site RAW Socket protocol > */ > private final Integer siteToSitePort; /** > the port that external clients should use to communicate with this node via > Site-to-Site HTTP protocol, > this can be HTTP API port or HTTPS API port depending on whether > siteToSiteSecure or not. > */ > private final Integer siteToSiteHttpApiPort; /** > whether or not site-to-site communications with this node are secure > */ > private final Boolean siteToSiteSecure; private final Set > nodeIdentities; > With the following fields being used to determine quality: > apiAddress > apiPort > socketAddress > socketPort > If for example the apiPort is changed by switching from 8080 for (http) to > 8443 (for https), the node will show up twice in the in the cluster UI ( > hostname:8443 --> connected and hostname:8080 --> disconnected). Having > these disconnected nodes will prevent changes to the UI. Worse yet is that > ZK may report as the elected Cluster coordinator and end up having > both the 8080 and 8443 node both being marked as the cluster coordinator. > Then you may not even be able to access the cluster because requests fails to > replicate to the Cluster coordinator because it is not connected. > Resolving this issue requires users to shutdown NiFi, delete the local state > directory contents, and restart NiFi. > Downside to this resolution is any local state retained for NiFi components > (for example processors) is lost as well. > Suggested solution here is for NiFi to retain current node identifier field > configuration values. If on restart loaded configurations show any change to > those values, NiFi should clear out the previous retained node ids and create > all new node Ids. > Might also make sense to move the stored out of local state to make manual > removal of this information possible without affecting state stored by > components found in the flow.xml. > > The following Jira only addressed what specific configuration change that can > result in this issue: > https://jira.apache.org/jira/browse/NIFI-5672 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7268) Change library implementation of Bcrypt
[ https://issues.apache.org/jira/browse/NIFI-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061345#comment-17061345 ] ASF subversion and git services commented on NIFI-7268: --- Commit f91d6c420d78aa000f1cc894636f56b533b589b8 in nifi's branch refs/heads/master from M Tien [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f91d6c4 ] NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151) * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library. Updated LICENSE and NOTICE files to reflect changes. Updated unit tests. Co-authored-by: Andy LoPresto * NIFI-7268 Fixed typo in Javadoc. Co-authored-by: Andy LoPresto > Change library implementation of Bcrypt > --- > > Key: NIFI-7268 > URL: https://issues.apache.org/jira/browse/NIFI-7268 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.11.3 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: base64, dependancy, library, security > Time Spent: 50m > Remaining Estimate: 0h > > The jBcrypt library has not been updated since 2015, has known issues, and > doesn't provide utility methods for custom Radix64 operations. The Bcrypt > library available at [https://github.com/patrickfav/bcrypt] is an > enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7268) Change library implementation of Bcrypt
[ https://issues.apache.org/jira/browse/NIFI-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7268: Status: Patch Available (was: Open) > Change library implementation of Bcrypt > --- > > Key: NIFI-7268 > URL: https://issues.apache.org/jira/browse/NIFI-7268 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.11.3 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: base64, dependancy, library, security > Time Spent: 50m > Remaining Estimate: 0h > > The jBcrypt library has not been updated since 2015, has known issues, and > doesn't provide utility methods for custom Radix64 operations. The Bcrypt > library available at [https://github.com/patrickfav/bcrypt] is an > enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7268) Change library implementation of Bcrypt
[ https://issues.apache.org/jira/browse/NIFI-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7268: Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Change library implementation of Bcrypt > --- > > Key: NIFI-7268 > URL: https://issues.apache.org/jira/browse/NIFI-7268 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.11.3 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: base64, dependancy, library, security > Fix For: 1.12.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The jBcrypt library has not been updated since 2015, has known issues, and > doesn't provide utility methods for custom Radix64 operations. The Bcrypt > library available at [https://github.com/patrickfav/bcrypt] is an > enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7268) Change library implementation of Bcrypt
[ https://issues.apache.org/jira/browse/NIFI-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061343#comment-17061343 ] ASF subversion and git services commented on NIFI-7268: --- Commit f91d6c420d78aa000f1cc894636f56b533b589b8 in nifi's branch refs/heads/master from M Tien [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f91d6c4 ] NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151) * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library. Updated LICENSE and NOTICE files to reflect changes. Updated unit tests. Co-authored-by: Andy LoPresto * NIFI-7268 Fixed typo in Javadoc. Co-authored-by: Andy LoPresto > Change library implementation of Bcrypt > --- > > Key: NIFI-7268 > URL: https://issues.apache.org/jira/browse/NIFI-7268 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.11.3 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: base64, dependancy, library, security > Time Spent: 50m > Remaining Estimate: 0h > > The jBcrypt library has not been updated since 2015, has known issues, and > doesn't provide utility methods for custom Radix64 operations. The Bcrypt > library available at [https://github.com/patrickfav/bcrypt] is an > enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7268) Change library implementation of Bcrypt
[ https://issues.apache.org/jira/browse/NIFI-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061344#comment-17061344 ] ASF subversion and git services commented on NIFI-7268: --- Commit f91d6c420d78aa000f1cc894636f56b533b589b8 in nifi's branch refs/heads/master from M Tien [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f91d6c4 ] NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151) * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library. Updated LICENSE and NOTICE files to reflect changes. Updated unit tests. Co-authored-by: Andy LoPresto * NIFI-7268 Fixed typo in Javadoc. Co-authored-by: Andy LoPresto > Change library implementation of Bcrypt > --- > > Key: NIFI-7268 > URL: https://issues.apache.org/jira/browse/NIFI-7268 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.11.3 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: base64, dependancy, library, security > Time Spent: 50m > Remaining Estimate: 0h > > The jBcrypt library has not been updated since 2015, has known issues, and > doesn't provide utility methods for custom Radix64 operations. The Bcrypt > library available at [https://github.com/patrickfav/bcrypt] is an > enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] alopresto merged pull request #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa…
alopresto merged pull request #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… URL: https://github.com/apache/nifi/pull/4151 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on issue #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa…
alopresto commented on issue #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… URL: https://github.com/apache/nifi/pull/4151#issuecomment-600397049 Ran `contrib-check` and all tests pass. +1, merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa…
alopresto commented on a change in pull request #4151: NIFI-7268 Removed
org.mindrot.jBcrypt library and replaced with at.fa…
URL: https://github.com/apache/nifi/pull/4151#discussion_r394075738
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/crypto/BcryptCipherProvider.java
##
@@ -160,9 +162,54 @@ private String formatSaltForBcrypt(byte[] salt) {
}
}
+/**
+ * Returns the full salt in aa {@code byte[]} for this cipher provider
(i.e. {@code $2a$10$abcdef...} format).
Review comment:
Typo in Javadoc here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on issue #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa…
alopresto commented on issue #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… URL: https://github.com/apache/nifi/pull/4151#issuecomment-600383809 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mtien-apache opened a new pull request #4151: NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa…
mtien-apache opened a new pull request #4151: NIFI-7268 Removed
org.mindrot.jBcrypt library and replaced with at.fa…
URL: https://github.com/apache/nifi/pull/4151
…vre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.
Co-authored-by: Andy LoPresto
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Removed legacy library and replaced with modern version._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [x] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061295#comment-17061295 ] Manuel Loayza commented on NIFI-7266: - Thanks [~joewitt] for your quick response. We have been running those clusters for at least 2,5 years without any issue. But we need to upgrade our JDK for security reasons, and when I checked to use NIFI versions bigger than 1.4.0. I see there are many component that we would need to update in our configuration, and also some new cool features but we need to do this quick. What would we do to enable in NIFI to check what it is failing? do we need to enable some specific logger to see more details in the log files when this event happens? What have changed in NIFI since 1.1.2 that is not capable to handle a similar traffic without any change in the configuration and the flow? > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7268) Change library implementation of Bcrypt
M Tien created NIFI-7268: Summary: Change library implementation of Bcrypt Key: NIFI-7268 URL: https://issues.apache.org/jira/browse/NIFI-7268 Project: Apache NiFi Issue Type: Sub-task Components: Core Framework Affects Versions: 1.11.3 Reporter: M Tien Assignee: M Tien The jBcrypt library has not been updated since 2015, has known issues, and doesn't provide utility methods for custom Radix64 operations. The Bcrypt library available at [https://github.com/patrickfav/bcrypt] is an enhancement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
[ https://issues.apache.org/jira/browse/NIFI-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061238#comment-17061238 ] ASF subversion and git services commented on NIFI-7267: --- Commit 7105bf36a8e191cacf43ed8ac6171be64a8c2e02 in nifi's branch refs/heads/master from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7105bf3 ] NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150) Signed-off-by: Andy LoPresto > Upgrade spring-data-redis in Redis bundle > - > > Key: NIFI-7267 > URL: https://issues.apache.org/jira/browse/NIFI-7267 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to > 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] alopresto merged pull request #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle
alopresto merged pull request #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle URL: https://github.com/apache/nifi/pull/4150 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on issue #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle
alopresto commented on issue #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle URL: https://github.com/apache/nifi/pull/4150#issuecomment-600338618 Reviewed the code, created a flow that used `PutDistributedCacheMap` and `FetchDistributedCacheMap` with a `RedisDistributedMapCacheClientService` connecting to a Docker container running `redis` on `localhost`. Everything worked fine with the new dependency. +1, merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on issue #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle
alopresto commented on issue #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle URL: https://github.com/apache/nifi/pull/4150#issuecomment-600324336 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
[ https://issues.apache.org/jira/browse/NIFI-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-7267: - Status: Patch Available (was: Open) > Upgrade spring-data-redis in Redis bundle > - > > Key: NIFI-7267 > URL: https://issues.apache.org/jira/browse/NIFI-7267 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to > 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pvillard31 opened a new pull request #4150: NIFI-7267 - Upgrade spring-data-redis in Redis bundle
pvillard31 opened a new pull request #4150: NIFI-7267 - Upgrade
spring-data-redis in Redis bundle
URL: https://github.com/apache/nifi/pull/4150
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-7257) Add Hadoop-based DBCPConnectionPool
[ https://issues.apache.org/jira/browse/NIFI-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-7257: -- Status: Patch Available (was: In Progress) > Add Hadoop-based DBCPConnectionPool > --- > > Key: NIFI-7257 > URL: https://issues.apache.org/jira/browse/NIFI-7257 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Major > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Although our DBCPConnectionPool has options for doing Kerberos > authentication, drivers that under the covers rely on Hadoop's > UserGroupInformation won't see the logged in user from our service. It would > be helpful if we had a service similar to DBCPConnectionPool, but with > hadoop-common on the classpath so that we can handle the the UGI related > stuff. > This new service would be similar to a combination of DBCPConnectionPool and > some of the stuff done in Hive3ConnectionPool, but we would not bundle any > Hadoop dependency, and instead the user must specify hadoop-common as part of > the driver dependencies, or a shaded jar that includes it (as phoenix does). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] bbende opened a new pull request #4149: NIFI-7257 Added HadoopDBCPConnectionPool
bbende opened a new pull request #4149: NIFI-7257 Added HadoopDBCPConnectionPool
URL: https://github.com/apache/nifi/pull/4149
- Updated InstanceClassLoader to resolve files that are in the instance urls
or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for
ControllerServices
- Added unit test for HadoopDBCPConnectionPool
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (NIFI-7267) Upgrade spring-data-redis in Redis bundle
Pierre Villard created NIFI-7267: Summary: Upgrade spring-data-redis in Redis bundle Key: NIFI-7267 URL: https://issues.apache.org/jira/browse/NIFI-7267 Project: Apache NiFi Issue Type: Improvement Reporter: Pierre Villard Assignee: Pierre Villard The spring-data-redis dependency can be upgraded from 2.1.0.RELEASE to 2.1.16.RELEASE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-6875) Nifi Zookeeper Cluster_Mode broken in 1.10.0
[ https://issues.apache.org/jira/browse/NIFI-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061166#comment-17061166 ] Dharmeshkumar Patel edited comment on NIFI-6875 at 3/17/20, 8:29 PM: - [~pvillard] [~wyllys] [~joewitt] I am seeing below issue with nifi 1.11.3 version. zookeeper basically gave up, is there any fixed for this. I am using internal zookeeper.version=3.5.6 2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at
[jira] [Commented] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
[ https://issues.apache.org/jira/browse/NIFI-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061167#comment-17061167 ] Joe Witt commented on NIFI-7266: These are extremely old versions of NiFi here and very limited information on the flow/configuration details. Hard to respond meaningfully. Seeing a sudden drop in performance could be related to iops throttling. Could be related to heap being close to full (which really depends on flow specifics as to whether that should happen). Or other items. Would require a lot more details/logs/etc.. to have a chance to be helpful. > NIFI 1.4.0 gets unresponsive after heavy load > - > > Key: NIFI-7266 > URL: https://issues.apache.org/jira/browse/NIFI-7266 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Manuel Loayza >Priority: Trivial > Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png > > > We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 > + Linux CentOS > The traffic get divided between those 2 clusters: > 1. TPS: 2700 - EAST cluster > 2. TPS: 980. - WEST cluster > We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster > with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it > web console got unresponsive. > I've tried many things to fix this thing, but only thing I got was to > increase the time from 4 to 6 hours before it fails > Our current instances are running on AWS and each EC2 instances has 8 cpus > (c5.2xlarge), and 16GB RAM. > I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the > same outcome. > I don't have a clue to figure it out what the issue is. Also I have a > datadog dashboard to track some java head metrics but everything looks normal. > What should I do to find why those new better instances are failing? is it > memory or disk space or threads got stuck? Why an old NIFI cluster conf > works better than a new NIFI? > Hope you can help me with this. > Thanks > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-6875) Nifi Zookeeper Cluster_Mode broken in 1.10.0
[ https://issues.apache.org/jira/browse/NIFI-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061166#comment-17061166 ] Dharmeshkumar Patel edited comment on NIFI-6875 at 3/17/20, 8:20 PM: - [~pvillard] [~wyllys] [~joewitt] I am seeing below issue with nifi 1.11.3 version. zookeeper basically gave up, is there any fixed for this. I am using internal Zookeeper 5.3.6 2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at
[jira] [Commented] (NIFI-6875) Nifi Zookeeper Cluster_Mode broken in 1.10.0
[ https://issues.apache.org/jira/browse/NIFI-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061166#comment-17061166 ] Dharmeshkumar Patel commented on NIFI-6875: --- [~pvillard] [~wyllys] I am seeing below issue with nifi 1.11.3 version. zookeeper basically gave up, is there any fixed for this. I am using internal Zookeeper 5.3.6 2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2020-03-17 19:58:44,143 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at
[jira] [Created] (NIFI-7266) NIFI 1.4.0 gets unresponsive after heavy load
Manuel Loayza created NIFI-7266: --- Summary: NIFI 1.4.0 gets unresponsive after heavy load Key: NIFI-7266 URL: https://issues.apache.org/jira/browse/NIFI-7266 Project: Apache NiFi Issue Type: Bug Components: Configuration Affects Versions: 1.4.0, 1.3.0, 1.2.0 Reporter: Manuel Loayza Attachments: Screen Shot 2020-03-17 at 3.18.27 PM.png We have 2 clusters (6 instances each one) running with NIFI 1.1.2 + JDK 8u121 + Linux CentOS The traffic get divided between those 2 clusters: 1. TPS: 2700 - EAST cluster 2. TPS: 980. - WEST cluster We have tried to migrate to NIFI 1.2.0, 1.3.0, and 1.4.0, but the cluster with higher TPS (EAST) got stuck after 4 hours of intensive traffic. Also it web console got unresponsive. I've tried many things to fix this thing, but only thing I got was to increase the time from 4 to 6 hours before it fails Our current instances are running on AWS and each EC2 instances has 8 cpus (c5.2xlarge), and 16GB RAM. I've tried to use c5.4xlarge (it doubles the cpu and ram), but I got the same outcome. I don't have a clue to figure it out what the issue is. Also I have a datadog dashboard to track some java head metrics but everything looks normal. What should I do to find why those new better instances are failing? is it memory or disk space or threads got stuck? Why an old NIFI cluster conf works better than a new NIFI? Hope you can help me with this. Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061068#comment-17061068 ] Mark Payne commented on NIFI-7265: -- [~ganeshb] I think we would need more information to understand the problem that you're running into. I attempted to replicate this but failed to do so. When I stopped ZooKeeper, I did see the errors noted above spewing into the logs. However, when I restarted ZooKeeper, within a second or two all nodes in my cluster managed to reconnect and continue on correctly. > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > Labels: Linux, kubernetes > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up >
[GitHub] [nifi] mattyb149 commented on a change in pull request #4147: NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
mattyb149 commented on a change in pull request #4147: NIFI-7260: Fix error
handling and re-evaluate Module Directory property on changed for scripted
controller services
URL: https://github.com/apache/nifi/pull/4147#discussion_r393826015
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/AbstractScriptedRecordFactory.java
##
@@ -42,9 +42,13 @@ public void setup() {
if (scriptNeedsReload.get() || recordFactory.get() == null) {
if
(ScriptingComponentHelper.isFile(scriptingComponentHelper.getScriptPath())) {
-reloadScriptFile(scriptingComponentHelper.getScriptPath());
+if
(!reloadScriptFile(scriptingComponentHelper.getScriptPath())) {
Review comment:
The exception was supposed to prevent the CS from enabling, Andy pointed out
the same issue. I thought I'd tested that but apparently did not, will take a
closer look
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] ChrisSamo632 commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable
ChrisSamo632 commented on a change in pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable URL: https://github.com/apache/nifi/pull/4148#discussion_r393802393 ## File path: nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/JsonPathEvaluator.java ## @@ -46,7 +47,14 @@ public JsonPathEvaluator(final Evaluator subject, final Evaluator
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061030#comment-17061030 ] Otto Fowler commented on NIFI-7264: --- I submitted a PR that changes the log level explicitly for PathNotFound to reduce this. If we think that having a default value version of the function ( so you don't have to do more processors and handling down the line if you need something other than empty ), that can be another jira. > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Assignee: Otto Fowler >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7243) SFTP Processors - Exhausted available authentication methods
[ https://issues.apache.org/jira/browse/NIFI-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061026#comment-17061026 ] Raúl commented on NIFI-7243: [~mattyb149] You can help us with this problem. Thanks in advance. > SFTP Processors - Exhausted available authentication methods > > > Key: NIFI-7243 > URL: https://issues.apache.org/jira/browse/NIFI-7243 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.10.0, 1.11.0, 1.11.1, 1.11.3 > Environment: PRO >Reporter: Raúl >Priority: Critical > Labels: SFTP, listsftp > > In version 1.8 this works correctly. In superiors no. > > Any of the processors: ListSFTP, FetchSFTP, GetSFTP, PutSFTP when connecting > to an sftp shows the error: > 2020-03-10 15:12:51,596 ERROR [Timer-Driven Process Thread-3] > o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=af72555e-0170-1000-c80d-7e992c9de544] Failed to perform listing > on remote host due to Exhausted available authentication methods: > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methods > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methods > at net.schmizz.sshj.SSHClient.auth(SSHClient.java:230) > > These tests that I commented have been made from nifi, both in windows and > linux environment and fails in both. > 1 - Strict Host Key Checking: false > 2 - Strict Host Key Checking: true > Host Key File: C:\Users\usuario\.ssh\known_hosts --> (ssh-rsa, > ssh-dss and ecdsa-sha2-nistp256) > > From the machines, both in windows and in linux, if an sftp is made from the > command line it connects without problems. But with Nifi he fails to connect. > > Any idea about how can I resolve this error? > > Thank you in advance, > log > - > 2020-03-10 15:12:51,596 ERROR [Timer-Driven Process Thread-3] > o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=af72555e-0170-1000-c80d-7e992c9de544] Failed to perform listing > on remote host due to Exhausted available authentication methods: > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methods2020-03-10 15:12:51,596 ERROR [Timer-Driven Process > Thread-3] o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=af72555e-0170-1000-c80d-7e992c9de544] Failed to perform listing > on remote host due to Exhausted available authentication methods: > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methodsnet.schmizz.sshj.userauth.UserAuthException: Exhausted > available authentication methods at > net.schmizz.sshj.SSHClient.auth(SSHClient.java:230) at > org.apache.nifi.processors.standard.util.SFTPTransfer.getSFTPClient(SFTPTransfer.java:606) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:233) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:196) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106) > at > org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:146) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:472) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:414) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)2020-03-10 15:12:51,601 DEBUG > [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=af72555e-0170-1000-c80d-7e992c9de544] has chosen to yield its > resources; will not be scheduled to run again for 1000
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #713: MINIFICPP-1119 MINIFICPP-1154 unify win/posix sockets + fix bugs
szaszm commented on a change in pull request #713: MINIFICPP-1119
MINIFICPP-1154 unify win/posix sockets + fix bugs
URL: https://github.com/apache/nifi-minifi-cpp/pull/713#discussion_r393795914
##
File path: libminifi/src/io/ClientSocket.cpp
##
@@ -0,0 +1,625 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "io/ClientSocket.h"
+#ifndef WIN32
+#include
+#include
+#include
+#include
+#include
+#else
+#include
+#pragma comment(lib, "Ws2_32.lib")
+#endif /* !WIN32 */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "io/validation.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/GeneralUtils.h"
+
+namespace util = org::apache::nifi::minifi::utils;
+
+namespace {
+
+std::string get_last_err_str() {
+#ifdef WIN32
+ const auto error_code = WSAGetLastError();
+#else
+ const auto error_code = errno;
+#endif /* WIN32 */
+ return std::system_category().message(error_code);
+}
+
+std::string get_last_getaddrinfo_err_str(int getaddrinfo_result) {
+#ifdef WIN32
+ (void)getaddrinfo_result; // against unused warnings on windows
+ return get_last_err_str();
+#else
+ return gai_strerror(getaddrinfo_result);
+#endif /* WIN32 */
+}
+
+bool valid_sock_fd(org::apache::nifi::minifi::io::SocketDescriptor fd) {
+#ifdef WIN32
+ return fd != INVALID_SOCKET && fd >= 0;
+#else
+ return fd >= 0;
+#endif /* WIN32 */
+}
+
+std::string sockaddr_ntop(const sockaddr* const sa) {
+ std::string result;
+ if (sa->sa_family == AF_INET) {
+sockaddr_in sa_in{};
+std::memcpy(reinterpret_cast(_in), sa, sizeof(sockaddr_in));
+result.resize(INET_ADDRSTRLEN);
+if (inet_ntop(AF_INET, _in.sin_addr, [0], INET_ADDRSTRLEN) ==
nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else if (sa->sa_family == AF_INET6) {
+sockaddr_in6 sa_in6{};
+std::memcpy(reinterpret_cast(_in6), sa, sizeof(sockaddr_in6));
+result.resize(INET6_ADDRSTRLEN);
+if (inet_ntop(AF_INET6, _in6.sin6_addr, [0], INET6_ADDRSTRLEN)
== nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else {
+throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
"sockaddr_ntop: unknown address family" };
+ }
+ result.resize(strlen(result.c_str())); // discard remaining null bytes at
the end
+ return result;
+}
+
+template
Review comment:
Ok, I will inline the function
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] shawnweeks commented on issue #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable
shawnweeks commented on issue #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable URL: https://github.com/apache/nifi/pull/4148#issuecomment-600142921 +1 from me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFIREG-371) Perform Release Manager activities for 0.6.0
Arpad Boda created NIFIREG-371: -- Summary: Perform Release Manager activities for 0.6.0 Key: NIFIREG-371 URL: https://issues.apache.org/jira/browse/NIFIREG-371 Project: NiFi Registry Issue Type: Task Reporter: Arpad Boda Release 0.6.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-7264: -- Status: Patch Available (was: Open) > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Assignee: Otto Fowler >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] ottobackwards opened a new pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable
ottobackwards opened a new pull request #4148: NIFI-7264 Make jsonPath Expression Logging More Reasonable URL: https://github.com/apache/nifi/pull/4148 add special handling of PathNotFoundExceptions to log to debug Thank you for submitting a contribution to Apache NiFi. Please provide a short description of the PR here: In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically `master`)? - [x] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [x] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [-] Have you written or updated unit tests to verify your changes? - [-] Have you verified that the full build is successful on both JDK 8 and JDK 11? - [- ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [-] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ganesh Banda updated NIFI-7265: --- Component/s: (was: Docker) > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > Labels: Linux, kubernetes > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at >
[jira] [Updated] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ganesh Banda updated NIFI-7265: --- Labels: Linux kubernetes (was: ) > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework, Docker >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > Labels: Linux, kubernetes > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) >
[jira] [Updated] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ganesh Banda updated NIFI-7265: --- Component/s: Docker > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework, Docker >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at >
[jira] [Updated] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
[ https://issues.apache.org/jira/browse/NIFI-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ganesh Banda updated NIFI-7265: --- Component/s: Core Framework Configuration Management > [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: > SUSPENDED and KeepeerError > > > Key: NIFI-7265 > URL: https://issues.apache.org/jira/browse/NIFI-7265 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration Management, Core Framework >Affects Versions: 1.11.3 >Reporter: Ganesh Banda >Priority: Critical > > I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the > Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow > error and never join with the cluster. I need to restart the Nifi each node > to to form cluster back again. Could you please help here ? I think for the > production system restarting is not a good option I feel. Tried to increase > zookeeper timeout to high values but didn't worked. > Logs: > 2020-03-17 14:02:19,708 INFO [main-EventThread] > o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 > Connection State changed to SUSPENDED > 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] > o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e > Connection State changed to SUSPENDED > . > . > . > 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up > org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = > ConnectionLoss > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] > o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss > at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) > at >
[jira] [Created] (NIFI-7265) [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError
Ganesh Banda created NIFI-7265: -- Summary: [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED and KeepeerError Key: NIFI-7265 URL: https://issues.apache.org/jira/browse/NIFI-7265 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.11.3 Reporter: Ganesh Banda I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow error and never join with the cluster. I need to restart the Nifi each node to to form cluster back again. Could you please help here ? I think for the production system restarting is not a good option I feel. Tried to increase zookeeper timeout to high values but didn't worked. Logs: 2020-03-17 14:02:19,708 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 Connection State changed to SUSPENDED 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e Connection State changed to SUSPENDED . . . 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
[GitHub] [nifi] MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
MikeThomsen commented on issue #4104: NIFI-7159 - Add support for some data types in various processors URL: https://github.com/apache/nifi/pull/4104#issuecomment-600106854 As a rule of thumb, if the Linux build passes we can merge. Will try to find some time to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060953#comment-17060953 ] Otto Fowler commented on NIFI-7264: --- Or log at debug or trace > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Assignee: Otto Fowler >Priority: Trivial > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6875) Nifi Zookeeper Cluster_Mode broken in 1.10.0
[ https://issues.apache.org/jira/browse/NIFI-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060948#comment-17060948 ] Ganesh Banda commented on NIFI-6875: Hi Team, I am using Nifi 1.11.3 with external Zookeeper 5.3.6. Able to start the Nifi in a cluster mode. When I made complete ZK down, Nifi throwing bellow error and never join with the cluster. I need to restart the Nifi each node to to form cluster back again. Could you please help here ? I think for the production system restarting is not a good option I feel. Tried to increase zookeeper timeout to high values but didn't worked. Logs: 2020-03-17 14:02:19,708 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2020-03-17 14:02:19,709 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@a032996 Connection State changed to SUSPENDED 2020-03-17 14:02:19,710 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@e3cbc0e Connection State changed to SUSPENDED . . . 2020-03-17 14:19:00,044 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-17 14:19:00,045 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-17 14:19:00,098 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[
https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060949#comment-17060949
]
Otto Fowler commented on NIFI-7264:
---
1. Based on the documentation, I think that the correct change for that
function is to catch path not found and not log, but log other exceptions as it
is now.
${record_json:jsonPathWithDefault('$.pass.reissue_date'),'undefined'} would be
an example.
> Make jsonPath Expression Logging More Reasonable
>
>
> Key: NIFI-7264
> URL: https://issues.apache.org/jira/browse/NIFI-7264
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Assignee: Otto Fowler
>Priority: Trivial
>
> Currently JsonPathEvaluator logs on all errors despite some errors like the
> PathNotFoundException that might only need to be logged at the debug level.
> Since these kind of errors aren't returned to the UI we're just filling up
> the logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #713: MINIFICPP-1119 MINIFICPP-1154 unify win/posix sockets + fix bugs
bakaid commented on a change in pull request #713: MINIFICPP-1119
MINIFICPP-1154 unify win/posix sockets + fix bugs
URL: https://github.com/apache/nifi-minifi-cpp/pull/713#discussion_r393716853
##
File path: libminifi/src/io/ClientSocket.cpp
##
@@ -0,0 +1,625 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "io/ClientSocket.h"
+#ifndef WIN32
+#include
+#include
+#include
+#include
+#include
+#else
+#include
+#pragma comment(lib, "Ws2_32.lib")
+#endif /* !WIN32 */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "io/validation.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/GeneralUtils.h"
+
+namespace util = org::apache::nifi::minifi::utils;
+
+namespace {
+
+std::string get_last_err_str() {
+#ifdef WIN32
+ const auto error_code = WSAGetLastError();
+#else
+ const auto error_code = errno;
+#endif /* WIN32 */
+ return std::system_category().message(error_code);
+}
+
+std::string get_last_getaddrinfo_err_str(int getaddrinfo_result) {
+#ifdef WIN32
+ (void)getaddrinfo_result; // against unused warnings on windows
+ return get_last_err_str();
+#else
+ return gai_strerror(getaddrinfo_result);
+#endif /* WIN32 */
+}
+
+bool valid_sock_fd(org::apache::nifi::minifi::io::SocketDescriptor fd) {
+#ifdef WIN32
+ return fd != INVALID_SOCKET && fd >= 0;
+#else
+ return fd >= 0;
+#endif /* WIN32 */
+}
+
+std::string sockaddr_ntop(const sockaddr* const sa) {
+ std::string result;
+ if (sa->sa_family == AF_INET) {
+sockaddr_in sa_in{};
+std::memcpy(reinterpret_cast(_in), sa, sizeof(sockaddr_in));
+result.resize(INET_ADDRSTRLEN);
+if (inet_ntop(AF_INET, _in.sin_addr, [0], INET_ADDRSTRLEN) ==
nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else if (sa->sa_family == AF_INET6) {
+sockaddr_in6 sa_in6{};
+std::memcpy(reinterpret_cast(_in6), sa, sizeof(sockaddr_in6));
+result.resize(INET6_ADDRSTRLEN);
+if (inet_ntop(AF_INET6, _in6.sin6_addr, [0], INET6_ADDRSTRLEN)
== nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else {
+throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
"sockaddr_ntop: unknown address family" };
+ }
+ result.resize(strlen(result.c_str())); // discard remaining null bytes at
the end
+ return result;
+}
+
+template
Review comment:
Iterating on a linked list is one of the basic CS concepts, one of the first
things people learn. Almost everyone used it, C developers use it all the time,
even C++ developers who interface with OS functions should be pretty familiar
with it. It is easily readable and prevalent in C/C++ codebases.
The point of using templates, at least in my view, is adding generic,
reusable functionality. This template sits in a cpp file, and is used in a
single function, so it is not reused, nor it is very much reusable.
I agree that separating logic from control, in the style of the Algorithms
library can be beneficial, because it can help readbility by enabling the
reader to focus on the main business logic, and it helps avoiding mistakes in
implementing the control flows.
In this case however, the extracted control flow is a simple linked list
iterating loop. To achieve this, a template - with a 2 full line "minimal
concept emulation", which is an eyesore, in my opinion - is created, multiple
lambdas are manufactured and then used for calling the template function.
It is much harder to read than a straightforward loop would be, and since
the control flow it replaces is a trivial one, it does not help too much on
that front either.
On the whole, this abstraction feels completely gratuitous, and reminds me
of the Jurassic Park quote: "Your scientists were so preoccupied with whether
or not they could, they didn’t stop to think if they should."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[
https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060947#comment-17060947
]
Shawn Weeks commented on NIFI-7264:
---
>From the documentation it sounds like returning empty string was intended.
>What are some examples for jsonPathWithDefault that can't already be handled
>by checking if the jsonPath isEmpty and then going for there?
{code:java}
An empty string is generated if the Subject does not contain valid JSON, the
jsonPath is invalid, or the path does not exist in the Subject.
{code}
> Make jsonPath Expression Logging More Reasonable
>
>
> Key: NIFI-7264
> URL: https://issues.apache.org/jira/browse/NIFI-7264
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Assignee: Otto Fowler
>Priority: Trivial
>
> Currently JsonPathEvaluator logs on all errors despite some errors like the
> PathNotFoundException that might only need to be logged at the debug level.
> Since these kind of errors aren't returned to the UI we're just filling up
> the logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #743: Minificpp 1169 - Simplify C2 metrics collection and reporting
msharee9 commented on a change in pull request #743: Minificpp 1169 - Simplify
C2 metrics collection and reporting
URL: https://github.com/apache/nifi-minifi-cpp/pull/743#discussion_r393707126
##
File path: extensions/http-curl/tests/C2JstackTest.cpp
##
@@ -16,152 +16,63 @@
* limitations under the License.
*/
-#include
#undef NDEBUG
-#include
-#include
-#include
-#include
-#include
#include
-#include
-#include
-#include
-#include
-#include
-#include "HTTPClient.h"
-#include "InvokeHTTP.h"
#include "TestBase.h"
-#include "utils/StringUtils.h"
-#include "core/Core.h"
-#include "core/logging/Logger.h"
-#include "core/ProcessGroup.h"
-#include "core/yaml/YamlConfiguration.h"
-#include "FlowController.h"
-#include "properties/Configure.h"
-#include "unit/ProvenanceTestHelper.h"
-#include "io/StreamFactory.h"
-#include "c2/C2Agent.h"
-#include "CivetServer.h"
-#include
-#include "protocols/RESTSender.h"
+#include "HTTPIntegrationBase.h"
+#include "HTTPHandlers.h"
-void waitToVerifyProcessor() {
- std::this_thread::sleep_for(std::chrono::seconds(10));
-}
-
-
-class ConfigHandler : public CivetHandler {
+class VerifyC2DescribeJstack : public VerifyC2Describe {
public:
- ConfigHandler() {
-calls_ = 0;
- }
- bool handlePost(CivetServer *server, struct mg_connection *conn) {
-calls_++;
-std::string heartbeat_response = "{\"operation\" :
\"heartbeat\",\"requested_operations\": [ {"
- "\"operation\" : \"describe\", "
- "\"operationid\" : \"8675309\", "
- "\"name\": \"jstack\""
- "}]}";
- mg_printf(conn, "HTTP/1.1 200 OK\r\nContent-Type: "
-"text/plain\r\nContent-Length: %lu\r\nConnection:
close\r\n\r\n",
-heartbeat_response.length());
- mg_printf(conn, "%s", heartbeat_response.c_str());
-
-
-return true;
+ explicit VerifyC2DescribeJstack(bool isSecure)
+ : VerifyC2Describe(isSecure) {
}
- bool handleGet(CivetServer *server, struct mg_connection *conn) {
-std::ifstream myfile(test_file_location_.c_str());
-
-if (myfile.is_open()) {
- std::stringstream buffer;
- buffer << myfile.rdbuf();
- std::string str = buffer.str();
- myfile.close();
- mg_printf(conn, "HTTP/1.1 200 OK\r\nContent-Type: "
-"text/plain\r\nContent-Length: %lu\r\nConnection:
close\r\n\r\n",
-str.length());
- mg_printf(conn, "%s", str.c_str());
-} else {
- mg_printf(conn, "HTTP/1.1 500 Internal Server Error\r\n");
-}
-
-return true;
+ virtual void runAssertions() {
+assert(LogTestController::getInstance().contains("SchedulingAgent") ==
true);
}
- std::string test_file_location_;
- std::atomic calls_;
};
-int main(int argc, char **argv) {
- mg_init_library(0);
- LogTestController::getInstance().setInfo();
- LogTestController::getInstance().setDebug();
- LogTestController::getInstance().setDebug();
- LogTestController::getInstance().setTrace();
+class DescribeJstackHandler : public HeartbeatHandler {
+ public:
+ explicit DescribeJstackHandler(bool isSecure)
+ : HeartbeatHandler(isSecure) {
+ }
- const char *options[] = { "document_root", ".", "listening_ports", "0", 0 };
- std::vector cpp_options;
- for (int i = 0; i < (sizeof(options) / sizeof(options[0]) - 1); i++) {
-cpp_options.push_back(options[i]);
+ virtual void handleHeartbeat(const rapidjson::Document& root, struct
mg_connection * conn) {
+sendHeartbeatResponse("DESCRIBE", "jstack", "889398", conn);
}
- CivetServer server(cpp_options);
+ virtual void handleAcknowledge(const rapidjson::Document& root) {
+assert(root.HasMember("Flowcontroller threadpool #0") == true);
+ }
- std::string port_str = std::to_string(server.getListeningPorts()[0]);
+};
- ConfigHandler h_ex;
- server.addHandler("/update", h_ex);
- std::string key_dir, test_file_location;
+int main(int argc, char **argv) {
+ std::string key_dir, test_file_location, url;
+ url = "http://localhost:0/api/heartbeat;;
if (argc > 1) {
-h_ex.test_file_location_ = test_file_location = argv[1];
-key_dir = argv[2];
+test_file_location = argv[1];
+if (argc > 2) {
+ url = "https://localhost:0/api/heartbeat;;
+ key_dir = argv[2];
+}
}
+ bool isSecure = false;
+ if (url.find("https") != std::string::npos) {
+isSecure = true;
+ }
- std::shared_ptr configuration =
std::make_shared();
-
- std::string c2_rest_url = "http://localhost:; + port_str + "/update";
-
- configuration->set("c2.rest.url", c2_rest_url);
- configuration->set("c2.agent.heartbeat.period", "1000");
-
- std::shared_ptr test_repo =
std::make_shared();
- std::shared_ptr test_flow_repo =
std::make_shared();
-
- configuration->set(minifi::Configure::nifi_flow_configuration_file,
test_file_location);
-
- std::shared_ptr stream_factory =
minifi::io::StreamFactory::getInstance(configuration);
- std::shared_ptr content_repo =
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #713: MINIFICPP-1119 MINIFICPP-1154 unify win/posix sockets + fix bugs
szaszm commented on a change in pull request #713: MINIFICPP-1119
MINIFICPP-1154 unify win/posix sockets + fix bugs
URL: https://github.com/apache/nifi-minifi-cpp/pull/713#discussion_r393702174
##
File path: libminifi/src/io/ClientSocket.cpp
##
@@ -0,0 +1,625 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "io/ClientSocket.h"
+#ifndef WIN32
+#include
+#include
+#include
+#include
+#include
+#else
+#include
+#pragma comment(lib, "Ws2_32.lib")
+#endif /* !WIN32 */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "io/validation.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/GeneralUtils.h"
+
+namespace util = org::apache::nifi::minifi::utils;
+
+namespace {
+
+std::string get_last_err_str() {
+#ifdef WIN32
+ const auto error_code = WSAGetLastError();
+#else
+ const auto error_code = errno;
+#endif /* WIN32 */
+ return std::system_category().message(error_code);
+}
+
+std::string get_last_getaddrinfo_err_str(int getaddrinfo_result) {
+#ifdef WIN32
+ (void)getaddrinfo_result; // against unused warnings on windows
+ return get_last_err_str();
+#else
+ return gai_strerror(getaddrinfo_result);
+#endif /* WIN32 */
+}
+
+bool valid_sock_fd(org::apache::nifi::minifi::io::SocketDescriptor fd) {
+#ifdef WIN32
+ return fd != INVALID_SOCKET && fd >= 0;
+#else
+ return fd >= 0;
+#endif /* WIN32 */
+}
+
+std::string sockaddr_ntop(const sockaddr* const sa) {
+ std::string result;
+ if (sa->sa_family == AF_INET) {
+sockaddr_in sa_in{};
+std::memcpy(reinterpret_cast(_in), sa, sizeof(sockaddr_in));
+result.resize(INET_ADDRSTRLEN);
+if (inet_ntop(AF_INET, _in.sin_addr, [0], INET_ADDRSTRLEN) ==
nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else if (sa->sa_family == AF_INET6) {
+sockaddr_in6 sa_in6{};
+std::memcpy(reinterpret_cast(_in6), sa, sizeof(sockaddr_in6));
+result.resize(INET6_ADDRSTRLEN);
+if (inet_ntop(AF_INET6, _in6.sin6_addr, [0], INET6_ADDRSTRLEN)
== nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else {
+throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
"sockaddr_ntop: unknown address family" };
+ }
+ result.resize(strlen(result.c_str())); // discard remaining null bytes at
the end
+ return result;
+}
+
+template
Review comment:
No, almost nothing is "absolutely necessary". I wanted to name the logic
there, so that instead of a for loop, we have something with a descriptive name.
Looking up "abomination" gave me this: "something regarded with disgust or
hatred". Do you not like the fact that I extracted the logic or the minimal
concept emulation on the next line? Could you give more details?
In simple terms: please write down what you want/don't want and why.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #713: MINIFICPP-1119 MINIFICPP-1154 unify win/posix sockets + fix bugs
szaszm commented on a change in pull request #713: MINIFICPP-1119
MINIFICPP-1154 unify win/posix sockets + fix bugs
URL: https://github.com/apache/nifi-minifi-cpp/pull/713#discussion_r393702174
##
File path: libminifi/src/io/ClientSocket.cpp
##
@@ -0,0 +1,625 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "io/ClientSocket.h"
+#ifndef WIN32
+#include
+#include
+#include
+#include
+#include
+#else
+#include
+#pragma comment(lib, "Ws2_32.lib")
+#endif /* !WIN32 */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "io/validation.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/GeneralUtils.h"
+
+namespace util = org::apache::nifi::minifi::utils;
+
+namespace {
+
+std::string get_last_err_str() {
+#ifdef WIN32
+ const auto error_code = WSAGetLastError();
+#else
+ const auto error_code = errno;
+#endif /* WIN32 */
+ return std::system_category().message(error_code);
+}
+
+std::string get_last_getaddrinfo_err_str(int getaddrinfo_result) {
+#ifdef WIN32
+ (void)getaddrinfo_result; // against unused warnings on windows
+ return get_last_err_str();
+#else
+ return gai_strerror(getaddrinfo_result);
+#endif /* WIN32 */
+}
+
+bool valid_sock_fd(org::apache::nifi::minifi::io::SocketDescriptor fd) {
+#ifdef WIN32
+ return fd != INVALID_SOCKET && fd >= 0;
+#else
+ return fd >= 0;
+#endif /* WIN32 */
+}
+
+std::string sockaddr_ntop(const sockaddr* const sa) {
+ std::string result;
+ if (sa->sa_family == AF_INET) {
+sockaddr_in sa_in{};
+std::memcpy(reinterpret_cast(_in), sa, sizeof(sockaddr_in));
+result.resize(INET_ADDRSTRLEN);
+if (inet_ntop(AF_INET, _in.sin_addr, [0], INET_ADDRSTRLEN) ==
nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else if (sa->sa_family == AF_INET6) {
+sockaddr_in6 sa_in6{};
+std::memcpy(reinterpret_cast(_in6), sa, sizeof(sockaddr_in6));
+result.resize(INET6_ADDRSTRLEN);
+if (inet_ntop(AF_INET6, _in6.sin6_addr, [0], INET6_ADDRSTRLEN)
== nullptr) {
+ throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
get_last_err_str() };
+}
+ } else {
+throw minifi::Exception{ minifi::ExceptionType::GENERAL_EXCEPTION,
"sockaddr_ntop: unknown address family" };
+ }
+ result.resize(strlen(result.c_str())); // discard remaining null bytes at
the end
+ return result;
+}
+
+template
Review comment:
No, almost nothing is "absolutely necessary". I wanted to name the logic
there, so that instead of a for loop, we have something with a descriptive name.
Looking up "abomination" gave me this: "something regarded with disgust or
hatred". Do you not like the fact that I extracted the logic or the minimal
concept emulation on the next line? Could you give more details?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060917#comment-17060917 ] Otto Fowler commented on NIFI-7264: --- [~Chris S], [~Absolutesantaja] The options, from what I can see are some combination of: 1. leave it as it is 2. have it _not_ log if the exception is pathnotfound 3. have it log debug if the exception is pathnotfound, log error otherwise 4. add new function jsonPathWithDefault that allows you to set the default if path not found > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Assignee: Otto Fowler >Priority: Trivial > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-7264: - Assignee: Otto Fowler > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Assignee: Otto Fowler >Priority: Trivial > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
Shawn Weeks created NIFI-7264: - Summary: Make jsonPath Expression Logging More Reasonable Key: NIFI-7264 URL: https://issues.apache.org/jira/browse/NIFI-7264 Project: Apache NiFi Issue Type: Improvement Reporter: Shawn Weeks Currently JsonPathEvaluator logs on all errors despite some errors like the PathNotFoundException that might only need to be logged at the debug level. Since these kind of errors aren't returned to the UI we're just filling up the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7264) Make jsonPath Expression Logging More Reasonable
[ https://issues.apache.org/jira/browse/NIFI-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060909#comment-17060909 ] Shawn Weeks commented on NIFI-7264: --- 8:17 the code just catches Exception Shawn Weeks 8:17 AM I can't actually make my expected use case fail in 12 snapsho 8:17 I wonder if something has changed recently ottO 8:18 AM if the type is specific enough, then maybe the handling could be better Chris S 8:21 AM com.jayway.jsonpath.PathNotFoundException: No results for path: ... Shawn Weeks 8:22 AM It only fails on nested paths for missing paths and no error is shown in the UI 8:22 So right now we are logging and not throwing ottO 8:22 AM Right, I found the code Chris S 8:22 AM March 17th 2020, 13:11:30 at org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:48)   March 17th 2020, 13:11:30 at org.apache.nifi.attribute.expression.language.evaluation.functions.IsEmptyEvaluator.evaluate(IsEmptyEvaluator.java:35)   March 17th 2020, 13:11:30 at org.apache.nifi.attribute.expression.language.evaluation.functions.AndEvaluator.evaluate(AndEvaluator.java:54)   March 17th 2020, 13:11:30 com.jayway.jsonpath.PathNotFoundException: No results for path: $['...']['...'] Shawn Weeks 8:22 AM So the end user has no visbility unless they check the log. Chris S 8:22 AM (stack trace components reversed as they're chipped to Elasticsearch) new messages ottO 8:23 AM What would you expect to happen with the api as it is? Not Log? Fail Hard? Shawn Weeks 8:24 AM I'd say if it's a normally expected scenario that we wouldn't log it and sparsely populate JSON is a normal scenario. Right now we're logging all errors from JsonPathEvaluator and not raising any of them lol ottO 8:24 AM log debug? Shawn Weeks 8:24 AM That would work Chris S 8:25 AM I'd struggle with failures unless there were a way to accept "path not found" - it's fine (in my use case) for the path to not be present Kart 8:25 AM Shall we start a thread if its fine. so all the communication will be under one thread. 4 replies Last reply today at 8:28 AMView thread ottO 8:25 AM I don’t know how that function is documented Shawn Weeks 8:26 AM @Kart I can't figure out how to start a thread ottO 8:27 AM how about @Shawn Weeks or @Chris S create a jira and post the number here, and we discuss there 8:28 i’m willing to do a pr if it turns out not to be crazy Shawn Weeks 8:28 AM @ottO I'll put a jira in now :+1: 1 > Make jsonPath Expression Logging More Reasonable > > > Key: NIFI-7264 > URL: https://issues.apache.org/jira/browse/NIFI-7264 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Shawn Weeks >Priority: Trivial > > Currently JsonPathEvaluator logs on all errors despite some errors like the > PathNotFoundException that might only need to be logged at the debug level. > Since these kind of errors aren't returned to the UI we're just filling up > the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7263) Add a No tracking Strategy to ListFile/ListFTP
Jens M Kofoed created NIFI-7263: --- Summary: Add a No tracking Strategy to ListFile/ListFTP Key: NIFI-7263 URL: https://issues.apache.org/jira/browse/NIFI-7263 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Jens M Kofoed The Listfile/ListFTP has 2 Listing Strategies: Tracking Timestamps and Tracking Entities. It would be very very nice if the List process also could have a No Tracking (fix it your self) strategy If running NIFI in a cluster the List/Fetch is the perfect solution instead of using a GetFile. But we have had many caces where files in the pickup folder has old timestamps, so here we have to use Tracking Entities. The issue is in cases where you are not allowed to delete files but you have to make a change to the file filter. The tracking entities start all over, and list all files again. In other situations we need to resent all data, and would like to clear the state of the Tracking Entities. But you can't. So I have to make a small flow for detecting duplicates. And in some cases just ignore duplicates and in other caces open up for sending duplicates. But it is a pain in the ... to use the Tracking Entities. So a NO STRATEGY would be very very nice -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7262) Add detailed logging to PutCassandraRecord
Arpad Boda created NIFI-7262: Summary: Add detailed logging to PutCassandraRecord Key: NIFI-7262 URL: https://issues.apache.org/jira/browse/NIFI-7262 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.11.3 Reporter: Arpad Boda PutCassandraRecord doesn't provide any logging to log the queries generated. Some debug lines would help integration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
arpadboda commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st… URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393647823 ## File path: extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h ## @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef __UNORDERED_MAP_KEY_VALUE_STORE_SERVICE_H__ Review comment: Okay, I'm convinced. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
arpadboda commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393646027
##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -760,140 +713,83 @@ void ListSFTP::listByTrackingTimestamps(
}
}
-bool ListSFTP::persistTrackingEntitiesCache(const std::string& hostname, const
std::string& username, const std::string& remote_path) {
- std::ofstream file(tracking_entities_state_filename_);
- if (!file.is_open()) {
-logger_->log_error("Failed to store Tracking Entities state to state file
\"%s\"", tracking_entities_state_filename_.c_str());
-return false;
- }
- file << "hostname=" << hostname << "\n";
- file << "username=" << username << "\n";
- file << "remote_path=" << remote_path << "\n";
- file << "json_state_file=" << tracking_entities_state_json_filename_ << "\n";
- file.close();
-
- std::ofstream json_file(tracking_entities_state_json_filename_);
- if (!json_file.is_open()) {
-logger_->log_error("Failed to store Tracking Entities state to state json
file \"%s\"", tracking_entities_state_json_filename_.c_str());
-return false;
- }
-
- rapidjson::Document entities(rapidjson::kObjectType);
- rapidjson::Document::AllocatorType& alloc = entities.GetAllocator();
- for (const auto& already_listed_entity : already_listed_entities_) {
-rapidjson::Value entity(rapidjson::kObjectType);
-entity.AddMember("timestamp", already_listed_entity.second.timestamp,
alloc);
-entity.AddMember("size", already_listed_entity.second.size, alloc);
-entities.AddMember(rapidjson::Value(already_listed_entity.first.c_str(),
alloc), std::move(entity), alloc);
+bool ListSFTP::persistTrackingEntitiesCache(const
std::shared_ptr& context, const std::string& hostname,
const std::string& username, const std::string& remote_path) {
+ std::unordered_map state;
+ state["listing_strategy"] = listing_strategy_;
+ state["hostname"] = hostname;
+ state["username"] = username;
+ state["remote_path"] = remote_path;
+ size_t i = 0;
+ for (const auto _listed_entity : already_listed_entities_) {
+state["entity." + std::to_string(i) + ".name"] =
already_listed_entity.first;
+state["entity." + std::to_string(i) + ".timestamp"] =
std::to_string(already_listed_entity.second.timestamp);
+state["entity." + std::to_string(i) + ".size"] =
std::to_string(already_listed_entity.second.size);
+++i;
}
-
- rapidjson::OStreamWrapper osw(json_file);
- rapidjson::Writer writer(osw);
- entities.Accept(writer);
-
- return true;
+ return state_manager_->set(state);
}
-bool ListSFTP::updateFromTrackingEntitiesCache(const std::string& hostname,
const std::string& username, const std::string& remote_path) {
- std::ifstream file(tracking_entities_state_filename_);
- if (!file.is_open()) {
-logger_->log_error("Failed to open Tracking Entities state file \"%s\"",
tracking_entities_state_filename_.c_str());
-return false;
- }
+bool ListSFTP::updateFromTrackingEntitiesCache(const
std::shared_ptr& context, const std::string& hostname,
const std::string& username, const std::string& remote_path) {
+ std::string state_listing_strategy;
std::string state_hostname;
std::string state_username;
std::string state_remote_path;
- std::string state_json_state_file;
-
- std::string line;
- while (std::getline(file, line)) {
-size_t separator_pos = line.find('=');
-if (separator_pos == std::string::npos) {
- logger_->log_warn("None key-value line found in Tracking Entities state
file \"%s\": \"%s\"", tracking_entities_state_filename_.c_str(), line.c_str());
- continue;
-}
-std::string key = line.substr(0, separator_pos);
-std::string value = line.substr(separator_pos + 1);
-if (key == "hostname") {
- state_hostname = std::move(value);
-} else if (key == "username") {
- state_username = std::move(value);
-} else if (key == "remote_path") {
- state_remote_path = std::move(value);
-} else if (key == "json_state_file") {
- state_json_state_file = std::move(value);
-} else {
- logger_->log_warn("Unknown key found in Tracking Entities state file
\"%s\": \"%s\"", tracking_entities_state_filename_.c_str(), key.c_str());
-}
- }
- file.close();
-
- if (state_hostname != hostname ||
- state_username != username ||
- state_remote_path != remote_path) {
-logger_->log_error("Tracking Entities state file \"%s\" was created with
different settings than the current ones, ignoring. "
- "Hostname: \"%s\" vs. \"%s\", "
- "Username: \"%s\" vs. \"%s\", "
- "Remote Path: \"%s\" vs. \"%s\"",
- tracking_entities_state_filename_.c_str(),
- state_hostname, hostname,
- state_username, username,
-
[jira] [Updated] (NIFI-7261) Automatic reload of truststore
[ https://issues.apache.org/jira/browse/NIFI-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens M Kofoed updated NIFI-7261: Summary: Automatic reload of truststore (was: Automatic reload of trustsore) > Automatic reload of truststore > -- > > Key: NIFI-7261 > URL: https://issues.apache.org/jira/browse/NIFI-7261 > Project: Apache NiFi > Issue Type: Improvement > Components: Security >Reporter: Jens M Kofoed >Priority: Blocker > Labels: truststore > > When a new remote connection is going to be etablish between new systems. > Both clusters in each end has to restarted after the public keys has been > added to the truststore. It would be very very good if the system automatic > could reload the truststore instead -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7261) Automatic reload of trustsore
Jens M Kofoed created NIFI-7261: --- Summary: Automatic reload of trustsore Key: NIFI-7261 URL: https://issues.apache.org/jira/browse/NIFI-7261 Project: Apache NiFi Issue Type: Improvement Components: Security Reporter: Jens M Kofoed When a new remote connection is going to be etablish between new systems. Both clusters in each end has to restarted after the public keys has been added to the truststore. It would be very very good if the system automatic could reload the truststore instead -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] shawnweeks commented on a change in pull request #4147: NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
shawnweeks commented on a change in pull request #4147: NIFI-7260: Fix error
handling and re-evaluate Module Directory property on changed for scripted
controller services
URL: https://github.com/apache/nifi/pull/4147#discussion_r393620016
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/AbstractScriptedRecordFactory.java
##
@@ -42,9 +42,13 @@ public void setup() {
if (scriptNeedsReload.get() || recordFactory.get() == null) {
if
(ScriptingComponentHelper.isFile(scriptingComponentHelper.getScriptPath())) {
-reloadScriptFile(scriptingComponentHelper.getScriptPath());
+if
(!reloadScriptFile(scriptingComponentHelper.getScriptPath())) {
Review comment:
I'm not sure we need to raise an exception here since reloadScriptFile and
reloadScriptBody already add a validation failure and logs it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st… URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393557670 ## File path: extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h ## @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef __UNORDERED_MAP_KEY_VALUE_STORE_SERVICE_H__ Review comment: I can very well imagine having multiple volatile implementations of the `KeyValueStoreService` in the future, with different properties than this, and I didn't want to designate this **the** `VolatileKeyValueStoreService`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st… URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393557670 ## File path: extensions/standard-processors/controllers/UnorderedMapKeyValueStoreService.h ## @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef __UNORDERED_MAP_KEY_VALUE_STORE_SERVICE_H__ Review comment: I can very well imagine having multiple volatile implementations of the KeyValueStoreService in the future, with different properties than this, and I didn't want to designate this *the* VolatileKeyValueStoreService. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393555662
##
File path: extensions/windows-event-log/Bookmark.cpp
##
@@ -222,7 +184,7 @@ bool Bookmark::getBookmarkXmlFromFile(std::wstring&
bookmarkXml) {
if (std::wstring::npos == pos) {
logger_->log_error("No '!' in bookmarXml '%ls'", bookmarkXml.c_str());
bookmarkXml.clear();
-return createEmptyBookmarkXmlFile();
+ return false;
Review comment:
Thanks, since these are Windows-only, I edited them in VS when migrating
them, and well... this is the result.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393554128
##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -760,140 +713,83 @@ void ListSFTP::listByTrackingTimestamps(
}
}
-bool ListSFTP::persistTrackingEntitiesCache(const std::string& hostname, const
std::string& username, const std::string& remote_path) {
- std::ofstream file(tracking_entities_state_filename_);
- if (!file.is_open()) {
-logger_->log_error("Failed to store Tracking Entities state to state file
\"%s\"", tracking_entities_state_filename_.c_str());
-return false;
- }
- file << "hostname=" << hostname << "\n";
- file << "username=" << username << "\n";
- file << "remote_path=" << remote_path << "\n";
- file << "json_state_file=" << tracking_entities_state_json_filename_ << "\n";
- file.close();
-
- std::ofstream json_file(tracking_entities_state_json_filename_);
- if (!json_file.is_open()) {
-logger_->log_error("Failed to store Tracking Entities state to state json
file \"%s\"", tracking_entities_state_json_filename_.c_str());
-return false;
- }
-
- rapidjson::Document entities(rapidjson::kObjectType);
- rapidjson::Document::AllocatorType& alloc = entities.GetAllocator();
- for (const auto& already_listed_entity : already_listed_entities_) {
-rapidjson::Value entity(rapidjson::kObjectType);
-entity.AddMember("timestamp", already_listed_entity.second.timestamp,
alloc);
-entity.AddMember("size", already_listed_entity.second.size, alloc);
-entities.AddMember(rapidjson::Value(already_listed_entity.first.c_str(),
alloc), std::move(entity), alloc);
+bool ListSFTP::persistTrackingEntitiesCache(const
std::shared_ptr& context, const std::string& hostname,
const std::string& username, const std::string& remote_path) {
+ std::unordered_map state;
+ state["listing_strategy"] = listing_strategy_;
+ state["hostname"] = hostname;
+ state["username"] = username;
+ state["remote_path"] = remote_path;
+ size_t i = 0;
+ for (const auto _listed_entity : already_listed_entities_) {
+state["entity." + std::to_string(i) + ".name"] =
already_listed_entity.first;
+state["entity." + std::to_string(i) + ".timestamp"] =
std::to_string(already_listed_entity.second.timestamp);
+state["entity." + std::to_string(i) + ".size"] =
std::to_string(already_listed_entity.second.size);
+++i;
}
-
- rapidjson::OStreamWrapper osw(json_file);
- rapidjson::Writer writer(osw);
- entities.Accept(writer);
-
- return true;
+ return state_manager_->set(state);
}
-bool ListSFTP::updateFromTrackingEntitiesCache(const std::string& hostname,
const std::string& username, const std::string& remote_path) {
- std::ifstream file(tracking_entities_state_filename_);
- if (!file.is_open()) {
-logger_->log_error("Failed to open Tracking Entities state file \"%s\"",
tracking_entities_state_filename_.c_str());
-return false;
- }
+bool ListSFTP::updateFromTrackingEntitiesCache(const
std::shared_ptr& context, const std::string& hostname,
const std::string& username, const std::string& remote_path) {
+ std::string state_listing_strategy;
std::string state_hostname;
std::string state_username;
std::string state_remote_path;
- std::string state_json_state_file;
-
- std::string line;
- while (std::getline(file, line)) {
-size_t separator_pos = line.find('=');
-if (separator_pos == std::string::npos) {
- logger_->log_warn("None key-value line found in Tracking Entities state
file \"%s\": \"%s\"", tracking_entities_state_filename_.c_str(), line.c_str());
- continue;
-}
-std::string key = line.substr(0, separator_pos);
-std::string value = line.substr(separator_pos + 1);
-if (key == "hostname") {
- state_hostname = std::move(value);
-} else if (key == "username") {
- state_username = std::move(value);
-} else if (key == "remote_path") {
- state_remote_path = std::move(value);
-} else if (key == "json_state_file") {
- state_json_state_file = std::move(value);
-} else {
- logger_->log_warn("Unknown key found in Tracking Entities state file
\"%s\": \"%s\"", tracking_entities_state_filename_.c_str(), key.c_str());
-}
- }
- file.close();
-
- if (state_hostname != hostname ||
- state_username != username ||
- state_remote_path != remote_path) {
-logger_->log_error("Tracking Entities state file \"%s\" was created with
different settings than the current ones, ignoring. "
- "Hostname: \"%s\" vs. \"%s\", "
- "Username: \"%s\" vs. \"%s\", "
- "Remote Path: \"%s\" vs. \"%s\"",
- tracking_entities_state_filename_.c_str(),
- state_hostname, hostname,
- state_username, username,
-
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on a change in pull request #605: MINIFICPP-550 - Implement
RocksDB controller service and component st…
URL: https://github.com/apache/nifi-minifi-cpp/pull/605#discussion_r393551822
##
File path:
extensions/standard-processors/controllers/UnorderedMapPersistableKeyValueStoreService.h
##
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef __UNORDERED_MAP_PERSISTABLE_KEY_VALUE_STORE_SERVICE_H__
+#define __UNORDERED_MAP_PERSISTABLE_KEY_VALUE_STORE_SERVICE_H__
+
+#include "controllers/keyvalue/AbstractAutoPersistingKeyValueStoreService.h"
+#include "UnorderedMapKeyValueStoreService.h"
+#include "core/Core.h"
+#include "properties/Configure.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/Resource.h"
+
+#include
+#include
+#include
+#include
+#include
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace controllers {
+
+class UnorderedMapPersistableKeyValueStoreService : public
AbstractAutoPersistingKeyValueStoreService,
Review comment:
The motivation is twofold: to provide an implementation without any third
party requirements, and to provide a serialized format which is human-readable
and easily editable.
As for the first motivation: we don't want to make it a hard requirement to
have rocksdb to use stateful processors, because that would hinder
low-footprint usages, nor do I want to make any other third party a dependency
for this: it would, again, introduce an unwanted requirement for low-footprint
usages and we would have to find one that supports all target platforms, and
maintain it, which is not a negligible maintenance burden.
For the second one: so far users could easily clear a state by deleting the
state file or directory. With the rocksdb state storage this is no longer a
trivial task. The C2 methods introduced here attempt to solve this problem, but
until they are finalized (or if someone does not want to use, or can't use C2),
they aren't really useful.
A solution for this is using this state storage method: its serialized
format is trivial and editable in text format without any third party tools,
making it easy to clear (or even modify) the state of processors.
Finally for the support argument: this is a trivial, and, if you take a look
at `PersistableKeyValueStoreServiceTest.cpp`, well-tested implementation, that
I don't think would introduce a significant maintenance burden going forward.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (NIFI-7238) Conduct Apache NiFi 1.11.4 Release
[ https://issues.apache.org/jira/browse/NIFI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060738#comment-17060738 ] Pierre Villard commented on NIFI-7238: -- Hi [~jmkofoed] - hopefully, we could have 1.11.4 by the end of the week or early next week if everything goes as planned. > Conduct Apache NiFi 1.11.4 Release > -- > > Key: NIFI-7238 > URL: https://issues.apache.org/jira/browse/NIFI-7238 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.11.4 > > > NIFI-7223 7374361b5c6398a92577479d297e55dbe7ea2f63, > f9d75056fa293acf7c2965d9cdba62f2b151886e, > 69b6c231fddb4c547905e85c4087af18ea77c38f, > 3feb85a030c6766941e64875ff825560878d54cc > NIFI-7208 ace575452415a092fb6923d947e010eeb1e41c96 > NIFI-7256 0ad58200af01c5c07e25770acd967e7321800547 > NIFI-7250 561be89a2174542f2e089e72a64225d8300a623b > NIFI-7249 798a8eeb509209015e619f0ec4b6826f2754634f > NIFI-7229 dc95cff3b75dbaba5a95044c07da7ce0936522b3 > NIFI-7251 dbda8c7f7eb4de035414bb5c478d6969c387d3e3 > NIFI-6742 decb5d062ea20714b666906e1414e0c9a8e4064c > NIFI-7244 97e250cdaa4731e9d2b8cca9ed041848b1410e7a > NIFI-7248 dfaef3880515c4018727589a3f5e681a34b55e5e > NIFI-7241 d4a2afc25ce35570dff73e6cb02a6820dd7062e1 > NIFI-7119 290bd378d5e219dabac8f3ecf2bf9c69451f1c3c > NIFI-7242 d68720920fa10155eff81e1c1314a9286ba7a51c > NIFI-7236 cd83e70b91fb1e07381ec55bd86605c376b8db44, > 67676ba5b48c72c5b3dc9994d386e99832e708df > NIFI-7245 3de3ad40290ccd4c9e09d5c4fd03e83a6cbf0d86 > NIFI-7050 89d8b877f9adc3f474983c4dec45bf4586a6baef > NIFI-7239 9cb85a4fd0b42d0b78e4c9a957cd6f659e1c91d0 > NIFI-4970 a679e88b6f4f0e5f800aba5a6891494bdbdfb193 > NIFI-7200 afad982e91debd1109a6ec6d1865a77e8b3470ee > NIFI-7210 12c8402ac3e648dd63b959b415cea61099b10864 > NIFI-7233 a33a5e35b3e56ed66c7befb82e2b7d97e9c8131b, > 57947cd2a205a522b3d929fde8c0cf102a7a59c6 > NIFI-7195 0f775f3a578ad855d4768619d361ff30a0f380ab > NIFI-7224 ac4d52b6ca7c4a11a8f3a303aebac3e3ebed6144 > NIFI-7197 578430c9d9d52cabb0d60dd4a21634475ba2db9e > NIFI-7231 f4b65afb6462dc52420fe93eaca55f0a319ec0ae > NIFI-7191 cca54f7ff22709debe9f3b3b9b97eee92e1e0193 > NIFI-7232 20dda05f2622fe9153c50989d36d7fa0268136e9, > f283c1191ca207217f3e22c24574e340b10db580 > NIFI-7226 7c57e75da42bf9ca8b8c0b609717314c84124970 > NIFI-7222 040c8a0af9f2058188dacd5818ec7acbe2635788 > NIFI-7055 f1c6e92df58bf24eb5199cdcb1784cbc438946db, > b82fec41d96ccf808ab060ac68433cb95421212b > NIFI-7227 7773681eeaa82f9c4099a3191d1f7f784f91bf7a > NIFI-7208 74b1b2fc596f43389b9a7629e4c8544e9e008997 > NIFI-4970 bad254ec5b3ea6ab70a9aed673158a21ec40c378 > NIFI-7121 0b2816baa4a44ac8e2128e4730796cc193d8bced > NIFI-7218 cb08382da4cb888c1d8cdfab078a57c0c2986501 > NIFI-5644 2cf4fde6868aa7148148fa45371c255b75579faf > NIFI-6856 23b04ae96863fc3b0fd3b268bcb7d155edf17f26 > NIFI-7206,NIFI-7205 4cd63c99e8f412650caa2f3b775a9647f6414a98 > NIFI-7201 bad0f10a52de4822147dfb8a13f35eccdbc0bb3c > NIFI-7164 c092a23bdf787f68fc061aa4c546d87a05b5753c > NIFI-7183 9a8a551e03753c8bd9b256613572e42b7d831e8f > NIFI-7185 5722b99432ce075f93659c168ba367d38006fdf1c > NIFI-7178 e631851e0f7e82b1e4bd33eb214fbd875a0d6cb1 > Insert these commits from bottom to top, left to right - to match order they > appear on master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors
Xeinn commented on issue #4104: NIFI-7159 - Add support for some data types in various processors URL: https://github.com/apache/nifi/pull/4104#issuecomment-599916838 @MikeThomsen @tpalfy Seem to have some issues reported from the Windows build. Not sure if they are related to something I have done or results of commits I've rebased from master. However I've pushed the changes as suggested above. Would appreciate you checking them again especially the hbase changes. Question. Should I look to add parameters to affected processors to allow users to select to turn on handling of Decimal type and try where not enabled to maintain current behaviour? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
