[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1281: MINIFICPP-1769 Add propertyValue to agent manifest metadata
szaszm commented on a change in pull request #1281:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1281#discussion_r831782664
##
File path: libminifi/include/properties/ConfigurationReader.h
##
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+namespace org::apache::nifi::minifi {
+
+class ConfigurationReader {
+ public:
+ virtual bool get(const std::string& key, std::string& value) const = 0;
+ virtual bool get(const std::string& key, const std::string& alternate_key,
std::string& value) const = 0;
+ virtual std::optional get(const std::string& key) const = 0;
+};
Review comment:
I mean all I'm seeing in AgentInformation is setters for random objects,
and I don't think it makes a difference if that object is a custom class versus
a `std::function`, a setter is a setter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] marcus41495 commented on a change in pull request #5887: NIFI-8124 Added cookie redirect handling to InvokeHTTP
marcus41495 commented on a change in pull request #5887:
URL: https://github.com/apache/nifi/pull/5887#discussion_r831779683
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
##
@@ -516,6 +519,15 @@
)
.build();
+public static final PropertyDescriptor PROP_ENABLE_COOKIE_REDIRECTS = new
PropertyDescriptor.Builder()
+.name("enable-cookie-redirects")
+.description("If true, when an HTTP server responds to a request
with a redirect, the provided cookie will be copied to the following request.")
+.displayName("Enable Cookie Redirects")
+.required(true)
+.defaultValue("false")
+.allowableValues("true", "false")
Review comment:
You make good points, thanks for your input @exceptionfactory! I went
ahead and implemented the changes you suggested. Let me know if they look
alright.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] github-actions[bot] commented on pull request #4754: NIFI-7417: GetAzureCosmosDBRecord processor
github-actions[bot] commented on pull request #4754: URL: https://github.com/apache/nifi/pull/4754#issuecomment-1074546504 We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5887: NIFI-8124 Added cookie redirect handling to InvokeHTTP
exceptionfactory commented on a change in pull request #5887:
URL: https://github.com/apache/nifi/pull/5887#discussion_r831602716
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
##
@@ -516,6 +519,15 @@
)
.build();
+public static final PropertyDescriptor PROP_ENABLE_COOKIE_REDIRECTS = new
PropertyDescriptor.Builder()
+.name("enable-cookie-redirects")
+.description("If true, when an HTTP server responds to a request
with a redirect, the provided cookie will be copied to the following request.")
+.displayName("Enable Cookie Redirects")
+.required(true)
+.defaultValue("false")
+.allowableValues("true", "false")
Review comment:
The configured cookie policy applies to all types of responses, not just
redirects. For that reason, adjusting the property name would be helpful.
Rather than having a `Boolean` value, it would be helpful to define this
property as a strategy, defaulting to `Accept None`. This would provide the
opportunity for additional strategies down the road. A new `CookieStrategy`
enum could be defined with values of `DISABLED` and `ACCEPT_ALL`.
```suggestion
public static final PropertyDescriptor PROP_COOKIE_STRATEGY = new
PropertyDescriptor.Builder()
.name("cookie-strategy")
.description("Strategy for accepting and persisting HTTP
cookies. Accepting cookies enables persistence across multiple requests.")
.displayName("Cookie Strategy")
.required(true)
.defaultValue(CookieStrategy.DISABLED)
.allowableValues(CookieStrategy.values())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on a change in pull request #5862: NIFI-8106 Added an "Are you sure ?" prompt when delete a processor group
thenatog commented on a change in pull request #5862:
URL: https://github.com/apache/nifi/pull/5862#discussion_r831579776
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-actions.js
##
@@ -1153,42 +1153,75 @@
if (selection.size() === 1) {
var selectionData = selection.datum();
var revision = nfClient.getRevision(selectionData);
-
-$.ajax({
-type: 'DELETE',
-url: selectionData.uri + '?' + $.param({
-'version': revision.version,
-'clientId': revision.clientId,
-'disconnectedNodeAcknowledged':
nfStorage.isDisconnectionAcknowledged()
-}),
-dataType: 'json'
-}).done(function (response) {
-// remove the component/connection in question
-
nfCanvasUtils.getComponentByType(selectionData.type).remove(selectionData.id);
-
-// if the selection is a connection, reload the source
and destination accordingly
-if (nfCanvasUtils.isConnection(selection) === false) {
-var connections =
nfConnection.getComponentConnections(selectionData.id);
-if (connections.length > 0) {
-var ids = [];
-$.each(connections, function (_, connection) {
-ids.push(connection.id);
-});
-
-// remove the corresponding connections
-nfConnection.remove(ids);
-}
-}
-
-// update URL deep linking params
-nfCanvasUtils.setURLParameters();
-
-// refresh the birdseye
-nfBirdseye.refresh();
-// inform Angular app values have changed
-nfNgBridge.digest();
-}).fail(nfErrorHandler.handleAjaxError);
-} else {
+
+ if
(nfCanvasUtils.isProcessGroup(selection) === true) {
Review comment:
The whitespacing on this section of code has gone wrong. Can we update
the whitespace to align with the rest of the file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7230) Improve EL to support microseconds/nanoseconds precision
[
https://issues.apache.org/jira/browse/NIFI-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-7230:
-
Status: Patch Available (was: In Progress)
> Improve EL to support microseconds/nanoseconds precision
>
>
> Key: NIFI-7230
> URL: https://issues.apache.org/jira/browse/NIFI-7230
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Lehel Boér
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Using the EL, there is no way to convert a date like "2020-02-26
> 20:23:53.439134" into its numerical representation (timestamp with
> microsecond/nanosecond precision).
> At the moment, if trying to do:
>
> {code:java}
> ${ts:toDate("-MM-dd HH:mm:ss.SS"):toNumber()}
> {code}
> It'll give a timestamp with milliseconds precision.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] Lehel44 opened a new pull request #5888: NIFI-7230: Created toInstant() and formatInstant() EL functions
Lehel44 opened a new pull request #5888:
URL: https://github.com/apache/nifi/pull/5888
…e functions
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
https://issues.apache.org/jira/browse/NIFI-7230
Added **toInstant** and **formatInstant** functions to EL to support
nanosecond precision datetime.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-7230) Improve EL to support microseconds/nanoseconds precision
[
https://issues.apache.org/jira/browse/NIFI-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér reassigned NIFI-7230:
Assignee: Lehel Boér (was: Mani Jindal)
> Improve EL to support microseconds/nanoseconds precision
>
>
> Key: NIFI-7230
> URL: https://issues.apache.org/jira/browse/NIFI-7230
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Lehel Boér
>Priority: Major
>
> Using the EL, there is no way to convert a date like "2020-02-26
> 20:23:53.439134" into its numerical representation (timestamp with
> microsecond/nanosecond precision).
> At the moment, if trying to do:
>
> {code:java}
> ${ts:toDate("-MM-dd HH:mm:ss.SS"):toNumber()}
> {code}
> It'll give a timestamp with milliseconds precision.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] marcus41495 opened a new pull request #5887: NIFI-8124 Added cookie redirect handling to InvokeHTTP
marcus41495 opened a new pull request #5887:
URL: https://github.com/apache/nifi/pull/5887
Description of PR
NIFI-8124 Adds support for persistent cookies within the InvokeHTTP
processor. Previously, cookies weren't being carried over during redirects.
This addition adds a new property "Enable Cookie Redirects", which will create
a persistent cookie jar file that will properly handle cookies during HTTP
redirects. By default, this is set to false, which will produce the current
behavior of the processor.
See https://issues.apache.org/jira/browse/NIFI-8124
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [x] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8106) An "Are you sure?" prompt when deleting a PG
[ https://issues.apache.org/jira/browse/NIFI-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-8106: -- Status: Patch Available (was: Open) > An "Are you sure?" prompt when deleting a PG > > > Key: NIFI-8106 > URL: https://issues.apache.org/jira/browse/NIFI-8106 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.12.1, 1.11.4, 1.11.3, 1.11.1 >Reporter: Itay v >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Deleting a process group from the UI is a bit too easy - you select one and > press delete. The problem is that it can be very dangerous to delete an > entire process group, and without having a prompt in the UI asking if you are > sure about deleting it, a lot of time and resources could be wasted on a > single accidental press of one button -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9780) Conduct 1.16.0 Release
[ https://issues.apache.org/jira/browse/NIFI-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510109#comment-17510109 ] ASF subversion and git services commented on NIFI-9780: --- Commit 0419dc2939d726c9bc489d7eca8f7eed085c0e89 in nifi's branch refs/heads/NIFI-9780-RC3 from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0419dc2 ] NIFI-9780-RC3 prepare for next development iteration > Conduct 1.16.0 Release > -- > > Key: NIFI-9780 > URL: https://issues.apache.org/jira/browse/NIFI-9780 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9780) Conduct 1.16.0 Release
[ https://issues.apache.org/jira/browse/NIFI-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510108#comment-17510108 ] ASF subversion and git services commented on NIFI-9780: --- Commit b019a9191f1c83bc7f547dc02c1b679b8936acee in nifi's branch refs/heads/NIFI-9780-RC3 from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b019a91 ] NIFI-9780-RC3 prepare release nifi-1.16.0-RC3 > Conduct 1.16.0 Release > -- > > Key: NIFI-9780 > URL: https://issues.apache.org/jira/browse/NIFI-9780 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] mcgilman commented on pull request #5862: NIFI-8106 Added an "Are you sure ?" prompt when delete a processor group
mcgilman commented on pull request #5862: URL: https://github.com/apache/nifi/pull/5862#issuecomment-1074403894 Will review... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510099#comment-17510099 ] David Handermann commented on NIFI-9820: When upgrading from 1.14.0 and later, the existing value would carry through to a new version. However, when upgrading from something older, such as 1.13.2, that instance would get the new default value. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9391) MergeRecord does not work in stateless flows
[
https://issues.apache.org/jira/browse/NIFI-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-9391:
-
Fix Version/s: 1.16.0
> MergeRecord does not work in stateless flows
>
>
> Key: NIFI-9391
> URL: https://issues.apache.org/jira/browse/NIFI-9391
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Labels: stateless
> Fix For: 1.16.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The stateless runtime triggers a processor up until FFs are available in the
> upstream queue.
> MergeRecord needs an extra onTrigger() call where session.get() yields no
> result in order to get notified that no more FFs are available and it is time
> to send the merged FF downstream. This extra call will never happen in
> stateless environment and MergeRecord cannot send out FF in this way.
> Modify MergeRecord to run session.get() in a loop within a single onTrigger()
> call like MergeContent does:
> {code:java}
> while (binManager.getBinCount() <=
> context.getProperty(MAX_BIN_COUNT).asInteger()) {
> if (!isScheduled()) {
> break;
> }
> final ProcessSession session = sessionFactory.createSession();
> final List flowFiles = session.get(1000);
> if (flowFiles.isEmpty()) {
> break;
> }
> ...
> {code}
> NIFI-9390 also affects MergeContent when it is not in the starting position
> in the flow but this jira needs to be fixed first (otherwise the other cannot
> pop up).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9793) System Integration Test Failure with HTTP 409
[
https://issues.apache.org/jira/browse/NIFI-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Grey updated NIFI-9793:
Fix Version/s: 1.17.0
> System Integration Test Failure with HTTP 409
> -
>
> Key: NIFI-9793
> URL: https://issues.apache.org/jira/browse/NIFI-9793
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.17.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> System Integration tests for NiFi reported the following failure in a recent
> automated build on GitHub:
> {noformat}
> Error:
> org.apache.nifi.tests.system.verification.ClusteredVerifiableControllerServiceSystemIT.testVerificationWithValidConfigWhenComponentRunning
> Time elapsed: 1.096 s <<< ERROR!
> org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error
> deleting Controller Service: Node localhost:5671 is unable to fulfill this
> request due to: Controller Service 85f6b66c-017f-1000--68457893
> cannot be deleted because it is not disabled
> Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
> Error:
> ClusteredVerifiableControllerServiceSystemIT>NiFiSystemIT.teardown:126->NiFiSystemIT.destroyFlow:177
> » NiFiClient
> {noformat}
> The impacted portion of the test should be adjusted to retry when receiving a
> response with HTTP 409.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9793) System Integration Test Failure with HTTP 409
[
https://issues.apache.org/jira/browse/NIFI-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Grey updated NIFI-9793:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> System Integration Test Failure with HTTP 409
> -
>
> Key: NIFI-9793
> URL: https://issues.apache.org/jira/browse/NIFI-9793
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> System Integration tests for NiFi reported the following failure in a recent
> automated build on GitHub:
> {noformat}
> Error:
> org.apache.nifi.tests.system.verification.ClusteredVerifiableControllerServiceSystemIT.testVerificationWithValidConfigWhenComponentRunning
> Time elapsed: 1.096 s <<< ERROR!
> org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error
> deleting Controller Service: Node localhost:5671 is unable to fulfill this
> request due to: Controller Service 85f6b66c-017f-1000--68457893
> cannot be deleted because it is not disabled
> Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
> Error:
> ClusteredVerifiableControllerServiceSystemIT>NiFiSystemIT.teardown:126->NiFiSystemIT.destroyFlow:177
> » NiFiClient
> {noformat}
> The impacted portion of the test should be adjusted to retry when receiving a
> response with HTTP 409.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9793) System Integration Test Failure with HTTP 409
[
https://issues.apache.org/jira/browse/NIFI-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510094#comment-17510094
]
ASF subversion and git services commented on NIFI-9793:
---
Commit 5928d2048ee9e9f8b4572832a12816bde5419e23 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5928d20 ]
NIFI-9793 Added wait for Controller Service run status in tests
- Added check for controller-level service run status in
disableControllerLevelServices to avoid HTTP 409 Conflict response when
attempting to delete services
This closes #5863
Signed-off-by: Paul Grey
> System Integration Test Failure with HTTP 409
> -
>
> Key: NIFI-9793
> URL: https://issues.apache.org/jira/browse/NIFI-9793
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> System Integration tests for NiFi reported the following failure in a recent
> automated build on GitHub:
> {noformat}
> Error:
> org.apache.nifi.tests.system.verification.ClusteredVerifiableControllerServiceSystemIT.testVerificationWithValidConfigWhenComponentRunning
> Time elapsed: 1.096 s <<< ERROR!
> org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error
> deleting Controller Service: Node localhost:5671 is unable to fulfill this
> request due to: Controller Service 85f6b66c-017f-1000--68457893
> cannot be deleted because it is not disabled
> Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
> Error:
> ClusteredVerifiableControllerServiceSystemIT>NiFiSystemIT.teardown:126->NiFiSystemIT.destroyFlow:177
> » NiFiClient
> {noformat}
> The impacted portion of the test should be adjusted to retry when receiving a
> response with HTTP 409.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] greyp9 closed pull request #5863: NIFI-9793 Add Wait for Controller Service Run Status in Tests
greyp9 closed pull request #5863: URL: https://github.com/apache/nifi/pull/5863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9793) System Integration Test Failure with HTTP 409
[
https://issues.apache.org/jira/browse/NIFI-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510083#comment-17510083
]
David Handermann commented on NIFI-9793:
The ReportingTaskIT also appears to fail with a similar error for
Controller-level Service:
{noformat}
Error: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.492
s <<< FAILURE! - in org.apache.nifi.tests.system.reportingtask.ReportingTaskIT
Error:
org.apache.nifi.tests.system.reportingtask.ReportingTaskIT.testReportingTaskDependingOnControllerService
Time elapsed: 52.371 s <<< ERROR!
org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error
enabling or disabling controller service: 9fbbce78-017f-1000-2f3d-72705ac13954
cannot be disabled because it is referenced by 1 components that are currently
running: [9fbbcf28-017f-1000-838b-46236e76dd9e]
Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
Error:
ReportingTaskIT>NiFiSystemIT.teardown:126->NiFiSystemIT.destroyFlow:175 »
NiFiClient
{noformat}
> System Integration Test Failure with HTTP 409
> -
>
> Key: NIFI-9793
> URL: https://issues.apache.org/jira/browse/NIFI-9793
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> System Integration tests for NiFi reported the following failure in a recent
> automated build on GitHub:
> {noformat}
> Error:
> org.apache.nifi.tests.system.verification.ClusteredVerifiableControllerServiceSystemIT.testVerificationWithValidConfigWhenComponentRunning
> Time elapsed: 1.096 s <<< ERROR!
> org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException: Error
> deleting Controller Service: Node localhost:5671 is unable to fulfill this
> request due to: Controller Service 85f6b66c-017f-1000--68457893
> cannot be deleted because it is not disabled
> Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
> Error:
> ClusteredVerifiableControllerServiceSystemIT>NiFiSystemIT.teardown:126->NiFiSystemIT.destroyFlow:177
> » NiFiClient
> {noformat}
> The impacted portion of the test should be adjusted to retry when receiving a
> response with HTTP 409.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9800) Unwrap SQLExceptions in PutDatabaseRecord when table does not exist
[ https://issues.apache.org/jira/browse/NIFI-9800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-9800: --- Fix Version/s: 1.16.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Unwrap SQLExceptions in PutDatabaseRecord when table does not exist > --- > > Key: NIFI-9800 > URL: https://issues.apache.org/jira/browse/NIFI-9800 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.16.0 > > Time Spent: 40m > Remaining Estimate: 0h > > As of NIFI-5757, SQLExceptions in PutDatabaseRecord's code to determine table > information have been wrapped in a ProcessException as the code was moved > into a lambda. Until NIFI-8146, this introduced a regression as the flowfile > would be rolled back (even if Rollback on Failure is false) rather than being > routed to failure. > The refactor of PutDatabaseRecord in NIFI-8146 changed the logic to correctly > route to failure; however the exception is still wrapped in a > ProcessException and should be unwrapped before logging. Behavior (transfer > to failure) should remain the same and thus avoid another regression. > To test, simply send a flowfile to PutDatabaseRecord to a table that doesn't > exist. In NiFi 1.8, it will be routed to failure. In NiFi 1.9-1.12, it will > be rolled back. In 1.13+ it is again routed to failure. After this Jira, it > should still be routed to failure but logged as a SQLException rather than a > ProcessException with a SQLException cause. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9800) Unwrap SQLExceptions in PutDatabaseRecord when table does not exist
[ https://issues.apache.org/jira/browse/NIFI-9800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510054#comment-17510054 ] ASF subversion and git services commented on NIFI-9800: --- Commit c84f43878205726a4e7b891ebd64068516bcfe68 in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c84f438 ] NIFI-9800: Unwrap SQLException in PutDatabaseRecord when table does not exist Signed-off-by: Matthew Burgess This closes #5871 > Unwrap SQLExceptions in PutDatabaseRecord when table does not exist > --- > > Key: NIFI-9800 > URL: https://issues.apache.org/jira/browse/NIFI-9800 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > As of NIFI-5757, SQLExceptions in PutDatabaseRecord's code to determine table > information have been wrapped in a ProcessException as the code was moved > into a lambda. Until NIFI-8146, this introduced a regression as the flowfile > would be rolled back (even if Rollback on Failure is false) rather than being > routed to failure. > The refactor of PutDatabaseRecord in NIFI-8146 changed the logic to correctly > route to failure; however the exception is still wrapped in a > ProcessException and should be unwrapped before logging. Behavior (transfer > to failure) should remain the same and thus avoid another regression. > To test, simply send a flowfile to PutDatabaseRecord to a table that doesn't > exist. In NiFi 1.8, it will be routed to failure. In NiFi 1.9-1.12, it will > be rolled back. In 1.13+ it is again routed to failure. After this Jira, it > should still be routed to failure but logged as a SQLException rather than a > ProcessException with a SQLException cause. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] mattyb149 closed pull request #5871: NIFI-9800: Unwrap SQLException in PutDatabaseRecord when table does not exist
mattyb149 closed pull request #5871: URL: https://github.com/apache/nifi/pull/5871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8521) Replace/improve InfluxDB extension.
[ https://issues.apache.org/jira/browse/NIFI-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510046#comment-17510046 ] ASF subversion and git services commented on NIFI-8521: --- Commit 2dc23e7309c6b15231b66d933fe0d77ac5db083d in nifi's branch refs/heads/main from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2dc23e7 ] NIFI-8521 Removed nifi-influxdb-nar package from nifi-assembly This closes #5880 Signed-off-by: David Handermann > Replace/improve InfluxDB extension. > --- > > Key: NIFI-8521 > URL: https://issues.apache.org/jira/browse/NIFI-8521 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: GAbor Bilicz >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The builtin InfluxDB Processor requires well formatted data. But no > Processor/Controller service offered, to do this easily from a JSON for > example. > On this link they created an extension for this. I don't know, what's the > process, and howto discuss it with them, etc. It's just an idea, but I think > it would be cool. They also supporting InfluxDB 1.x and InlfuxDB 2.x > [https://github.com/influxdata/nifi-influxdb-bundle > BR, > Gabor|https://github.com/influxdata/nifi-influxdb-bundle] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-8521) Replace/improve InfluxDB extension.
[ https://issues.apache.org/jira/browse/NIFI-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-8521. Fix Version/s: 1.16.0 Resolution: Fixed Removed nifi-influxdb-nar from nifi-assembly. > Replace/improve InfluxDB extension. > --- > > Key: NIFI-8521 > URL: https://issues.apache.org/jira/browse/NIFI-8521 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: GAbor Bilicz >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The builtin InfluxDB Processor requires well formatted data. But no > Processor/Controller service offered, to do this easily from a JSON for > example. > On this link they created an extension for this. I don't know, what's the > process, and howto discuss it with them, etc. It's just an idea, but I think > it would be cool. They also supporting InfluxDB 1.x and InlfuxDB 2.x > [https://github.com/influxdata/nifi-influxdb-bundle > BR, > Gabor|https://github.com/influxdata/nifi-influxdb-bundle] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9816) Ensure "lang" attribute is set in tags
[ https://issues.apache.org/jira/browse/NIFI-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Conti-Vock updated NIFI-9816: -- Component/s: (was: NiFi Registry) > Ensure "lang" attribute is set in tags > - > > Key: NIFI-9816 > URL: https://issues.apache.org/jira/browse/NIFI-9816 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.3 >Reporter: Jonathan Conti-Vock >Assignee: Jonathan Conti-Vock >Priority: Minor > > Accessibility standards require every tag to have a "lang" attribute > where the value represents the primary language of the document. From the > [WCAG 2.1 requirement's > description|[https://www.w3.org/WAI/WCAG21/Techniques/html/H57]], identifying > the language of the document is important for a number of reasons: > * It allows braille translation software to substitute control codes for > accented characters, and insert control codes necessary to prevent erroneous > creation of Grade 2 braille contractions. > * Speech synthesizers that support multiple languages will be able to orient > and adapt to the pronunciation and syntax that are specific to the language > of the page, speaking the text in the appropriate accent with proper > pronunciation. > * Marking the language can benefit future developments in technology, for > example users who are unable to translate between languages themselves will > be able to use machines to translate unfamiliar languages. > * Marking the language can also assist user agents in providing definitions > using a dictionary. > While reviewing the backlog for existing tickets related to localization, I > found there's already been some significant (though stale) work done to > implement localization/multilingualization (NIFI-4785). Given that work, I > would suggest compatible JSTL changes be made to the existing JSPs and > otherwise hardcode lang="en" in existing HTML pages until further > localization improvements are made. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory closed pull request #5880: NIFI-8521 Removed influxdb package.
exceptionfactory closed pull request #5880: URL: https://github.com/apache/nifi/pull/5880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-7096) Reporting task for ElasticSearch
[ https://issues.apache.org/jira/browse/NIFI-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved NIFI-7096. Resolution: Won't Do The recommended flow for reporting tasks to a data store is to use a remote process group on another instance or cluster and have that cluster use the standard bundles where appropriate to add the data to the data store. > Reporting task for ElasticSearch > > > Key: NIFI-7096 > URL: https://issues.apache.org/jira/browse/NIFI-7096 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Xun REN >Priority: Major > > Hi, > There is no complet NiFi reporting tasks for ElasticSearch. > It is good to have this feature. > Inspired from another project, I created a new one to report almost all the > information/metrics to ElasticSearch. Like Bulletin info, JVM Metrics, Status > etc. I could also add provenance repository information later. > The project can be found here: > [https://github.com/renxunsaky/nifi-elastic-reporting] > > If you find it is useful, I could further improve it and create a PR in order > to integrate it into the Elasticsearch bundle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-7113) Developing NiFi custom processors from maven archetype in NetBeans 8.2 fails
[ https://issues.apache.org/jira/browse/NIFI-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved NIFI-7113. Resolution: Won't Fix Problems with NetBeans need to be referred to the NetBeans development team. > Developing NiFi custom processors from maven archetype in NetBeans 8.2 fails > > > Key: NIFI-7113 > URL: https://issues.apache.org/jira/browse/NIFI-7113 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: nifi-nar-maven-plugin-1.3.1, 1.11.0 > Environment: Windows 10 > NetBeans 8.2 >Reporter: Valentina Ivanova >Priority: Major > Attachments: project-settings.png, result.png, select-archetype.png > > > Cannot create a project from a maven Archetype. > Steps to reproduce in NetBeans 8.2: > File-> New Project -> Maven -> Project from Maven Archetype and then > selecting nifi-processor-bundle-archetype from the available archetypes. > (screenshots attached) > Related info: > After some investigation I figured out that file can't be downloaded: > http://repo.maven.apache.org/maven2/org/apache/nifi/nifi/1.11.0/nifi-1.11.0.pom > as the protocol needs to be set to HTTPS. So I created a custom repository > (and built the index) which uses the HTTPS protocol and try to create the > project using it. However, it seems the HTTP version is still used, so > creating the project failed again. > When I browsed the Maven repositories in the Services tab and tried to > download certain org.apache.nifi poms I got error messages that some > dependencies are broken which is not clear how to fix. > I figured out further that the protocol has not been changed to https for the > nifi-nar-maven-plugin. The path to it is: > Expand Services tab, expand Maven Repository, Central Repository and > org.apache.nifi under it. > Expand the nifi-nar-maven-plugin and download the 1.3.1 jar. > The pom for which the protocol has not been changed is under > META-INF.maven.org.apache.nifi.nifi-nar-maven-plugin . -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509972#comment-17509972 ] Josef Zahner commented on NIFI-9820: Everything is better than the value today :). Correct, an upgrade to a newer NiFi should not change the property or have an impact for existing users. However this isn't the case anyway - right? Because as soon as the processor is on the canvas, I would expect that the flow.xml.gz file stores the value in any case. So an existing processor shouldn't be impacted if NiFi changes the default > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] tamas-horvath edited a comment on pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
tamas-horvath edited a comment on pull request #5884: URL: https://github.com/apache/nifi/pull/5884#issuecomment-1074069696 Sample exception log: ``` 2022-03-21 16:49:41,127 DEBUG [Validate Components Thread-2] o.a.nifi.security.util.KeyStoreUtils Keystore [file:/Users/tamashorvath/repos/cloudera-truststore.jks] Type [JKS] load failed java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.nifi.security.util.KeyStoreUtils.isStoreValid(KeyStoreUtils.java:399) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststoreFile(StandardSSLContextService.java:495) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststore(StandardSSLContextService.java:459) at org.apache.nifi.ssl.StandardSSLContextService.validateStore(StandardSSLContextService.java:376) at org.apache.nifi.ssl.StandardSSLContextService.customValidate(StandardSSLContextService.java:197) at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130) at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:741) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:708) at org.apache.nifi.controller.service.StandardControllerServiceNode.performValidation(StandardControllerServiceNode.java:500) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:722) at org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52) at org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ... 24 common frames omitted``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-9786) KeyStoreUtils.isStoreValid() suppresses exceptions without logging when trying to open a keystore
[ https://issues.apache.org/jira/browse/NIFI-9786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9786: --- Fix Version/s: 1.16.0 Resolution: Fixed Status: Resolved (was: Patch Available) > KeyStoreUtils.isStoreValid() suppresses exceptions without logging when > trying to open a keystore > - > > Key: NIFI-9786 > URL: https://issues.apache.org/jira/browse/NIFI-9786 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Peter Gyori >Assignee: Tamas Horvath >Priority: Minor > Fix For: 1.16.0 > > Time Spent: 50m > Remaining Estimate: 0h > > KeyStoreUtils.isStoreValid() simply returns false if the keystore cannot be > opened e.g. because the password is wrong. There is no log entry about why > the keystore is not valid. > As a result when SslContextFactory.getTrustManagers() method attempts to read > a trust store file using the configured properties, if the password is wrong, > the method is not informed why, and only throws a TlsException stating "The > truststore properties are not valid". > It would be useful to add debug-level logging to KeyStoreUtils.isStoreValid() > to log the reason for the keystore being invalid. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9786) KeyStoreUtils.isStoreValid() suppresses exceptions without logging when trying to open a keystore
[ https://issues.apache.org/jira/browse/NIFI-9786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509963#comment-17509963 ] ASF subversion and git services commented on NIFI-9786: --- Commit a8cd7e5200332b38006a99a55e600de3650377a1 in nifi's branch refs/heads/main from Tamas Horvath [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a8cd7e5 ] NIFI-9786 Added debug to KeyStoreUtils.isStoreValid This closes #5884 Signed-off-by: David Handermann > KeyStoreUtils.isStoreValid() suppresses exceptions without logging when > trying to open a keystore > - > > Key: NIFI-9786 > URL: https://issues.apache.org/jira/browse/NIFI-9786 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Peter Gyori >Assignee: Tamas Horvath >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > KeyStoreUtils.isStoreValid() simply returns false if the keystore cannot be > opened e.g. because the password is wrong. There is no log entry about why > the keystore is not valid. > As a result when SslContextFactory.getTrustManagers() method attempts to read > a trust store file using the configured properties, if the password is wrong, > the method is not informed why, and only throws a TlsException stating "The > truststore properties are not valid". > It would be useful to add debug-level logging to KeyStoreUtils.isStoreValid() > to log the reason for the keystore being invalid. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] tamas-horvath edited a comment on pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
tamas-horvath edited a comment on pull request #5884: URL: https://github.com/apache/nifi/pull/5884#issuecomment-1074069696 Sample exception log: ```2022-03-21 16:49:41,127 DEBUG [Validate Components Thread-2] o.a.nifi.security.util.KeyStoreUtils Keystore [file:/Users/tamashorvath/repos/cloudera-truststore.jks] Type [JKS] load failed java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.nifi.security.util.KeyStoreUtils.isStoreValid(KeyStoreUtils.java:399) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststoreFile(StandardSSLContextService.java:495) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststore(StandardSSLContextService.java:459) at org.apache.nifi.ssl.StandardSSLContextService.validateStore(StandardSSLContextService.java:376) at org.apache.nifi.ssl.StandardSSLContextService.customValidate(StandardSSLContextService.java:197) at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130) at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:741) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:708) at org.apache.nifi.controller.service.StandardControllerServiceNode.performValidation(StandardControllerServiceNode.java:500) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:722) at org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52) at org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ... 24 common frames omitted``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
exceptionfactory closed pull request #5884: URL: https://github.com/apache/nifi/pull/5884 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tamas-horvath edited a comment on pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
tamas-horvath edited a comment on pull request #5884: URL: https://github.com/apache/nifi/pull/5884#issuecomment-1074069696 Sample exception: ```2022-03-21 16:49:41,127 DEBUG [Validate Components Thread-2] o.a.nifi.security.util.KeyStoreUtils Keystore [file:/Users/tamashorvath/repos/cloudera-truststore.jks] Type [JKS] load failed java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.nifi.security.util.KeyStoreUtils.isStoreValid(KeyStoreUtils.java:399) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststoreFile(StandardSSLContextService.java:495) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststore(StandardSSLContextService.java:459) at org.apache.nifi.ssl.StandardSSLContextService.validateStore(StandardSSLContextService.java:376) at org.apache.nifi.ssl.StandardSSLContextService.customValidate(StandardSSLContextService.java:197) at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130) at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:741) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:708) at org.apache.nifi.controller.service.StandardControllerServiceNode.performValidation(StandardControllerServiceNode.java:500) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:722) at org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52) at org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ... 24 common frames omitted``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tamas-horvath commented on pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
tamas-horvath commented on pull request #5884: URL: https://github.com/apache/nifi/pull/5884#issuecomment-1074069696 Sample exception: `2022-03-21 16:49:41,127 DEBUG [Validate Components Thread-2] o.a.nifi.security.util.KeyStoreUtils Keystore [file:/Users/tamashorvath/repos/cloudera-truststore.jks] Type [JKS] load failed java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.nifi.security.util.KeyStoreUtils.isStoreValid(KeyStoreUtils.java:399) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststoreFile(StandardSSLContextService.java:495) at org.apache.nifi.ssl.StandardSSLContextService.validateTruststore(StandardSSLContextService.java:459) at org.apache.nifi.ssl.StandardSSLContextService.validateStore(StandardSSLContextService.java:376) at org.apache.nifi.ssl.StandardSSLContextService.customValidate(StandardSSLContextService.java:197) at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130) at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:741) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:708) at org.apache.nifi.controller.service.StandardControllerServiceNode.performValidation(StandardControllerServiceNode.java:500) at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:722) at org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52) at org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ... 24 common frames omitted` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1281: MINIFICPP-1769 Add propertyValue to agent manifest metadata
szaszm commented on a change in pull request #1281:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1281#discussion_r831254268
##
File path: libminifi/include/properties/ConfigurationReader.h
##
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+namespace org::apache::nifi::minifi {
+
+class ConfigurationReader {
+ public:
+ virtual bool get(const std::string& key, std::string& value) const = 0;
+ virtual bool get(const std::string& key, const std::string& alternate_key,
std::string& value) const = 0;
+ virtual std::optional get(const std::string& key) const = 0;
+};
Review comment:
What do you mean by consistency with other setters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9820: --- Status: Patch Available (was: Open) > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory opened a new pull request #5886: NIFI-9820 Reduce Kudu Client Worker Count default setting
exceptionfactory opened a new pull request #5886:
URL: https://github.com/apache/nifi/pull/5886
Description of PR
NIFI-9820 Reduces the `Kudu Client Worker Count` default value in the
`PutKudu` Processor. The current default value is based on the JVM
runtime-reported available processors, multiplied by 2. This value was based on
the internal default value from the Kudu Client library. The updated value
removes the multiplication by 2, setting the default value to match the number
of runtime-reported processors.
This approach improves the default memory-usage behavior of the `PutKudu` in
flows with numerous instances of the Processor. The new default value also
minimizes the impact from older versions of `PutKudu` that do not have the
`Kudu Client Worker Count` property, and relied on the internal default value
from the Kudu Client library.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Reopened] (NIFI-7360) Publish documentation for non-included components
[ https://issues.apache.org/jira/browse/NIFI-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen reopened NIFI-7360: > Publish documentation for non-included components > - > > Key: NIFI-7360 > URL: https://issues.apache.org/jira/browse/NIFI-7360 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Pierre Villard >Priority: Major > > Some components are not part of the binary we release for NiFi. Such > components are built through specific profiles and the NARs can be downloaded > from the internet. > However from a user perspective and for a better visibility, it'd be great to > have the associated documentation available on the NiFi website: > [https://nifi.apache.org/docs.html] > Maybe there could be a specific way/icon to display such components in the > list and add a "disclaimer" saying that the components are not available out > of the bow but can be downloaded from public repositories. > It could also open the door to include documentation of "community" > components that we don't include but which would definitely be beneficial for > the community. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7360) Publish documentation for non-included components
[ https://issues.apache.org/jira/browse/NIFI-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509946#comment-17509946 ] Mike Thomsen commented on NIFI-7360: Might be worthwhile for me to extend the documentation I wrote to include direct download links then. > Publish documentation for non-included components > - > > Key: NIFI-7360 > URL: https://issues.apache.org/jira/browse/NIFI-7360 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Pierre Villard >Priority: Major > > Some components are not part of the binary we release for NiFi. Such > components are built through specific profiles and the NARs can be downloaded > from the internet. > However from a user perspective and for a better visibility, it'd be great to > have the associated documentation available on the NiFi website: > [https://nifi.apache.org/docs.html] > Maybe there could be a specific way/icon to display such components in the > list and add a "disclaimer" saying that the components are not available out > of the bow but can be downloaded from public repositories. > It could also open the door to include documentation of "community" > components that we don't include but which would definitely be beneficial for > the community. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-8391) Nifi to support .RAR format in UnpackContent
[ https://issues.apache.org/jira/browse/NIFI-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved NIFI-8391. Resolution: Won't Do I did some research on this, and apparently there aren't any good pure Java libraries that support RAR. You can get around this issue for opening rars at least by using ExecuteStreamCommand. Marking resolved because AFAICT there's no path forward for full support. > Nifi to support .RAR format in UnpackContent > > > Key: NIFI-8391 > URL: https://issues.apache.org/jira/browse/NIFI-8391 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration >Affects Versions: 1.12.1 >Reporter: Abha Kabra >Priority: Major > > Nifi should add .rar in Unpack Content processor configuration. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] thenatog opened a new pull request #5885: NIFI-9701 - Updated the ListSFTP NO_TRACKING listing strategy to crea…
thenatog opened a new pull request #5885:
URL: https://github.com/apache/nifi/pull/5885
…te a single flow file when using a record writer.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1280: MINIFICPP-1770 Add C2 NO_OPERATION response if property update is NoOp
szaszm commented on a change in pull request #1280:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1280#discussion_r831206143
##
File path: extensions/http-curl/tests/HTTPHandlers.h
##
@@ -792,20 +792,40 @@ class C2AcknowledgeHandler : public ServerAwareHandler {
rapidjson::Document root;
root.Parse(req.data(), req.size());
if (root.IsObject() && root.HasMember("operationId")) {
- std::lock_guard guard(mtx_);
+ std::lock_guard guard(ack_operations_mtx_);
Review comment:
`ack_operations_mtx_` is released before locking `apply_count_mtx_`, so
they are never locked at the same time. Since only one is held at a time, it's
not possible that this function and another thread are waiting for each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509924#comment-17509924 ] David Handermann commented on NIFI-9820: I agree that the NiFi use case is different than the default value set in the Kudu Client library. Using a small default value is probably a safer approach, and would follow a similar pattern to other processors, as well as the default number of concurrent tasks. On the other hand, for flows that have existing PutKudu processors, the challenge is to avoid introducing a negative impact. For example, if an existing flow has a larger number of concurrent tasks, upgrading to a version of NiFi that defaults to the Worker Client Count to 1 or 2 would have a negative impact on performance after the upgrade. That was part of the reason for setting the default value to match the internal value from the Kudu Client library. For this reason, changing the default value to the number of CPU cores, versus the number multiplied by 2, seems a middle way forward. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] adam-markovics commented on a change in pull request #1280: MINIFICPP-1770 Add C2 NO_OPERATION response if property update is NoOp
adam-markovics commented on a change in pull request #1280:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1280#discussion_r831157243
##
File path: extensions/http-curl/tests/HTTPHandlers.h
##
@@ -792,20 +792,40 @@ class C2AcknowledgeHandler : public ServerAwareHandler {
rapidjson::Document root;
root.Parse(req.data(), req.size());
if (root.IsObject() && root.HasMember("operationId")) {
- std::lock_guard guard(mtx_);
+ std::lock_guard guard(ack_operations_mtx_);
Review comment:
With locking two mutexes, you always have to be careful. Is it possible
to have another thread locking them in the reverse order than here? So, first
`apply_count_mtx_` and then `ack_operations_mtx_`? That could result in a
deadlock. It's possible by calling `isAcknowledged` first and `getApplyCount`
second, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-9788) Update Apache Commons Codec to 1.15
[ https://issues.apache.org/jira/browse/NIFI-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-9788. Fix Version/s: 1.16.0 Resolution: Fixed > Update Apache Commons Codec to 1.15 > --- > > Key: NIFI-9788 > URL: https://issues.apache.org/jira/browse/NIFI-9788 > Project: Apache NiFi > Issue Type: Task >Reporter: Jon Shoemaker >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.16.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9788) Update Apache Commons Codec to 1.15
[ https://issues.apache.org/jira/browse/NIFI-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509912#comment-17509912 ] ASF subversion and git services commented on NIFI-9788: --- Commit 2bb1a06a8c5efa9eb95ebd9c26983190d09b4211 in nifi's branch refs/heads/main from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2bb1a06 ] NIFI-9788 Updated commons-codec to 1.15 across all modules. This closes #5881 Signed-off-by: David Handermann > Update Apache Commons Codec to 1.15 > --- > > Key: NIFI-9788 > URL: https://issues.apache.org/jira/browse/NIFI-9788 > Project: Apache NiFi > Issue Type: Task >Reporter: Jon Shoemaker >Assignee: Mike Thomsen >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory closed pull request #5881: NIFI-9788 Updated commons-codec to 1.15 across all modules.
exceptionfactory closed pull request #5881: URL: https://github.com/apache/nifi/pull/5881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1281: MINIFICPP-1769 Add propertyValue to agent manifest metadata
lordgamez commented on a change in pull request #1281:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1281#discussion_r831141775
##
File path: libminifi/include/properties/ConfigurationReader.h
##
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+namespace org::apache::nifi::minifi {
+
+class ConfigurationReader {
+ public:
+ virtual bool get(const std::string& key, std::string& value) const = 0;
+ virtual bool get(const std::string& key, const std::string& alternate_key,
std::string& value) const = 0;
+ virtual std::optional get(const std::string& key) const = 0;
+};
Review comment:
I decided to keep the interface for now. It's more consistent with the
other setters of the agent information nodes and in my opinion it's cleaner to
see what that member represents.
Also good catch, I missed the virtual destructor, added in last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509910#comment-17509910 ] Josef Zahner commented on NIFI-9820: Btw. we haven't tested what happens if we use the default value (128) in our case. We have the issue that we have to test the behaviour in our production system. Wouldn't it be great to have a save value to prevent a high memory issue per default? Other values are per default as well very small, like the number of tasks of a processor (defaults to 1). That's why I wrote a very small number in my initial post. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5881: NIFI-9788 Updated commons-codec to 1.15 across all modules.
exceptionfactory commented on a change in pull request #5881: URL: https://github.com/apache/nifi/pull/5881#discussion_r831147244 ## File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ## @@ -151,6 +151,7 @@ commons-codec commons-codec +1.15 Review comment: Thanks, that makes some sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509905#comment-17509905 ] Josef Zahner edited comment on NIFI-9820 at 3/21/22, 2:05 PM: -- It's really a big advantage that we can change the value with a property since 1.14.0, however I could imagine that the internal default value from the Kudu Client library doesn't expect to have multiple clients (aka processors in nifi) on one node. So it makes sense to use the number of CPUs there. In NiFi however it's a different case, it's very likely that you don't have just one client. The question is, why should we use a "dynamic" calculated value as any other property (eg. FlowFiles per Batch) is as well just on a fixed value. The user has to test/find anyway a good value in his setup. was (Author: jzahner): It's really a big advantage that we can change the value with a property since 1.14.0, however I could imagine that the internal default value from the Kudu Client library doesn't expect to have multiple clients (aka processors in nifi) on one node. So it makes sense to use the number of CPUs there. In NiFi however it's a different case, it's very likely that you don't have just one client. The question is, why should we use a "dynamic" calculated value as any other property (eg. FlowFiles per Batch) is as well just on a fixed value. The use has to test/find anyway a good value in his setup. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509905#comment-17509905 ] Josef Zahner commented on NIFI-9820: It's really a big advantage that we can change the value with a property since 1.14.0, however I could imagine that the internal default value from the Kudu Client library doesn't expect to have multiple clients (aka processors in nifi) on one node. So it makes sense to use the number of CPUs there. In NiFi however it's a different case, it's very likely that you don't have just one client. The question is, why should we use a "dynamic" calculated value as any other property (eg. FlowFiles per Batch) is as well just on a fixed value. The use has to test/find anyway a good value in his setup. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1281: MINIFICPP-1769 Add propertyValue to agent manifest metadata
lordgamez commented on a change in pull request #1281:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1281#discussion_r831141775
##
File path: libminifi/include/properties/ConfigurationReader.h
##
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+namespace org::apache::nifi::minifi {
+
+class ConfigurationReader {
+ public:
+ virtual bool get(const std::string& key, std::string& value) const = 0;
+ virtual bool get(const std::string& key, const std::string& alternate_key,
std::string& value) const = 0;
+ virtual std::optional get(const std::string& key) const = 0;
+};
Review comment:
I decided to keep the interface for now. It's more consistent with the
other setters of the agent information nodes and in my opinion it's cleaner to
see what that member represents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509900#comment-17509900 ] David Handermann commented on NIFI-9820: Thanks for raising this issue [~jzahner]. The current default value derives from the internal default value used within the Kudu Client library. As you have noted, however, configuring multiple PutKudu Processors results in memory usage problems. Some NiFi flow configurations can benefit from a larger number, where only a couple PutKudu Processors may be configured. Setting the property based on the number of concurrent tasks is a good approach in general. Changing the default value to the number of reported CPU cores is one simple way forward. Another option could be setting the default value to half the number of CPU cores, with a minimum value of 1. I am inclined to go with the first option, and I can put forward a pull request for further consideration. > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] adam-markovics commented on a change in pull request #1284: MINIFICPP-1776 - Parallel heartbeat and flow update can cause crash
adam-markovics commented on a change in pull request #1284:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1284#discussion_r831131778
##
File path: extensions/http-curl/tests/C2ClearCoreComponentStateTest.cpp
##
@@ -51,10 +51,21 @@ class VerifyC2ClearCoreComponentState : public VerifyC2Base
{
protected:
void updateProperties(minifi::FlowController& flow_controller) override {
-
dynamic_cast(flow_controller.getComponents("TailFile1")[0])
-->getProcessor()->setProperty(minifi::processors::TailFile::FileName,
test_file_1_);
-
dynamic_cast(flow_controller.getComponents("TailFile2")[0])
-->getProcessor()->setProperty(minifi::processors::TailFile::FileName,
test_file_2_);
+size_t componentIdx = 0;
+auto setFileName = [] (const std::string& fileName,
minifi::state::StateController* component){
Review comment:
Not present anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-9820: -- Assignee: David Handermann > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Assignee: David Handermann >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1280: MINIFICPP-1770 Add C2 NO_OPERATION response if property update is NoOp
lordgamez commented on a change in pull request #1280:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1280#discussion_r831104636
##
File path: libminifi/src/Configure.cpp
##
@@ -30,6 +30,14 @@ namespace nifi {
namespace minifi {
bool Configure::get(const std::string& key, std::string& value) const {
+ const std::string_view log_prefix = "nifi.log.";
Review comment:
Good idea, updated in 50839c9804f63ec6a588c765821dce6b170b3af0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1280: MINIFICPP-1770 Add C2 NO_OPERATION response if property update is NoOp
lordgamez commented on a change in pull request #1280:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1280#discussion_r831104449
##
File path: libminifi/include/c2/C2Agent.h
##
@@ -150,10 +150,16 @@ class C2Agent : public state::UpdateController {
*/
void handle_describe(const C2ContentResponse );
+
+ enum class UpdateResult {
+NO_UPDATE,
+UPDATE_SUCCESFUL,
Review comment:
Good catch, fixed in 50839c9804f63ec6a588c765821dce6b170b3af0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
[ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josef Zahner updated NIFI-9820: --- Priority: Minor (was: Major) > Change PutKudu Property "Kudu Client Worker Count" Default Value > > > Key: NIFI-9820 > URL: https://issues.apache.org/jira/browse/NIFI-9820 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.15.3 >Reporter: Josef Zahner >Priority: Minor > > The PutKudu processor property "Kudu Client Worker Count" has a suboptimal > value. Please don't use the current "number of CPUs multiplied by 2" > behaviour as it leads to a massive amount of workers in our case with > physical servers. We have a 8-node cluster where each server has 64 CPUs. We > have about 30 PutKudu processors configured -> a lot of worker threads per > default just for kudu. > We have changed the number of worker threads in our case to the number of > concurrent tasks. I don't know, maybe it would be great to set it a bit > higher than that, but to be honest, I don't exactly understand the impact. It > looks still fast with the current config. > *To sum it up, please set a low default value (eg. 4 or 8) for the property > "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu > processor.* > Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value
Josef Zahner created NIFI-9820: -- Summary: Change PutKudu Property "Kudu Client Worker Count" Default Value Key: NIFI-9820 URL: https://issues.apache.org/jira/browse/NIFI-9820 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.15.3 Reporter: Josef Zahner The PutKudu processor property "Kudu Client Worker Count" has a suboptimal value. Please don't use the current "number of CPUs multiplied by 2" behaviour as it leads to a massive amount of workers in our case with physical servers. We have a 8-node cluster where each server has 64 CPUs. We have about 30 PutKudu processors configured -> a lot of worker threads per default just for kudu. We have changed the number of worker threads in our case to the number of concurrent tasks. I don't know, maybe it would be great to set it a bit higher than that, but to be honest, I don't exactly understand the impact. It looks still fast with the current config. *To sum it up, please set a low default value (eg. 4 or 8) for the property "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu processor.* Btw. are there any suggestions how big the number should be? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] MikeThomsen commented on pull request #5830: NIFI-4307 Add support for Kotlin to the scripting bundle
MikeThomsen commented on pull request #5830: URL: https://github.com/apache/nifi/pull/5830#issuecomment-1073895638 The inability to get the `processor` object from the interpreted script appears to be a bug and worth bringing to JetBrains's attention. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a change in pull request #5881: NIFI-9788 Updated commons-codec to 1.15 across all modules.
MikeThomsen commented on a change in pull request #5881: URL: https://github.com/apache/nifi/pull/5881#discussion_r831101316 ## File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ## @@ -151,6 +151,7 @@ commons-codec commons-codec +1.15 Review comment: IIRC it was. I ripped out ALL of them and 4 modules still required it. IIRC, it was due to test vs compile scoping issues with `` in the root pom. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
exceptionfactory commented on a change in pull request #5884:
URL: https://github.com/apache/nifi/pull/5884#discussion_r831096975
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -401,6 +401,7 @@ public static boolean isStoreValid(final URL keystore,
final KeystoreType keysto
return true;
} catch (Exception e) {
+logger.debug("Keystore could not be loaded", e);
Review comment:
Recommend including the keystore location and keystore type for
additional debugging:
```suggestion
logger.debug("Keystore [{}] Type [{}] load failed", keystore,
keystoreType, e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9786) KeyStoreUtils.isStoreValid() suppresses exceptions without logging when trying to open a keystore
[ https://issues.apache.org/jira/browse/NIFI-9786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Horvath updated NIFI-9786: Status: Patch Available (was: In Progress) > KeyStoreUtils.isStoreValid() suppresses exceptions without logging when > trying to open a keystore > - > > Key: NIFI-9786 > URL: https://issues.apache.org/jira/browse/NIFI-9786 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Peter Gyori >Assignee: Tamas Horvath >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > KeyStoreUtils.isStoreValid() simply returns false if the keystore cannot be > opened e.g. because the password is wrong. There is no log entry about why > the keystore is not valid. > As a result when SslContextFactory.getTrustManagers() method attempts to read > a trust store file using the configured properties, if the password is wrong, > the method is not informed why, and only throws a TlsException stating "The > truststore properties are not valid". > It would be useful to add debug-level logging to KeyStoreUtils.isStoreValid() > to log the reason for the keystore being invalid. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] tamas-horvath opened a new pull request #5884: NIFI-9786 Add debug logs if keystore cannot be loaded
tamas-horvath opened a new pull request #5884:
URL: https://github.com/apache/nifi/pull/5884
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables logging for KeyStoreUtils isStoreValid validation failure; closed
improvement NIFI-9786._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [/] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [/] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [/] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [/] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [/] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [/] Have you verified that the full build is successful on JDK 8?
- [/] Have you verified that the full build is successful on JDK 11?
- [/] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [/] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [/] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [/] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [/] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5881: NIFI-9788 Updated commons-codec to 1.15 across all modules.
exceptionfactory commented on a change in pull request #5881: URL: https://github.com/apache/nifi/pull/5881#discussion_r831091623 ## File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ## @@ -151,6 +151,7 @@ commons-codec commons-codec +1.15 Review comment: Is this specific version necessary with the version already specified in the root Maven configuration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8435) PutKudu 1.13.2 Memory Leak
[ https://issues.apache.org/jira/browse/NIFI-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509834#comment-17509834 ] Josef Zahner commented on NIFI-8435: Just for reference. We have upgraded from NiFi 1.13.2 to NiFi 1.15.3 and adapted the property "Kudu Client Worker Count" to the number of "Concurrent Tasks" accordingly. So far no leaks or high memory situations anymore. However it's not ideal to have the property "Kudu Client Worker Count" as high as the number of CPUs as we have bladeservers with 64 CPUs and about 30 PutKudus in place -> leads to a lot of kudu client workers. > PutKudu 1.13.2 Memory Leak > -- > > Key: NIFI-8435 > URL: https://issues.apache.org/jira/browse/NIFI-8435 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.13.2 > Environment: NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu > 1.10.0 >Reporter: Josef Zahner >Assignee: Peter Gyori >Priority: Critical > Labels: kudu, nifi, oom > Fix For: 1.14.0 > > Attachments: Screenshot 2021-04-20 at 14.27.11.png, > grafana_heap_overview.png, kudu_inserts_per_sec.png, > putkudu_processor_config.png, visualvm_bytes_detail_view.png, > visualvm_total_bytes_used.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with > PutKudu. > PutKudu on the 1.13.2 eats up all the heap memory and garbage collection > can't anymore free up the memory. We allow Java to use 31GB memory and as you > can see with NiFi 1.11.4 it will be used like it should with GC. However with > NiFi 1.13.2 with our actual load it fills up the memory relatively fast. > Manual GC via visualvm tool didn't help at all to free up memory. > !grafana_heap_overview.png! > > Visual VM shows the following culprit: !visualvm_total_bytes_used.png! > !visualvm_bytes_detail_view.png! > The bytes array shows millions of char data which isn't cleaned up. In fact > here 14,9GB memory (heapdump has been taken after a while of full load). If > we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a > few hundred MBs. > As you could imagine we can't upload the heap dump as currently we have only > productive data on the system. But don't hesitate to ask questions about the > heapdump if you need more information. > I haven't done any screenshot of the processor config, but I can do that if > you wish (we are back to NiFi 1.11.4 at the moment). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory commented on pull request #5845: NIFI-9768: Fix indeterminate map ordering in testSchemaToAttribute
exceptionfactory commented on pull request #5845: URL: https://github.com/apache/nifi/pull/5845#issuecomment-1073878384 > Thank you for the feedback. I am working to fix the issue. Do you think, instead of the present adjustment, I should update the assertions in the test to withstand HashMap's unpredictable behavior? I implement the other solution in another branch, and all automated checks passes for that one. Thanks for the reply @MarcyGO. If the failing test relies on incorrect assumptions about the behavior of HashMap, then it sounds like that particular test should be changed. If you have a solution, please commit the change and we can evaluate it together with the current set of changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-9819) SAML should have option to autocreate users.
[
https://issues.apache.org/jira/browse/NIFI-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wiktor Kubicki updated NIFI-9819:
-
Description:
*Prolog:*
In SSO i set roles that give the user access to the company's systems or
logical areas.
*Case:*
Using saml i can set nifi.security.user.saml.group.attribute.name and link SSO
roles to groups in NIFI, then i don't have to add users to groups in NiFi.
But what if user in NiFi doesn't exist, even if have NiFi group set up in SSO?
Then he receiving "Unknown user with identity..." alert. So before a user with
aproperiate roles in SSO logs in to NiFi, you have toadd him separatly.
This lack is confirmed in comment:
{code:java}
The real issue is "Unknown user with identity 'user2'"... all of the users and
groups still need to be known to NiFi's authorization, the only part that does
not need to be known is the actual group membership since that is coming from
the SAML response.{code}
~[http://disq.us/p/2g2fdie]
*Workaroung:*
I can create one user without privileges, and map user name to the new one in
{_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
flow history what gives me user accountability..
*Expected behavior:*
There should be an option in nifi.properties,
nifi.security.user.saml.create.user which, when is set to true, will add
"empty" (without privileges or groups) user. Then, if the user has the right
samla group, he will have access to the platform.
or..
In this situation give user access and privileges even without creating user in
users.xml file.
{*}Extra value{*}:
There may be extra option _nifi.security.user.saml.new.user.default.group_
which allowed to link new user to existing ( ! ) group, one or more.
was:
*Prolog:*
In SSO i set roles that give the user access to the company's systems or
logical areas.
*Case:*
Using saml i can set nifi.security.user.saml.group.attribute.name and link SSO
roles to groups in NIFI, then i don't have to add users to groups in NiFi.
But what if user in NiFi doesn't exist, even if have NiFi group set up in SSO?
Then he receiving "Unknown user with identity..." alert. So before a user with
aproperiate roles in SSO logs in to NiFi, you have toadd him separatly.
This lack is confirmed in comment:
{code:java}
The real issue is "Unknown user with identity 'user2'"... all of the users and
groups still need to be known to NiFi's authorization, the only part that does
not need to be known is the actual group membership since that is coming from
the SAML response.{code}
~[http://disq.us/p/2g2fdie]
*Workaroung:*
I can create one user without privileges, and map user name to the new one in
{_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
flow history what gives me user accountability..
*Expected behavior:*
There should be an option in nifi.properties,
nifi.security.user.saml.create.user which, when is set to true, will add
"empty" (without privilages or groups) user. Then, if the user has the right
samla group, he will have access to the platform.
{*}Extra value{*}:
There may be extra option _nifi.security.user.saml.new.user.default.group_
which allowed to link new user to existing ( ! ) group, one or more.
> SAML should have option to autocreate users.
>
>
> Key: NIFI-9819
> URL: https://issues.apache.org/jira/browse/NIFI-9819
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Security
>Affects Versions: 1.15.3
>Reporter: Wiktor Kubicki
>Priority: Major
>
> *Prolog:*
> In SSO i set roles that give the user access to the company's systems or
> logical areas.
> *Case:*
> Using saml i can set nifi.security.user.saml.group.attribute.name and link
> SSO roles to groups in NIFI, then i don't have to add users to groups in NiFi.
> But what if user in NiFi doesn't exist, even if have NiFi group set up in
> SSO? Then he receiving "Unknown user with identity..." alert. So before a
> user with aproperiate roles in SSO logs in to NiFi, you have toadd him
> separatly.
> This lack is confirmed in comment:
> {code:java}
> The real issue is "Unknown user with identity 'user2'"... all of the users
> and groups still need to be known to NiFi's authorization, the only part that
> does not need to be known is the actual group membership since that is coming
> from the SAML response.{code}
> ~[http://disq.us/p/2g2fdie]
> *Workaroung:*
> I can create one user without privileges, and map user name to the new one in
> {_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
> flow history what gives me user accountability..
> *Expected behavior:*
> There should be an option in nifi.properties,
> nifi.security.user.saml.create.user which, when is set to true, will add
> "empty"
[jira] [Commented] (NIFI-7360) Publish documentation for non-included components
[ https://issues.apache.org/jira/browse/NIFI-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509825#comment-17509825 ] Bryan Bende commented on NIFI-7360: --- [~mike.thomsen] I think this ticket and NIFI-9765 are really two different things. For this one, I don't think we are talking about building source code at all, most users won't actually do that. We are talking about documentation that makes it easy for users to know that there are extra NARs that are released to Maven central on every release, and can be downloaded and added to their lib directory. We answer this question all the time in Slack/email, so I think the intent was to preemptively answer it with docs. > Publish documentation for non-included components > - > > Key: NIFI-7360 > URL: https://issues.apache.org/jira/browse/NIFI-7360 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Pierre Villard >Priority: Major > > Some components are not part of the binary we release for NiFi. Such > components are built through specific profiles and the NARs can be downloaded > from the internet. > However from a user perspective and for a better visibility, it'd be great to > have the associated documentation available on the NiFi website: > [https://nifi.apache.org/docs.html] > Maybe there could be a specific way/icon to display such components in the > list and add a "disclaimer" saying that the components are not available out > of the bow but can be downloaded from public repositories. > It could also open the door to include documentation of "community" > components that we don't include but which would definitely be beneficial for > the community. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1281: MINIFICPP-1769 Add propertyValue to agent manifest metadata
szaszm commented on a change in pull request #1281:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1281#discussion_r831070760
##
File path: libminifi/include/properties/ConfigurationReader.h
##
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+namespace org::apache::nifi::minifi {
+
+class ConfigurationReader {
+ public:
+ virtual bool get(const std::string& key, std::string& value) const = 0;
+ virtual bool get(const std::string& key, const std::string& alternate_key,
std::string& value) const = 0;
+ virtual std::optional get(const std::string& key) const = 0;
+};
Review comment:
I think `std::optional get(std::string_view key)` would
cover all use cases. Alternate keys could be handled with `utils::orElse`.
Consider removing this interface and replacing it with a
`std::function(std::string_view)>` or templates with
a similar callback signature, in the name of simplicity.
If you decide to keep it like this, please add a virtual destructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1280: MINIFICPP-1770 Add C2 NO_OPERATION response if property update is NoOp
szaszm commented on a change in pull request #1280:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1280#discussion_r831057966
##
File path: libminifi/include/c2/C2Agent.h
##
@@ -150,10 +150,16 @@ class C2Agent : public state::UpdateController {
*/
void handle_describe(const C2ContentResponse );
+
+ enum class UpdateResult {
+NO_UPDATE,
+UPDATE_SUCCESFUL,
Review comment:
typo
```suggestion
UPDATE_SUCCESSFUL,
```
##
File path: libminifi/src/Configure.cpp
##
@@ -30,6 +30,14 @@ namespace nifi {
namespace minifi {
bool Configure::get(const std::string& key, std::string& value) const {
+ const std::string_view log_prefix = "nifi.log.";
Review comment:
Could you make this delegate to the `std::optional`-returning overload,
to reduce code duplication? And the prefix could be `static constexpr` to force
the construction to compile-time.
```suggestion
static constexpr std::string_view log_prefix = "nifi.log.";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-9818) Flaky tests due to HashMap/HashSet in nifi-record
[
https://issues.apache.org/jira/browse/NIFI-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen resolved NIFI-9818.
Fix Version/s: 1.16.0
Resolution: Fixed
> Flaky tests due to HashMap/HashSet in nifi-record
> --
>
> Key: NIFI-9818
> URL: https://issues.apache.org/jira/browse/NIFI-9818
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Jiazhen Xu
>Priority: Major
> Fix For: 1.16.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Four tests in *nifi-record*
> (org.apache.nifi.serialization.record.TestMapRecord#testDefaultValueWithAliasValue
>
> org.apache.nifi.serialization.record.TestMapRecord#testAliasConflictingAliasValues,
>
> org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapStringKeys,
> and
> org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapNonStringKeys)
> can fail due to the non-deterministic order of iteration of _HashMap_ and
> {_}HashSet{_}.
> One can manifest this problem using
> [[NonDex]([https://github.com/TestingResearchIllinois/NonDex)]]
> {code:java}
> mvn install -pl nifi-commons/nifi-record -am
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestMapRecord#testAliasConflictingAliasValues
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestMapRecord#testDefaultValueWithAliasValue
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapStringKeys
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapNonStringKe
> {code}
> Use LinkedHashMap/LinkedHashSet instead of HashMap/HashSet in the test can
> solve the problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9818) Flaky tests due to HashMap/HashSet in nifi-record
[
https://issues.apache.org/jira/browse/NIFI-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509815#comment-17509815
]
ASF subversion and git services commented on NIFI-9818:
---
Commit 8d5de6b292981818e81ffd6cb3b8773c710884ca in nifi's branch
refs/heads/main from 在红尘中成仙
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8d5de6b ]
NIFI-9818 fix flaky tests
This closes #5883
Signed-off-by: Mike Thomsen
> Flaky tests due to HashMap/HashSet in nifi-record
> --
>
> Key: NIFI-9818
> URL: https://issues.apache.org/jira/browse/NIFI-9818
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Jiazhen Xu
>Priority: Major
>
> Four tests in *nifi-record*
> (org.apache.nifi.serialization.record.TestMapRecord#testDefaultValueWithAliasValue
>
> org.apache.nifi.serialization.record.TestMapRecord#testAliasConflictingAliasValues,
>
> org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapStringKeys,
> and
> org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapNonStringKeys)
> can fail due to the non-deterministic order of iteration of _HashMap_ and
> {_}HashSet{_}.
> One can manifest this problem using
> [[NonDex]([https://github.com/TestingResearchIllinois/NonDex)]]
> {code:java}
> mvn install -pl nifi-commons/nifi-record -am
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestMapRecord#testAliasConflictingAliasValues
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestMapRecord#testDefaultValueWithAliasValue
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapStringKeys
> mvn -pl nifi-commons/nifi-record
> edu.illinois:nondex-maven-plugin:1.1.2:nondex
> -Dtest=org.apache.nifi.serialization.record.TestDataTypeUtils#testInferTypeWithMapNonStringKe
> {code}
> Use LinkedHashMap/LinkedHashSet instead of HashMap/HashSet in the test can
> solve the problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] asfgit closed pull request #5883: NIFI-9818: fix four flaky tests in nifi-record
asfgit closed pull request #5883: URL: https://github.com/apache/nifi/pull/5883 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-7360) Publish documentation for non-included components
[ https://issues.apache.org/jira/browse/NIFI-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved NIFI-7360. Resolution: Duplicate > Publish documentation for non-included components > - > > Key: NIFI-7360 > URL: https://issues.apache.org/jira/browse/NIFI-7360 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Pierre Villard >Priority: Major > > Some components are not part of the binary we release for NiFi. Such > components are built through specific profiles and the NARs can be downloaded > from the internet. > However from a user perspective and for a better visibility, it'd be great to > have the associated documentation available on the NiFi website: > [https://nifi.apache.org/docs.html] > Maybe there could be a specific way/icon to display such components in the > list and add a "disclaimer" saying that the components are not available out > of the bow but can be downloaded from public repositories. > It could also open the door to include documentation of "community" > components that we don't include but which would definitely be beneficial for > the community. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7360) Publish documentation for non-included components
[ https://issues.apache.org/jira/browse/NIFI-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509808#comment-17509808 ] Mike Thomsen commented on NIFI-7360: [~pvillard] I think 9765 addresses the lion's share of this request. I'm going to close this ticket and raise the rest of it to a discussion point on dev since [~stevenmatison] (think I got the right user handle there) is spear-heading a redesign of the site. > Publish documentation for non-included components > - > > Key: NIFI-7360 > URL: https://issues.apache.org/jira/browse/NIFI-7360 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Pierre Villard >Priority: Major > > Some components are not part of the binary we release for NiFi. Such > components are built through specific profiles and the NARs can be downloaded > from the internet. > However from a user perspective and for a better visibility, it'd be great to > have the associated documentation available on the NiFi website: > [https://nifi.apache.org/docs.html] > Maybe there could be a specific way/icon to display such components in the > list and add a "disclaimer" saying that the components are not available out > of the bow but can be downloaded from public repositories. > It could also open the door to include documentation of "community" > components that we don't include but which would definitely be beneficial for > the community. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831007669
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+ std::invocable
auto serializer) {
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed", disk_stat_data.getReadsCompleted());
+ serializer("Reads Merged", disk_stat_data.getReadsMerged());
+ serializer("Sectors Read", disk_stat_data.getSectorsRead());
+ serializer("Writes Completed", disk_stat_data.getWritesCompleted());
+ serializer("Writes Merged", disk_stat_data.getWritesMerged());
+ serializer("Sectors Written", disk_stat_data.getSectorsWritten());
+ serializer("IO-s in progress", disk_stat_data.getIosInProgress());
+}
+
+void SerializeDiskStatDataPerSec(const DiskStatData& disk_stat_data,
+ const std::chrono::duration duration,
+ std::invocable
auto serializer) {
+ gsl_Expects(duration > 0ms);
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed/sec",
disk_stat_data.getReadsCompleted()/duration.count());
+ serializer("Reads Merged/sec",
disk_stat_data.getReadsMerged()/duration.count());
+ serializer("Sectors Read/sec",
disk_stat_data.getSectorsRead()/duration.count());
+ serializer("Writes Completed/sec",
disk_stat_data.getWritesCompleted()/duration.count());
+ serializer("Writes Merged/sec",
disk_stat_data.getWritesMerged()/duration.count());
+ serializer("Sectors Written/sec",
disk_stat_data.getSectorsWritten()/duration.count());
+ serializer("IO-s in progress",
disk_stat_data.getIosInProgress()/duration.count());
+}
+
+void SerializeMemInfo(const MemInfo& mem_info,
+ std::invocable auto
serializer) {
+ serializer("MemTotal",
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831006768
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+ std::invocable
auto serializer) {
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed", disk_stat_data.getReadsCompleted());
+ serializer("Reads Merged", disk_stat_data.getReadsMerged());
+ serializer("Sectors Read", disk_stat_data.getSectorsRead());
+ serializer("Writes Completed", disk_stat_data.getWritesCompleted());
+ serializer("Writes Merged", disk_stat_data.getWritesMerged());
+ serializer("Sectors Written", disk_stat_data.getSectorsWritten());
+ serializer("IO-s in progress", disk_stat_data.getIosInProgress());
Review comment:
agree, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-cb54b002ac8341fabeaff2d9dbbf25274c8a38a9bcb0e9e3a61e67b76f373dbdR75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831006580
##
File path: extensions/procfs/ProcFsJsonSerialization.h
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+#include "ProcFsSerialization.h"
+#include "rapidjson/stream.h"
+#include "rapidjson/document.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+namespace details {
+class UInt64Serializer {
+ public:
+ UInt64Serializer(rapidjson::Value& json, rapidjson::Document::AllocatorType&
alloc) :
+ json_(json), alloc_(alloc) {
+ }
+ void operator()(const char()[], const uint64_t value) {
+json_.AddMember(rapidjson::StringRef(key), value, alloc_);
+ }
+ private:
+ rapidjson::Value& json_;
+ rapidjson::Document::AllocatorType& alloc_;
+};
+
+class DoubleSerializer {
+ public:
+ DoubleSerializer(rapidjson::Value& json, rapidjson::Document::AllocatorType&
alloc) :
+ json_(json), alloc_(alloc) {
+ }
+ void operator()(const char()[], const double value) {
+json_.AddMember(rapidjson::StringRef(key), value, alloc_);
+ }
+ private:
+ rapidjson::Value& json_;
+ rapidjson::Document::AllocatorType& alloc_;
+};
+
+class StringSerializer {
+ public:
+ StringSerializer(rapidjson::Value& json, rapidjson::Document::AllocatorType&
alloc) :
+ json_(json), alloc_(alloc) {
+ }
+ void operator()(const char()[], const std::string_view& value) {
+rapidjson::Value value_json(value.data(), value.size(), alloc_);
+json_.AddMember(rapidjson::StringRef(key), value_json, alloc_);
+ }
+ private:
+ rapidjson::Value& json_;
+ rapidjson::Document::AllocatorType& alloc_;
+};
+} // namespace details
+
+void addCPUStatToJson(const std::string& cpu_name,
+ const CpuStatData& cpu_stat,
+ rapidjson::Value& cpu_root,
+ rapidjson::Document::AllocatorType& alloc) {
+ rapidjson::Value cpu_key(cpu_name.c_str(), cpu_name.length(), alloc);
+ cpu_root.AddMember(cpu_key.Move(), rapidjson::kObjectType, alloc);
+ rapidjson::Value& cpu_stat_json = cpu_root[cpu_name.c_str()];
+ SerializeCPUStatData(cpu_stat,
+ details::UInt64Serializer(cpu_stat_json, alloc));
+}
+
+void addCPUStatPeriodToJson(const std::string& cpu_name,
+const CpuStatData& start,
+const CpuStatData& end,
+rapidjson::Value& cpu_root,
+rapidjson::Document::AllocatorType& alloc) {
+ rapidjson::Value cpu_key(cpu_name.c_str(), cpu_name.length(), alloc);
+ cpu_root.AddMember(cpu_key.Move(), rapidjson::kObjectType, alloc);
+ rapidjson::Value& cpu_stat_json = cpu_root[cpu_name.c_str()];
+ SerializeNormalizedCPUStat(end-start,
+ details::DoubleSerializer(cpu_stat_json, alloc));
+}
+
+void addDiskStatToJson(const std::string& disk_name,
+ const DiskStatData& disk_stat,
+ rapidjson::Value& disk_root,
+ rapidjson::Document::AllocatorType& alloc) {
+ rapidjson::Value disk_key(disk_name.c_str(), disk_name.length(), alloc);
+ disk_root.AddMember(disk_key.Move(), rapidjson::kObjectType, alloc);
+ rapidjson::Value& disk_json = disk_root[disk_name.c_str()];
+ SerializeDiskStatData(disk_stat,
+details::UInt64Serializer(disk_json, alloc));
+}
+
+void addDiskStatPerSecToJson(const std::string& disk_name,
+ const DiskStatData disk_stat,
+ const std::chrono::duration duration,
+ rapidjson::Value& disk_root,
+ rapidjson::Document::AllocatorType& alloc) {
+ rapidjson::Value disk_key(disk_name.c_str(), disk_name.length(), alloc);
+ disk_root.AddMember(disk_key.Move(), rapidjson::kObjectType, alloc);
+ rapidjson::Value& disk_json = disk_root[disk_name.c_str()];
+ SerializeDiskStatDataPerSec(disk_stat,
+ duration,
+ details::DoubleSerializer(disk_json, alloc));
+}
+
+
+void
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831004999
##
File path: extensions/procfs/ProcFsJsonSerialization.h
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+
+#include "ProcFsSerialization.h"
+#include "rapidjson/stream.h"
+#include "rapidjson/document.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+namespace details {
+class UInt64Serializer {
+ public:
+ UInt64Serializer(rapidjson::Value& json, rapidjson::Document::AllocatorType&
alloc) :
+ json_(json), alloc_(alloc) {
+ }
+ void operator()(const char()[], const uint64_t value) {
Review comment:
Yeah, it was leftover from when the literals where static constexpr
members of the classes (it didnt work with rapidjson otherwise).
But entirely unnecessary now, so changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-8a4db62b36fb7340e48e062700fa2ae5b9c3851563ef91e310a51b84ba0e44c4R35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831003794
##
File path: extensions/procfs/ProcFs.cpp
##
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ProcFs.h"
+#include
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+namespace {
+bool is_number(const std::string& s) {
+ return !s.empty() && std::all_of(s.begin(), s.end(), ::isdigit);
+}
+} // namespace
+
+std::unordered_map ProcFs::getProcessStats() const {
+ std::unordered_map process_stats;
+ for (const auto : std::filesystem::directory_iterator(root_path_)) {
+if (entry.is_directory() && is_number(entry.path().filename())) {
+ auto stat_file_path = entry.path() / STAT_FILE;
+ std::ifstream stat_file(stat_file_path);
+ if (auto process_stat_data =
ProcessStatData::parseProcessStatFile(stat_file)) {
+process_stats.emplace(process_stat_data->getPid(),
ProcessStat(*process_stat_data, page_size_));
+ }
+}
+ }
+ return process_stats;
+}
+
+std::unordered_map ProcFs::getCpuStats() const {
+ std::unordered_map cpu_stats;
+ auto stat_file_path = root_path_ / STAT_FILE;
+ std::ifstream stat_file;
+ stat_file.open(stat_file_path);
+ std::string line;
+ while (std::getline(stat_file, line)) {
+std::istringstream iss(line);
+std::string entry_name;
+iss >> entry_name;
+if (entry_name.starts_with("cpu")) {
+ if (auto cpu_stat_data = CpuStatData::parseCpuStatLine(iss))
+cpu_stats.emplace(entry_name, *cpu_stat_data);
+}
+ }
+ return cpu_stats;
+}
+
+std::optional ProcFs::getMemInfo() const {
+ auto mem_info_file_path = root_path_ / MEMINFO_FILE;
+ std::ifstream mem_info_file(mem_info_file_path);
+ return MemInfo::parseMemInfoFile(mem_info_file);
+}
+
+std::unordered_map ProcFs::getNetDevs() const {
+ std::unordered_mapnet_devs;
+ auto stat_file_path = root_path_ / NET_DEV_FILE;
+ std::ifstream stat_file;
+ stat_file.open(stat_file_path);
+ std::string line;
+ std::getline(stat_file, line);
+ std::getline(stat_file, line);
+ while (std::getline(stat_file, line)) {
+std::istringstream iss(line);
+std::string entry_name;
+iss >> entry_name;
+if (iss.fail())
+ continue;
+entry_name.pop_back();
+if (auto net_dev_data = NetDevData::parseNetDevLine(iss))
+ net_devs.emplace(entry_name, *net_dev_data);
Review comment:
Good idea :+1: , i've added log_errors in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-2b6e743856f10f8108cb9791fda43ac483fc3e7b8a8b6d2d4bbf488fe057b299R38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831003429
##
File path: extensions/procfs/ProcFs.cpp
##
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ProcFs.h"
+#include
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+namespace {
+bool is_number(const std::string& s) {
+ return !s.empty() && std::all_of(s.begin(), s.end(), ::isdigit);
+}
+} // namespace
+
+std::unordered_map ProcFs::getProcessStats() const {
+ std::unordered_map process_stats;
+ for (const auto : std::filesystem::directory_iterator(root_path_)) {
+if (entry.is_directory() && is_number(entry.path().filename())) {
+ auto stat_file_path = entry.path() / STAT_FILE;
+ std::ifstream stat_file(stat_file_path);
+ if (auto process_stat_data =
ProcessStatData::parseProcessStatFile(stat_file)) {
+process_stats.emplace(process_stat_data->getPid(),
ProcessStat(*process_stat_data, page_size_));
+ }
+}
+ }
+ return process_stats;
+}
+
+std::unordered_map ProcFs::getCpuStats() const {
+ std::unordered_map cpu_stats;
+ auto stat_file_path = root_path_ / STAT_FILE;
+ std::ifstream stat_file;
+ stat_file.open(stat_file_path);
+ std::string line;
+ while (std::getline(stat_file, line)) {
+std::istringstream iss(line);
+std::string entry_name;
+iss >> entry_name;
+if (entry_name.starts_with("cpu")) {
+ if (auto cpu_stat_data = CpuStatData::parseCpuStatLine(iss))
+cpu_stats.emplace(entry_name, *cpu_stat_data);
+}
+ }
+ return cpu_stats;
+}
+
+std::optional ProcFs::getMemInfo() const {
+ auto mem_info_file_path = root_path_ / MEMINFO_FILE;
+ std::ifstream mem_info_file(mem_info_file_path);
+ return MemInfo::parseMemInfoFile(mem_info_file);
+}
+
+std::unordered_map ProcFs::getNetDevs() const {
+ std::unordered_mapnet_devs;
+ auto stat_file_path = root_path_ / NET_DEV_FILE;
+ std::ifstream stat_file;
+ stat_file.open(stat_file_path);
+ std::string line;
+ std::getline(stat_file, line);
+ std::getline(stat_file, line);
+ while (std::getline(stat_file, line)) {
+std::istringstream iss(line);
+std::string entry_name;
+iss >> entry_name;
+if (iss.fail())
+ continue;
+entry_name.pop_back();
Review comment:
makes sense, I've added the comment in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-bb84b43a558560608fd5680317d7f8585c7d18d6f1b6b4c84f3036020667R47
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831003077
##
File path: extensions/procfs/ProcFs.cpp
##
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ProcFs.h"
+#include
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+namespace {
+bool is_number(const std::string& s) {
+ return !s.empty() && std::all_of(s.begin(), s.end(), ::isdigit);
+}
+} // namespace
+
+std::unordered_map ProcFs::getProcessStats() const {
+ std::unordered_map process_stats;
+ for (const auto : std::filesystem::directory_iterator(root_path_)) {
+if (entry.is_directory() && is_number(entry.path().filename())) {
+ auto stat_file_path = entry.path() / STAT_FILE;
+ std::ifstream stat_file(stat_file_path);
+ if (auto process_stat_data =
ProcessStatData::parseProcessStatFile(stat_file)) {
+process_stats.emplace(process_stat_data->getPid(),
ProcessStat(*process_stat_data, page_size_));
+ }
+}
+ }
+ return process_stats;
+}
+
+std::unordered_map ProcFs::getCpuStats() const {
+ std::unordered_map cpu_stats;
+ auto stat_file_path = root_path_ / STAT_FILE;
+ std::ifstream stat_file;
+ stat_file.open(stat_file_path);
+ std::string line;
+ while (std::getline(stat_file, line)) {
+std::istringstream iss(line);
+std::string entry_name;
+iss >> entry_name;
+if (entry_name.starts_with("cpu")) {
+ if (auto cpu_stat_data = CpuStatData::parseCpuStatLine(iss))
+cpu_stats.emplace(entry_name, *cpu_stat_data);
+}
+ }
+ return cpu_stats;
+}
+
+std::optional ProcFs::getMemInfo() const {
+ auto mem_info_file_path = root_path_ / MEMINFO_FILE;
+ std::ifstream mem_info_file(mem_info_file_path);
+ return MemInfo::parseMemInfoFile(mem_info_file);
+}
+
+std::unordered_map ProcFs::getNetDevs() const {
+ std::unordered_mapnet_devs;
Review comment:
thanks :+1:
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-2b6e743856f10f8108cb9791fda43ac483fc3e7b8a8b6d2d4bbf488fe057b299R73
(maybe the linter gets confused with the template parameters?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831002022
##
File path: extensions/procfs/NetDev.cpp
##
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "NetDev.h"
+
+using namespace std::literals::chrono_literals;
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+std::optional NetDevData::parseNetDevLine(std::istream& iss) {
+ NetDevData net_dev_data;
+ iss >> net_dev_data.bytes_received_ >> net_dev_data.packets_received_ >>
net_dev_data.errs_received_ >> net_dev_data.drop_errors_received_
+ >> net_dev_data.fifo_errors_received_ >>
net_dev_data.frame_errors_received_ >> net_dev_data.compressed_packets_received_
+ >> net_dev_data.multicast_frames_received_>>
net_dev_data.bytes_transmitted_ >> net_dev_data.packets_transmitted_
+ >> net_dev_data.errs_transmitted_ >>
net_dev_data.drop_errors_transmitted_ >> net_dev_data.fifo_errors_transmitted_
+ >> net_dev_data.collisions_transmitted_ >>
net_dev_data.carrier_losses_transmitted_ >>
net_dev_data.compressed_packets_transmitted_;
Review comment:
makes sense, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-bb84b43a558560608fd5680317d7f8585c7d18d6f1b6b4c84f3036020667R26-R43
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831001542
##
File path: extensions/procfs/DiskStat.h
##
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+
+using namespace std::literals::chrono_literals;
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+class DiskStatData {
+ DiskStatData() = default;
+
+ public:
+ struct MonotonicIncreasingMembers {
+uint64_t reads_completed_;
+uint64_t reads_merged_;
+uint64_t sectors_read_;
+uint64_t milliseconds_spent_reading_;
+uint64_t writes_completed_;
+uint64_t writes_merges_;
Review comment:
thanks, changed this and the other occurrences in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152: URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831001765 ## File path: extensions/procfs/MemInfo.h ## @@ -0,0 +1,46 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include Review comment: fixed it in https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-6fed75c9f3cfbff3e0321e0e3d12e62ab961bfc77fc3f38ebcc758a9507cf35fR20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831001015
##
File path: extensions/procfs/CpuStat.h
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "SystemClockDuration.h"
+
+using namespace std::literals::chrono_literals;
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+class CpuStatData {
+ CpuStatData() = default;
Review comment:
makes sense, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r831000748
##
File path: extensions/procfs/DiskStat.cpp
##
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "DiskStat.h"
+#include
+#include "utils/gsl.h"
+
+using namespace std::literals::chrono_literals;
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+DiskStatData::MonotonicIncreasingMembers
DiskStatData::MonotonicIncreasingMembers::operator-(const
MonotonicIncreasingMembers& rhs) const {
+ MonotonicIncreasingMembers diff;
+ diff.reads_completed_ = reads_completed_ - rhs.reads_completed_;
+ diff.reads_merged_ = reads_merged_ - rhs.reads_merged_;
+ diff.sectors_read_ = sectors_read_ - rhs.sectors_read_;
+ diff.milliseconds_spent_reading_ = milliseconds_spent_reading_ -
rhs.milliseconds_spent_reading_;
+ diff.writes_completed_ = writes_completed_ - rhs.writes_completed_;
+ diff.writes_merges_ = writes_merges_ - rhs.writes_merges_;
+ diff.sectors_written_ = sectors_written_ - rhs.sectors_written_;
+ diff.milliseconds_spent_writing_ = milliseconds_spent_writing_ -
rhs.milliseconds_spent_writing_;
+ diff.milliseconds_spent_io_ = milliseconds_spent_io_ -
rhs.milliseconds_spent_io_;
+ diff.weighted_milliseconds_spent_io_ = weighted_milliseconds_spent_io_ -
rhs.weighted_milliseconds_spent_io_;
+ return diff;
+}
+
+std::optional>
DiskStatData::parseDiskStatLine(std::istream& iss) {
+ DiskStatData disk_stat_data;
+ std::string disk_name;
+ iss >> disk_stat_data.major_device_number_ >>
disk_stat_data.minor_device_number_ >> disk_name >>
disk_stat_data.monotonic_increasing_members_.reads_completed_
+ >> disk_stat_data.monotonic_increasing_members_.reads_merged_ >>
disk_stat_data.monotonic_increasing_members_.sectors_read_
+ >>
disk_stat_data.monotonic_increasing_members_.milliseconds_spent_reading_ >>
disk_stat_data.monotonic_increasing_members_.writes_completed_
+ >> disk_stat_data.monotonic_increasing_members_.writes_merges_ >>
disk_stat_data.monotonic_increasing_members_.sectors_written_
+ >>
disk_stat_data.monotonic_increasing_members_.milliseconds_spent_reading_ >>
disk_stat_data.monotonic_increasing_members_.writes_completed_
+ >> disk_stat_data.monotonic_increasing_members_.writes_merges_ >>
disk_stat_data.monotonic_increasing_members_.sectors_written_
+ >>
disk_stat_data.monotonic_increasing_members_.milliseconds_spent_writing_ >>
disk_stat_data.ios_in_progress_ >> disk_stat_data.ios_in_progress_
+ >> disk_stat_data.monotonic_increasing_members_.milliseconds_spent_io_
>> disk_stat_data.monotonic_increasing_members_.weighted_milliseconds_spent_io_;
Review comment:
wow awesome catch, fixed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-9e8cddde6f61ed0902194bc13cdfbd1d5e6635c96874a12cba648e2d5d1ae43fR44-R57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r83147
##
File path: libminifi/test/SingleProcessorTestController.h
##
@@ -54,6 +51,12 @@ class SingleInputTestController : public TestController {
return result;
}
+ auto trigger(const std::string_view input_flow_file_content,
std::unordered_map input_flow_file_attributes = {}) {
+const auto new_flow_file = createFlowFile(input_flow_file_content,
std::move(input_flow_file_attributes));
+input_->put(new_flow_file);
+return trigger();
+ }
Review comment:
Good idea, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/84e464c7b9501293616bcae170c4a12a710f7a6d#diff-a019248b6c700d2a0bc69681c6ea011e5afe07861a32a021f992375eeda11847R54
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r830995850
##
File path: extensions/procfs/ProcessStat.cpp
##
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ProcessStat.h"
+#include
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+std::optional
ProcessStatData::parseProcessStatFile(std::istream& stat_file) {
+ std::stringstream stat_stream;
+ copy(std::istreambuf_iterator(stat_file),
+ std::istreambuf_iterator(),
+ std::ostreambuf_iterator(stat_stream));
+
+ ProcessStatData process_stat_data;
+ stat_stream >> process_stat_data.pid_;
+
+ std::string stat_stream_string = stat_stream.str();
+ size_t comm_start = stat_stream_string.find_first_of('(');
+ size_t comm_end = stat_stream_string.find_last_of(')');
+ if (comm_start == std::string::npos || comm_end == std::string::npos)
+return std::nullopt;
+ process_stat_data.comm_ = stat_stream_string.substr(comm_start+1,
comm_end-comm_start-1);
+ stat_stream.seekg(comm_end+2);
Review comment:
comm fields can contain whitespaces, so this wouldn't work unfortunately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9819) SAML should have option to autocreate users.
[
https://issues.apache.org/jira/browse/NIFI-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wiktor Kubicki updated NIFI-9819:
-
Description:
*Prolog:*
In SSO i set roles that give the user access to the company's systems or
logical areas.
*Case:*
Using saml i can set nifi.security.user.saml.group.attribute.name and link SSO
roles to groups in NIFI, then i don't have to add users to groups in NiFi.
But what if user in NiFi doesn't exist, even if have NiFi group set up in SSO?
Then he receiving "Unknown user with identity..." alert. So before a user with
aproperiate roles in SSO logs in to NiFi, you have toadd him separatly.
This lack is confirmed in comment:
{code:java}
The real issue is "Unknown user with identity 'user2'"... all of the users and
groups still need to be known to NiFi's authorization, the only part that does
not need to be known is the actual group membership since that is coming from
the SAML response.{code}
~[http://disq.us/p/2g2fdie]
*Workaroung:*
I can create one user without privileges, and map user name to the new one in
{_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
flow history what gives me user accountability..
*Expected behavior:*
There should be an option in nifi.properties,
nifi.security.user.saml.create.user which, when is set to true, will add
"empty" (without privilages or groups) user. Then, if the user has the right
samla group, he will have access to the platform.
{*}Extra value{*}:
There may be extra option _nifi.security.user.saml.new.user.default.group_
which allowed to link new user to existing ( ! ) group, one or more.
was:
*Prolog:*
In SSO i set roles that give the user access to the company's systems or
logical areas.
*Case:*
Using saml i can set nifi.security.user.saml.group.attribute.name and link SSO
roles to groups in NIFI, then i don't have to add users to groups in NiFi.
But what if user in NiFi doesn't exist, even if have NiFi group set up in SSO?
Then he receiving "Unknown user with identity..." alert. So before a user with
aproperiate roles in SSO logs in to NiFi, you have toadd him separatly.
This lack is confirmed in comment:
{code:java}
The real issue is "Unknown user with identity 'user2'"... all of the users and
groups still need to be known to NiFi's authorization, the only part that does
not need to be known is the actual group membership since that is coming from
the SAML response.{code}
~http://disq.us/p/2g2fdie
*Workaroung:*
I can create one user without privileges, and map user name to the new one in
{_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
flow history what gives me user accountability..
*Expected behavior:*
There should be an option in nifi.properties,
nifi.security.user.saml.create.user which, when is set to true, will add
"empty" (without privilages or groups) user. Then, if the user has the right
samla group, he will have access to the platform.
{*}Extra value{*}:
There may be extra option _nifi.security.user.saml.new.user.default.group_
which allowed to link new user to existing(!) group, one or more.
> SAML should have option to autocreate users.
>
>
> Key: NIFI-9819
> URL: https://issues.apache.org/jira/browse/NIFI-9819
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Security
>Affects Versions: 1.15.3
>Reporter: Wiktor Kubicki
>Priority: Major
>
> *Prolog:*
> In SSO i set roles that give the user access to the company's systems or
> logical areas.
> *Case:*
> Using saml i can set nifi.security.user.saml.group.attribute.name and link
> SSO roles to groups in NIFI, then i don't have to add users to groups in NiFi.
> But what if user in NiFi doesn't exist, even if have NiFi group set up in
> SSO? Then he receiving "Unknown user with identity..." alert. So before a
> user with aproperiate roles in SSO logs in to NiFi, you have toadd him
> separatly.
> This lack is confirmed in comment:
> {code:java}
> The real issue is "Unknown user with identity 'user2'"... all of the users
> and groups still need to be known to NiFi's authorization, the only part that
> does not need to be known is the actual group membership since that is coming
> from the SAML response.{code}
> ~[http://disq.us/p/2g2fdie]
> *Workaroung:*
> I can create one user without privileges, and map user name to the new one in
> {_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
> flow history what gives me user accountability..
> *Expected behavior:*
> There should be an option in nifi.properties,
> nifi.security.user.saml.create.user which, when is set to true, will add
> "empty" (without privilages or groups) user. Then, if the user has the right
> samla group, he will have access
[jira] [Created] (NIFI-9819) SAML should have option to autocreate users.
Wiktor Kubicki created NIFI-9819:
Summary: SAML should have option to autocreate users.
Key: NIFI-9819
URL: https://issues.apache.org/jira/browse/NIFI-9819
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework, Security
Affects Versions: 1.15.3
Reporter: Wiktor Kubicki
*Prolog:*
In SSO i set roles that give the user access to the company's systems or
logical areas.
*Case:*
Using saml i can set nifi.security.user.saml.group.attribute.name and link SSO
roles to groups in NIFI, then i don't have to add users to groups in NiFi.
But what if user in NiFi doesn't exist, even if have NiFi group set up in SSO?
Then he receiving "Unknown user with identity..." alert. So before a user with
aproperiate roles in SSO logs in to NiFi, you have toadd him separatly.
This lack is confirmed in comment:
{code:java}
The real issue is "Unknown user with identity 'user2'"... all of the users and
groups still need to be known to NiFi's authorization, the only part that does
not need to be known is the actual group membership since that is coming from
the SAML response.{code}
~http://disq.us/p/2g2fdie
*Workaroung:*
I can create one user without privileges, and map user name to the new one in
{_}nifi.security.identity.mapping.value.dn{_}, but i will lost user names in
flow history what gives me user accountability..
*Expected behavior:*
There should be an option in nifi.properties,
nifi.security.user.saml.create.user which, when is set to true, will add
"empty" (without privilages or groups) user. Then, if the user has the right
samla group, he will have access to the platform.
{*}Extra value{*}:
There may be extra option _nifi.security.user.saml.new.user.default.group_
which allowed to link new user to existing(!) group, one or more.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
szaszm commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r830895453
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+ std::invocable
auto serializer) {
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed", disk_stat_data.getReadsCompleted());
+ serializer("Reads Merged", disk_stat_data.getReadsMerged());
+ serializer("Sectors Read", disk_stat_data.getSectorsRead());
+ serializer("Writes Completed", disk_stat_data.getWritesCompleted());
+ serializer("Writes Merged", disk_stat_data.getWritesMerged());
+ serializer("Sectors Written", disk_stat_data.getSectorsWritten());
+ serializer("IO-s in progress", disk_stat_data.getIosInProgress());
+}
+
+void SerializeDiskStatDataPerSec(const DiskStatData& disk_stat_data,
+ const std::chrono::duration duration,
+ std::invocable
auto serializer) {
+ gsl_Expects(duration > 0ms);
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed/sec",
disk_stat_data.getReadsCompleted()/duration.count());
+ serializer("Reads Merged/sec",
disk_stat_data.getReadsMerged()/duration.count());
+ serializer("Sectors Read/sec",
disk_stat_data.getSectorsRead()/duration.count());
+ serializer("Writes Completed/sec",
disk_stat_data.getWritesCompleted()/duration.count());
+ serializer("Writes Merged/sec",
disk_stat_data.getWritesMerged()/duration.count());
+ serializer("Sectors Written/sec",
disk_stat_data.getSectorsWritten()/duration.count());
+ serializer("IO-s in progress",
disk_stat_data.getIosInProgress()/duration.count());
+}
+
+void SerializeMemInfo(const MemInfo& mem_info,
+ std::invocable auto
serializer) {
+ serializer("MemTotal",
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
szaszm commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r830895453
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+ std::invocable
auto serializer) {
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed", disk_stat_data.getReadsCompleted());
+ serializer("Reads Merged", disk_stat_data.getReadsMerged());
+ serializer("Sectors Read", disk_stat_data.getSectorsRead());
+ serializer("Writes Completed", disk_stat_data.getWritesCompleted());
+ serializer("Writes Merged", disk_stat_data.getWritesMerged());
+ serializer("Sectors Written", disk_stat_data.getSectorsWritten());
+ serializer("IO-s in progress", disk_stat_data.getIosInProgress());
+}
+
+void SerializeDiskStatDataPerSec(const DiskStatData& disk_stat_data,
+ const std::chrono::duration duration,
+ std::invocable
auto serializer) {
+ gsl_Expects(duration > 0ms);
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed/sec",
disk_stat_data.getReadsCompleted()/duration.count());
+ serializer("Reads Merged/sec",
disk_stat_data.getReadsMerged()/duration.count());
+ serializer("Sectors Read/sec",
disk_stat_data.getSectorsRead()/duration.count());
+ serializer("Writes Completed/sec",
disk_stat_data.getWritesCompleted()/duration.count());
+ serializer("Writes Merged/sec",
disk_stat_data.getWritesMerged()/duration.count());
+ serializer("Sectors Written/sec",
disk_stat_data.getSectorsWritten()/duration.count());
+ serializer("IO-s in progress",
disk_stat_data.getIosInProgress()/duration.count());
+}
+
+void SerializeMemInfo(const MemInfo& mem_info,
+ std::invocable auto
serializer) {
+ serializer("MemTotal",
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
szaszm commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r830895453
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+ std::invocable
auto serializer) {
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed", disk_stat_data.getReadsCompleted());
+ serializer("Reads Merged", disk_stat_data.getReadsMerged());
+ serializer("Sectors Read", disk_stat_data.getSectorsRead());
+ serializer("Writes Completed", disk_stat_data.getWritesCompleted());
+ serializer("Writes Merged", disk_stat_data.getWritesMerged());
+ serializer("Sectors Written", disk_stat_data.getSectorsWritten());
+ serializer("IO-s in progress", disk_stat_data.getIosInProgress());
+}
+
+void SerializeDiskStatDataPerSec(const DiskStatData& disk_stat_data,
+ const std::chrono::duration duration,
+ std::invocable
auto serializer) {
+ gsl_Expects(duration > 0ms);
+ serializer("Major Device Number", disk_stat_data.getMajorDeviceNumber());
+ serializer("Minor Device Number", disk_stat_data.getMinorDeviceNumber());
+ serializer("Reads Completed/sec",
disk_stat_data.getReadsCompleted()/duration.count());
+ serializer("Reads Merged/sec",
disk_stat_data.getReadsMerged()/duration.count());
+ serializer("Sectors Read/sec",
disk_stat_data.getSectorsRead()/duration.count());
+ serializer("Writes Completed/sec",
disk_stat_data.getWritesCompleted()/duration.count());
+ serializer("Writes Merged/sec",
disk_stat_data.getWritesMerged()/duration.count());
+ serializer("Sectors Written/sec",
disk_stat_data.getSectorsWritten()/duration.count());
+ serializer("IO-s in progress",
disk_stat_data.getIosInProgress()/duration.count());
+}
+
+void SerializeMemInfo(const MemInfo& mem_info,
+ std::invocable auto
serializer) {
+ serializer("MemTotal",
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
szaszm commented on a change in pull request #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r827730482
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
Review comment:
I don't see the point of passing a string as a reference unless it's
used to detect the extent of the literal. I would either pass them as ordinary
`const char*` and rely on the optimizer a bit more, or parameterize the
implementation on the extent, but the second is not easy to write correct
concept checks for.
In the second parameter type check, I don't think const makes a difference
when passing by value.
##
File path: extensions/procfs/ProcFsSerialization.h
##
@@ -0,0 +1,163 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+#include
+
+#include "CpuStat.h"
+#include "DiskStat.h"
+#include "MemInfo.h"
+#include "NetDev.h"
+#include "ProcessStat.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+void SerializeCPUStatData(const CpuStatData& cpu_stat_data,
+ std::invocable auto
serializer) {
+ serializer("user time", cpu_stat_data.getUser().count());
+ serializer("nice time", cpu_stat_data.getNice().count());
+ serializer("system time", cpu_stat_data.getSystem().count());
+ serializer("idle time", cpu_stat_data.getIdle().count());
+ serializer("io wait time", cpu_stat_data.getIoWait().count());
+ serializer("irq time", cpu_stat_data.getIrq().count());
+ serializer("soft irq time", cpu_stat_data.getSoftIrq().count());
+ serializer("steal time", cpu_stat_data.getSteal().count());
+ serializer("guest time", cpu_stat_data.getGuest().count());
+ serializer("guest nice time", cpu_stat_data.getGuestNice().count());
+}
+
+void SerializeNormalizedCPUStat(const CpuStatData& cpu_stat_data,
+std::invocable
auto serializer) {
+ gsl_Expects(cpu_stat_data.getTotal() > 0ms);
+ serializer("user time %", cpu_stat_data.getUser()/cpu_stat_data.getTotal());
+ serializer("nice time %", cpu_stat_data.getNice()/cpu_stat_data.getTotal());
+ serializer("system time %",
cpu_stat_data.getSystem()/cpu_stat_data.getTotal());
+ serializer("idle time %", cpu_stat_data.getIdle()/cpu_stat_data.getTotal());
+ serializer("io wait time %",
cpu_stat_data.getIoWait()/cpu_stat_data.getTotal());
+ serializer("irq time %", cpu_stat_data.getIrq()/cpu_stat_data.getTotal());
+ serializer("soft irq %",
cpu_stat_data.getSoftIrq()/cpu_stat_data.getTotal());
+ serializer("steal time %",
cpu_stat_data.getSteal()/cpu_stat_data.getTotal());
+ serializer("guest time %",
cpu_stat_data.getGuest()/cpu_stat_data.getTotal());
+ serializer("guest nice time %",
cpu_stat_data.getGuestNice()/cpu_stat_data.getTotal());
+}
+
+void SerializeDiskStatData(const DiskStatData& disk_stat_data,
+
