[jira] [Commented] (NIFI-8521) Replace/improve InfluxDB extension.
[ https://issues.apache.org/jira/browse/NIFI-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522064#comment-17522064 ] Jakub Bednar commented on NIFI-8521: Hi [~exceptionfactory] and [~gbilicz], I am a maintainer of [https://github.com/influxdata/nifi-influxdb-bundle.] I would like to ask if Is there a place where we can promote our bundle... documentation or plugin registry? Regards > Replace/improve InfluxDB extension. > --- > > Key: NIFI-8521 > URL: https://issues.apache.org/jira/browse/NIFI-8521 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: GAbor Bilicz >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The builtin InfluxDB Processor requires well formatted data. But no > Processor/Controller service offered, to do this easily from a JSON for > example. > On this link they created an extension for this. I don't know, what's the > process, and howto discuss it with them, etc. It's just an idea, but I think > it would be cool. They also supporting InfluxDB 1.x and InlfuxDB 2.x > [https://github.com/influxdata/nifi-influxdb-bundle > BR, > Gabor|https://github.com/influxdata/nifi-influxdb-bundle] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9923) Component Logs sometimes missing Throwable Causes
[
https://issues.apache.org/jira/browse/NIFI-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9923:
---
Description:
As a result of improvements to correct component stack trace logging in
NIFI-9871, some calls to {{SimpleProcessLogger}} through {{ComponentLog}} do
not include the {{Throwable}} summary of causes in Bulletin Messages.
The {{nifi-app.log}} contains the correct formatting and stack trace, and this
issue does not impact any released versions.
The problem is the result of adding the summary of causes as an argument to the
Log Repository array of arguments, instead of replacing the {{Throwable}} with
the summary of causes.
was:
As a result of improvements to correct component stack trace logging in
NIFI-9871, some calls to {{SimpleProcessLogger}} through {{ComponentLog}} do
not include the {{Throwable}} summary of causes in Bulletin Messages.
The {{nifi-app.log}} contains the correct formatting and stack trace, and this
issue does not impact and released versions.
The problem is the result of adding the summary of causes as an argument to the
Log Repository array of arguments, instead of replacing the {{Throwable}} with
the summary of causes.
> Component Logs sometimes missing Throwable Causes
> -
>
> Key: NIFI-9923
> URL: https://issues.apache.org/jira/browse/NIFI-9923
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As a result of improvements to correct component stack trace logging in
> NIFI-9871, some calls to {{SimpleProcessLogger}} through {{ComponentLog}} do
> not include the {{Throwable}} summary of causes in Bulletin Messages.
> The {{nifi-app.log}} contains the correct formatting and stack trace, and
> this issue does not impact any released versions.
> The problem is the result of adding the summary of causes as an argument to
> the Log Repository array of arguments, instead of replacing the {{Throwable}}
> with the summary of causes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9923) Component Logs sometimes missing Throwable Causes
[
https://issues.apache.org/jira/browse/NIFI-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9923:
---
Status: Patch Available (was: Open)
> Component Logs sometimes missing Throwable Causes
> -
>
> Key: NIFI-9923
> URL: https://issues.apache.org/jira/browse/NIFI-9923
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As a result of improvements to correct component stack trace logging in
> NIFI-9871, some calls to {{SimpleProcessLogger}} through {{ComponentLog}} do
> not include the {{Throwable}} summary of causes in Bulletin Messages.
> The {{nifi-app.log}} contains the correct formatting and stack trace, and
> this issue does not impact and released versions.
> The problem is the result of adding the summary of causes as an argument to
> the Log Repository array of arguments, instead of replacing the {{Throwable}}
> with the summary of causes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] exceptionfactory opened a new pull request, #5965: NIFI-9923 Correct SimpleProcessLogger handling of Throwable causes
exceptionfactory opened a new pull request, #5965:
URL: https://github.com/apache/nifi/pull/5965
Description of PR
NIFI-9923 Corrects the behavior of `SimpleProcessLogger` to format the array
of arguments for Bulletin Messages in order to log the summary of Throwable
causes.
The correction updates log methods that accept a `String` and `Object` array
to replace the last `Throwable` argument with a formatted summary of causes
when calling `LogRepository.addLogMessage()`.
This corrects behavior introduced for NIFI-9871 in PR #5945. The problem and
resolution can be observed on processors such as `EvaluateXPath`, which write
error logs with one format argument and an exception.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-9923) Component Logs sometimes missing Throwable Causes
David Handermann created NIFI-9923:
--
Summary: Component Logs sometimes missing Throwable Causes
Key: NIFI-9923
URL: https://issues.apache.org/jira/browse/NIFI-9923
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: David Handermann
Assignee: David Handermann
Fix For: 1.16.1
As a result of improvements to correct component stack trace logging in
NIFI-9871, some calls to {{SimpleProcessLogger}} through {{ComponentLog}} do
not include the {{Throwable}} summary of causes in Bulletin Messages.
The {{nifi-app.log}} contains the correct formatting and stack trace, and this
issue does not impact and released versions.
The problem is the result of adding the summary of causes as an argument to the
Log Repository array of arguments, instead of replacing the {{Throwable}} with
the summary of causes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (NIFI-9922) Parameter Context parameter usage in UpdateAttribute Advance breaks syntax highlighting
Brandon Panna created NIFI-9922:
---
Summary: Parameter Context parameter usage in UpdateAttribute
Advance breaks syntax highlighting
Key: NIFI-9922
URL: https://issues.apache.org/jira/browse/NIFI-9922
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.16.0
Environment: Centos 7, Standalone NiFi node
Reporter: Brandon Panna
Referencing parameters #\{parameterName} in either conditions or actions fields
break syntax highlighting for the field. In addition, ctrl+space autocompletion
doesn't work.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (NIFI-9921) Parameter context referencing components not detecting UpdateAttribute advanced usage
Brandon Panna created NIFI-9921:
---
Summary: Parameter context referencing components not detecting
UpdateAttribute advanced usage
Key: NIFI-9921
URL: https://issues.apache.org/jira/browse/NIFI-9921
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.16.0
Environment: Centos 7, Standalone NiFi node
Reporter: Brandon Panna
Parameters defined in Parameter Context and used in the UpdateAttribute
Advanced section are not listed in the Parameter Context referencing components
list. No listing is produced by usage in either conditions or actions.
Issue 1.
Reproduce:
# Create two parameters in Parameter Context:
## conditionsTestParam:true
## actionsTestParam:myValue
# Create a rule in UpdateAttribute advanced
## Expression: ${#\{conditionsTestParam}:equals('true')}
## Actions: Att=outputFromAction Value=${#\{actionsTestParam}}
# Run flowfile though processor
# Flowfile attribute now has a new attribute outputFromAction with value
myValue
# Check Parameter Context panel
Expectations:
Referencing Components should list UpdateAttribute processor
A related issue, deleting the parameter doesn't cause the referencing
UpdateAttribute processor to disable.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9901) Implement Module for Common XML Utilities
[
https://issues.apache.org/jira/browse/NIFI-9901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9901:
---
Fix Version/s: 1.17.0
1.16.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Implement Module for Common XML Utilities
> -
>
> Key: NIFI-9901
> URL: https://issues.apache.org/jira/browse/NIFI-9901
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.16.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.17.0, 1.16.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The {{nifi-security-utils}} module includes several classes for working with
> various Java XML components. These classes are self-contained, and do not
> require the set of dependencies listed for {{{}nifi-security-utils{}}}.
> Refactoring these classes to a new module under {{nifi-commons}} would
> streamline the dependency tree for other modules that do not need Bouncy
> Castle or other dependencies of {{{}nifi-security-utils{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9901) Implement Module for Common XML Utilities
[
https://issues.apache.org/jira/browse/NIFI-9901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521982#comment-17521982
]
ASF subversion and git services commented on NIFI-9901:
---
Commit 15f7590f7ab27c16166200028d86f3ebd2974092 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=15f7590f7a ]
NIFI-9901 Added nifi-xml-processing to nifi-commons
- Refactored XML parsing to use providers from nifi-xml-processing
- Configured spotbugs-maven-plugin with findsecbugs-plugin in
nifi-xml-processing
- Disabled Validate DTD in default configuration for EvaluateXPath and
EvaluateXQuery
- Replaced configuration of DocumentBuilder and streaming XML Readers with
shared components
- Removed XML utilities from nifi-security-utils
- Moved Commons Configuration classes to nifi-lookup-services
This closes #5962
Signed-off-by: Paul Grey
> Implement Module for Common XML Utilities
> -
>
> Key: NIFI-9901
> URL: https://issues.apache.org/jira/browse/NIFI-9901
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.16.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The {{nifi-security-utils}} module includes several classes for working with
> various Java XML components. These classes are self-contained, and do not
> require the set of dependencies listed for {{{}nifi-security-utils{}}}.
> Refactoring these classes to a new module under {{nifi-commons}} would
> streamline the dependency tree for other modules that do not need Bouncy
> Castle or other dependencies of {{{}nifi-security-utils{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] greyp9 closed pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
greyp9 closed pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons URL: https://github.com/apache/nifi/pull/5962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-9920) On active ProcessGroup Bulletin, drag of process group on canvas causes bulletin to disappear
Paul Grey created NIFI-9920: --- Summary: On active ProcessGroup Bulletin, drag of process group on canvas causes bulletin to disappear Key: NIFI-9920 URL: https://issues.apache.org/jira/browse/NIFI-9920 Project: Apache NiFi Issue Type: Bug Reporter: Paul Grey Just noticed this on HEAD. Had a process group containing several processors. One of the processors alerted, causing red bulletin icon to appear in upper right of processor widget on canvas. Exited child process group to root PG, which displayed the same bulletin icon on the affected PG. I dragged the PG to rearrange things on the canvas, and I noticed the bulletin icon disappear. Canvas refresh causes bulletin to reappear. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9919) RSA Private Key Authentication Fails for Azure Blob SFTP
David Handermann created NIFI-9919:
--
Summary: RSA Private Key Authentication Fails for Azure Blob SFTP
Key: NIFI-9919
URL: https://issues.apache.org/jira/browse/NIFI-9919
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.16.0, 1.15.0, 1.14.0
Reporter: David Handermann
Assignee: David Handermann
Microsoft [Azure Blob
Storage|https://azure.microsoft.com/en-us/services/storage/blobs/] supports
access using SFTP with either password or private key authentication. [SFTP
support for Azure Blob
Storage|https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support]
has a limited set of supported algorithms, including the following three
algorithms for Public Key authentication:
* ssh-rsa
* ecdsa-sha2-nistp256
* ecdsa-sha2-nistp384
The documentation lists sshj 0.27.0 as supported, but changes in sshj 0.30.0 to
support RSA SHA2 algorithms appear to have created problems with selection of
the client key algorithm during the negotiation process. This issue persists in
sshj 0.32.0, but appears to be resolved in the current development branch of
sshj.
As a result of this issue, SFTP processors are unable to authenticate to Azure
Blob Storage SFTP and return the following error with a valid RSA Private Key:
{noformat}
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication
methods
{noformat}
It is possible to workaround the problem with RSA Private Keys using an ECDSA
Private Key, which can be generated using the following command on compatible
platforms:
{noformat}
ssh-keygen -t ecdsa
{noformat}
This issue may impact other SFTP servers that support RSA SHA2 host key
algorithms, but do not support that algorithm for Public Key authentication.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9903) LookupRecord doesn't add results to any Records if first Record doesn't have a match
[ https://issues.apache.org/jira/browse/NIFI-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-9903: --- Resolution: Fixed Status: Resolved (was: Patch Available) > LookupRecord doesn't add results to any Records if first Record doesn't have > a match > > > Key: NIFI-9903 > URL: https://issues.apache.org/jira/browse/NIFI-9903 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.17.0, 1.16.1 > > Time Spent: 40m > Remaining Estimate: 0h > > When using Lookup Record, if the "Routing Strategy" is set to "Route to > Success" and the first record in the incoming FlowFile doesn't match, no > Records in that FlowFile get enriched. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9903) LookupRecord doesn't add results to any Records if first Record doesn't have a match
[ https://issues.apache.org/jira/browse/NIFI-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521945#comment-17521945 ] ASF subversion and git services commented on NIFI-9903: --- Commit fa827649cf37457c6add35720f4ce58c23892779 in nifi's branch refs/heads/support/nifi-1.16 from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=fa827649cf ] NIFI-9903: This closes #5955. When using the 'success' relationship only for LookupRecord, lookup all records until a match is found, in order to determine the resultant schema. Refactored code to eliminate AbstractRouteRecord, because LookupRecord is the last processor that extended from it. Refactored code to use an inner interface to clean up code. Signed-off-by: Joe Witt > LookupRecord doesn't add results to any Records if first Record doesn't have > a match > > > Key: NIFI-9903 > URL: https://issues.apache.org/jira/browse/NIFI-9903 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.17.0, 1.16.1 > > Time Spent: 40m > Remaining Estimate: 0h > > When using Lookup Record, if the "Routing Strategy" is set to "Route to > Success" and the first record in the incoming FlowFile doesn't match, no > Records in that FlowFile get enriched. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9903) LookupRecord doesn't add results to any Records if first Record doesn't have a match
[ https://issues.apache.org/jira/browse/NIFI-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521944#comment-17521944 ] ASF subversion and git services commented on NIFI-9903: --- Commit 05f3d7510f24c24bd307461399c74195859d40e6 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=05f3d7510f ] NIFI-9903: This closes #5955. When using the 'success' relationship only for LookupRecord, lookup all records until a match is found, in order to determine the resultant schema. Refactored code to eliminate AbstractRouteRecord, because LookupRecord is the last processor that extended from it. Refactored code to use an inner interface to clean up code. Signed-off-by: Joe Witt > LookupRecord doesn't add results to any Records if first Record doesn't have > a match > > > Key: NIFI-9903 > URL: https://issues.apache.org/jira/browse/NIFI-9903 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.17.0, 1.16.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When using Lookup Record, if the "Routing Strategy" is set to "Route to > Success" and the first record in the incoming FlowFile doesn't match, no > Records in that FlowFile get enriched. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] asfgit closed pull request #5955: NIFI-9903: When using the 'success' relationship only for LookupRecor…
asfgit closed pull request #5955: NIFI-9903: When using the 'success' relationship only for LookupRecor… URL: https://github.com/apache/nifi/pull/5955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521937#comment-17521937 ] Otto Fowler commented on NIFI-9863: --- Yeah, this is fine in the backlog > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler updated NIFI-9863: -- Priority: Minor (was: Major) > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Minor > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9893) Ensure orderly cluster node removal on node delete via UI
[ https://issues.apache.org/jira/browse/NIFI-9893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-9893: --- Fix Version/s: 1.16.1 > Ensure orderly cluster node removal on node delete via UI > - > > Key: NIFI-9893 > URL: https://issues.apache.org/jira/browse/NIFI-9893 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Fix For: 1.17.0, 1.16.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a NiFi cluster node is deleted (via the UI), a series of steps is > executed to update the cluster state. If the node is being deleted due to > node process failure, then any attempt to communicate with that node will > fail, causing an exception. > In `NodeClusterCoordinator.removeNode()`, the sequence of steps could be > improved to perform the deletion before the node participants are notified. > This reduces the chance that node failure will prevent the UI operation from > completing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9893) Ensure orderly cluster node removal on node delete via UI
[ https://issues.apache.org/jira/browse/NIFI-9893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521934#comment-17521934 ] ASF subversion and git services commented on NIFI-9893: --- Commit ba16a1e196b6c2bd20e5a7d53a9a09a27719e0ef in nifi's branch refs/heads/support/nifi-1.16 from greyp9 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ba16a1e196 ] NIFI-9893 - Ensure orderly cluster node removal on node delete via UI (#5946) > Ensure orderly cluster node removal on node delete via UI > - > > Key: NIFI-9893 > URL: https://issues.apache.org/jira/browse/NIFI-9893 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Fix For: 1.17.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a NiFi cluster node is deleted (via the UI), a series of steps is > executed to update the cluster state. If the node is being deleted due to > node process failure, then any attempt to communicate with that node will > fail, causing an exception. > In `NodeClusterCoordinator.removeNode()`, the sequence of steps could be > improved to perform the deletion before the node participants are notified. > This reduces the chance that node failure will prevent the UI operation from > completing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9863) Controller Service for managing custom Grok patterns
[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521931#comment-17521931 ] David Handermann commented on NIFI-9863: Thanks [~otto], it sounds like scoping out the features of a new Grok-based Reader would be a first step for a potential new implementation. > Controller Service for managing custom Grok patterns > > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Otto Fowler >Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] patalwell commented on pull request #5905: NiFi-9817 Add a Validator for the PutCloudWatchMetric Processor's Unit Field
patalwell commented on PR #5905: URL: https://github.com/apache/nifi/pull/5905#issuecomment-1098457832 @nandorsoma ok made the changes, if you like them I'll rebase. Forgive my ignorance here, but everytime i attempt to rebase I get a boatload of commits from other folks. What's the best way to keep this clean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on PR #5962: URL: https://github.com/apache/nifi/pull/5962#issuecomment-1098442359 Thanks for the initial feedback @greyp9! Will plan on updating the one exception message noted, as well as anything else, following additional feedback from testing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849849414
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java:
##
@@ -137,7 +138,7 @@ public class EvaluateXPath extends AbstractProcessor {
.description("Specifies whether or not the XML content should be
validated against the DTD.")
.required(true)
.allowableValues("true", "false")
-.defaultValue("true")
Review Comment:
As mentioned for `EvaluateXQuery`, disabling validation of the Document Type
Declaration in the default configuration provides a more secure starting point
for new instances of the Processor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849846738
##
nifi-commons/nifi-xml-processing/src/main/java/org/apache/nifi/xml/processing/validation/StandardSchemaValidator.java:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml.processing.validation;
+
+import org.apache.nifi.xml.processing.ProcessingException;
+import org.apache.nifi.xml.processing.ProcessingFeature;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.transform.Source;
+import javax.xml.validation.Schema;
+import javax.xml.validation.Validator;
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Standard implementation of XML Schema Validator with secure processing
enabled
+ */
+public class StandardSchemaValidator implements SchemaValidator {
+/**
+ * Validate Source using Schema
+ *
+ * @param schema Schema source for Validator
+ * @param source Source to be validated
+ */
+@Override
+public void validate(final Schema schema, final Source source) {
Review Comment:
Yes, the `Schema` object provides the `newValidator()` method, so this
interface encapsulates that operation and sets standard properties on the
`Validator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849845697
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java:
##
@@ -156,7 +152,7 @@ public class EvaluateXQuery extends AbstractProcessor {
.description("Specifies whether or not the XML content should be
validated against the DTD.")
.required(true)
.allowableValues("true", "false")
-.defaultValue("true")
+.defaultValue("false")
Review Comment:
Disabling Document Type Validation in the default configuration provides a
more secure starting point for new instances of the Processor. The
implementation in `StandardDocumentProvider` provides standard security
restrictions on Document Type Validation, so enabling the `Validate DTD`
property is not the optimal configuration. Changing the default value to
`false` retains the property for deployments where embedded DTD validation is
desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849843776
##
nifi-framework-api/src/main/java/org/apache/nifi/authorization/AbstractPolicyBasedAuthorizer.java:
##
@@ -427,8 +429,7 @@ private PoliciesUsersAndGroups
parsePoliciesUsersAndGroups(final String fingerpr
final byte[] fingerprintBytes =
fingerprint.getBytes(StandardCharsets.UTF_8);
try (final ByteArrayInputStream in = new
ByteArrayInputStream(fingerprintBytes)) {
-final DocumentBuilder docBuilder = createSafeDocumentBuilder();
-final Document document = docBuilder.parse(in);
+final Document document = parseFingerprint(in);
Review Comment:
Good question! This is only remaining location with a local reference to the
`DocumentBuilderFactory`. PR #5514 for NIFI-9069 included the direct
implementation, as opposed to `XmlUtils` due to issues with class loading
between `nifi-framework-api` and dependent modules. It might be possible to
revisit the problem, but this change maintains a limited scope for
`nifi-framework-api` module dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849840721
##
nifi-commons/nifi-xml-processing/src/main/java/org/apache/nifi/xml/processing/sax/StandardInputSourceParser.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml.processing.sax;
+
+import org.apache.nifi.xml.processing.ProcessingException;
+import org.apache.nifi.xml.processing.ProcessingFeature;
+import org.xml.sax.ContentHandler;
+import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
+import org.xml.sax.XMLReader;
+
+import javax.xml.XMLConstants;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.parsers.SAXParser;
+import javax.xml.parsers.SAXParserFactory;
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Standard implementation of Input Source Parser with secure processing
enabled
+ */
+public class StandardInputSourceParser implements InputSourceParser {
+private boolean namespaceAware;
+
+/**
+ * Set Namespace Aware status on SAXParserFactory
+ *
+ * @param namespaceAware Namespace Aware status
+ */
+public void setNamespaceAware(final boolean namespaceAware) {
+this.namespaceAware = namespaceAware;
+}
+
+/**
+ * Parse Input Source using Content Handler
+ *
+ * @param inputSource Input Source to be parsed
+ * @param contentHandler Content Handler used during parsing
+ */
+@Override
+public void parse(final InputSource inputSource, final ContentHandler
contentHandler) {
+Objects.requireNonNull(inputSource, "InputSource required");
+Objects.requireNonNull(contentHandler, "ContentHandler required");
+
+try {
+parseInputSource(inputSource, contentHandler);
+} catch (final ParserConfigurationException|SAXException e) {
+throw new ProcessingException("Parser Configuration failed", e);
Review Comment:
That's a good point, will adjust to a more generalized `Parsing failed`
message for the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
exceptionfactory commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849840041
##
nifi-commons/nifi-xml-processing/src/main/java/org/apache/nifi/xml/processing/parsers/StandardDocumentProvider.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml.processing.parsers;
+
+import org.apache.nifi.xml.processing.ProcessingException;
+import org.apache.nifi.xml.processing.ProcessingFeature;
+import org.w3c.dom.Document;
+import org.xml.sax.ErrorHandler;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.validation.Schema;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+/**
+ * Standard implementation of Document Provider with secure processing enabled
+ */
+public class StandardDocumentProvider implements DocumentProvider {
+private boolean namespaceAware;
+
+private Schema schema;
+
+private ErrorHandler errorHandler;
+
+/**
+ * Set Error Handler
+ *
+ * @param errorHandler Error Handler
+ */
+public void setErrorHandler(final ErrorHandler errorHandler) {
+this.errorHandler = errorHandler;
+}
+
+/**
+ * Set Namespace Aware status on DocumentBuilderFactory
+ *
+ * @param namespaceAware Namespace Awareness
+ */
+public void setNamespaceAware(final boolean namespaceAware) {
+this.namespaceAware = namespaceAware;
+}
+
+/**
+ * Set Namespace Aware status on DocumentBuilderFactory
+ *
+ * @param schema Schema for validation or null to disable validation
+ */
+public void setSchema(final Schema schema) {
+this.schema = schema;
+}
+
+@Override
+public Document newDocument() {
+final DocumentBuilderFactory documentBuilderFactory =
getDocumentBuilderFactory();
+
+try {
+
documentBuilderFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING,
ProcessingFeature.SECURE_PROCESSING.isEnabled());
Review Comment:
The purpose of the `ProcessingFeature.SECURE_PROCESSING` reference was to
define the property value in a central location that could be reused in
multiple classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] greyp9 commented on a diff in pull request #5962: NIFI-9901 Add nifi-xml-processing to nifi-commons
greyp9 commented on code in PR #5962:
URL: https://github.com/apache/nifi/pull/5962#discussion_r849740132
##
nifi-commons/nifi-xml-processing/src/main/java/org/apache/nifi/xml/processing/sax/StandardInputSourceParser.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml.processing.sax;
+
+import org.apache.nifi.xml.processing.ProcessingException;
+import org.apache.nifi.xml.processing.ProcessingFeature;
+import org.xml.sax.ContentHandler;
+import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
+import org.xml.sax.XMLReader;
+
+import javax.xml.XMLConstants;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.parsers.SAXParser;
+import javax.xml.parsers.SAXParserFactory;
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Standard implementation of Input Source Parser with secure processing
enabled
+ */
+public class StandardInputSourceParser implements InputSourceParser {
+private boolean namespaceAware;
+
+/**
+ * Set Namespace Aware status on SAXParserFactory
+ *
+ * @param namespaceAware Namespace Aware status
+ */
+public void setNamespaceAware(final boolean namespaceAware) {
+this.namespaceAware = namespaceAware;
+}
+
+/**
+ * Parse Input Source using Content Handler
+ *
+ * @param inputSource Input Source to be parsed
+ * @param contentHandler Content Handler used during parsing
+ */
+@Override
+public void parse(final InputSource inputSource, final ContentHandler
contentHandler) {
+Objects.requireNonNull(inputSource, "InputSource required");
+Objects.requireNonNull(contentHandler, "ContentHandler required");
+
+try {
+parseInputSource(inputSource, contentHandler);
+} catch (final ParserConfigurationException|SAXException e) {
+throw new ProcessingException("Parser Configuration failed", e);
Review Comment:
This would encompass both the parser configuration and the parse operation.
`Parser Configuration / Parse Operation failed`
##
nifi-commons/nifi-xml-processing/src/main/java/org/apache/nifi/xml/processing/parsers/StandardDocumentProvider.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.xml.processing.parsers;
+
+import org.apache.nifi.xml.processing.ProcessingException;
+import org.apache.nifi.xml.processing.ProcessingFeature;
+import org.w3c.dom.Document;
+import org.xml.sax.ErrorHandler;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.validation.Schema;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+/**
+ * Standard implementation of Document Provider with secure processing enabled
+ */
+public class StandardDocumentProvider implements DocumentProvider {
+private boolean namespaceAware;
+
+private Schema schema;
+
+private ErrorHandler errorHandler;
+
+/**
+ * Set Error Handler
+ *
+ * @param errorHandler Error Handler
+ */
+public void setErrorHandler(final ErrorHandler errorHandler) {
+this.errorHandler = errorHandler;
+}
+
+/**
+ * Set Namespace Aware status on
[GitHub] [nifi] markap14 commented on a diff in pull request #5958: NIFI-9895 Allow parameters to reference controller services
markap14 commented on code in PR #5958:
URL: https://github.com/apache/nifi/pull/5958#discussion_r849445707
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/AuthorizeControllerServiceReference.java:
##
@@ -88,9 +91,17 @@ public static void
authorizeControllerServiceReferences(final Map {
// Verify READ and WRITE permissions for user, for the
Parameter Context itself
-final Authorizable parameterContext =
lookup.getParameterContext(contextId);
+final ParameterContext parameterContext =
lookup.getParameterContext(contextId);
parameterContext.authorize(authorizer, RequestAction.READ,
user);
parameterContext.authorize(authorizer, RequestAction.WRITE,
user);
// Verify READ and WRITE permissions for user, for every
component that is affected
affectedComponents.forEach(component ->
authorizeAffectedComponent(component, lookup, user, true, true));
+
+
requestEntity.getComponent().getParameters().forEach(parameterEntity -> {
+String parameterName =
parameterEntity.getParameter().getName();
+List
referencedControllerServiceDataSet = parameterContext
+.getParameterReferenceManager()
+.getReferencedControllerServiceData(parameterContext,
parameterName);
+
+Set>
referencedControllerServiceTypes = referencedControllerServiceDataSet
+.stream()
+
.map(ParameterReferencedControllerServiceData::getReferencedControllerServiceType)
+.collect(Collectors.toSet());
+
+if (referencedControllerServiceTypes.size() > 1) {
+throw new IllegalStateException("Parameter is used by
multiple different types of controller service references");
+} else if (!referencedControllerServiceTypes.isEmpty()) {
+Optional.ofNullable(parameterEntity)
+.map(ParameterEntity::getParameter)
+.map(ParameterDTO::getName)
+.flatMap(parameterContext::getParameter)
+.map(Parameter::getValue)
+.map(lookup::getControllerService)
+.map(ComponentAuthorizable::getAuthorizable)
+.ifPresent(controllerServiceAuthorizable -> {
+
controllerServiceAuthorizable.authorize(authorizer, RequestAction.READ, user);
+
controllerServiceAuthorizable.authorize(authorizer, RequestAction.WRITE, user);
+});
+
+Optional.ofNullable(parameterEntity)
+.map(ParameterEntity::getParameter)
+.map(ParameterDTO::getValue)
+.map(lookup::getControllerService)
+.map(ComponentAuthorizable::getAuthorizable)
+.ifPresent(controllerServiceAuthorizable -> {
+
controllerServiceAuthorizable.authorize(authorizer, RequestAction.READ, user);
+
controllerServiceAuthorizable.authorize(authorizer, RequestAction.WRITE, user);
+
+if (
+!referencedControllerServiceTypes
+.stream()
+.findFirst()
+.get()
+.isAssignableFrom(
+((StandardControllerServiceNode)
controllerServiceAuthorizable).getComponent().getClass()
+)
+) {
+throw new IllegalArgumentException("New
Parameter value attempts to reference an incompatible controller service");
+}
+});
+}
+});
Review Comment:
We should be preferring procedural/imperative coding over functional coding
where it makes sense. As a general rule of thumb, a lambda should not exceed
3-5 lines of code.
See Google's Guava's recommendations/explanations here as to why:
https://github.com/google/guava/wiki/FunctionalExplained
As well as Effective Java by Joshua Bloch, where he discusses the some
reasons for using functional style judiciously and keeping lambdas to only a
few lines.
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java:
##
@@ -460,18 +455,14 @@ private void
[jira] [Updated] (NIFI-9893) Ensure orderly cluster node removal on node delete via UI
[ https://issues.apache.org/jira/browse/NIFI-9893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-9893: - Fix Version/s: 1.17.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Ensure orderly cluster node removal on node delete via UI > - > > Key: NIFI-9893 > URL: https://issues.apache.org/jira/browse/NIFI-9893 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Fix For: 1.17.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a NiFi cluster node is deleted (via the UI), a series of steps is > executed to update the cluster state. If the node is being deleted due to > node process failure, then any attempt to communicate with that node will > fail, causing an exception. > In `NodeClusterCoordinator.removeNode()`, the sequence of steps could be > improved to perform the deletion before the node participants are notified. > This reduces the chance that node failure will prevent the UI operation from > completing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9893) Ensure orderly cluster node removal on node delete via UI
[ https://issues.apache.org/jira/browse/NIFI-9893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521849#comment-17521849 ] ASF subversion and git services commented on NIFI-9893: --- Commit 3034f2637a99dca92fc7ab200e7d8a938f57916f in nifi's branch refs/heads/main from greyp9 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3034f2637a ] NIFI-9893 - Ensure orderly cluster node removal on node delete via UI (#5946) > Ensure orderly cluster node removal on node delete via UI > - > > Key: NIFI-9893 > URL: https://issues.apache.org/jira/browse/NIFI-9893 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > When a NiFi cluster node is deleted (via the UI), a series of steps is > executed to update the cluster state. If the node is being deleted due to > node process failure, then any attempt to communicate with that node will > fail, causing an exception. > In `NodeClusterCoordinator.removeNode()`, the sequence of steps could be > improved to perform the deletion before the node participants are notified. > This reduces the chance that node failure will prevent the UI operation from > completing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] markap14 commented on pull request #5946: NIFI-9893 - Ensure orderly cluster node removal on node delete via UI
markap14 commented on PR #5946: URL: https://github.com/apache/nifi/pull/5946#issuecomment-1098320355 Thanks @greyp9 changes look good to me. +1 merged to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 merged pull request #5946: NIFI-9893 - Ensure orderly cluster node removal on node delete via UI
markap14 merged PR #5946: URL: https://github.com/apache/nifi/pull/5946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9903) LookupRecord doesn't add results to any Records if first Record doesn't have a match
[ https://issues.apache.org/jira/browse/NIFI-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521846#comment-17521846 ] Stephen Jeffrey Hindmarch commented on NIFI-9903: - My fault, I was building against wrong branch. > LookupRecord doesn't add results to any Records if first Record doesn't have > a match > > > Key: NIFI-9903 > URL: https://issues.apache.org/jira/browse/NIFI-9903 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.17.0, 1.16.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When using Lookup Record, if the "Routing Strategy" is set to "Route to > Success" and the first record in the incoming FlowFile doesn't match, no > Records in that FlowFile get enriched. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (MINIFICPP-1799) PutGCSObject wrong property name
[
https://issues.apache.org/jira/browse/MINIFICPP-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1799:
---
Status: Patch Available (was: In Progress)
https://github.com/apache/nifi-minifi-cpp/pull/1303
> PutGCSObject wrong property name
>
>
> Key: MINIFICPP-1799
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1799
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> const core::Property PutGCSObject::Key(
> core::PropertyBuilder::createProperty("Name of the object.")
> ->withDescription("Name of the object.")
> ->withDefaultValue("${filename}")
> ->supportsExpressionLanguage(true)
> ->build()); {code}
> should be
> {code:java}
> const core::Property PutGCSObject::Key(
> core::PropertyBuilder::createProperty("Key")
> ->withDescription("Name of the object.")
> ->withDefaultValue("${filename}")
> ->supportsExpressionLanguage(true)
> ->build()); {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (MINIFICPP-1536) Deprecated ProcessMetrics.h and SystemMetrics.h
[ https://issues.apache.org/jira/browse/MINIFICPP-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1536: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1302 > Deprecated ProcessMetrics.h and SystemMetrics.h > --- > > Key: MINIFICPP-1536 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1536 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Trivial > Labels: MiNiFi-CPP-Hygiene > Time Spent: 10m > Remaining Estimate: 0h > > There are couple unused ResponseNodes (i.e ProcessMetrics, SystemMetrics) > which contain more or less the same data as DeviceInfoNode and > AgentInformation. > e.g.: > systeminfo.systemInfo.vCores, systeminfo.systemInfo.physicalMem vs > deviceInfo.systemInfo.vCores, deviceInfo.systemInfo.physicalMem > The deprecated classes are not properly platform independent and provide > minimal added information (only involuntary context switches) > In my opinion we should remove these classes to avoid misusage. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (MINIFICPP-1799) PutGCSObject wrong property name
[
https://issues.apache.org/jira/browse/MINIFICPP-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1799:
---
Description:
{code:java}
const core::Property PutGCSObject::Key(
core::PropertyBuilder::createProperty("Name of the object.")
->withDescription("Name of the object.")
->withDefaultValue("${filename}")
->supportsExpressionLanguage(true)
->build()); {code}
should be
{code:java}
const core::Property PutGCSObject::Key(
core::PropertyBuilder::createProperty("Key")
->withDescription("Name of the object.")
->withDefaultValue("${filename}")
->supportsExpressionLanguage(true)
->build()); {code}
was:
{code:java}
const core::Property PutGCSObject::Key(
core::PropertyBuilder::createProperty("Name of the object.")
->withDescription("Name of the object.")
->withDefaultValue("${filename}")
->supportsExpressionLanguage(true)
->build()); {code}
> PutGCSObject wrong property name
>
>
> Key: MINIFICPP-1799
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1799
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> const core::Property PutGCSObject::Key(
> core::PropertyBuilder::createProperty("Name of the object.")
> ->withDescription("Name of the object.")
> ->withDefaultValue("${filename}")
> ->supportsExpressionLanguage(true)
> ->build()); {code}
> should be
> {code:java}
> const core::Property PutGCSObject::Key(
> core::PropertyBuilder::createProperty("Key")
> ->withDescription("Name of the object.")
> ->withDefaultValue("${filename}")
> ->supportsExpressionLanguage(true)
> ->build()); {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1303: MINIFICPP-1799 Fix PutGCSObject::Key property name
martinzink opened a new pull request, #1303: URL: https://github.com/apache/nifi-minifi-cpp/pull/1303 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
martinzink commented on code in PR #1294: URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849604941 ## docker/test/integration/minifi/core/ImageStore.py: ## @@ -181,6 +185,12 @@ def __build_mqtt_broker_image(self): def __build_splunk_image(self): return self.__build_image_by_path(self.test_dir + "/resources/splunk-hec", 'minifi-splunk') +def __build_syslog_udp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/udp", 'syslog-udp-client') + +def __build_syslog_tcp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/tcp", 'syslog-tcp-client') Review Comment: Replaced it with a simpler version, based on our conversation :+1: https://github.com/apache/nifi-minifi-cpp/pull/1294/commits/29d3690e8bd030a1315f6c72fbac0f1a3d0c2cc8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
martinzink commented on code in PR #1294: URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849588662 ## docker/test/integration/minifi/core/ImageStore.py: ## @@ -181,6 +185,12 @@ def __build_mqtt_broker_image(self): def __build_splunk_image(self): return self.__build_image_by_path(self.test_dir + "/resources/splunk-hec", 'minifi-splunk') +def __build_syslog_udp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/udp", 'syslog-udp-client') + +def __build_syslog_tcp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/tcp", 'syslog-tcp-client') Review Comment: Good idea. I've changed this in https://github.com/apache/nifi-minifi-cpp/pull/1294/commits/26053fde5f0a6d4ed055ee98558d9653fd89254a ## docker/test/integration/resources/syslog-client/tcp/Dockerfile: ## @@ -0,0 +1,2 @@ +FROM ubuntu:latest Review Comment: Good idea. I've changed this in https://github.com/apache/nifi-minifi-cpp/pull/1294/commits/26053fde5f0a6d4ed055ee98558d9653fd89254a -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] hindmasj commented on pull request #5955: NIFI-9903: When using the 'success' relationship only for LookupRecor…
hindmasj commented on PR #5955: URL: https://github.com/apache/nifi/pull/5955#issuecomment-1098151594 Encountered CERT_UNTRUSTED error when executing this stage. `Running 'npm install bower' in /home/sjh/workspace/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory` Had to disable SSL checking in npm to continue. `npm config set strict-ssl false` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5937: NIFI-9862: Update JsonTreeReader to read Records from a Nested Array
exceptionfactory commented on code in PR #5937:
URL: https://github.com/apache/nifi/pull/5937#discussion_r849574880
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/JsonTreeReader.java:
##
@@ -59,32 +61,62 @@
+ "If an array is encountered, each element in that array will be
treated as a separate record. "
+ "If the schema that is configured contains a field that is not
present in the JSON, a null value will be used. If the JSON contains "
+ "a field that is not present in the schema, that field will be
skipped. "
-+ "See the Usage of the Controller Service for more information and
examples.")
++ "See the Usage of the Controller Service for more information and
examples.")
@SeeAlso(JsonPathReader.class)
public class JsonTreeReader extends SchemaRegistryService implements
RecordReaderFactory {
private volatile String dateFormat;
private volatile String timeFormat;
private volatile String timestampFormat;
+private volatile String startingFieldName;
+private volatile StartingFieldStrategy startingFieldStrategy;
+
+public static final PropertyDescriptor STARTING_FIELD_STRATEGY = new
PropertyDescriptor.Builder()
+.name("starting-field-strategy")
+.displayName("Starting Field Strategy")
+.description("Start processing from the root node or from a
specified nested node.")
+.required(true)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.defaultValue(StartingFieldStrategy.ROOT_NODE.name())
+.allowableValues(
+
Arrays.stream(StartingFieldStrategy.values()).map(startingStrategy ->
+new AllowableValue(startingStrategy.name(),
startingStrategy.name(), startingStrategy.getDescription())
Review Comment:
Recommend adding a `getDisplayName()` and `StartingFieldStrategy` and
referencing it when constructing the AllowableValue to make it more readable:
```suggestion
new AllowableValue(startingStrategy.name(),
startingStrategy.getDisplayName(), startingStrategy.getDescription())
```
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/StartingFieldStrategy.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.json;
+
+public enum StartingFieldStrategy {
+ROOT_NODE("Begins processing from the root node."),
Review Comment:
Recommend adding a display name:
```suggestion
ROOT_NODE("Root Node", "Begins processing from the root node"),
```
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/StartingFieldStrategy.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.json;
+
+public enum StartingFieldStrategy {
+ROOT_NODE("Begins processing from the root node."),
+NESTED_NODE("Skips forward to the given nested JSON field (array or
object) to begin processing.");
Review Comment:
```suggestion
NESTED_FIELD("Nested Field", "Skips forward to the given nested JSON
field
[GitHub] [nifi] hindmasj commented on pull request #5955: NIFI-9903: When using the 'success' relationship only for LookupRecor…
hindmasj commented on PR #5955: URL: https://github.com/apache/nifi/pull/5955#issuecomment-1098133714 I am building this for myself but seem to have hit [NIFI-3554](https://issues.apache.org/jira/browse/NIFI-3554). Manually editing nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml and changing "jBcrypt" to "jBCrypt" allows me to progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1799) PutGCSObject wrong property name
Martin Zink created MINIFICPP-1799:
--
Summary: PutGCSObject wrong property name
Key: MINIFICPP-1799
URL: https://issues.apache.org/jira/browse/MINIFICPP-1799
Project: Apache NiFi MiNiFi C++
Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink
{code:java}
const core::Property PutGCSObject::Key(
core::PropertyBuilder::createProperty("Name of the object.")
->withDescription("Name of the object.")
->withDefaultValue("${filename}")
->supportsExpressionLanguage(true)
->build()); {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9918) XMLReader: XML attribute parsing should be optional, configurable
[ https://issues.apache.org/jira/browse/NIFI-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Gyori updated NIFI-9918: -- Status: Patch Available (was: In Progress) > XMLReader: XML attribute parsing should be optional, configurable > - > > Key: NIFI-9918 > URL: https://issues.apache.org/jira/browse/NIFI-9918 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Major > Labels: reader, xml > Time Spent: 10m > Remaining Estimate: 0h > > In older versions of NiFi, XMLReader ignored XML attributes. > Currently, XMLReader parses XML attributes and adds them to the record as a > new field. > This is not backward compatible with the previous behavior, and some users > might experience that new fields appear in their records after upgrading NiFi > - especially if they use schema inference with XML data. > To introduce compatibility with the previous behavior, a new property should > be added to XMLReader that makes it possible to turn off parsing XML > attributes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] pgyori opened a new pull request, #5964: NIFI-9918: 'Parse XML Attributes' property added to XMLReader
pgyori opened a new pull request, #5964:
URL: https://github.com/apache/nifi/pull/5964
https://issues.apache.org/jira/browse/NIFI-9918
Description of PR
Adds an extra property to XMLReader, called 'Parse XML Attributes'. It is a
boolean property that affects whether XML attributes are considered when
parsing XML data.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1796) Log properties are not retrieved properly with getString from Configure
[ https://issues.apache.org/jira/browse/MINIFICPP-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi reassigned MINIFICPP-1796: Assignee: Gábor Gyimesi > Log properties are not retrieved properly with getString from Configure > --- > > Key: MINIFICPP-1796 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1796 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > getString should retrieve properties from the Configure object in raw form, > without decrypting them. Because getString is implemented in Properties and > the Configure object contains and additional Properties object for the log > properties, the getString call made through a Configure object is not > forwarded properly to the log Properties object. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (MINIFICPP-1796) Log properties are not retrieved properly with getString from Configure
[ https://issues.apache.org/jira/browse/MINIFICPP-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi updated MINIFICPP-1796: - Status: Patch Available (was: In Progress) > Log properties are not retrieved properly with getString from Configure > --- > > Key: MINIFICPP-1796 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1796 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > getString should retrieve properties from the Configure object in raw form, > without decrypting them. Because getString is implemented in Properties and > the Configure object contains and additional Properties object for the log > properties, the getString call made through a Configure object is not > forwarded properly to the log Properties object. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-9917) Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and PublishKafkaRecord*
[ https://issues.apache.org/jira/browse/NIFI-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Soma Abonyi updated NIFI-9917: - Status: Patch Available (was: In Progress) > Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and > PublishKafkaRecord* > > > Key: NIFI-9917 > URL: https://issues.apache.org/jira/browse/NIFI-9917 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nandor Soma Abonyi >Assignee: Nandor Soma Abonyi >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > > The tooltip of `Best effort` option of `Delivery Guarantee` property is a bit > misleading, because it's not clear what successfully means in that context: > "FlowFile will be routed to success after successfully writing the content to > a Kafka node, without waiting for a response. This provides the best > performance but may result in data loss." At the end of the first sentence it > mentions that "without waiting for a response", but it doesn't catch the eye. > Probably "the producer will not wait for any acknowledgment from the server > at all." from the Kafka guide is more expedient: > [https://kafka.apache.org/documentation/#producerconfigs_acks] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] nandorsoma opened a new pull request, #5963: NIFI-9917 Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and PublishKafkaRecord*
nandorsoma opened a new pull request, #5963:
URL: https://github.com/apache/nifi/pull/5963
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
The tooltip of `Best effort` option of `Delivery Guarantee` property is a
bit misleading, because it's not clear what successfully means in that context:
"FlowFile will be routed to success after successfully writing the content to a
Kafka node, without waiting for a response. This provides the best performance
but may result in data loss." At the end of the first sentence it mentions that
"without waiting for a response", but it doesn't catch the eye. Probably "the
producer will not wait for any acknowledgment from the server at all." from the
Kafka guide is more expedient:
https://kafka.apache.org/documentation/#producerconfigs_acks
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9917) Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and PublishKafkaRecord*
[ https://issues.apache.org/jira/browse/NIFI-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Soma Abonyi updated NIFI-9917: - Description: The tooltip of `Best effort` option of `Delivery Guarantee` property is a bit misleading, because it's not clear what successfully means in that context: "FlowFile will be routed to success after successfully writing the content to a Kafka node, without waiting for a response. This provides the best performance but may result in data loss." At the end of the first sentence it mentions that "without waiting for a response", but it doesn't catch the eye. Probably "the producer will not wait for any acknowledgment from the server at all." from the Kafka guide is more expedient: [https://kafka.apache.org/documentation/#producerconfigs_acks] was: The tooltip of `Best effort` is a bit misleading, because it's not clear what successfully means in that context: "FlowFile will be routed to success after successfully writing the content to a Kafka node, without waiting for a response. This provides the best performance but may result in data loss." At the end of the first sentence it mentions that "without waiting for a response", but it doesn't catch the eye. Probably "the producer will not wait for any acknowledgment from the server at all." from the Kafka guide is more expedient: [https://kafka.apache.org/documentation/#producerconfigs_acks] > Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and > PublishKafkaRecord* > > > Key: NIFI-9917 > URL: https://issues.apache.org/jira/browse/NIFI-9917 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nandor Soma Abonyi >Assignee: Nandor Soma Abonyi >Priority: Minor > > > The tooltip of `Best effort` option of `Delivery Guarantee` property is a bit > misleading, because it's not clear what successfully means in that context: > "FlowFile will be routed to success after successfully writing the content to > a Kafka node, without waiting for a response. This provides the best > performance but may result in data loss." At the end of the first sentence it > mentions that "without waiting for a response", but it doesn't catch the eye. > Probably "the producer will not wait for any acknowledgment from the server > at all." from the Kafka guide is more expedient: > [https://kafka.apache.org/documentation/#producerconfigs_acks] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1302: MINIFICPP-1536 Remove deprecated ProcessMetrics and SystemMetrics
martinzink opened a new pull request, #1302: URL: https://github.com/apache/nifi-minifi-cpp/pull/1302 ProcessMetrics and SystemMetrics are superseded by AgentInformation and DeviceInfoNode. These deprecated classes are not platform independent and they provide no additional information that AgentInformation and DeviceInfoNode doesnt already provide. They are also not used in any C2 implementation as far as I could tell. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
lordgamez commented on code in PR #1294: URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849461803 ## docker/test/integration/resources/syslog-client/tcp/Dockerfile: ## @@ -0,0 +1,2 @@ +FROM ubuntu:latest Review Comment: Please use a specific version instead of latest as the latest tag could refer to another version later. ## docker/test/integration/minifi/core/ImageStore.py: ## @@ -181,6 +185,12 @@ def __build_mqtt_broker_image(self): def __build_splunk_image(self): return self.__build_image_by_path(self.test_dir + "/resources/splunk-hec", 'minifi-splunk') +def __build_syslog_udp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/udp", 'syslog-udp-client') + +def __build_syslog_tcp_client_image(self): +return self.__build_image_by_path(self.test_dir + "/resources/syslog-client/tcp", 'syslog-tcp-client') Review Comment: Do these need to be built as new images? Looking at the Dockerfiles it seems to me that we could use the ubuntu image and define the entrypoint parameter with the command in the image. This way we could remove the images, and have only 1 container with 2 different commands for TCP and UDP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
martinzink commented on code in PR #1294:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849460551
##
extensions/standard-processors/processors/ListenSyslog.cpp:
##
@@ -17,318 +14,272 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
+
#include "ListenSyslog.h"
-#include
-#include
-#include
-#include
-#include
-#include
-#include "utils/TimeUtil.h"
-#include "utils/StringUtils.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
-#include "core/TypedValues.h"
#include "core/Resource.h"
-namespace org {
-namespace apache {
-namespace nifi {
-namespace minifi {
-namespace processors {
-#ifndef WIN32
-core::Property ListenSyslog::RecvBufSize(
-core::PropertyBuilder::createProperty("Receive Buffer
Size")->withDescription("The size of each buffer used to receive Syslog
messages.")->
-withDefaultValue("65507 B")->build());
-
-core::Property ListenSyslog::MaxSocketBufSize(
-core::PropertyBuilder::createProperty("Max Size of Socket
Buffer")->withDescription("The maximum size of the socket buffer that should be
used.")->withDefaultValue("1 MB")
-->build());
+namespace org::apache::nifi::minifi::processors {
-core::Property ListenSyslog::MaxConnections(
-core::PropertyBuilder::createProperty("Max Number of TCP
Connections")->withDescription("The maximum number of concurrent connections to
accept Syslog messages in TCP mode.")
-->withDefaultValue(2)->build());
+const core::Property ListenSyslog::Port(
+core::PropertyBuilder::createProperty("Listening Port")
+->withDescription("The port for Syslog communication.")
+->isRequired(true)
+->withDefaultValue(514,
core::StandardValidators::get().LISTEN_PORT_VALIDATOR)->build());
-core::Property ListenSyslog::MaxBatchSize(
-core::PropertyBuilder::createProperty("Max Batch
Size")->withDescription("The maximum number of Syslog events to add to a single
FlowFile.")->withDefaultValue(1)->build());
+const core::Property ListenSyslog::ProtocolProperty(
+core::PropertyBuilder::createProperty("Protocol")
+->withDescription("The protocol for Syslog communication.")
+->isRequired(true)
+->withAllowableValues(Protocol::values())
+->withDefaultValue(toString(Protocol::UDP))
+->build());
-core::Property ListenSyslog::MessageDelimiter(
-core::PropertyBuilder::createProperty("Message
Delimiter")->withDescription("Specifies the delimiter to place between Syslog
messages when multiple "
-
"messages are bundled together (see
core::Property).")->withDefaultValue("\n")->build());
+const core::Property ListenSyslog::MaxBatchSize(
+core::PropertyBuilder::createProperty("Max Batch Size")
+->withDescription("The maximum number of Syslog events to process at a
time.")
+->withDefaultValue(500,
std::make_shared("Greater or equal than 1
validator", 1))
+->build());
-core::Property ListenSyslog::ParseMessages(
-core::PropertyBuilder::createProperty("Parse
Messages")->withDescription("Indicates if the processor should parse the Syslog
messages. If set to false, each outgoing FlowFile will only.")
+const core::Property ListenSyslog::ParseMessages(
+core::PropertyBuilder::createProperty("Parse Messages")
+->withDescription("Indicates if the processor should parse the Syslog
messages. "
+ "If set to false, each outgoing FlowFile will only
contain the sender, protocol, and port, and no additional attributes.")
->withDefaultValue(false)->build());
-core::Property ListenSyslog::Protocol(
-core::PropertyBuilder::createProperty("Protocol")->withDescription("The
protocol for Syslog
communication.")->withAllowableValue("UDP")->withAllowableValue("TCP")->withDefaultValue(
-"UDP")->build());
+const core::Property ListenSyslog::MaxQueueSize(
+core::PropertyBuilder::createProperty("Max Size of Message Queue")
+->withDescription("Maximum number of Syslog messages allowed to be
buffered before processing them when the processor is triggered. "
+ "If the buffer full, the message is ignored. If set
to zero the buffer is unlimited.")
+->withDefaultValue(0)->build());
+
+const core::Relationship ListenSyslog::Success("success", "Incoming messages
that match the expected format when parsing will be sent to this relationship. "
+ "When Parse Messages
is set to false, all incoming message will be sent to this relationship.");
+const core::Relationship ListenSyslog::Invalid("invalid", "Incoming messages
that do not match the expected format when parsing will be sent to this
relationship.");
+
-core::Property ListenSyslog::Port(
-
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
martinzink commented on code in PR #1294: URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849458708 ## PROCESSORS.md: ## @@ -1002,27 +1002,52 @@ In the list below, the names of required properties appear in bold. Any other pr ### Description -Listens for Syslog messages being sent to a given port over TCP or UDP. Incoming messages are checked against regular expressions for RFC5424 and RFC3164 formatted messages. The format of each message is: ()(VERSION )(TIMESTAMP) (HOSTNAME) (BODY) where version is optional. The timestamp can be an RFC5424 timestamp with a format of "-MM-dd'T'HH:mm:ss.SZ" or "-MM-dd'T'HH:mm:ss.S+hh:mm", or it can be an RFC3164 timestamp with a format of "MMM d HH:mm:ss". If an incoming messages matches one of these patterns, the message will be parsed and the individual pieces will be placed in FlowFile attributes, with the original message in the content of the FlowFile. If an incoming message does not match one of these patterns it will not be parsed and the syslog.valid attribute will be set to false with the original message in the content of the FlowFile. Valid messages will be transferred on the success relationship, and invalid messages will be transferred on the invalid relat ionship. +Listens for Syslog messages being sent to a given port over TCP or UDP. +Incoming messages are optionally checked against regular expressions for RFC5424 and RFC3164 formatted messages. +With parsing enabled the individual parts of the message will be placed as FlowFile attributes and valid messages will be transferred to success relationship, while invalid messages will be transferred to invalid relationship. +With parsing disabled all message will be routed to the success relationship, but they will only contain the sender, protocol, and port attributes. + + ### Properties In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -|Max Batch Size|1||The maximum number of Syslog events to add to a single FlowFile.| -|Max Number of TCP Connections|2||The maximum number of concurrent connections to accept Syslog messages in TCP mode.| -|Max Size of Socket Buffer|1 MB||The maximum size of the socket buffer that should be used.| -|Message Delimiter|\n||Specifies the delimiter to place between Syslog messages when multiple messages are bundled together (see core::Property).| -|Parse Messages|false||Indicates if the processor should parse the Syslog messages. If set to false, each outgoing FlowFile will only.| -|Port|514||The port for Syslog communication| -|Protocol|UDP|UDPTCP|The protocol for Syslog communication.| -|Receive Buffer Size|65507 B||The size of each buffer used to receive Syslog messages.| +| Name | Default Value | Allowable Values | Description | +|---|---|--|--| +| Listening Port| 514 | | The port for Syslog communication. | Review Comment: Good idea, I've changed it in https://github.com/apache/nifi-minifi-cpp/pull/1294/commits/251865e006f1e53e45c33fdaa184a9259f7bf13d -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-9917) Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and PublishKafkaRecord*
[ https://issues.apache.org/jira/browse/NIFI-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Soma Abonyi updated NIFI-9917: - Summary: Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and PublishKafkaRecord* (was: Fix DELIVERY_BEST_EFFORT's tooltip in PublishKafka* and PublishKafkaRecord*) > Fix tooltip of "delivery guarantee/best effort" in PublishKafka* and > PublishKafkaRecord* > > > Key: NIFI-9917 > URL: https://issues.apache.org/jira/browse/NIFI-9917 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nandor Soma Abonyi >Assignee: Nandor Soma Abonyi >Priority: Minor > > > The tooltip of `Best effort` is a bit misleading, because it's not clear what > successfully means in that context: "FlowFile will be routed to success after > successfully writing the content to a Kafka node, without waiting for a > response. This provides the best performance but may result in data loss." At > the end of the first sentence it mentions that "without waiting for a > response", but it doesn't catch the eye. Probably "the producer will not wait > for any acknowledgment from the server at all." from the Kafka guide is more > expedient: [https://kafka.apache.org/documentation/#producerconfigs_acks] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] markap14 commented on pull request #5958: NIFI-9895 Allow parameters to reference controller services
markap14 commented on PR #5958: URL: https://github.com/apache/nifi/pull/5958#issuecomment-1098016608 Thanks @tpalfy , will review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-9918) XMLReader: XML attribute parsing should be optional, configurable
Peter Gyori created NIFI-9918: - Summary: XMLReader: XML attribute parsing should be optional, configurable Key: NIFI-9918 URL: https://issues.apache.org/jira/browse/NIFI-9918 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Peter Gyori Assignee: Peter Gyori In older versions of NiFi, XMLReader ignored XML attributes. Currently, XMLReader parses XML attributes and adds them to the record as a new field. This is not backward compatible with the previous behavior, and some users might experience that new fields appear in their records after upgrading NiFi - especially if they use schema inference with XML data. To introduce compatibility with the previous behavior, a new property should be added to XMLReader that makes it possible to turn off parsing XML attributes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849428772
##
extensions/procfs/tests/CMakeLists.txt:
##
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+file(GLOB PROCFS_TESTS "*.cpp")
+
+SET(PROCFS_TEST_COUNT 0)
+FOREACH(testfile ${PROCFS_TESTS})
+get_filename_component(testfilename "${testfile}" NAME_WE)
+add_executable("${testfilename}" "${testfile}")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/procfs")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/libminifi/test/")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors")
+createTests("${testfilename}")
+target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
+target_link_libraries(${testfilename} minifi-procfs)
+target_link_libraries(${testfilename} minifi-standard-processors)
+MATH(EXPR PROCFS_TEST_COUNT "${PROCFS_TEST_COUNT}+1")
+add_test(NAME "${testfilename}" COMMAND "${testfilename}")
+ENDFOREACH()
+file(COPY "${CMAKE_SOURCE_DIR}/extensions/procfs/tests/mockprocfs_t0"
DESTINATION "${CMAKE_BINARY_DIR}/extensions/procfs/tests/")
+file(COPY "${CMAKE_SOURCE_DIR}/extensions/procfs/tests/mockprocfs_t1"
DESTINATION "${CMAKE_BINARY_DIR}/extensions/procfs/tests/")
Review Comment:
good idea, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/6d1985dea3427458200b68e5082df02b15e8d3c9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-9917) Fix DELIVERY_BEST_EFFORT's tooltip in PublishKafka* and PublishKafkaRecord*
Nandor Soma Abonyi created NIFI-9917: Summary: Fix DELIVERY_BEST_EFFORT's tooltip in PublishKafka* and PublishKafkaRecord* Key: NIFI-9917 URL: https://issues.apache.org/jira/browse/NIFI-9917 Project: Apache NiFi Issue Type: Improvement Reporter: Nandor Soma Abonyi Assignee: Nandor Soma Abonyi The tooltip of `Best effort` is a bit misleading, because it's not clear what successfully means in that context: "FlowFile will be routed to success after successfully writing the content to a Kafka node, without waiting for a response. This provides the best performance but may result in data loss." At the end of the first sentence it mentions that "without waiting for a response", but it doesn't catch the eye. Probably "the producer will not wait for any acknowledgment from the server at all." from the Kafka guide is more expedient: [https://kafka.apache.org/documentation/#producerconfigs_acks] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (MINIFICPP-1798) Change LogAttribute default Maximum Payload Line Length to 0
Marton Szasz created MINIFICPP-1798: --- Summary: Change LogAttribute default Maximum Payload Line Length to 0 Key: MINIFICPP-1798 URL: https://issues.apache.org/jira/browse/MINIFICPP-1798 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Marton Szasz -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #1298: MINIFICPP-1708 - Route flowfiles to Failure on decompression error
arpadboda closed pull request #1298: MINIFICPP-1708 - Route flowfiles to Failure on decompression error URL: https://github.com/apache/nifi-minifi-cpp/pull/1298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1797) Rewrite the bootstrap script in python
Marton Szasz created MINIFICPP-1797: --- Summary: Rewrite the bootstrap script in python Key: MINIFICPP-1797 URL: https://issues.apache.org/jira/browse/MINIFICPP-1797 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Marton Szasz Make it cross-platform. Maybe a pysimplegui-based GUI? More robust package version checks? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1301: MINIFICPP-1796 Fix getting raw value of log properties
lordgamez opened a new pull request, #1301: URL: https://github.com/apache/nifi-minifi-cpp/pull/1301 https://issues.apache.org/jira/browse/MINIFICPP-1796 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1675) Stack overflow in ExtractText for long matches on regular expressions
[
https://issues.apache.org/jira/browse/MINIFICPP-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz updated MINIFICPP-1675:
Status: Patch Available (was: Open)
https://github.com/apache/nifi-minifi-cpp/pull/1300
> Stack overflow in ExtractText for long matches on regular expressions
> -
>
> Key: MINIFICPP-1675
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1675
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Major
> Attachments: core.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Regex search can crash in ExtractText::ReadCallback::process at
> ExtractText.cpp:184 when calling
> {code:java}
> while (std::regex_search(workStr, matches, rgx)) {
> {code}
> due to stack overflow in long line matches.
> Possible options to handle the issue suggested by [~szaszm]:
> # Check out if {{}} {{{}regcomp{}}}/{{{}regexec{}}} works better.
> Revert [https://github.com/apache/nifi-minifi-cpp/pull/1159/files] and change
> to always use regcomp/regexec on libstdc++ (or on Linux if it's not possible
> to check)
> # Check if using POSIX regex instead of ECMAScript avoids the issue with
> std::regex.
> # Add option to limit the matched string size. I would make this a property
> in minifi.properties rather than the processor, since this is a standard
> implementation-specific issue, not a processor-specific one.
> An example backtrace with over 2000 frames:
> {code:java}
> #2010 0x7f073a7ae987 in
> std::__detail::_Executor<__gnu_cxx::__normal_iterator std::__cxx11::basic_string, std::allocator
> > >, std::allocator const*, std::__cxx11::basic_string,
> std::allocator > > > >, std::__cxx11::regex_traits,
> true>::_M_search_from_first (this=0x7f07350ce6b0) at
> /usr/include/c++/10.2.1/bits/regex_executor.h:101
> #2011 0x7f073565469c in
> std::__detail::_Executor<__gnu_cxx::__normal_iterator std::__cxx11::basic_string, std::allocator
> > >, std::allocator const*, std::__cxx11::basic_string,
> std::allocator > > > >, std::__cxx11::regex_traits,
> true>::_M_search (this=0x7f07350ce6b0) at
> /usr/include/c++/10.2.1/bits/regex_executor.tcc:42
> #2012 0x7f073565364f in
> std::__detail::__regex_algo_impl<__gnu_cxx::__normal_iterator std::__cxx11::basic_string, std::allocator
> > >, std::allocator const*, std::__cxx11::basic_string,
> std::allocator > > > >, char, std::__cxx11::regex_traits,
> (std::__detail::_RegexExecutorPolicy)0, false> (__s=..., __e=..., __m=...,
> __re=..., __flags=0) at /usr/include/c++/10.2.1/bits/regex.tcc:82
> #2013 0x7f0735651ad4 in
> std::regex_search<__gnu_cxx::__normal_iterator std::__cxx11::basic_string, std::allocator
> > >, std::allocator const*, std::__cxx11::basic_string,
> std::allocator > > > >, char, std::__cxx11::regex_traits >
> (__s=..., __e=..., __m=..., __re=..., __flags=0) at
> /usr/include/c++/10.2.1/bits/regex.h:2337#2014 0x7f073565075a in
> std::regex_search, std::allocator,
> std::allocator const*, std::__cxx11::basic_string,
> std::allocator > > > >, char, std::__cxx11::regex_traits >
> (__s=..., __m=..., __e=..., __f=0) at
> /usr/include/c++/10.2.1/bits/regex.h:2443
> #2015 0x7f073564d3ac in
> org::apache::nifi::minifi::processors::ExtractText::ReadCallback::process
> (this=0x7f07350ced40, stream=...) at
> /opt/minifi/extensions/standard-processors/processors/ExtractText.cpp:184
> #2016 0x7f073a6ffc36 in
> org::apache::nifi::minifi::core::ProcessSession::read (this=0x7f07346509a0,
> flow=..., callback=0x7f07350ced40) at
> /opt/minifi/libminifi/src/core/ProcessSession.cpp:317
> #2017 0x7f073564cc5a in
> org::apache::nifi::minifi::processors::ExtractText::onTrigger
> (this=0x7f0735365b40, context=0x7f07357842f0, session=0x7f07346509a0) at
> /opt/minifi/extensions/standard-processors/processors/ExtractText.cpp:111
> #2018 0x7f073a71bd4e in
> org::apache::nifi::minifi::core::Processor::onTrigger (this=0x7f0735365b40,
> context=..., session=...) at
> /opt/minifi/libminifi/include/core/Processor.h:222
> #2019 0x7f073a71a4fb in
> org::apache::nifi::minifi::core::Processor::onTrigger (this=0x7f0735365b40,
> context=..., sessionFactory=...) at
> /opt/minifi/libminifi/src/core/Processor.cpp:244
> #2020 0x7f073a63c51c in
> org::apache::nifi::minifi::SchedulingAgent::onTrigger (this=0x7f0739793b60,
> processor=..., processContext=..., sessionFactory=...) at
> /opt/minifi/libminifi/src/SchedulingAgent.cpp:120
> #2021 0x7f073a5c3290 in
> org::apache::nifi::minifi::EventDrivenSchedulingAgent::run
> (this=0x7f0739793b60, processor=..., processContext=..., sessionFactory=...)
> at /opt/minifi/libminifi/src/EventDrivenSchedulingAgent.cpp:45
> #2022
[GitHub] [nifi-minifi-cpp] arpadboda commented on a diff in pull request #1294: MINIFICPP-1771: Reworked ListenSyslog
arpadboda commented on code in PR #1294:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1294#discussion_r849376987
##
PROCESSORS.md:
##
@@ -1002,27 +1002,52 @@ In the list below, the names of required properties
appear in bold. Any other pr
### Description
-Listens for Syslog messages being sent to a given port over TCP or UDP.
Incoming messages are checked against regular expressions for RFC5424 and
RFC3164 formatted messages. The format of each message is: ()(VERSION
)(TIMESTAMP) (HOSTNAME) (BODY) where version is optional. The timestamp can be
an RFC5424 timestamp with a format of "-MM-dd'T'HH:mm:ss.SZ" or
"-MM-dd'T'HH:mm:ss.S+hh:mm", or it can be an RFC3164 timestamp with a
format of "MMM d HH:mm:ss". If an incoming messages matches one of these
patterns, the message will be parsed and the individual pieces will be placed
in FlowFile attributes, with the original message in the content of the
FlowFile. If an incoming message does not match one of these patterns it will
not be parsed and the syslog.valid attribute will be set to false with the
original message in the content of the FlowFile. Valid messages will be
transferred on the success relationship, and invalid messages will be
transferred on the invalid relat
ionship.
+Listens for Syslog messages being sent to a given port over TCP or UDP.
+Incoming messages are optionally checked against regular expressions for
RFC5424 and RFC3164 formatted messages.
+With parsing enabled the individual parts of the message will be placed as
FlowFile attributes and valid messages will be transferred to success
relationship, while invalid messages will be transferred to invalid
relationship.
+With parsing disabled all message will be routed to the success relationship,
but they will only contain the sender, protocol, and port attributes.
+
+
### Properties
In the list below, the names of required properties appear in bold. Any other
properties (not in bold) are considered optional. The table also indicates any
default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-| - | - | - | - |
-|Max Batch Size|1||The maximum number of Syslog events to add to a single
FlowFile.|
-|Max Number of TCP Connections|2||The maximum number of concurrent connections
to accept Syslog messages in TCP mode.|
-|Max Size of Socket Buffer|1 MB||The maximum size of the socket buffer that
should be used.|
-|Message Delimiter|\n||Specifies the delimiter to place between Syslog
messages when multiple messages are bundled together (see
core::Property).|
-|Parse Messages|false||Indicates if the processor should parse the Syslog
messages. If set to false, each outgoing FlowFile will only.|
-|Port|514||The port for Syslog communication|
-|Protocol|UDP|UDPTCP|The protocol for Syslog communication.|
-|Receive Buffer Size|65507 B||The size of each buffer used to receive Syslog
messages.|
+| Name | Default Value | Allowable Values | Description
|
+|---|---|--|--|
+| Listening Port| 514 | | The port for
Syslog communication.
|
Review Comment:
I think a warning here would be useful about the required root priviliges in
case of using the default port.
##
extensions/standard-processors/processors/ListenSyslog.cpp:
##
@@ -17,318 +14,272 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
+
#include "ListenSyslog.h"
-#include
-#include
-#include
-#include
-#include
-#include
-#include "utils/TimeUtil.h"
-#include "utils/StringUtils.h"
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
-#include "core/TypedValues.h"
#include "core/Resource.h"
-namespace org {
-namespace apache {
-namespace nifi {
-namespace minifi {
-namespace processors {
-#ifndef WIN32
-core::Property ListenSyslog::RecvBufSize(
-core::PropertyBuilder::createProperty("Receive Buffer
Size")->withDescription("The size of each buffer used to receive Syslog
messages.")->
-withDefaultValue("65507 B")->build());
-
-core::Property ListenSyslog::MaxSocketBufSize(
-core::PropertyBuilder::createProperty("Max Size of Socket
Buffer")->withDescription("The maximum size of the socket buffer that should be
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849392840
##
extensions/procfs/processors/ProcFsMonitor.h:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "../ProcFs.h"
+#include "core/Processor.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/logging/Logger.h"
+#include "utils/Enum.h"
+
+#include "rapidjson/stream.h"
+#include "rapidjson/document.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+class ProcFsMonitor : public core::Processor {
+ public:
+ explicit ProcFsMonitor(const std::string& name, utils::Identifier uuid =
utils::Identifier())
+ : Processor(name, uuid) {
+ }
+ ProcFsMonitor(const ProcFsMonitor&) = delete;
+ ProcFsMonitor(ProcFsMonitor&&) = delete;
+ ProcFsMonitor& operator=(const ProcFsMonitor&) = delete;
+ ProcFsMonitor& operator=(ProcFsMonitor&&) = delete;
+ ~ProcFsMonitor() override = default;
Review Comment:
thanks that's good to know, less boilerplate
removed them in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/df0ddde8ad2fa559dee75a3c51752a6e0d2f0eb0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849392192
##
extensions/procfs/tests/CPUStatTests.cpp:
##
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Catch.h"
+#include "ProcFs.h"
+
+#include "rapidjson/stream.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFs;
+using org::apache::nifi::minifi::extensions::procfs::CpuStatData;
+using org::apache::nifi::minifi::extensions::procfs::SystemClockDuration;
+
+void cpu_stat_period_total_should_be_one(const CpuStatData& cpu_stat) {
+ double percentage = 0;
+ percentage += cpu_stat.getUser()/cpu_stat.getTotal();
+ percentage += cpu_stat.getNice()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSystem()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIdle()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIoWait()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIrq()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSoftIrq()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSteal()/cpu_stat.getTotal();
+ percentage += cpu_stat.getGuest()/cpu_stat.getTotal();
+ percentage += cpu_stat.getGuestNice()/cpu_stat.getTotal();
+ REQUIRE(percentage == Approx(1.0));
+}
+
+TEST_CASE("ProcFSTest stat test with mock", "[procfsstatmockabsolutetest]") {
+ ProcFs proc_fs_t0("./mockprocfs_t0");
+ auto cpu_stats_t0 = proc_fs_t0.getCpuStats();
+ REQUIRE(cpu_stats_t0.size() == 13);
+ for (const auto& [cpu_name, cpu_stat] : cpu_stats_t0) {
+cpu_stat_period_total_should_be_one(cpu_stat);
+ }
+
+ REQUIRE(cpu_stats_t0[4].first == "cpu3");
+ CHECK(cpu_stats_t0[4].second.getUser() == SystemClockDuration(1299551));
+ CHECK(cpu_stats_t0[4].second.getNice() == SystemClockDuration(732));
+ CHECK(cpu_stats_t0[4].second.getSystem() == SystemClockDuration(316295));
+ CHECK(cpu_stats_t0[4].second.getIdle() == SystemClockDuration(4498383));
+ CHECK(cpu_stats_t0[4].second.getIoWait() == SystemClockDuration(1428));
+ CHECK(cpu_stats_t0[4].second.getIrq() == SystemClockDuration(22491));
+ CHECK(cpu_stats_t0[4].second.getSoftIrq() == SystemClockDuration(7022));
+ CHECK(cpu_stats_t0[4].second.getSteal() == SystemClockDuration(0));
+ CHECK(cpu_stats_t0[4].second.getGuest() == SystemClockDuration(0));
+ CHECK(cpu_stats_t0[4].second.getGuestNice() == SystemClockDuration(0));
+
+ ProcFs proc_fs_t1("./mockprocfs_t1");
+ auto cpu_stats_t1 = proc_fs_t1.getCpuStats();
+ REQUIRE(cpu_stats_t1.size() == 13);
+ for (const auto& [cpu_name, cpu_stat] : cpu_stats_t1) {
+cpu_stat_period_total_should_be_one(cpu_stat);
+ }
+
+ for (size_t i = 0; i < cpu_stats_t1.size(); ++i) {
+const auto& cpu_name_t0 = cpu_stats_t0[i].first;
+const auto& cpu_stat_t0 = cpu_stats_t0[i].second;
+const auto& cpu_name_t1 = cpu_stats_t1[i].first;
+const auto& cpu_stat_t1 = cpu_stats_t1[i].second;
Review Comment:
changed these in in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/f06938c35f0943a66c4bd1d7b895a02da8c0a83f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849391978
##
extensions/procfs/tests/DiskStatTests.cpp:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Catch.h"
+#include "ProcFs.h"
+#include "DiskStat.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFs;
+
+TEST_CASE("ProcFSTest DiskStat with mock", "[procfsdiskstatmocktest]") {
+ ProcFs proc_fs_t0("./mockprocfs_t0");
+ auto disk_stats_t0 = proc_fs_t0.getDiskStats();
+ REQUIRE(disk_stats_t0.size() == 17);
+
+ REQUIRE(disk_stats_t0[3].first == "nvme0n1p3");
+ CHECK(disk_stats_t0[3].second.getMajorDeviceNumber() == 259);
+ CHECK(disk_stats_t0[3].second.getMinorDeviceNumber() == 3);
+ CHECK(disk_stats_t0[3].second.getReadsCompleted() == 59);
+ CHECK(disk_stats_t0[3].second.getReadsMerged() == 1);
+ CHECK(disk_stats_t0[3].second.getSectorsRead() == 4526);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentReading() == 30);
+ CHECK(disk_stats_t0[3].second.getWritesCompleted() == 510);
+ CHECK(disk_stats_t0[3].second.getWritesMerged() == 3533);
+ CHECK(disk_stats_t0[3].second.getSectorsWritten() == 32344);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentWriting() == 15021);
+ CHECK(disk_stats_t0[3].second.getIosInProgress() == 0);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentIo() == 6150);
+ CHECK(disk_stats_t0[3].second.getWeightedMillisecondsSpentIo() == 15052);
+
+
+ ProcFs proc_fs_t1("./mockprocfs_t1");
+ auto disk_stats_t1 = proc_fs_t1.getDiskStats();
+ REQUIRE(disk_stats_t1.size() == 17);
+
+ for (const auto& disk_stat_t0 : disk_stats_t0) {
+auto disk_stat_t1 = std::find_if(disk_stats_t1.begin(),
disk_stats_t1.end(), [_stat_t0](auto& it){return it.first ==
disk_stat_t0.first;});
Review Comment:
makes sense, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/f06938c35f0943a66c4bd1d7b895a02da8c0a83f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
martinzink commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849391618
##
extensions/procfs/tests/ProcFsMonitorTests.cpp:
##
@@ -0,0 +1,157 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "SingleProcessorTestController.h"
+#include "Catch.h"
+#include "processors/ProcFsMonitor.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFsMonitor;
+
+TEST_CASE("ProcFsMonitorTests", "[procfsmonitortests]") {
+ std::shared_ptr proc_fs_monitor =
std::make_shared("ProcFsMonitor");
+ org::apache::nifi::minifi::test::SingleProcessorTestController
test_controller_{proc_fs_monitor};
+
+ SECTION("Absolute JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("CPU"));
+CHECK(document["CPU"].HasMember("cpu"));
+CHECK(document.HasMember("Disk"));
+CHECK(document.HasMember("Network"));
+CHECK(document.HasMember("Process"));
+CHECK(document.HasMember("Memory"));
+ }
+
+ SECTION("Absolute OpenTelemetry") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "OpenTelemetry");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("Body"));
+REQUIRE(document["Body"].HasMember("CPU"));
+CHECK(document["Body"]["CPU"].HasMember("cpu"));
+CHECK(document["Body"].HasMember("Disk"));
+CHECK(document["Body"].HasMember("Network"));
+CHECK(document["Body"].HasMember("Process"));
+CHECK(document["Body"].HasMember("Memory"));
+ }
+
+ SECTION("Relative JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Relative");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+{
+ const auto& result = test_controller_.trigger();
+
+ REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+ const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+ rapidjson::Document document;
+ auto content = test_controller_.plan->getContent(result_flow_file);
+ document.Parse(content.c_str());
+ REQUIRE(document.IsObject());
+ // First trigger has not enough information for relative output
+ CHECK_FALSE(document.HasMember("CPU"));
+ CHECK_FALSE(document.HasMember("Disk"));
+ CHECK_FALSE(document.HasMember("Network"));
+ CHECK_FALSE(document.HasMember("Process"));
+ CHECK(document.HasMember("Memory"));
+}
+sleep(1);
Review Comment:
good idea, changed it in
https://github.com/apache/nifi-minifi-cpp/pull/1152/commits/f06938c35f0943a66c4bd1d7b895a02da8c0a83f
##
extensions/procfs/tests/MemInfoTests.cpp:
##
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1271: MINIFICPP-1763 - Move extension inclusion logic into the extensions
adamdebreceni commented on code in PR #1271:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1271#discussion_r849389586
##
CONTRIB.md:
##
@@ -103,22 +103,29 @@ are contributing a custom Processor or Controller
Service, the mechanism to regi
To use this include REGISTER_RESOURCE(YourClassName); in your header file. The
default class loader will make instances of YourClassName available for
inclusion.
-The extensions sub-directory allows you to contribute conditionally built
extensions. An example of the GPS extension will provide an example. In this a
conditional
-allows flags to specify that your extension is to be include or excluded by
default. In this example -DENABLE_GPS=ON must be specified by the builder to
include it.
-The function call will then create an extension that will automatically be
while main is built. The first argument of createExtension will be the target
-reference that is automatically used for documentation and linking. The second
and third arguments are used for printing information on what was built or
linked in
-the consumer's build. The last two argument represent where the extension and
tests exist.
-
- if (ENABLE_ALL OR ENABLE_GPS)
- createExtension(GPS-EXTENSION "GPS EXTENSIONS" "Enables LibGPS
Functionality and the GetGPS processor." "extensions/gps"
"${TEST_DIR}/gps-tests")
- endif()
-
-
-Once the createExtension target is made in the root CMakeLists.txt , you may
load your dependencies and build your targets. Once you are finished defining
your build
-and link commands, you must set your target reference to a target within your
build. In this case, the previously mentioned GPS-EXTENSION will be assigned to
minifi-gps.
-The next call register_extension will ensure that minifi-gps is linked
appropriately for inclusion into the final binary.
-
- SET (GPS-EXTENSION minifi-gps PARENT_SCOPE)
- register_extension(minifi-gps)
-
-
+The extensions sub-directory allows you to contribute conditionally built
extensions. The system adds all subdirectories in `extensions/*` that contain
+a `CMakeLists.txt` file. It is up to the extension creator's discretion how
they handle cmake flags.
+It is important that `register_extension` be called at the end of the setup,
for the extension to be made available to other stages of the build process.
+
+```
+# extensions/gps/CMakeLists.txt
+
+# the author chooses to look for the explicit compilation request
+if (NOT ENABLE_GPS)
+ return()
+endif()
+
+#
+# extension definition goes here
+#
+
+# at the end we should announce our extension
+register_extension(minifi-gps "GPS EXTENSIONS" GPS-EXTENSION "Enables LibGPS
Functionality and the GetGPS processor." "${TEST_DIR}/gps-tests")
Review Comment:
changed it to kubernetes extension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1271: MINIFICPP-1763 - Move extension inclusion logic into the extensions
adamdebreceni commented on code in PR #1271:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1271#discussion_r849386040
##
libminifi/include/core/ConfigurableComponent.h:
##
@@ -56,6 +56,9 @@ class ConfigurableComponent {
ConfigurableComponent& operator=(const ConfigurableComponent ) =
delete;
ConfigurableComponent& operator=(ConfigurableComponent &) = delete;
+ template>>
+ std::optional getProperty(const Property& property) const;
Review Comment:
added commit message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849383995
##
extensions/procfs/tests/ProcFsMonitorTests.cpp:
##
@@ -0,0 +1,157 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "SingleProcessorTestController.h"
+#include "Catch.h"
+#include "processors/ProcFsMonitor.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFsMonitor;
+
+TEST_CASE("ProcFsMonitorTests", "[procfsmonitortests]") {
+ std::shared_ptr proc_fs_monitor =
std::make_shared("ProcFsMonitor");
+ org::apache::nifi::minifi::test::SingleProcessorTestController
test_controller_{proc_fs_monitor};
+
+ SECTION("Absolute JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("CPU"));
+CHECK(document["CPU"].HasMember("cpu"));
+CHECK(document.HasMember("Disk"));
+CHECK(document.HasMember("Network"));
+CHECK(document.HasMember("Process"));
+CHECK(document.HasMember("Memory"));
+ }
+
+ SECTION("Absolute OpenTelemetry") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "OpenTelemetry");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("Body"));
+REQUIRE(document["Body"].HasMember("CPU"));
+CHECK(document["Body"]["CPU"].HasMember("cpu"));
+CHECK(document["Body"].HasMember("Disk"));
+CHECK(document["Body"].HasMember("Network"));
+CHECK(document["Body"].HasMember("Process"));
+CHECK(document["Body"].HasMember("Memory"));
+ }
+
+ SECTION("Relative JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Relative");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+{
+ const auto& result = test_controller_.trigger();
+
+ REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+ const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+ rapidjson::Document document;
+ auto content = test_controller_.plan->getContent(result_flow_file);
+ document.Parse(content.c_str());
+ REQUIRE(document.IsObject());
+ // First trigger has not enough information for relative output
+ CHECK_FALSE(document.HasMember("CPU"));
+ CHECK_FALSE(document.HasMember("Disk"));
+ CHECK_FALSE(document.HasMember("Network"));
+ CHECK_FALSE(document.HasMember("Process"));
+ CHECK(document.HasMember("Memory"));
+}
+sleep(1);
Review Comment:
is this in seconds? could we lower the wait time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849382524
##
extensions/procfs/tests/ProcFsMonitorTests.cpp:
##
@@ -0,0 +1,157 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "SingleProcessorTestController.h"
+#include "Catch.h"
+#include "processors/ProcFsMonitor.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFsMonitor;
+
+TEST_CASE("ProcFsMonitorTests", "[procfsmonitortests]") {
+ std::shared_ptr proc_fs_monitor =
std::make_shared("ProcFsMonitor");
+ org::apache::nifi::minifi::test::SingleProcessorTestController
test_controller_{proc_fs_monitor};
+
+ SECTION("Absolute JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("CPU"));
+CHECK(document["CPU"].HasMember("cpu"));
+CHECK(document.HasMember("Disk"));
+CHECK(document.HasMember("Network"));
+CHECK(document.HasMember("Process"));
+CHECK(document.HasMember("Memory"));
+ }
+
+ SECTION("Absolute OpenTelemetry") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Absolute");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "OpenTelemetry");
+const auto& result = test_controller_.trigger();
+
+REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+rapidjson::Document document;
+auto content = test_controller_.plan->getContent(result_flow_file);
+document.Parse(content.c_str());
+REQUIRE(document.IsObject());
+REQUIRE(document.HasMember("Body"));
+REQUIRE(document["Body"].HasMember("CPU"));
+CHECK(document["Body"]["CPU"].HasMember("cpu"));
+CHECK(document["Body"].HasMember("Disk"));
+CHECK(document["Body"].HasMember("Network"));
+CHECK(document["Body"].HasMember("Process"));
+CHECK(document["Body"].HasMember("Memory"));
+ }
+
+ SECTION("Relative JSON") {
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::ResultRelativenessProperty.getName(), "Relative");
+test_controller_.plan->setProperty(proc_fs_monitor,
ProcFsMonitor::OutputFormatProperty.getName(), "JSON");
+{
+ const auto& result = test_controller_.trigger();
+
+ REQUIRE(result.at(ProcFsMonitor::Success).size() == 1);
+ const auto& result_flow_file = result.at(ProcFsMonitor::Success)[0];
+
+ rapidjson::Document document;
+ auto content = test_controller_.plan->getContent(result_flow_file);
+ document.Parse(content.c_str());
+ REQUIRE(document.IsObject());
+ // First trigger has not enough information for relative output
+ CHECK_FALSE(document.HasMember("CPU"));
+ CHECK_FALSE(document.HasMember("Disk"));
+ CHECK_FALSE(document.HasMember("Network"));
+ CHECK_FALSE(document.HasMember("Process"));
+ CHECK(document.HasMember("Memory"));
+}
+sleep(1);
Review Comment:
is this the linux `sleep` syscall? could we use
`std::this_thread::sleep_for` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1536) Deprecated ProcessMetrics.h and SystemMetrics.h
[ https://issues.apache.org/jira/browse/MINIFICPP-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink reassigned MINIFICPP-1536: -- Assignee: Martin Zink > Deprecated ProcessMetrics.h and SystemMetrics.h > --- > > Key: MINIFICPP-1536 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1536 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Trivial > Labels: MiNiFi-CPP-Hygiene > > There are couple unused ResponseNodes (i.e ProcessMetrics, SystemMetrics) > which contain more or less the same data as DeviceInfoNode and > AgentInformation. > e.g.: > systeminfo.systemInfo.vCores, systeminfo.systemInfo.physicalMem vs > deviceInfo.systemInfo.vCores, deviceInfo.systemInfo.physicalMem > The deprecated classes are not properly platform independent and provide > minimal added information (only involuntary context switches) > In my opinion we should remove these classes to avoid misusage. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849367062
##
extensions/procfs/tests/MemInfoTests.cpp:
##
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Catch.h"
+#include "ProcFs.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFs;
+using org::apache::nifi::minifi::extensions::procfs::MemInfo;
+
+TEST_CASE("ProcFSTest meminfo test with mock", "[procfmeminfomocktest]") {
+ ProcFs proc_fs("./mockprocfs_t0");
+ auto mem_info = proc_fs.getMemInfo();
+ REQUIRE(mem_info);
+ CHECK(mem_info->getTotalMemory() == 32895000LL*1024);
+ CHECK(mem_info->getFreeMemory() == 9870588LL*1024);
+ CHECK(mem_info->getAvailableMemory() == 22167288LL*1024);
+ CHECK(mem_info->getTotalSwap() == 9227464LL*1024);
+ CHECK(mem_info->getFreeSwap() == 9188320LL*1024);
Review Comment:
note: we have a `Literals.h` which has a `_KiB`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849365076
##
extensions/procfs/tests/DiskStatTests.cpp:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Catch.h"
+#include "ProcFs.h"
+#include "DiskStat.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFs;
+
+TEST_CASE("ProcFSTest DiskStat with mock", "[procfsdiskstatmocktest]") {
+ ProcFs proc_fs_t0("./mockprocfs_t0");
+ auto disk_stats_t0 = proc_fs_t0.getDiskStats();
+ REQUIRE(disk_stats_t0.size() == 17);
+
+ REQUIRE(disk_stats_t0[3].first == "nvme0n1p3");
+ CHECK(disk_stats_t0[3].second.getMajorDeviceNumber() == 259);
+ CHECK(disk_stats_t0[3].second.getMinorDeviceNumber() == 3);
+ CHECK(disk_stats_t0[3].second.getReadsCompleted() == 59);
+ CHECK(disk_stats_t0[3].second.getReadsMerged() == 1);
+ CHECK(disk_stats_t0[3].second.getSectorsRead() == 4526);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentReading() == 30);
+ CHECK(disk_stats_t0[3].second.getWritesCompleted() == 510);
+ CHECK(disk_stats_t0[3].second.getWritesMerged() == 3533);
+ CHECK(disk_stats_t0[3].second.getSectorsWritten() == 32344);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentWriting() == 15021);
+ CHECK(disk_stats_t0[3].second.getIosInProgress() == 0);
+ CHECK(disk_stats_t0[3].second.getMillisecondsSpentIo() == 6150);
+ CHECK(disk_stats_t0[3].second.getWeightedMillisecondsSpentIo() == 15052);
+
+
+ ProcFs proc_fs_t1("./mockprocfs_t1");
+ auto disk_stats_t1 = proc_fs_t1.getDiskStats();
+ REQUIRE(disk_stats_t1.size() == 17);
+
+ for (const auto& disk_stat_t0 : disk_stats_t0) {
+auto disk_stat_t1 = std::find_if(disk_stats_t1.begin(),
disk_stats_t1.end(), [_stat_t0](auto& it){return it.first ==
disk_stat_t0.first;});
Review Comment:
note: I think `it` here is a reference to the item not an iterator (I try to
reserve `it` to name an iterator, others might not have this convention)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849361225
##
extensions/procfs/tests/CPUStatTests.cpp:
##
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "Catch.h"
+#include "ProcFs.h"
+
+#include "rapidjson/stream.h"
+
+using org::apache::nifi::minifi::extensions::procfs::ProcFs;
+using org::apache::nifi::minifi::extensions::procfs::CpuStatData;
+using org::apache::nifi::minifi::extensions::procfs::SystemClockDuration;
+
+void cpu_stat_period_total_should_be_one(const CpuStatData& cpu_stat) {
+ double percentage = 0;
+ percentage += cpu_stat.getUser()/cpu_stat.getTotal();
+ percentage += cpu_stat.getNice()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSystem()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIdle()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIoWait()/cpu_stat.getTotal();
+ percentage += cpu_stat.getIrq()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSoftIrq()/cpu_stat.getTotal();
+ percentage += cpu_stat.getSteal()/cpu_stat.getTotal();
+ percentage += cpu_stat.getGuest()/cpu_stat.getTotal();
+ percentage += cpu_stat.getGuestNice()/cpu_stat.getTotal();
+ REQUIRE(percentage == Approx(1.0));
+}
+
+TEST_CASE("ProcFSTest stat test with mock", "[procfsstatmockabsolutetest]") {
+ ProcFs proc_fs_t0("./mockprocfs_t0");
+ auto cpu_stats_t0 = proc_fs_t0.getCpuStats();
+ REQUIRE(cpu_stats_t0.size() == 13);
+ for (const auto& [cpu_name, cpu_stat] : cpu_stats_t0) {
+cpu_stat_period_total_should_be_one(cpu_stat);
+ }
+
+ REQUIRE(cpu_stats_t0[4].first == "cpu3");
+ CHECK(cpu_stats_t0[4].second.getUser() == SystemClockDuration(1299551));
+ CHECK(cpu_stats_t0[4].second.getNice() == SystemClockDuration(732));
+ CHECK(cpu_stats_t0[4].second.getSystem() == SystemClockDuration(316295));
+ CHECK(cpu_stats_t0[4].second.getIdle() == SystemClockDuration(4498383));
+ CHECK(cpu_stats_t0[4].second.getIoWait() == SystemClockDuration(1428));
+ CHECK(cpu_stats_t0[4].second.getIrq() == SystemClockDuration(22491));
+ CHECK(cpu_stats_t0[4].second.getSoftIrq() == SystemClockDuration(7022));
+ CHECK(cpu_stats_t0[4].second.getSteal() == SystemClockDuration(0));
+ CHECK(cpu_stats_t0[4].second.getGuest() == SystemClockDuration(0));
+ CHECK(cpu_stats_t0[4].second.getGuestNice() == SystemClockDuration(0));
+
+ ProcFs proc_fs_t1("./mockprocfs_t1");
+ auto cpu_stats_t1 = proc_fs_t1.getCpuStats();
+ REQUIRE(cpu_stats_t1.size() == 13);
+ for (const auto& [cpu_name, cpu_stat] : cpu_stats_t1) {
+cpu_stat_period_total_should_be_one(cpu_stat);
+ }
+
+ for (size_t i = 0; i < cpu_stats_t1.size(); ++i) {
+const auto& cpu_name_t0 = cpu_stats_t0[i].first;
+const auto& cpu_stat_t0 = cpu_stats_t0[i].second;
+const auto& cpu_name_t1 = cpu_stats_t1[i].first;
+const auto& cpu_stat_t1 = cpu_stats_t1[i].second;
Review Comment:
we could use structured binding here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849356259
##
extensions/procfs/tests/CMakeLists.txt:
##
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+file(GLOB PROCFS_TESTS "*.cpp")
+
+SET(PROCFS_TEST_COUNT 0)
+FOREACH(testfile ${PROCFS_TESTS})
+get_filename_component(testfilename "${testfile}" NAME_WE)
+add_executable("${testfilename}" "${testfile}")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/procfs")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/libminifi/test/")
+target_include_directories(${testfilename} PRIVATE BEFORE
"${CMAKE_SOURCE_DIR}/extensions/standard-processors")
+createTests("${testfilename}")
+target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
+target_link_libraries(${testfilename} minifi-procfs)
+target_link_libraries(${testfilename} minifi-standard-processors)
+MATH(EXPR PROCFS_TEST_COUNT "${PROCFS_TEST_COUNT}+1")
+add_test(NAME "${testfilename}" COMMAND "${testfilename}")
+ENDFOREACH()
+file(COPY "${CMAKE_SOURCE_DIR}/extensions/procfs/tests/mockprocfs_t0"
DESTINATION "${CMAKE_BINARY_DIR}/extensions/procfs/tests/")
+file(COPY "${CMAKE_SOURCE_DIR}/extensions/procfs/tests/mockprocfs_t1"
DESTINATION "${CMAKE_BINARY_DIR}/extensions/procfs/tests/")
Review Comment:
are these test resources? sftp tests and script tests copy such resources to
somewhere under `${CMAKE_BINARY_DIR}/bin/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (MINIFICPP-1796) Log properties are not retrieved properly with getString from Configure
Gábor Gyimesi created MINIFICPP-1796: Summary: Log properties are not retrieved properly with getString from Configure Key: MINIFICPP-1796 URL: https://issues.apache.org/jira/browse/MINIFICPP-1796 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Gábor Gyimesi getString should retrieve properties from the Configure object in raw form, without decrypting them. Because getString is implemented in Properties and the Configure object contains and additional Properties object for the log properties, the getString call made through a Configure object is not forwarded properly to the log Properties object. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1152: MINIFICPP-1593 ProcFsMonitor to monitor /proc pseduo filesystem
adamdebreceni commented on code in PR #1152:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1152#discussion_r849306778
##
extensions/procfs/processors/ProcFsMonitor.h:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "../ProcFs.h"
+#include "core/Processor.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/logging/Logger.h"
+#include "utils/Enum.h"
+
+#include "rapidjson/stream.h"
+#include "rapidjson/document.h"
+
+namespace org::apache::nifi::minifi::extensions::procfs {
+
+class ProcFsMonitor : public core::Processor {
+ public:
+ explicit ProcFsMonitor(const std::string& name, utils::Identifier uuid =
utils::Identifier())
+ : Processor(name, uuid) {
+ }
+ ProcFsMonitor(const ProcFsMonitor&) = delete;
+ ProcFsMonitor(ProcFsMonitor&&) = delete;
+ ProcFsMonitor& operator=(const ProcFsMonitor&) = delete;
+ ProcFsMonitor& operator=(ProcFsMonitor&&) = delete;
+ ~ProcFsMonitor() override = default;
Review Comment:
I think the base class CoreComponent has these members deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1296: MINIFICPP-1794 Remove sensitive properties from agent manifest
lordgamez commented on code in PR #1296:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1296#discussion_r849251638
##
libminifi/src/core/state/nodes/SupportedOperations.cpp:
##
@@ -64,9 +65,12 @@ void
SupportedOperations::addProperty(SerializedResponseNode& properties, const
SupportedOperations::Metadata
SupportedOperations::buildUpdatePropertiesMetadata() const {
std::vector>
supported_config_updates;
- for (const auto& config_property :
minifi::Configuration::CONFIGURATION_PROPERTIES) {
-if (!update_policy_controller_ ||
-(update_policy_controller_ &&
update_policy_controller_->canUpdate(std::string(config_property.name {
+ for (const auto& config_property : Configuration::CONFIGURATION_PROPERTIES) {
+auto sensitive_properties =
Configuration::getSensitiveProperties(configuration_reader_);
+if (ranges::find(sensitive_properties, config_property.name) !=
ranges::end(sensitive_properties)) {
+ continue;
+}
+if (!update_policy_controller_ || (update_policy_controller_ &&
update_policy_controller_->canUpdate(std::string(config_property.name {
Review Comment:
Good point, updated in 8561e67e7453596a5158f2ab937fc724ec551974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1300: MINIFICPP-1675 Use regex.h instead of std::regex when using libstdc++
lordgamez opened a new pull request, #1300: URL: https://github.com/apache/nifi-minifi-cpp/pull/1300 Due to a bug in the libstdc++ implementation of std::regex it is unsafe to use std::regex_match and std::regex_search on large texts as the process can crash due to a stack overflow. In this PR std::regex usages are replaced with minifi::utils::Regex which uses regex.h if libstdc++ is used otherwise it uses std::regex. There are a few exceptions: - std::regex is still used for regex_replace calls - AppendHostInfo still uses std::regex as POSIX regex grammar is not sufficient for easily filtering interface names, and there is no risk of running into the bug as the interface names are small - In tests and mocks we still use std::regex as they are only run in our test environments with controlled inputs and sometimes use regex patterns not available in POSIX grammar - Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1296: MINIFICPP-1794 Remove sensitive properties from agent manifest
martinzink commented on code in PR #1296:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1296#discussion_r849197218
##
libminifi/src/core/state/nodes/SupportedOperations.cpp:
##
@@ -64,9 +65,12 @@ void
SupportedOperations::addProperty(SerializedResponseNode& properties, const
SupportedOperations::Metadata
SupportedOperations::buildUpdatePropertiesMetadata() const {
std::vector>
supported_config_updates;
- for (const auto& config_property :
minifi::Configuration::CONFIGURATION_PROPERTIES) {
-if (!update_policy_controller_ ||
-(update_policy_controller_ &&
update_policy_controller_->canUpdate(std::string(config_property.name {
+ for (const auto& config_property : Configuration::CONFIGURATION_PROPERTIES) {
+auto sensitive_properties =
Configuration::getSensitiveProperties(configuration_reader_);
+if (ranges::find(sensitive_properties, config_property.name) !=
ranges::end(sensitive_properties)) {
+ continue;
+}
+if (!update_policy_controller_ || (update_policy_controller_ &&
update_policy_controller_->canUpdate(std::string(config_property.name {
Review Comment:
```suggestion
if (!update_policy_controller_ ||
update_policy_controller_->canUpdate(std::string(config_property.name))) {
```
The second checking of update_policy_controller_ seems unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
