[jira] [Created] (NIFI-7610) Edit README.md for dockerhub to reflect 1.11.4 image
Chad Zobrisky created NIFI-7610: --- Summary: Edit README.md for dockerhub to reflect 1.11.4 image Key: NIFI-7610 URL: https://issues.apache.org/jira/browse/NIFI-7610 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.11.4 Reporter: Chad Zobrisky Fix For: 1.12.0 The readme.md for dockerhub still refers to 1.8.0. This Jira ticket is to reflect the latest version in main, 1.12.0 as the latest dockerhub image. Ref: [https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md] dockerhub: [https://hub.docker.com/r/apache/nifi/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7609) Edit README.md for dockerhub to reflect 1.11.4 image
[ https://issues.apache.org/jira/browse/NIFI-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chad Zobrisky resolved NIFI-7609. - Resolution: Invalid > Edit README.md for dockerhub to reflect 1.11.4 image > > > Key: NIFI-7609 > URL: https://issues.apache.org/jira/browse/NIFI-7609 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Affects Versions: 1.11.4 >Reporter: Chad Zobrisky >Assignee: Harleen Singh Mann >Priority: Minor > > The readme.md stil refers to 1.8.0. This Jira ticket is to reflect 1.9.0 as > the latest dockerhub image > > ref: > https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7609) Edit README.md for dockerhub to reflect 1.11.4 image
[ https://issues.apache.org/jira/browse/NIFI-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chad Zobrisky updated NIFI-7609: Affects Version/s: (was: 1.9.0) 1.11.4 > Edit README.md for dockerhub to reflect 1.11.4 image > > > Key: NIFI-7609 > URL: https://issues.apache.org/jira/browse/NIFI-7609 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Affects Versions: 1.11.4 >Reporter: Chad Zobrisky >Assignee: Harleen Singh Mann >Priority: Minor > > The readme.md stil refers to 1.8.0. This Jira ticket is to reflect 1.9.0 as > the latest dockerhub image > > ref: > https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7609) Edit README.md for dockerhub to reflect 1.11.4 image
Chad Zobrisky created NIFI-7609: --- Summary: Edit README.md for dockerhub to reflect 1.11.4 image Key: NIFI-7609 URL: https://issues.apache.org/jira/browse/NIFI-7609 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.9.0 Reporter: Chad Zobrisky Assignee: Harleen Singh Mann The readme.md stil refers to 1.8.0. This Jira ticket is to reflect 1.9.0 as the latest dockerhub image ref: https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/README.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] MikeThomsen commented on pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on pull request #4374: URL: https://github.com/apache/nifi/pull/4374#issuecomment-655166954 @markap14 I'll make some time tonight or tomorrow to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4309: NIFI-7497 Adding support for AWS Credentials Assume Role to be able t…
MikeThomsen commented on a change in pull request #4309:
URL: https://github.com/apache/nifi/pull/4309#discussion_r451174955
##
File path:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/credentials/provider/factory/CredentialPropertyDescriptors.java
##
@@ -177,6 +177,16 @@
.required(false)
.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
.sensitive(false)
-.description("Proxy pot for cross-account access, if needed within
your environment. This will configure a proxy to request for temporary access
keys into another AWS account")
+.description("Proxy port for cross-account access, if needed
within your environment. This will configure a proxy to request for temporary
access keys into another AWS account")
+.build();
+
+public static final PropertyDescriptor ASSUME_ROLE_STS_ENDPOINT = new
PropertyDescriptor.Builder()
+.name("assume-role-sts-endpoint")
+.displayName("Assume Role STS Endpoint")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(false)
+.description("STS Endpoint override for cross-account access, if
needed within your environment.")
Review comment:
@pvillard31 If that change is the only thing blocking this, I can make
it and merge. Let me know if you're +1 on that. We have a use case that really
needs this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7608) Create CLI command for replacing process group content
[ https://issues.apache.org/jira/browse/NIFI-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-7608: -- Status: Patch Available (was: Open) > Create CLI command for replacing process group content > -- > > Key: NIFI-7608 > URL: https://issues.apache.org/jira/browse/NIFI-7608 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > Create a CLI command that uses the /replace-requests endpoint in the > ProcessGroupResource. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] bbende opened a new pull request #4392: NIFI-7608 Add CLI command to replace contents of a process group
bbende opened a new pull request #4392:
URL: https://github.com/apache/nifi/pull/4392
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-7608) Create CLI command for replacing process group content
Bryan Bende created NIFI-7608: - Summary: Create CLI command for replacing process group content Key: NIFI-7608 URL: https://issues.apache.org/jira/browse/NIFI-7608 Project: Apache NiFi Issue Type: Improvement Reporter: Bryan Bende Assignee: Bryan Bende Create a CLI command that uses the /replace-requests endpoint in the ProcessGroupResource. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
[ https://issues.apache.org/jira/browse/NIFI-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-7607: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Controller Service can show that it is referenced by Processors that no > longer reference it > --- > > Key: NIFI-7607 > URL: https://issues.apache.org/jira/browse/NIFI-7607 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When looking at the configuration for a Controller Service, it can sometimes > show that a Processor (or another Controller Service or Reporting Task) > references it even when the Processor/component no longer does. In fact, the > component may not even exist anymore. When this happens, if the Controller > Service is already enabled, it cannot be disabled, and it cannot be enabled > with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] bbende merged pull request #4391: NIFI-7607: Fixed bug that caused the wrong Controller Service to be d…
bbende merged pull request #4391: URL: https://github.com/apache/nifi/pull/4391 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
[ https://issues.apache.org/jira/browse/NIFI-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153081#comment-17153081 ] ASF subversion and git services commented on NIFI-7607: --- Commit e17db80514870077ac379186b73762833dcacff1 in nifi's branch refs/heads/main from markap14 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e17db80 ] NIFI-7607: Fixed bug that caused the wrong Controller Service to be de-referenced when a component that used to reference a Controller Service is changed to reference a different Controller Service (#4391) > Controller Service can show that it is referenced by Processors that no > longer reference it > --- > > Key: NIFI-7607 > URL: https://issues.apache.org/jira/browse/NIFI-7607 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.12.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When looking at the configuration for a Controller Service, it can sometimes > show that a Processor (or another Controller Service or Reporting Task) > references it even when the Processor/component no longer does. In fact, the > component may not even exist anymore. When this happens, if the Controller > Service is already enabled, it cannot be disabled, and it cannot be enabled > with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] bbende commented on pull request #4391: NIFI-7607: Fixed bug that caused the wrong Controller Service to be d…
bbende commented on pull request #4391: URL: https://github.com/apache/nifi/pull/4391#issuecomment-655119444 Looks good, merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4384: NIFI-2702 Support named captures in ExtractText
ottobackwards commented on pull request #4384: URL: https://github.com/apache/nifi/pull/4384#issuecomment-655106630 updated the base to `main` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153069#comment-17153069 ] Otto Fowler commented on NIFI-2072: --- I'm more inclined to do the validation, since it think handling mixed, and scoped ( nested ) etc goes downhill real fast since java doesn't support it. Would would be nice is when I called group(string) i could also call getGroupIndex(string) so that I could mix and match, but you can't. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] kevdoran commented on pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
kevdoran commented on pull request #4344: URL: https://github.com/apache/nifi/pull/4344#issuecomment-655097808 Thanks for the contribution @mtien-apache, and for the reviews @thenatog, @alopresto. I cherry-picked this to `main`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] achristianson opened a new pull request #834: MINIFICPP-1282 Use GNUInstallDirs for more reliable determination of installed lib/lib64 directories
achristianson opened a new pull request #834: URL: https://github.com/apache/nifi-minifi-cpp/pull/834 Use `GNUInstallDirs` for more reliable determination of installed lib/lib64 directories. This fixes a bug :beetle: I ran into when attempting to build on Arch Linux. According to https://cmake.org/cmake/help/v3.4/module/GNUInstallDirs.html `GNUInstallDirs` `CMAKE_INSTALL_LIBDIR` will give the correct dir: "object code libraries (lib or lib64 or lib/ on Debian)" Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] kevdoran closed pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
kevdoran closed pull request #4344: URL: https://github.com/apache/nifi/pull/4344 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153064#comment-17153064
]
ASF subversion and git services commented on NIFI-7332:
---
Commit 43fb57e7bb4b3a3a7714578816ab57ece63349c8 in nifi's branch
refs/heads/main from mtien
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=43fb57e ]
NIFI-7332 Added method to log available claim names from the ID provider
response when the OIDC Identifying User claim is not found. Revised log message
to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output
to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent
initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve
the NiFi JWT.
Signed-off-by: Nathan Gough
This closes #4344.
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Fix For: 1.12.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153065#comment-17153065
]
ASF subversion and git services commented on NIFI-7332:
---
Commit 43fb57e7bb4b3a3a7714578816ab57ece63349c8 in nifi's branch
refs/heads/main from mtien
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=43fb57e ]
NIFI-7332 Added method to log available claim names from the ID provider
response when the OIDC Identifying User claim is not found. Revised log message
to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output
to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent
initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve
the NiFi JWT.
Signed-off-by: Nathan Gough
This closes #4344.
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Fix For: 1.12.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (MINIFICPP-1282) BUILD_BYPRODUCTS incorrectly set to lib64
Andrew Christianson created MINIFICPP-1282:
--
Summary: BUILD_BYPRODUCTS incorrectly set to lib64
Key: MINIFICPP-1282
URL: https://issues.apache.org/jira/browse/MINIFICPP-1282
Project: Apache NiFi MiNiFi C++
Issue Type: Bug
Reporter: Andrew Christianson
Assignee: Andrew Christianson
On updated arch linux as of 2020-07-07, getting the following build error:
{code:java}
make[2]: *** No rule to make target
'thirdparty/rocksdb-install/lib64/librocksdb.a', needed by 'main/minifi'. Stop.
{code}
Looking at the directory, the library was built, but its in {{lib/}}:
{noformat}
$ ls thirdparty/rocksdb-install/lib64/
ls: cannot access 'thirdparty/rocksdb-install/lib64/': No such file or directory
$ ls thirdparty/rocksdb-install/lib/
cmake librocksdb.a
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough updated NIFI-7332:
---
Fix Version/s: 1.12.0
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Fix For: 1.12.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough resolved NIFI-7332.
Resolution: Fixed
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153050#comment-17153050
]
ASF subversion and git services commented on NIFI-7332:
---
Commit aa741cc5967f62c3c38c2a47e712b7faa6fe19ff in nifi's branch
refs/heads/master from mtien
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=aa741cc ]
NIFI-7332 Added method to log available claim names from the ID provider
response when the OIDC Identifying User claim is not found. Revised log message
to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output
to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent
initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve
the NiFi JWT.
Signed-off-by: Nathan Gough
This closes #4344.
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7332) Improve communication to user when OIDC response does not contain usable claims
[
https://issues.apache.org/jira/browse/NIFI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153051#comment-17153051
]
ASF subversion and git services commented on NIFI-7332:
---
Commit aa741cc5967f62c3c38c2a47e712b7faa6fe19ff in nifi's branch
refs/heads/master from mtien
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=aa741cc ]
NIFI-7332 Added method to log available claim names from the ID provider
response when the OIDC Identifying User claim is not found. Revised log message
to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output
to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent
initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve
the NiFi JWT.
Signed-off-by: Nathan Gough
This closes #4344.
> Improve communication to user when OIDC response does not contain usable
> claims
> ---
>
> Key: NIFI-7332
> URL: https://issues.apache.org/jira/browse/NIFI-7332
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: M Tien
>Priority: Major
> Labels: oidc, security
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The messaging displayed to the user/admin does not clearly indicate the
> problem if the OIDC response does not contain a claim that NiFi is configured
> to use (i.e. NiFi expects an {{email}} claim but the user does not have an
> email configured on the OIDC IdP).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] thenatog commented on pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
thenatog commented on pull request #4344: URL: https://github.com/apache/nifi/pull/4344#issuecomment-655075490 Also, thank you for these changes, it should help with some work I am doing in NiFi Registry. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4377: NIFI-7568 - Fixing Kerberos mappings
thenatog commented on pull request #4377: URL: https://github.com/apache/nifi/pull/4377#issuecomment-655074591 I will verify if we want to apply these changes for OIDC. I am reviewing and merging mtien-apache's changes and will rebase this change on top for a start. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog edited a comment on pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
thenatog edited a comment on pull request #4344: URL: https://github.com/apache/nifi/pull/4344#issuecomment-655070816 Tested with G Suite OIDC provider, and ran a contrib check. All checks passing. +1. Will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
thenatog commented on pull request #4344: URL: https://github.com/apache/nifi/pull/4344#issuecomment-655070816 Tested with G Suite OIDC provider, and ran a contrib check. All checks passing. +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema
[ https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-5213: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Allow AvroReader with explicit schema to read files with embedded schema > > > Key: NIFI-5213 > URL: https://issues.apache.org/jira/browse/NIFI-5213 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > AvroReader allows the choice of schema access strategy from such options as > Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming > Avro files will have embedded schemas, then Use Embedded Schema is best > practice for the Avro Reader. However it is not intuitive that if the same > schema that is embedded in the file is specified by name (using a schema > registry) or explicitly via Schema Text, that errors can occur. This has been > noticed in QueryRecord for example, and the error is also not intuitive or > descriptive (it is often an ArrayIndexOutOfBoundsException). > To provide a better user experience, it would be an improvement for > AvroReader to be able to successfully process Avro files with embedded > schemas, even when the Schema Access Strategy is not "Use Embedded Schema". > Of course, the explicit schema would have to match the embedded schema, or an > error would be reported (and rightfully so). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema
[ https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-5213: --- Fix Version/s: 1.12.0 > Allow AvroReader with explicit schema to read files with embedded schema > > > Key: NIFI-5213 > URL: https://issues.apache.org/jira/browse/NIFI-5213 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > AvroReader allows the choice of schema access strategy from such options as > Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming > Avro files will have embedded schemas, then Use Embedded Schema is best > practice for the Avro Reader. However it is not intuitive that if the same > schema that is embedded in the file is specified by name (using a schema > registry) or explicitly via Schema Text, that errors can occur. This has been > noticed in QueryRecord for example, and the error is also not intuitive or > descriptive (it is often an ArrayIndexOutOfBoundsException). > To provide a better user experience, it would be an improvement for > AvroReader to be able to successfully process Avro files with embedded > schemas, even when the Schema Access Strategy is not "Use Embedded Schema". > Of course, the explicit schema would have to match the embedded schema, or an > error would be reported (and rightfully so). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema
[ https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152975#comment-17152975 ] ASF subversion and git services commented on NIFI-5213: --- Commit f040c6aadbedb0fb5e778e7e13069d66cc97af45 in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f040c6a ] NIFI-5213: Allow AvroReader to process files w embedded schema even when the access strategy is explicit schema NIFI-5213: Incorporated review comments This closes #2718 Signed-off-by: Mike Thomsen > Allow AvroReader with explicit schema to read files with embedded schema > > > Key: NIFI-5213 > URL: https://issues.apache.org/jira/browse/NIFI-5213 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > AvroReader allows the choice of schema access strategy from such options as > Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming > Avro files will have embedded schemas, then Use Embedded Schema is best > practice for the Avro Reader. However it is not intuitive that if the same > schema that is embedded in the file is specified by name (using a schema > registry) or explicitly via Schema Text, that errors can occur. This has been > noticed in QueryRecord for example, and the error is also not intuitive or > descriptive (it is often an ArrayIndexOutOfBoundsException). > To provide a better user experience, it would be an improvement for > AvroReader to be able to successfully process Avro files with embedded > schemas, even when the Schema Access Strategy is not "Use Embedded Schema". > Of course, the explicit schema would have to match the embedded schema, or an > error would be reported (and rightfully so). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #2718: NIFI-5213: Allow AvroReader to process files w embedded schema even when the access strategy is explicit schema
asfgit closed pull request #2718: URL: https://github.com/apache/nifi/pull/2718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema
[ https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152976#comment-17152976 ] ASF subversion and git services commented on NIFI-5213: --- Commit f040c6aadbedb0fb5e778e7e13069d66cc97af45 in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f040c6a ] NIFI-5213: Allow AvroReader to process files w embedded schema even when the access strategy is explicit schema NIFI-5213: Incorporated review comments This closes #2718 Signed-off-by: Mike Thomsen > Allow AvroReader with explicit schema to read files with embedded schema > > > Key: NIFI-5213 > URL: https://issues.apache.org/jira/browse/NIFI-5213 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > AvroReader allows the choice of schema access strategy from such options as > Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming > Avro files will have embedded schemas, then Use Embedded Schema is best > practice for the Avro Reader. However it is not intuitive that if the same > schema that is embedded in the file is specified by name (using a schema > registry) or explicitly via Schema Text, that errors can occur. This has been > noticed in QueryRecord for example, and the error is also not intuitive or > descriptive (it is often an ArrayIndexOutOfBoundsException). > To provide a better user experience, it would be an improvement for > AvroReader to be able to successfully process Avro files with embedded > schemas, even when the Schema Access Strategy is not "Use Embedded Schema". > Of course, the explicit schema would have to match the embedded schema, or an > error would be reported (and rightfully so). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] tpalfy closed pull request #4390: NIFI-6934 In PutDatabaseRecord added Postgres UPSERT support
tpalfy closed pull request #4390: URL: https://github.com/apache/nifi/pull/4390 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
[ https://issues.apache.org/jira/browse/NIFI-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-7607: - Status: Patch Available (was: Open) > Controller Service can show that it is referenced by Processors that no > longer reference it > --- > > Key: NIFI-7607 > URL: https://issues.apache.org/jira/browse/NIFI-7607 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4, 1.11.3, 1.11.2, 1.11.1, 1.11.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When looking at the configuration for a Controller Service, it can sometimes > show that a Processor (or another Controller Service or Reporting Task) > references it even when the Processor/component no longer does. In fact, the > component may not even exist anymore. When this happens, if the Controller > Service is already enabled, it cannot be disabled, and it cannot be enabled > with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7571) Deleted processors in the Controller Service
[ https://issues.apache.org/jira/browse/NIFI-7571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-7571. -- Resolution: Duplicate > Deleted processors in the Controller Service > > > Key: NIFI-7571 > URL: https://issues.apache.org/jira/browse/NIFI-7571 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 >Reporter: Martin >Priority: Major > > I am not able to enable one of my controller services. If I want to check > inactive processors I get this. So far so good. > !https://files.slack.com/files-pri/T0L9SDNRZ-F015K0ZQB2N/image.png! > Now I click on these processors: > !https://files.slack.com/files-pri/T0L9SDNRZ-F015C8H6UFQ/image.png! > I then tried to find the processors with the search functionality in the UI > using the ID, no results. > !https://files.slack.com/files-pri/T0L9SDNRZ-F015DMYLZML/image.png! > I also checked the processors in the list: > !https://files.slack.com/files-pri/T0L9SDNRZ-F0155MBU079/image.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 opened a new pull request #4391: NIFI-7607: Fixed bug that caused the wrong Controller Service to be d…
markap14 opened a new pull request #4391:
URL: https://github.com/apache/nifi/pull/4391
…e-referenced when a component that used to reference a Controller Service
is changed to reference a different Controller Service
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
[ https://issues.apache.org/jira/browse/NIFI-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152959#comment-17152959 ] Mark Payne commented on NIFI-7607: -- To replicate: 1. Create a ConvertRecord processor. 2. For the Record Reader, choose to create a new Avro Record Reader named "AvroRecordReader 1" 3. Apply changes for the Processor. 4. Configure ConvertRecord processor again. 5. For the Record reader, change the value to create another new Avro Record Reader named "AvroRecordReader 2" 6. Apply changes for the Processor. 7. Configure the Process Group. 8. In the Controller Services tab, note that configuring AvroRecordReader 1 shows that ConvertRecord is still referencing, even though it is not. AvroRecordReader 2 will also show that ConvertRecord is referencing it (which is correct). > Controller Service can show that it is referenced by Processors that no > longer reference it > --- > > Key: NIFI-7607 > URL: https://issues.apache.org/jira/browse/NIFI-7607 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.12.0 > > > When looking at the configuration for a Controller Service, it can sometimes > show that a Processor (or another Controller Service or Reporting Task) > references it even when the Processor/component no longer does. In fact, the > component may not even exist anymore. When this happens, if the Controller > Service is already enabled, it cannot be disabled, and it cannot be enabled > with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] tpalfy opened a new pull request #4390: NIFI-6934 In PutDatabaseRecord added Postgres UPSERT support
tpalfy opened a new pull request #4390: URL: https://github.com/apache/nifi/pull/4390 https://issues.apache.org/jira/browse/NIFI-6934 Reopened https://github.com/apache/nifi/pull/4350 against `main` Implemented using `DatabaseAdapter` ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
[ https://issues.apache.org/jira/browse/NIFI-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-7607: - Affects Version/s: 1.11.0 1.11.1 1.11.2 1.11.3 1.11.4 > Controller Service can show that it is referenced by Processors that no > longer reference it > --- > > Key: NIFI-7607 > URL: https://issues.apache.org/jira/browse/NIFI-7607 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.12.0 > > > When looking at the configuration for a Controller Service, it can sometimes > show that a Processor (or another Controller Service or Reporting Task) > references it even when the Processor/component no longer does. In fact, the > component may not even exist anymore. When this happens, if the Controller > Service is already enabled, it cannot be disabled, and it cannot be enabled > with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7607) Controller Service can show that it is referenced by Processors that no longer reference it
Mark Payne created NIFI-7607: Summary: Controller Service can show that it is referenced by Processors that no longer reference it Key: NIFI-7607 URL: https://issues.apache.org/jira/browse/NIFI-7607 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.12.0 When looking at the configuration for a Controller Service, it can sometimes show that a Processor (or another Controller Service or Reporting Task) references it even when the Processor/component no longer does. In fact, the component may not even exist anymore. When this happens, if the Controller Service is already enabled, it cannot be disabled, and it cannot be enabled with its references. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog commented on pull request #4344: NIFI-7332 Added method to log available claim names from the ID provi…
thenatog commented on pull request #4344: URL: https://github.com/apache/nifi/pull/4344#issuecomment-655023627 Reviewing.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
ottobackwards commented on pull request #4376: URL: https://github.com/apache/nifi/pull/4376#issuecomment-655007142 OK, if there are no classes that are removed from what a 3rd party would see as the default set of classes, then there is 0 risk. Otherwise, I'm not sure you can assume that no 3rd party would break, since you can't assume the nar dependency. There might be other places in the documentation that talk about nar dependencies and mention jetty, they should all be updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152904#comment-17152904 ] Malthe Borch edited comment on NIFI-2072 at 7/7/20, 5:14 PM: - I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. I often enough see named captures mixed with unnamed ones, simply because the author has not bothered to use a non-capturing group. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). It would then be an error to use a regex that has unnamed capture groups. was (Author: malthe): I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152904#comment-17152904 ] Malthe Borch commented on NIFI-2072: I would be happy then with "Enable named group support". In terms of what happens if an unnamed capture group is used, I think it would be better to either: - Allow it. - Implement a validation step that scans the expression for unnamed capture groups (i.e. those that are not named and not non-capturing). > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 commented on pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
mattyb149 commented on pull request #4376: URL: https://github.com/apache/nifi/pull/4376#issuecomment-654995453 I'm still looking at various NARs to see how/if they're affected. Many NARs end up using nifi-standard-services-api-nar as a parent, which itself has the nifi-jetty-bundle as a parent, so there should be no change there. We're not removing Jetty, just decoupling NiFi framework stuff that also uses Jetty, so we can exclude things like the NiFi UI/REST interface but still support processors that use Jetty. Good catch on the doc, I didn't know that was in there :) Will update the doc as appropriate, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
ottobackwards commented on pull request #4376: URL: https://github.com/apache/nifi/pull/4376#issuecomment-654989630 "The @RequiresInstanceClassLoading annotation also provides an optional flag `cloneAncestorResources'. If set to true, the instance ClassLoader will include ancestor resources up to the first ClassLoader containing a controller service API referenced by the component, or up to the Jetty NAR. If set to false, or not specified, only the resources from the component’s NAR will be included." I think you need to update the documentation then. I understand about taking jetty out, in my fork of the bundle "system" I did the same ( since it isn't in nifi) . Saying that, I'm not super familiar with what is actually _in_ that nar. So there is no chance that this will break existing 3rd party nars which may have picked something up from the jetty NAR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
mattyb149 commented on pull request #4376: URL: https://github.com/apache/nifi/pull/4376#issuecomment-654986964 The PR removes nifi-jetty (which has the JettyServer class, an implementation of the NiFiServer interface) as the parent classloader for all NARs that don't have one. Some NARs still need Jetty itself (some HTTP processors, e.g.) so our JettyServer class and its dependencies are now in a nifi-server NAR, which has the "nifi-jetty-bundle" as a parent. The naming gets confusing here because NiFi code has a JettyServer class, but the nifi-jetty-bundle just has pure Jetty dependencies for those NARs that need them. For example all NARs with custom UIs (the Advanced button on the component dialog), components that need to start a server (ListenHttp, e.g.) have nifi-jetty-bundle as the top-level parent. This PR also includes refactoring the way the NiFiServer implementation is instantiated. It used to look for a hardcoded class name (for JettyServer) and use reflection to instantiate it. This PR removes that code and replaces it with a ServiceLoader that looks in the NARs for an implementation of NiFiServer. For the normal NiFi assembly, the nifi-server NAR (with JettyServer) will be there so it will get discovered as expected. But for other assemblies (such as MiNiFi Java), the nifi-server NAR would be replaced with a different implementation, and this PR offers one called MiNiFiServer in a minifi-server NAR. If you replace the nifi-server NAR with minifi-server, the UI/REST components are no longer available, and the flow controller and all components are created immediately and the flow is loaded/initialized. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-2072) Support named captures in ExtractText
[ https://issues.apache.org/jira/browse/NIFI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152886#comment-17152886 ] Pierre Villard commented on NIFI-2072: -- Sorry, I don't really have the time to look into it as much as I'd like right now. The only rule is that we can't make any breaking change: meaning that any existing flow should keep working the exact same way after an upgrade. That's why we usually provide a property to explicitly allow users to enable this new behavior. > Support named captures in ExtractText > - > > Key: NIFI-2072 > URL: https://issues.apache.org/jira/browse/NIFI-2072 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Otto Fowler >Priority: Major > Labels: extracttext > > ExtractText currently captures and creates attributes using numeric indices > (e.g, attribute.name.0, attribute.name.1, etc.) whether or not the capture > groups are named, i.e., patterns like (?\w+). > In addition to being more faithful to the provided regexes, named captures > could help simplify data flows because you wouldn't have to add superfluous > UpdateAttribute steps which are just renaming the indexed captures to more > interpretable names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450993400
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[jira] [Commented] (NIFI-7579) Create a GetS3Object Processor
[ https://issues.apache.org/jira/browse/NIFI-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152860#comment-17152860 ] Pierre Villard commented on NIFI-7579: -- I think the comment here is that GenerateFlowFile (where you can set flow files attributes as dynamic properties) -> FetchS3 would provide the exact feature you're looking for. Another option would be to make optional the incoming connection on FetchS3, but not sure this would provide a good user experience. > Create a GetS3Object Processor > -- > > Key: NIFI-7579 > URL: https://issues.apache.org/jira/browse/NIFI-7579 > Project: Apache NiFi > Issue Type: New Feature >Reporter: ArpStorm1 >Assignee: YoungGyu Chun >Priority: Major > > Sometimes the client needs to get only specific object or a subset of objects > from its bucket. Now, the only way to do it is using ListS3 Processor and > after that using FetchS3Object processor. Creating a GetS3Object processor > for such cases can be great -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450976855
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] ottobackwards commented on pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
ottobackwards commented on pull request #4376: URL: https://github.com/apache/nifi/pull/4376#issuecomment-654958033 So this changes the 'root' classloader from 'jetty' to the current classloader? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on a change in pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
ottobackwards commented on a change in pull request #4376:
URL: https://github.com/apache/nifi/pull/4376#discussion_r450959476
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java
##
@@ -192,7 +192,7 @@ public static NarThreadContextClassLoader getInstance() {
public static T createInstance(final ExtensionManager
extensionManager, final String implementationClassName, final Class
typeDefinition, final NiFiProperties nifiProperties)
throws InstantiationException, IllegalAccessException,
ClassNotFoundException {
final ClassLoader originalClassLoader =
Thread.currentThread().getContextClassLoader();
-
Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance());
+
//Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance());
Review comment:
OK, that makes sense, just wasn't sure with the commented out code ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450927873
##
File path: libminifi/src/core/Processor.cpp
##
@@ -289,6 +303,109 @@ bool Processor::isWorkAvailable() {
return hasWork;
}
+// must hold the graphMutex
+void Processor::updateReachability(const std::lock_guard&
graph_lock, bool force) {
+ bool didChange = force;
+ for (auto& outIt : out_going_connections_) {
+for (auto& outConn : outIt.second) {
+ auto connection = std::dynamic_pointer_cast(outConn);
+ if (!connection) {
+continue;
+ }
+ auto dest = std::dynamic_pointer_cast(connection->getDestination());
+ if (!dest) {
+continue;
+ }
+ if (reachable_processors_[connection].insert(dest).second) {
+didChange = true;
+ }
+ for (auto& reachedIt : dest->reachable_processors_) {
+for (auto _proc : reachedIt.second) {
Review comment:
I'm all for a O(better) solution but yeah it probably won't be the
bottleneck
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450927873
##
File path: libminifi/src/core/Processor.cpp
##
@@ -289,6 +303,109 @@ bool Processor::isWorkAvailable() {
return hasWork;
}
+// must hold the graphMutex
+void Processor::updateReachability(const std::lock_guard&
graph_lock, bool force) {
+ bool didChange = force;
+ for (auto& outIt : out_going_connections_) {
+for (auto& outConn : outIt.second) {
+ auto connection = std::dynamic_pointer_cast(outConn);
+ if (!connection) {
+continue;
+ }
+ auto dest = std::dynamic_pointer_cast(connection->getDestination());
+ if (!dest) {
+continue;
+ }
+ if (reachable_processors_[connection].insert(dest).second) {
+didChange = true;
+ }
+ for (auto& reachedIt : dest->reachable_processors_) {
+for (auto _proc : reachedIt.second) {
Review comment:
I'm all for a O(better) solution
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450924096
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450924096
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] mattyb149 commented on a change in pull request #4376: NIFI-7592: Allow NiFi to be started without a GUI/REST interface
mattyb149 commented on a change in pull request #4376:
URL: https://github.com/apache/nifi/pull/4376#discussion_r450919766
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java
##
@@ -192,7 +192,7 @@ public static NarThreadContextClassLoader getInstance() {
public static T createInstance(final ExtensionManager
extensionManager, final String implementationClassName, final Class
typeDefinition, final NiFiProperties nifiProperties)
throws InstantiationException, IllegalAccessException,
ClassNotFoundException {
final ClassLoader originalClassLoader =
Thread.currentThread().getContextClassLoader();
-
Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance());
+
//Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance());
Review comment:
It's on purpose, but I'll remove the dead code. We want the NAR's
classloader to be used to load things, not the singleton
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] tpalfy commented on a change in pull request #4369: NIFI-7581 Add CS-based credential settings support for ADLS processors
tpalfy commented on a change in pull request #4369:
URL: https://github.com/apache/nifi/pull/4369#discussion_r450917508
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
##
@@ -158,41 +173,51 @@
public static Collection
validateCredentialProperties(final ValidationContext validationContext) {
final List results = new ArrayList<>();
-final boolean useManagedIdentity =
validationContext.getProperty(USE_MANAGED_IDENTITY).asBoolean();
-final boolean accountKeyIsSet =
validationContext.getProperty(ACCOUNT_KEY).isSet();
-final boolean sasTokenIsSet =
validationContext.getProperty(SAS_TOKEN).isSet();
-
-int credential_config_found = 0;
-if(useManagedIdentity) credential_config_found++;
-if(accountKeyIsSet) credential_config_found++;
-if(sasTokenIsSet) credential_config_found++;
-
-if(credential_config_found == 0){
-final String msg = String.format(
-"At least one of ['%s', '%s', '%s'] should be set",
-ACCOUNT_KEY.getDisplayName(),
-SAS_TOKEN.getDisplayName(),
-USE_MANAGED_IDENTITY.getDisplayName()
-);
-results.add(new ValidationResult.Builder().subject("Credentials
config").valid(false).explanation(msg).build());
-} else if(credential_config_found > 1) {
-final String msg = String.format(
-"Only one of ['%s', '%s', '%s'] should be set",
-ACCOUNT_KEY.getDisplayName(),
-SAS_TOKEN.getDisplayName(),
-USE_MANAGED_IDENTITY.getDisplayName()
-);
-results.add(new ValidationResult.Builder().subject("Credentials
config").valid(false).explanation(msg).build());
+if
(!validationContext.getProperty(AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE).isSet())
{
Review comment:
Great, thanks, I did the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #4350: NIFI-6934 In PutDatabaseRecord added Postgres UPSERT support
mattyb149 commented on pull request #4350: URL: https://github.com/apache/nifi/pull/4350#issuecomment-654910951 Can you please change the base branch of this PR to `main` and rebase against the latest `main` branch? That's the one we'll be using from now on. Thanks in advance! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
arpadboda commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450916680
##
File path: libminifi/test/flow-tests/LoopTest.cpp
##
@@ -0,0 +1,102 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#undef NDEBUG
+#include "CustomProcessors.h"
+#include "FlowBuilder.h"
+
+// A flow with structure:
+// [Generator] ---> [A] ---|
+// ^_|
+const char* flowConfigurationYaml =
+R"(
+Flow Controller:
+ name: MiNiFi Flow
+ id: 2438e3c8-015a-1001-79ca-83af40ec1990
+Processors:
+ - name: Generator
+id: 2438e3c8-015a-1001-79ca-83af40ec1991
+class: org.apache.nifi.processors.TestFlowFileGenerator
+max concurrent tasks: 1
+scheduling strategy: TIMER_DRIVEN
+scheduling period: 100 ms
+penalization period: 300 ms
+yield period: 100 ms
+run duration nanos: 0
+auto-terminated relationships list:
+ - name: A
+id: 2438e3c8-015a-1001-79ca-83af40ec1992
+class: org.apache.nifi.processors.TestProcessor
+max concurrent tasks: 1
+scheduling strategy: TIMER_DRIVEN
+scheduling period: 100 ms
+penalization period: 300 ms
+yield period: 100 ms
+run duration nanos: 0
+auto-terminated relationships list:
+Properties:
+ AppleProbability: 100
+ BananaProbability: 0
+
+Connections:
+ - name: Gen
+id: 2438e3c8-015a-1001-79ca-83af40ec1993
+source name: Generator
+destination name: A
+source relationship name: success
+max work queue size: 1
+max work queue data size: 1 MB
+flowfile expiration: 0
+ - name: Loop
+id: 2438e3c8-015a-1001-79ca-83af40ec1994
+source name: A
+destination name: A
+source relationship name: apple
+max work queue size: 1
+max work queue data size: 1 MB
+flowfile expiration: 0
+
+Remote Processing Groups:
+)";
+
+TEST_CASE("Flow with a single loop", "[SingleLoopFlow]") {
+ TestController testController;
+
+ LogTestController::getInstance().setTrace();
+ LogTestController::getInstance().setTrace();
+ LogTestController::getInstance().setTrace();
+
+ char format[] = "/tmp/flow.XX";
+ std::string dir = testController.createTempDirectory(format);
+ std::string yamlPath = utils::file::FileUtils::concat_path(dir,
"config.yml");
+ std::ofstream{yamlPath} << flowConfigurationYaml;
+
+ Flow flow = createFlow(yamlPath);
+
+ auto procGenerator =
std::static_pointer_cast(flow.root_->findProcessor("Generator"));
+ auto procA =
std::static_pointer_cast(flow.root_->findProcessor("A"));
+
+ int tryCount = 0;
+ // wait for the procA to get triggered 15 times
+ while (tryCount++ < 10 && !(procA->trigger_count.load() >= 15)) {
Review comment:
Fair point :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
arpadboda commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450908717
##
File path: libminifi/src/core/Processor.cpp
##
@@ -289,6 +303,109 @@ bool Processor::isWorkAvailable() {
return hasWork;
}
+// must hold the graphMutex
+void Processor::updateReachability(const std::lock_guard&
graph_lock, bool force) {
+ bool didChange = force;
+ for (auto& outIt : out_going_connections_) {
+for (auto& outConn : outIt.second) {
+ auto connection = std::dynamic_pointer_cast(outConn);
+ if (!connection) {
+continue;
+ }
+ auto dest = std::dynamic_pointer_cast(connection->getDestination());
+ if (!dest) {
+continue;
+ }
+ if (reachable_processors_[connection].insert(dest).second) {
+didChange = true;
+ }
+ for (auto& reachedIt : dest->reachable_processors_) {
+for (auto _proc : reachedIt.second) {
Review comment:
O(scary), but good enough in practice. :D
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
arpadboda commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450907899
##
File path: libminifi/include/core/Processor.h
##
@@ -305,6 +311,19 @@ class Processor : public Connectable, public
ConfigurableComponent, public std::
Processor =(const Processor );
private:
+ static std::mutex& getGraphMutex() {
+static std::mutex mutex{};
+return mutex;
+ }
+
+ // must hold the graphMutex
+ void updateReachability(const std::lock_guard& graph_lock, bool
force = false);
Review comment:
Okay, I'm fine with this, thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #821: MINIFICPP-1251 - Implement and test RetryFlowFile processor
arpadboda commented on a change in pull request #821:
URL: https://github.com/apache/nifi-minifi-cpp/pull/821#discussion_r450889316
##
File path: extensions/standard-processors/processors/RetryFlowFile.cpp
##
@@ -0,0 +1,183 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RetryFlowFile.h"
+
+#include "core/PropertyValidation.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property
RetryFlowFile::RetryAttribute(core::PropertyBuilder::createProperty("Retry
Attribute")
+->withDescription(
+"The name of the attribute that contains the current retry count for
the FlowFile."
+"WARNING: If the name matches an attribute already on the FlowFile
that does not contain a numerical value, "
+"the processor will either overwrite that attribute with '1' or fail
based on configuration.")
+->withDefaultValue("flowfile.retries")
+->supportsExpressionLanguage(true)
+->build());
+
+core::Property
RetryFlowFile::MaximumRetries(core::PropertyBuilder::createProperty("Maximum
Retries")
+->withDescription("The maximum number of times a FlowFile can be retried
before being passed to the 'retries_exceeded' relationship.")
+->withDefaultValue(3)
Review comment:
This as well
##
File path: extensions/standard-processors/processors/RetryFlowFile.cpp
##
@@ -0,0 +1,183 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RetryFlowFile.h"
+
+#include "core/PropertyValidation.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property
RetryFlowFile::RetryAttribute(core::PropertyBuilder::createProperty("Retry
Attribute")
+->withDescription(
+"The name of the attribute that contains the current retry count for
the FlowFile."
+"WARNING: If the name matches an attribute already on the FlowFile
that does not contain a numerical value, "
+"the processor will either overwrite that attribute with '1' or fail
based on configuration.")
+->withDefaultValue("flowfile.retries")
Review comment:
Shouldn't this be required?
##
File path: extensions/standard-processors/processors/RetryFlowFile.cpp
##
@@ -0,0 +1,183 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "RetryFlowFile.h"
+
+#include "core/PropertyValidation.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property
RetryFlowFile::RetryAttribute(core::PropertyBuilder::createProperty("Retry
Attribute")
+->withDescription(
+
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450899762
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[jira] [Resolved] (MINIFICPP-1276) CAPITests should clean up the temporary files it creates
[
https://issues.apache.org/jira/browse/MINIFICPP-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Markovics resolved MINIFICPP-1276.
---
Resolution: Fixed
> CAPITests should clean up the temporary files it creates
>
>
> Key: MINIFICPP-1276
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1276
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ádám Markovics
>Assignee: Ádám Markovics
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Background:*
> Running tests on MiNiFi currently leaves untracked files in the working
> directory.
> {code:bash|title=Test example generating untracked files}
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183
> Your branch is up to date with 'origin/MINIFICPP-1183'.
> nothing to commit, working tree clean
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ ctest -j24 -R CAPITests
> Test project /home/adam/work/minifi-cpp/build
> Start 30: CAPITests
> 1/1 Test #30: CAPITests Passed0.01 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 0.01
> secadam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183Your branch is up to date with
> 'origin/MINIFICPP-1183'.
> Untracked files: (use "git add ..." to include in what will be
> committed)
> \ ../libminifi/test/1593438783981-9
> nothing added to commit but untracked files present (use "git add" to track)
> {code}
> *Proposal:*
> There are utility functions available that create temporary directories for
> all target platforms. Change the tests so that they use this functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450885343
##
File path: libminifi/test/flow-tests/CustomProcessors.h
##
@@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+#define NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include "TestBase.h"
+#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
Review comment:
done
##
File path: libminifi/src/core/Processor.cpp
##
@@ -101,14 +102,27 @@ void Processor::setScheduledState(ScheduledState state) {
}
bool Processor::addConnection(std::shared_ptr conn) {
- bool ret = false;
+ enum class DidSet{
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450884786
##
File path: libminifi/test/flow-tests/CustomProcessors.h
##
@@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+#define NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include "TestBase.h"
+#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+static core::Relationship Apple{"apple", ""};
+static core::Relationship Banana{"banana", ""};
+// The probability that this processor routes to Apple
+static core::Property AppleProbability =
core::PropertyBuilder::createProperty("AppleProbability")->withDefaultValue(100)->build();
+// The probability that this processor routes to Banana
+static core::Property BananaProbability =
core::PropertyBuilder::createProperty("BananaProbability")->withDefaultValue(0)->build();
+
+class ProcessorWithStatistics {
+ public:
+ std::atomic trigger_count{0};
+};
+
+class TestProcessor : public core::Processor, public ProcessorWithStatistics {
+ public:
+ TestProcessor(std::string name, utils::Identifier ) : Processor(name,
uuid) {}
+ TestProcessor(std::string name) : Processor(name) {}
+ void initialize() override {
+setSupportedProperties({AppleProbability, BananaProbability});
+setSupportedRelationships({Apple, Banana});
+ }
+ void onTrigger(const std::shared_ptr , const
std::shared_ptr ) override {
+++trigger_count;
+auto flowFile = session->get();
+if (!flowFile) return;
+std::random_device rd{};
+std::uniform_int_distribution dis(0, 100);
+int rand = dis(rd);
+if (rand <= apple_probability_) {
+ session->transfer(flowFile, Apple);
+ return;
+}
+rand -= apple_probability_;
+if (rand <= banana_probability_) {
+ session->transfer(flowFile, Banana);
+ return;
+}
+throw std::runtime_error("Couldn't route file");
+ }
+ void onSchedule(const std::shared_ptr , const
std::shared_ptr ) override {
+int apple;
+assert(context->getProperty(AppleProbability.getName(), apple));
+int banana;
+assert(context->getProperty(BananaProbability.getName(), banana));
Review comment:
done (although each test started with an `#undef NDEBUG`)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450884735
##
File path: libminifi/test/flow-tests/FlowBuilder.h
##
@@ -0,0 +1,73 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef NIFI_MINIFI_CPP_FLOWCREATOR_H
+#define NIFI_MINIFI_CPP_FLOWCREATOR_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include "TestBase.h"
+
+struct Flow{
+ Flow(std::shared_ptr&& controller,
std::shared_ptr&& root)
+ : controller_(std::move(controller)), root_(std::move(root)) {
+controller_->load(root_);
+controller_->start();
+ }
+ ~Flow() {
+controller_->stop(true);
+controller_->unload();
+ }
+ std::shared_ptr controller_;
+ std::shared_ptr root_;
+};
+
+Flow createFlow(const std::string& yamlPath) {
+ std::shared_ptr configuration =
std::make_shared();
+ std::shared_ptr prov_repo =
std::make_shared();
+ std::shared_ptr ff_repo =
std::make_shared();
+ std::shared_ptr content_repo =
std::make_shared();
+
+ configuration->set(minifi::Configure::nifi_flow_configuration_file,
yamlPath);
+
+ std::shared_ptr stream_factory =
minifi::io::StreamFactory::getInstance(configuration);
+ content_repo->initialize(configuration);
+
+ std::unique_ptr flow =
utils::make_unique(prov_repo, ff_repo, content_repo,
stream_factory, configuration, yamlPath);
+ std::shared_ptr root = flow->getRoot();
+
+ std::shared_ptr controller =
std::make_shared(
+ prov_repo, ff_repo, configuration,
+ std::move(flow),
+ content_repo, DEFAULT_ROOT_GROUP_NAME, true);
+
+ return Flow{std::move(controller), std::move(root)};
+}
+
+double calculateCongestion(const std::shared_ptr& conn) {
Review comment:
removed
##
File path: libminifi/test/flow-tests/CustomProcessors.h
##
@@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+#define NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include "TestBase.h"
+#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+static core::Relationship Apple{"apple", ""};
+static core::Relationship Banana{"banana", ""};
+// The probability that this processor routes to Apple
+static core::Property AppleProbability =
core::PropertyBuilder::createProperty("AppleProbability")->withDefaultValue(100)->build();
+// The probability that this processor routes to Banana
+static core::Property BananaProbability =
core::PropertyBuilder::createProperty("BananaProbability")->withDefaultValue(0)->build();
+
+class ProcessorWithStatistics {
+ public:
+ std::atomic trigger_count{0};
+};
+
+class TestProcessor : public core::Processor, public ProcessorWithStatistics {
+ public:
+ TestProcessor(std::string name, utils::Identifier ) : Processor(name,
uuid) {}
+ TestProcessor(std::string name) : Processor(name) {}
+ void initialize() override {
+setSupportedProperties({AppleProbability, BananaProbability});
+setSupportedRelationships({Apple, Banana});
+ }
+ void onTrigger(const std::shared_ptr , const
std::shared_ptr ) override {
+++trigger_count;
+auto flowFile = session->get();
+if (!flowFile) return;
+std::random_device rd{};
+
[jira] [Created] (MINIFICPP-1281) Decrease the time spent sleeping in MiNiFi unit/integration tests
Adam Hunyadi created MINIFICPP-1281: --- Summary: Decrease the time spent sleeping in MiNiFi unit/integration tests Key: MINIFICPP-1281 URL: https://issues.apache.org/jira/browse/MINIFICPP-1281 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Adam Hunyadi Assignee: Adam Hunyadi Fix For: 0.8.0 *Background:* Some tests wait for many seconds in order to guarantee that all the paralell processing happens by the time the test makes its assertions. Using sleep for synching is a very bad pattern resulting in tests taking unnecessarily long to execute. *Proposal:* An alternative for setting a run-time for the tests is polling for events we expect to happen during the test allowing for an earlier finish in most of the cases. We should make an effort replacing unnecessarily long sleeps if possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450881009
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450881264
##
File path: libminifi/test/flow-tests/LoopTest.cpp
##
@@ -0,0 +1,102 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#undef NDEBUG
+#include "CustomProcessors.h"
+#include "FlowBuilder.h"
+
+// A flow with structure:
+// [Generator] ---> [A] ---|
+// ^_|
+const char* flowConfigurationYaml =
+R"(
+Flow Controller:
+ name: MiNiFi Flow
+ id: 2438e3c8-015a-1001-79ca-83af40ec1990
+Processors:
+ - name: Generator
+id: 2438e3c8-015a-1001-79ca-83af40ec1991
+class: org.apache.nifi.processors.TestFlowFileGenerator
+max concurrent tasks: 1
+scheduling strategy: TIMER_DRIVEN
+scheduling period: 100 ms
+penalization period: 300 ms
+yield period: 100 ms
+run duration nanos: 0
+auto-terminated relationships list:
+ - name: A
+id: 2438e3c8-015a-1001-79ca-83af40ec1992
+class: org.apache.nifi.processors.TestProcessor
+max concurrent tasks: 1
+scheduling strategy: TIMER_DRIVEN
+scheduling period: 100 ms
+penalization period: 300 ms
+yield period: 100 ms
+run duration nanos: 0
+auto-terminated relationships list:
+Properties:
+ AppleProbability: 100
+ BananaProbability: 0
+
+Connections:
+ - name: Gen
+id: 2438e3c8-015a-1001-79ca-83af40ec1993
+source name: Generator
+destination name: A
+source relationship name: success
+max work queue size: 1
+max work queue data size: 1 MB
+flowfile expiration: 0
+ - name: Loop
+id: 2438e3c8-015a-1001-79ca-83af40ec1994
+source name: A
+destination name: A
+source relationship name: apple
+max work queue size: 1
+max work queue data size: 1 MB
+flowfile expiration: 0
+
+Remote Processing Groups:
+)";
+
+TEST_CASE("Flow with a single loop", "[SingleLoopFlow]") {
+ TestController testController;
+
+ LogTestController::getInstance().setTrace();
+ LogTestController::getInstance().setTrace();
+ LogTestController::getInstance().setTrace();
+
+ char format[] = "/tmp/flow.XX";
+ std::string dir = testController.createTempDirectory(format);
+ std::string yamlPath = utils::file::FileUtils::concat_path(dir,
"config.yml");
+ std::ofstream{yamlPath} << flowConfigurationYaml;
+
+ Flow flow = createFlow(yamlPath);
+
+ auto procGenerator =
std::static_pointer_cast(flow.root_->findProcessor("Generator"));
+ auto procA =
std::static_pointer_cast(flow.root_->findProcessor("A"));
+
+ int tryCount = 0;
+ // wait for the procA to get triggered 15 times
+ while (tryCount++ < 10 && !(procA->trigger_count.load() >= 15)) {
Review comment:
`tryCount` is our friend
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450873613
##
File path: libminifi/src/core/Processor.cpp
##
@@ -101,14 +102,27 @@ void Processor::setScheduledState(ScheduledState state) {
}
bool Processor::addConnection(std::shared_ptr conn) {
- bool ret = false;
+ enum class DidSet{
Review comment:
interesting because I recall that I first called it "SetAs" but the
commit history doesn't confirm this, will make the change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
adamdebreceni commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450869926
##
File path: libminifi/include/core/Processor.h
##
@@ -305,6 +311,19 @@ class Processor : public Connectable, public
ConfigurableComponent, public std::
Processor =(const Processor );
private:
+ static std::mutex& getGraphMutex() {
+static std::mutex mutex{};
+return mutex;
+ }
+
+ // must hold the graphMutex
+ void updateReachability(const std::lock_guard& graph_lock, bool
force = false);
Review comment:
if it turns out that there is a legit use-case where we would need to
expose this method, I'm more than happy to make it non-private but I couldn't
come up with a reason to do so
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450869008
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450866256
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] mattyb149 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
mattyb149 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450863175
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #822: MINIFICPP-1253 - Do not hang on full cycles
arpadboda commented on a change in pull request #822:
URL: https://github.com/apache/nifi-minifi-cpp/pull/822#discussion_r450858585
##
File path: libminifi/test/flow-tests/CustomProcessors.h
##
@@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+#define NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include "TestBase.h"
+#include "../../extensions/standard-processors/processors/GenerateFlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+static core::Relationship Apple{"apple", ""};
+static core::Relationship Banana{"banana", ""};
+// The probability that this processor routes to Apple
+static core::Property AppleProbability =
core::PropertyBuilder::createProperty("AppleProbability")->withDefaultValue(100)->build();
+// The probability that this processor routes to Banana
+static core::Property BananaProbability =
core::PropertyBuilder::createProperty("BananaProbability")->withDefaultValue(0)->build();
+
+class ProcessorWithStatistics {
+ public:
+ std::atomic trigger_count{0};
+};
+
+class TestProcessor : public core::Processor, public ProcessorWithStatistics {
+ public:
+ TestProcessor(std::string name, utils::Identifier ) : Processor(name,
uuid) {}
+ TestProcessor(std::string name) : Processor(name) {}
+ void initialize() override {
+setSupportedProperties({AppleProbability, BananaProbability});
+setSupportedRelationships({Apple, Banana});
+ }
+ void onTrigger(const std::shared_ptr , const
std::shared_ptr ) override {
+++trigger_count;
+auto flowFile = session->get();
+if (!flowFile) return;
+std::random_device rd{};
+std::uniform_int_distribution dis(0, 100);
+int rand = dis(rd);
+if (rand <= apple_probability_) {
+ session->transfer(flowFile, Apple);
+ return;
+}
+rand -= apple_probability_;
+if (rand <= banana_probability_) {
+ session->transfer(flowFile, Banana);
+ return;
+}
+throw std::runtime_error("Couldn't route file");
+ }
+ void onSchedule(const std::shared_ptr , const
std::shared_ptr ) override {
+int apple;
+assert(context->getProperty(AppleProbability.getName(), apple));
+int banana;
+assert(context->getProperty(BananaProbability.getName(), banana));
Review comment:
Without nodebug ifdef I'm not sure these assert actually ever do
something.
##
File path: libminifi/include/core/Processor.h
##
@@ -305,6 +311,19 @@ class Processor : public Connectable, public
ConfigurableComponent, public std::
Processor =(const Processor );
private:
+ static std::mutex& getGraphMutex() {
+static std::mutex mutex{};
+return mutex;
+ }
+
+ // must hold the graphMutex
+ void updateReachability(const std::lock_guard& graph_lock, bool
force = false);
Review comment:
Do we want to hide these functions from actual processor implementations?
I can accept that as a design decision, just interested in the motivation.
##
File path: libminifi/test/flow-tests/CustomProcessors.h
##
@@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+#define NIFI_MINIFI_CPP_CUSTOMPROCESSORS_H
+
+#include
+#include
+#include
+#include
+#include "core/Processor.h"
+#include
[GitHub] [nifi] mattyb149 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
mattyb149 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450859580
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450854247
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450852076
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4371: NIFI-7589 Fix path value when unpacking tar
MikeThomsen commented on a change in pull request #4371:
URL: https://github.com/apache/nifi/pull/4371#discussion_r450851316
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
##
@@ -309,7 +309,10 @@ public void process(final InputStream in) throws
IOException {
}
final File file = new File(tarEntry.getName());
final Path filePath = file.toPath();
-final String filePathString = filePath.getParent()
+ "/";
+String filePathString = "/";
+if(filePath.getParent() != null) {
+filePathString = filePath.getParent() + "/";
Review comment:
Ok, let's skip that then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
markap14 commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450849766
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] turcsanyip commented on a change in pull request #4370: NIFI-6128 UnpackContent: Store unpacked file data
turcsanyip commented on a change in pull request #4370:
URL: https://github.com/apache/nifi/pull/4370#discussion_r450830871
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileInfo.java
##
@@ -206,6 +207,32 @@ public Builder permissions(String permissions) {
return this;
}
+public Builder permissions(int fileModeOctal) {
+if(fileModeOctal > 0777 || fileModeOctal < 00) {
+throw new RuntimeException("Invalid mode numeral");
+}
+final int NUM_SYSTEM = 8;
+StringBuilder sb = new StringBuilder();
+while(fileModeOctal > 0) {
+int digit = fileModeOctal % NUM_SYSTEM;
+final String[] perms = new String[]{"x", "w", "r"};
+for (final String perm : perms) {
+if ((digit & 1) == 1) {
+sb.insert(0, perm);
+} else {
+sb.insert(0, "-");
+}
+digit >>= 1;
+}
+fileModeOctal /= NUM_SYSTEM;
+}
+if (sb.length() < 9) {
+IntStream.range(0, 9 - sb.length()).forEach(i -> sb.insert(0,
'-'));
+}
+this.permissions = sb.toString();
+return this;
+}
Review comment:
I think this algorithm could be simplified a lot, similar to this (not
tested thoroughly):
```
char[] PERM = "xwrxwrxwr".toCharArray();
StringBuilder sb = new StringBuilder();
for (char p : PERM) {
sb.append((fileModeOctal & 1) == 1 ? p : '-');
fileModeOctal >>= 1;
}
this.permissions = sb.reverse().toString();
```
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
##
@@ -321,6 +337,16 @@ public void process(final InputStream in) throws
IOException {
attributes.put(CoreAttributes.ABSOLUTE_PATH.key(), absPathString);
attributes.put(CoreAttributes.MIME_TYPE.key(),
OCTET_STREAM);
+{
+FileInfo info = new
FileInfo.Builder().permissions(tarEntry.getMode()).build();
+attributes.put(FILE_PERMISSIONS_ATTRIBUTE,
info.getPermissions());
+}
Review comment:
Why are these lines in an anonymous code block?
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
##
@@ -110,6 +118,12 @@
public static final String OCTET_STREAM = "application/octet-stream";
+public static final String FILE_LAST_MODIFY_TIME_ATTRIBUTE =
"file.lastModifiedTime";
Review comment:
Typo: MODIFIED
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UnpackContent.java
##
@@ -298,6 +312,8 @@ public TarUnpacker(Pattern fileFilter) {
public void unpack(final ProcessSession session, final FlowFile
source, final List unpacked) {
final String fragmentId = UUID.randomUUID().toString();
session.read(source, new InputStreamCallback() {
+private final DateTimeFormatter dateTimeFormatter =
DateTimeFormatter.ofPattern(FILE_MODIFY_DATE_ATTR_FORMAT).withZone(ZoneId.systemDefault());
Review comment:
You can define the DateTimeFormatter instance in a constant as it is
thread-safe. There's no need the create a new one for every FlowFile.
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileInfo.java
##
@@ -206,6 +207,32 @@ public Builder permissions(String permissions) {
return this;
}
+public Builder permissions(int fileModeOctal) {
Review comment:
I would consider to define a separate utility method for this conversion
instead of putting it into the builder.
Creating a FileInfo object just to convert the permissions seems to me a bit
strange.
Furthermore, one might need the other way conversion (rwx => octal) in the
future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #820: MINIFICPP-1183 - Cleanup C2 Update tests
arpadboda commented on a change in pull request #820:
URL: https://github.com/apache/nifi-minifi-cpp/pull/820#discussion_r450837057
##
File path: extensions/coap/tests/CoapIntegrationBase.h
##
@@ -27,8 +27,7 @@ int log_message(const struct mg_connection *conn, const char
*message) {
return 1;
}
-int ssl_enable(void *ssl_context, void *user_data) {
- struct ssl_ctx_st *ctx = (struct ssl_ctx_st *) ssl_context;
+int ssl_enable(void *ssl_context, void *) {
Review comment:
I think the unused argument here is going to generate a warning.
##
File path: extensions/http-curl/tests/HttpGetIntegrationTest.cpp
##
@@ -17,46 +17,31 @@
*/
#define CURLOPT_SSL_VERIFYPEER_DISABLE 1
-#include
#undef NDEBUG
#include
#include
#include
-#include
#include
#include
#include
-#include
#include
#include "HTTPClient.h"
#include "InvokeHTTP.h"
#include "TestServer.h"
#include "TestBase.h"
#include "utils/StringUtils.h"
-#include "core/Core.h"
-#include "core/logging/Logger.h"
-#include "core/ProcessGroup.h"
#include "core/yaml/YamlConfiguration.h"
#include "FlowController.h"
#include "properties/Configure.h"
#include "unit/ProvenanceTestHelper.h"
#include "io/StreamFactory.h"
-#include "processors/InvokeHTTP.h"
-#include "processors/ListenHTTP.h"
#include "processors/LogAttribute.h"
+#include "HTTPIntegrationBase.h"
-void waitToVerifyProcessor() {
- std::this_thread::sleep_for(std::chrono::seconds(10));
-}
-
-int log_message(const struct mg_connection *conn, const char *message) {
- puts(message);
- return 1;
-}
-
-int ssl_enable(void* /*ssl_context*/, void* /*user_data*/) {
- puts("Enable ssl");
- return 0;
+namespace {
+ void waitToVerifyProcessor() {
+std::this_thread::sleep_for(std::chrono::seconds(10));
Review comment:
I would move that to the scope of a follow-up ticket, our tests
definitely need improvement in this aspect, and this is not the only one.
##
File path: extensions/http-curl/tests/C2NullConfiguration.cpp
##
@@ -74,23 +53,23 @@ class VerifyC2Server : public CoapIntegrationBase {
file.close();
}
- void cleanup() {
+ void cleanup() override {
unlink(ss.str().c_str());
Review comment:
Do we need it? It's created in a tempdir, which is created by the
testcontroller, I suppose this should be removed automatically.
##
File path: extensions/http-curl/tests/TimeoutHTTPSiteToSiteTests.cpp
##
@@ -152,25 +137,18 @@ void run_timeout_variance(std::string test_file_location,
bool isSecure, std::st
harness.run(test_file_location);
- assert(LogTestController::getInstance().contains("limit (200ms) reached,
terminating connection") == true);
+ assert(LogTestController::getInstance().contains("limit (200ms) reached,
terminating connection"));
LogTestController::getInstance().reset();
}
int main(int argc, char **argv) {
transaction_id = 0;
transaction_id_output = 0;
- std::string key_dir, test_file_location, url;
- if (argc > 1) {
-test_file_location = argv[1];
-key_dir = argv[2];
-url = argv[3];
- }
+ const cmd_args args = parse_cmdline_args_with_url(argc, argv);
- bool isSecure = false;
- if (url.find("https") != std::string::npos) {
-isSecure = true;
- }
+ // check https prefix
+ const bool isSecure = args.url.rfind("https://;, 0) == 0;
Review comment:
This could be a member function of args or some other helper function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450843336
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[GitHub] [nifi] MikeThomsen commented on a change in pull request #4374: NIFI-7572: Added ScriptedTransformRecord processor. Addressed a coupl…
MikeThomsen commented on a change in pull request #4374:
URL: https://github.com/apache/nifi/pull/4374#discussion_r450839641
##
File path:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/ScriptedTransformRecord.java
##
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Restriction;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.components.RequiredPermission;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.script.ScriptingComponentHelper;
+import org.apache.nifi.script.ScriptingComponentUtils;
+import org.apache.nifi.search.SearchContext;
+import org.apache.nifi.search.SearchResult;
+import org.apache.nifi.search.Searchable;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.python.jsr223.PyScriptEngine;
+
+import javax.script.Bindings;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptException;
+import javax.script.SimpleBindings;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+
+@EventDriven
+@SupportsBatching
+@SideEffectFree
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"record", "transform", "script", "groovy", "jython", "python",
"update", "modify", "filter"})
+@Restricted(restrictions = {
+@Restriction(requiredPermission = RequiredPermission.EXECUTE_CODE,
+explanation = "Provides operator the ability to execute arbitrary code
assuming all permissions that NiFi has.")
+})
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer"),
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile"),
+@WritesAttribute(attribute = "record.error.message", description = "This
attribute provides on failure the error message encountered by the Reader or
Writer.")
+})
+@CapabilityDescription("Provides the ability to evaluate a simple script
against each record in an incoming FlowFile. The script may transform the
record in some way, filter the record, or fork " +
+"additional records. See Processor's Additional
[jira] [Resolved] (MINIFICPP-1274) Flow restart could double-spend flowfiles
[ https://issues.apache.org/jira/browse/MINIFICPP-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Debreceni resolved MINIFICPP-1274. --- Resolution: Fixed > Flow restart could double-spend flowfiles > - > > Key: MINIFICPP-1274 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1274 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Adam Debreceni >Assignee: Adam Debreceni >Priority: Major > Labels: selected-for-development > Time Spent: 3h > Remaining Estimate: 0h > > Flowfiles are async-deleted from the FlowFileRepository and no flush happens > after shutdown, leaving these marked files in the repository. If we restart > the agent, this allows these zombie files to be resurrected and be put back > into their last connections. This could cause files to be processed multiple > times even if we marked them for deletion. > Solution proposal: flush the FlowFileRepository after shutdown, so all marked > files are actually deleted (this won't save us from double-processing > flowfiles after a crash) > (also make sure that the FlowFileRepository shutdown happens after no more > processors are running) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1273) Clear connections on flow update
[ https://issues.apache.org/jira/browse/MINIFICPP-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Debreceni resolved MINIFICPP-1273. --- Assignee: Adam Debreceni Resolution: Fixed > Clear connections on flow update > > > Key: MINIFICPP-1273 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1273 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Adam Debreceni >Assignee: Adam Debreceni >Priority: Major > Labels: selected-for-development > Time Spent: 3.5h > Remaining Estimate: 0h > > Upon a flow update all the previous connection and processors remain in > memory as they circularly reference each other through shared_ptr-s. This > wouldn't cause much of a problem as they are in the order of kb-s, but the > flowfiles in these connections are also leaked which could be a significant > sum. > We should drain the connections on flow shutdown. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7542) Upgrade jackson-databind dependency version
[ https://issues.apache.org/jira/browse/NIFI-7542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-7542: --- Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade jackson-databind dependency version > --- > > Key: NIFI-7542 > URL: https://issues.apache.org/jira/browse/NIFI-7542 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: dependency > Fix For: 1.12.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Upgrade jackson-databind dependency version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #826: MINIFICPP-1274 - Commit delete operation before shutdown
arpadboda closed pull request #826: URL: https://github.com/apache/nifi-minifi-cpp/pull/826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #827: MINIFICPP-1273 - Drain connections on flow shutdown
arpadboda closed pull request #827: URL: https://github.com/apache/nifi-minifi-cpp/pull/827 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #830: MINIFICPP-1276 - CAPITests should not leave any trash after running
arpadboda closed pull request #830: URL: https://github.com/apache/nifi-minifi-cpp/pull/830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4362: NIFI 7542 Upgrade additional jackson-databind versions.
asfgit closed pull request #4362: URL: https://github.com/apache/nifi/pull/4362 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7542) Upgrade jackson-databind dependency version
[ https://issues.apache.org/jira/browse/NIFI-7542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152712#comment-17152712 ] ASF subversion and git services commented on NIFI-7542: --- Commit f43dc8063f4f7c3ddd7efe700c9c9be4da4179fb in nifi's branch refs/heads/main from mtien [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f43dc80 ] NIFI-7542 Upgrade additional jackson-databind versions. This close #4362 Signed-off-by: Mike Thomsen > Upgrade jackson-databind dependency version > --- > > Key: NIFI-7542 > URL: https://issues.apache.org/jira/browse/NIFI-7542 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: M Tien >Assignee: M Tien >Priority: Major > Labels: dependency > Time Spent: 50m > Remaining Estimate: 0h > > Upgrade jackson-databind dependency version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] MikeThomsen commented on pull request #4362: NIFI 7542 Upgrade additional jackson-databind versions.
MikeThomsen commented on pull request #4362: URL: https://github.com/apache/nifi/pull/4362#issuecomment-654815669 Did a final pass with `mvn dependency:tree | grep databind` and it looks like you got everything. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1276) CAPITests should clean up the temporary files it createes
[
https://issues.apache.org/jira/browse/MINIFICPP-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpad Boda updated MINIFICPP-1276:
--
Summary: CAPITests should clean up the temporary files it createes (was:
CAPITests should not leave any trash after running)
> CAPITests should clean up the temporary files it createes
> -
>
> Key: MINIFICPP-1276
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1276
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ádám Markovics
>Assignee: Ádám Markovics
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Background:*
> Running tests on MiNiFi currently leaves untracked files in the working
> directory.
> {code:bash|title=Test example generating untracked files}
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183
> Your branch is up to date with 'origin/MINIFICPP-1183'.
> nothing to commit, working tree clean
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ ctest -j24 -R CAPITests
> Test project /home/adam/work/minifi-cpp/build
> Start 30: CAPITests
> 1/1 Test #30: CAPITests Passed0.01 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 0.01
> secadam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183Your branch is up to date with
> 'origin/MINIFICPP-1183'.
> Untracked files: (use "git add ..." to include in what will be
> committed)
> \ ../libminifi/test/1593438783981-9
> nothing added to commit but untracked files present (use "git add" to track)
> {code}
> *Proposal:*
> There are utility functions available that create temporary directories for
> all target platforms. Change the tests so that they use this functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1276) CAPITests should clean up the temporary files it creates
[
https://issues.apache.org/jira/browse/MINIFICPP-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpad Boda updated MINIFICPP-1276:
--
Summary: CAPITests should clean up the temporary files it creates (was:
CAPITests should clean up the temporary files it createes)
> CAPITests should clean up the temporary files it creates
>
>
> Key: MINIFICPP-1276
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1276
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ádám Markovics
>Assignee: Ádám Markovics
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Background:*
> Running tests on MiNiFi currently leaves untracked files in the working
> directory.
> {code:bash|title=Test example generating untracked files}
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183
> Your branch is up to date with 'origin/MINIFICPP-1183'.
> nothing to commit, working tree clean
> adam@amarkovics-Linux:~/work/minifi-cpp/build$ ctest -j24 -R CAPITests
> Test project /home/adam/work/minifi-cpp/build
> Start 30: CAPITests
> 1/1 Test #30: CAPITests Passed0.01 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 0.01
> secadam@amarkovics-Linux:~/work/minifi-cpp/build$ git st
> On branch MINIFICPP-1183Your branch is up to date with
> 'origin/MINIFICPP-1183'.
> Untracked files: (use "git add ..." to include in what will be
> committed)
> \ ../libminifi/test/1593438783981-9
> nothing added to commit but untracked files present (use "git add" to track)
> {code}
> *Proposal:*
> There are utility functions available that create temporary directories for
> all target platforms. Change the tests so that they use this functionality.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
