[jira] [Commented] (NIFI-7505) Add InvokeHTTPRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793510#comment-17793510
]
Ryan Miller commented on NIFI-7505:
---
Continuing from the previous PR [1], I've had a go at implementing the
following changes to the InvokeHTTPRecord Processor:
* Use {{WebClientServiceProvider}} instead of manually instantiating an HTTP
client inside {{{}InvokeHTTPRecord{}}}. This will make the code simpler, reduce
the number of properties and make it easier to write tests.
* Remove the functionality for splitting records by success/failure, and just
have the ability to store the success/failure flag on each record processed. If
we want to split records, we can do that with a separate {{SplitRecord}}
processor.
I've created a new PR [2] with these changes for the purposes of feedback and
additional discussion.
[1] [https://github.com/apache/nifi/pull/7532]
[2]
[https://github.com/apache/nifi/pull/8127|https://github.com/apache/nifi/pull/7532]
> Add InvokeHTTPRecord processor
> --
>
> Key: NIFI-7505
> URL: https://issues.apache.org/jira/browse/NIFI-7505
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Priority: Major
> Labels: Record, http, processor, rest
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Some users have recently requested being able to invoke a specific URL via
> {{InvokeHTTP}} on every record in a flowfile. Currently, the {{InvokeHTTP}}
> processor can only handle one piece of data per flowfile. There are some
> workarounds available for specific patterns with {{LookupRecord}} +
> {{RestLookupService}}, but this is not a complete solution. I propose
> introducing an {{InvokeHTTPRecord}} processor, providing the {{InvokeHTTP}}
> functionality in conjunction with the record processing behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-7505 Add InvokeHTTPRecord processor [nifi]
Aerilym opened a new pull request, #8127: URL: https://github.com/apache/nifi/pull/8127 # Summary [NIFI-7505](https://issues.apache.org/jira/browse/NIFI-7505) Continuation of PR #7532 This PR isn't complete yet, but I've created it in case anybody has any feedback on the approach I'm taking. I've kept the commit from the previous PR for tracking purposes, but they will be merged before the PR merge to conform to the PR rules. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12453) Allow obtaining a node's cluster status via nifi.sh
[ https://issues.apache.org/jira/browse/NIFI-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12453: Resolution: Fixed Status: Resolved (was: Patch Available) > Allow obtaining a node's cluster status via nifi.sh > --- > > Key: NIFI-12453 > URL: https://issues.apache.org/jira/browse/NIFI-12453 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There are often times when we want to check the cluster status of a > particular node (i.e., is it CONNECTING, CONNECTED, DISCONNECTING, > DISCONNECTED, etc.) Currently the only way to obtain this information is via > the REST API or the full diagnostic dump. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12454) Allow decommissioning a node without shutdown
[ https://issues.apache.org/jira/browse/NIFI-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12454: Resolution: Fixed Status: Resolved (was: Patch Available) > Allow decommissioning a node without shutdown > - > > Key: NIFI-12454 > URL: https://issues.apache.org/jira/browse/NIFI-12454 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > > When a node is decommissioned, it takes the following steps: > * Disconnect Node from cluster > * Trigger data offload > * Wait for data offload > * Remove node from cluster > * Shutdown > It would be helpful to allow taking the node out of the cluster without > completely terminating the process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12454) Allow decommissioning a node without shutdown
[ https://issues.apache.org/jira/browse/NIFI-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793499#comment-17793499 ] ASF subversion and git services commented on NIFI-12454: Commit 1d06185f13cda1e613f7bf29135584ffdf16639b in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1d06185f13 ] NIFI-12453 Added cluster-status command to nifi.sh NIFI-12454 Allow decommissioning cluster node without shutdown This closes #8100 Signed-off-by: David Handermann > Allow decommissioning a node without shutdown > - > > Key: NIFI-12454 > URL: https://issues.apache.org/jira/browse/NIFI-12454 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > > When a node is decommissioned, it takes the following steps: > * Disconnect Node from cluster > * Trigger data offload > * Wait for data offload > * Remove node from cluster > * Shutdown > It would be helpful to allow taking the node out of the cluster without > completely terminating the process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12453) Allow obtaining a node's cluster status via nifi.sh
[ https://issues.apache.org/jira/browse/NIFI-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793498#comment-17793498 ] ASF subversion and git services commented on NIFI-12453: Commit 1d06185f13cda1e613f7bf29135584ffdf16639b in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1d06185f13 ] NIFI-12453 Added cluster-status command to nifi.sh NIFI-12454 Allow decommissioning cluster node without shutdown This closes #8100 Signed-off-by: David Handermann > Allow obtaining a node's cluster status via nifi.sh > --- > > Key: NIFI-12453 > URL: https://issues.apache.org/jira/browse/NIFI-12453 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There are often times when we want to check the cluster status of a > particular node (i.e., is it CONNECTING, CONNECTED, DISCONNECTING, > DISCONNECTED, etc.) Currently the only way to obtain this information is via > the REST API or the full diagnostic dump. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12453, NIFI-12454: Allow easily determining cluster state of a n… [nifi]
exceptionfactory closed pull request #8100: NIFI-12453, NIFI-12454: Allow easily determining cluster state of a n… URL: https://github.com/apache/nifi/pull/8100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12475) Bypass Validation property set to True returns Authorization error.
[
https://issues.apache.org/jira/browse/NIFI-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-12475:
---
Assignee: David Handermann
> Bypass Validation property set to True returns Authorization error.
> ---
>
> Key: NIFI-12475
> URL: https://issues.apache.org/jira/browse/NIFI-12475
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.16.1
> Environment: based on standard container apache/nifi from docker hub
> no customer processors.
>Reporter: Patrick A. Mol
>Assignee: David Handermann
>Priority: Minor
>
> Across a few versions of NiFi and Mongo,
> the use of PutMongoRecord suddenly stopped working and returned an error.
> Using a flowfile with one valid record, and on PutMongoRecord failure,
> sending the flowfile to SplitRecord and feeding it to PutMongo, using the
> same standard mongodb controller service, the insert would work without error.
> Turns out that the default setting for the PutMongoRecord property Bypass
> Validation is {_}True{_}, which requires elevated privileges in Mongo.
> Changing the property to False allows insert without error.
> The error text is
> {noformat}
> PutMongoRecord[id=018b1026-a670-1590-7941-b6978c972dc6] PutMongoRecord failed
> with error:: com.mongodb.MongoCommandException: Command failed with error 13
> (Unauthorized): 'not authorized on MONGO_DATABASE_NAME to execute command {
> insert: "COLLECTION_NAME", ordered: false, bypassDocumentValidation: true,
> txnNumber: 1, $db: "MONGO_DATABASE_NAME", $clusterTime: { clusterTime:
> Timestamp(1701736623, 1), signature: { hash: BinData(0,
> 62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
> lsid: { id: UUID("722347df-2349-4ec5-88f2-867a946d9614") } }' on server
> MONGODB_URI_WITH_PORTNUMBER. The full response is {"ok": 0.0, "errmsg": "not
> authorized on MONGO_DATABASE_NAME to execute command { insert:
> \"COLLECTION_NAME\", ordered: false, bypassDocumentValidation: true,
> txnNumber: 1, $db: \"MONGO_DATABASE_NAME\", $clusterTime: { clusterTime:
> Timestamp(1701736623, 1), signature: { hash: BinData(0,
> 62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
> lsid: { id: UUID(\"722347df-2349-4ec5-88f2-867a946d9614\") } }", "code": 13,
> "codeName": "Unauthorized", "$clusterTime": {"clusterTime": {"$timestamp":
> {"t": 1701736623, "i": 1}}, "signature": {"hash": {"$binary": {"base64":
> "YrJKNoaaf68Hx3mAGfByzHZOip0=", "subType": "00"}}, "keyId":
> 7264286828646105928}}, "operationTime": {"$timestamp": {"t": 1701736623, "i":
> 1}}}
> {noformat}
> Apparently, PutMongo does not use the same setting for the bypass document
> validation flag, so there is an inconsistency.
> Other libraries/tools, e.g. pymongo insert_many(), also default to False.
> Details regarding the privilege in MongoDB are here
> https://www.mongodb.com/docs/manual/reference/privilege-actions/#mongodb-authaction-bypassDocumentValidation
> With the privilege requiring a custom role in MongoDB, it is debatable
> whether the default setting to True is a bug or changing it to False is an
> improvement.
> At least the error and resolution is recorded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12475) Bypass Validation property set to True returns Authorization error.
[
https://issues.apache.org/jira/browse/NIFI-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793467#comment-17793467
]
Patrick A. Mol commented on NIFI-12475:
---
By the way.
The PutMongo processor does not have a similar Bypass Validation property.
> Bypass Validation property set to True returns Authorization error.
> ---
>
> Key: NIFI-12475
> URL: https://issues.apache.org/jira/browse/NIFI-12475
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.16.1
> Environment: based on standard container apache/nifi from docker hub
> no customer processors.
>Reporter: Patrick A. Mol
>Priority: Minor
>
> Across a few versions of NiFi and Mongo,
> the use of PutMongoRecord suddenly stopped working and returned an error.
> Using a flowfile with one valid record, and on PutMongoRecord failure,
> sending the flowfile to SplitRecord and feeding it to PutMongo, using the
> same standard mongodb controller service, the insert would work without error.
> Turns out that the default setting for the PutMongoRecord property Bypass
> Validation is {_}True{_}, which requires elevated privileges in Mongo.
> Changing the property to False allows insert without error.
> The error text is
> {noformat}
> PutMongoRecord[id=018b1026-a670-1590-7941-b6978c972dc6] PutMongoRecord failed
> with error:: com.mongodb.MongoCommandException: Command failed with error 13
> (Unauthorized): 'not authorized on MONGO_DATABASE_NAME to execute command {
> insert: "COLLECTION_NAME", ordered: false, bypassDocumentValidation: true,
> txnNumber: 1, $db: "MONGO_DATABASE_NAME", $clusterTime: { clusterTime:
> Timestamp(1701736623, 1), signature: { hash: BinData(0,
> 62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
> lsid: { id: UUID("722347df-2349-4ec5-88f2-867a946d9614") } }' on server
> MONGODB_URI_WITH_PORTNUMBER. The full response is {"ok": 0.0, "errmsg": "not
> authorized on MONGO_DATABASE_NAME to execute command { insert:
> \"COLLECTION_NAME\", ordered: false, bypassDocumentValidation: true,
> txnNumber: 1, $db: \"MONGO_DATABASE_NAME\", $clusterTime: { clusterTime:
> Timestamp(1701736623, 1), signature: { hash: BinData(0,
> 62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
> lsid: { id: UUID(\"722347df-2349-4ec5-88f2-867a946d9614\") } }", "code": 13,
> "codeName": "Unauthorized", "$clusterTime": {"clusterTime": {"$timestamp":
> {"t": 1701736623, "i": 1}}, "signature": {"hash": {"$binary": {"base64":
> "YrJKNoaaf68Hx3mAGfByzHZOip0=", "subType": "00"}}, "keyId":
> 7264286828646105928}}, "operationTime": {"$timestamp": {"t": 1701736623, "i":
> 1}}}
> {noformat}
> Apparently, PutMongo does not use the same setting for the bypass document
> validation flag, so there is an inconsistency.
> Other libraries/tools, e.g. pymongo insert_many(), also default to False.
> Details regarding the privilege in MongoDB are here
> https://www.mongodb.com/docs/manual/reference/privilege-actions/#mongodb-authaction-bypassDocumentValidation
> With the privilege requiring a custom role in MongoDB, it is debatable
> whether the default setting to True is a bug or changing it to False is an
> improvement.
> At least the error and resolution is recorded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12475) Bypass Validation property set to True returns Authorization error.
Patrick A. Mol created NIFI-12475:
-
Summary: Bypass Validation property set to True returns
Authorization error.
Key: NIFI-12475
URL: https://issues.apache.org/jira/browse/NIFI-12475
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.16.1
Environment: based on standard container apache/nifi from docker hub
no customer processors.
Reporter: Patrick A. Mol
Across a few versions of NiFi and Mongo,
the use of PutMongoRecord suddenly stopped working and returned an error.
Using a flowfile with one valid record, and on PutMongoRecord failure, sending
the flowfile to SplitRecord and feeding it to PutMongo, using the same standard
mongodb controller service, the insert would work without error.
Turns out that the default setting for the PutMongoRecord property Bypass
Validation is {_}True{_}, which requires elevated privileges in Mongo.
Changing the property to False allows insert without error.
The error text is
{noformat}
PutMongoRecord[id=018b1026-a670-1590-7941-b6978c972dc6] PutMongoRecord failed
with error:: com.mongodb.MongoCommandException: Command failed with error 13
(Unauthorized): 'not authorized on MONGO_DATABASE_NAME to execute command {
insert: "COLLECTION_NAME", ordered: false, bypassDocumentValidation: true,
txnNumber: 1, $db: "MONGO_DATABASE_NAME", $clusterTime: { clusterTime:
Timestamp(1701736623, 1), signature: { hash: BinData(0,
62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
lsid: { id: UUID("722347df-2349-4ec5-88f2-867a946d9614") } }' on server
MONGODB_URI_WITH_PORTNUMBER. The full response is {"ok": 0.0, "errmsg": "not
authorized on MONGO_DATABASE_NAME to execute command { insert:
\"COLLECTION_NAME\", ordered: false, bypassDocumentValidation: true, txnNumber:
1, $db: \"MONGO_DATABASE_NAME\", $clusterTime: { clusterTime:
Timestamp(1701736623, 1), signature: { hash: BinData(0,
62B24A36869A7FAF07C7798019F072CC764E8A9D), keyId: 7264286828646105928 } },
lsid: { id: UUID(\"722347df-2349-4ec5-88f2-867a946d9614\") } }", "code": 13,
"codeName": "Unauthorized", "$clusterTime": {"clusterTime": {"$timestamp":
{"t": 1701736623, "i": 1}}, "signature": {"hash": {"$binary": {"base64":
"YrJKNoaaf68Hx3mAGfByzHZOip0=", "subType": "00"}}, "keyId":
7264286828646105928}}, "operationTime": {"$timestamp": {"t": 1701736623, "i":
1}}}
{noformat}
Apparently, PutMongo does not use the same setting for the bypass document
validation flag, so there is an inconsistency.
Other libraries/tools, e.g. pymongo insert_many(), also default to False.
Details regarding the privilege in MongoDB are here
https://www.mongodb.com/docs/manual/reference/privilege-actions/#mongodb-authaction-bypassDocumentValidation
With the privilege requiring a custom role in MongoDB, it is debatable whether
the default setting to True is a bug or changing it to False is an improvement.
At least the error and resolution is recorded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12418) Identity Provider Groups Missing in Refreshed Bearer Token
[ https://issues.apache.org/jira/browse/NIFI-12418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12418: Status: Patch Available (was: Open) > Identity Provider Groups Missing in Refreshed Bearer Token > -- > > Key: NIFI-12418 > URL: https://issues.apache.org/jira/browse/NIFI-12418 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.24.0, 2.0.0-M1 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: backport-needed > Time Spent: 10m > Remaining Estimate: 0h > > The OIDC Bearer Token Refresh Filter is responsible for renewing application > Bearer Tokens when NiFi is integrated with an OpenID Connect Identity > Provider that supports the Refresh Token Grant Type. > NiFi 1.23.0 introduced changes for handling group membership information > supplied from an Identity Provider, passing the groups in the application > Bearer Token instead of persisting the groups in the local database > repository. > As a result of these handling changes, the Identity Provider group membership > information is not retained when the OIDC Bearer Token Refresh Filter > generates a new token. In deployments where the configured User Group > Provider does not provide the group information, this behavior can result in > authorization failures after refreshing the token. > The Bearer Token Refresh Filter should be corrected to retrieve group > membership information from the new Identity Provider token. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12418 Correct Provider Groups Missing in Refreshed Tokens [nifi]
exceptionfactory opened a new pull request, #8126: URL: https://github.com/apache/nifi/pull/8126 # Summary [NIFI-12418](https://issues.apache.org/jira/browse/NIFI-12418) Corrects OpenID Connect Bearer Token refresh processing, preserving Identity Provider Group information after a token refresh occurs. Following refactoring in NiFi 1.24.0 to pass Identity Provider Group membership in the `groups` claim of Application Bearer Tokens, the group membership is lost when an automatic Bearer Token refresh occurs. This applies only to OpenID Connect integration, and only impacts users with Identity Provider Group membership that is not supplied through a User Group Provider. The correction streamlines the `LoginAuthenticationToken` class, removing unnecessary constructors to avoid ambiguity. It also introduces an Issuer Provider to return a standard application Bearer Token Issuer based on the configured host and port properties. Further changes to `LoginAuthenticationToken` include changing parameters to use `java.time.Instant` for greater clarity of the expiration time, as opposed to the primitive `long` number of seconds before expiration. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12474: remove TriggerWhenEmpty for DeleteHDFS [nifi]
emiliosetiadarma opened a new pull request, #8125: URL: https://github.com/apache/nifi/pull/8125 # Summary [NIFI-12474](https://issues.apache.org/jira/browse/NIFI-12474) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12474) Remove TriggerWhenEmpty annotation in DeleteHDFS
Emilio Setiadarma created NIFI-12474: Summary: Remove TriggerWhenEmpty annotation in DeleteHDFS Key: NIFI-12474 URL: https://issues.apache.org/jira/browse/NIFI-12474 Project: Apache NiFi Issue Type: Sub-task Reporter: Emilio Setiadarma Assignee: Emilio Setiadarma The DeleteHDFS processor has a call to `getUserGroupInformation()` which performs all the necessary Kerberos related stuff (logging in, and refreshing tickets). So @TriggerWhenEmpty could be safely removed from this processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11677) DeleteHDFS runs tasks with empty incoming connection
[
https://issues.apache.org/jira/browse/NIFI-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793441#comment-17793441
]
Emilio Setiadarma commented on NIFI-11677:
--
I think `@TriggerWhenEmpty` can be removed from `DeleteHDFS`. `DeleteHDFS` does
have a call to `getUserGroupInformation()` which does all the Kerberos stuff.
Will remove `@TriggerWhenEmpty` in a followup Jira.
> DeleteHDFS runs tasks with empty incoming connection
>
>
> Key: NIFI-11677
> URL: https://issues.apache.org/jira/browse/NIFI-11677
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.18.0, 1.19.1
> Environment: 64bit redhat linux, java 1.8.0_352, 3 node cluster
>Reporter: Isha Lamboo
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 2.0.0-M1, 1.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The DeleteHDFS processor can operate either with or without an incoming
> connection. When operating *with* an incoming connection that has no
> flowfiles, it still registers as executing tasks and taking a small amount of
> time (in the range of 50-100 millis on our clusters). In our case this
> results in ~200 processors each running 800-1000 tasks per 5 minutes on a 3
> node cluster with 16 CPUs per node.
> This is in contrast to the PutHDFS with an incoming connection which shows as
> 0 tasks/0 millis.
> Reproduce the situation by connecting a (stopped) ListHDFS processor to a
> DeleteHDFS processor with a variable for the path. Start the DeleteHDFS and
> refresh for statistics update.
> Having compared the code for PutHDFS and DeleteHDFS I think I see the cause:
> PutHDFS has this in its OnTrigger method:
> {code:java}
> final FlowFile flowFile = session.get();
> if (flowFile == null) {
> return;
> } {code}
> DeleteHDFS has this in the OnTrigger method:
> {code:java}
> final FlowFile originalFlowFile = session.get();
> // If this processor has an incoming connection, then do not run
> unless a
> // FlowFile is actually sent through
> if (originalFlowFile == null && context.hasIncomingConnection()) {
> context.yield();
> return;
> } {code}
> I'm guessing that the extra context.yield() causes the framework to execute
> some administration tasks and update counters, where it does not for PutHDFS.
> I don't know how much overhead this causes, probably very little, but I'm
> trying to improve performance on a cluster with thread starvation issues and
> high CPU usage and these keep popping up in the summaries looking bad when I
> think they shouldn't.
> The likely solution is to only perform the context.yield() when
> hasIncomingConnection() returns false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12452 Improve support for DescribedValue [nifi]
markap14 commented on PR #8102: URL: https://github.com/apache/nifi/pull/8102#issuecomment-1841666326 My preference is to keep the name `defaultValue` and add the override. I would avoid allowing for a `defaultValue` that accepts an `Object`. While this could be a change that breaks backward compatibility for a few, I think it's a pretty niche concern and still fair game for a 2.0, as long as it's mentioned in the Migration Guide. Also of note, custom code could be updated to use `.defaultValue( (String) null )` even in 1.x so there would be no need to maintain two different versions of the code, etc. So I think it's fair game. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12473 Delete removeByPattern Methods from Cache Services [nifi]
joewitt commented on PR #8124: URL: https://github.com/apache/nifi/pull/8124#issuecomment-1841659341 Thank you for cleaning this up. +1 assuming the checks run clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12473) Delete Pattern-based Remove Methods from Cache Client Services
[ https://issues.apache.org/jira/browse/NIFI-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12473: Status: Patch Available (was: Open) > Delete Pattern-based Remove Methods from Cache Client Services > -- > > Key: NIFI-12473 > URL: https://issues.apache.org/jira/browse/NIFI-12473 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 2.0.0-M1 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The DistributedMapCacheClient Controller Service interface has methods to > remove elements based on cache keys matching a regular expression. Cache > Service implementations have implemented this method at different levels, but > no Processors or other components use these interface methods. With the only > reference to these methods contained in test classes, the removeByPattern > methods should be removed from the Controller Service interface and from > implementation classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12473 Delete removeByPattern Methods from Cache Services [nifi]
exceptionfactory opened a new pull request, #8124: URL: https://github.com/apache/nifi/pull/8124 # Summary [NIFI-12473](https://issues.apache.org/jira/browse/NIFI-12473) Deletes the `removeByPattern` methods from the `DistributedCacheClientService` interface and implementing classes. Some Cache Client Services do not support the method, and instead throw exceptions. Current components do not use this method, so removing it does not have any impact on current references to Cache Client Services. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12331) Introduce a PublishSlack processor
[ https://issues.apache.org/jira/browse/NIFI-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12331: Resolution: Fixed Status: Resolved (was: Patch Available) > Introduce a PublishSlack processor > -- > > Key: NIFI-12331 > URL: https://issues.apache.org/jira/browse/NIFI-12331 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > The Slack API provides multiple different ways to publish messages to a Slack > channel. NiFi already has two Processors for pushing to Slack - PostSlack and > PutSlack. These processors have slightly different nuances, and the > documentation does not articulate when to which one. One of them is oriented > more toward sending FlowFile contents as an attachment while the other is > oriented toward posting a message based on a property value. We should > consolidate both of these Processors into a single Processor that is capable > of sending a message and optionally providing the FlowFile content as an > attachment. > Both PostSlack and PutSlack make use of WebHooks instead of using the > official Slack SDK. This means that rather than simply specifying the name of > the Channel to post to, in order to send a message in Slack, the creator of > the Slack App must explicitly add a Webhook for the desired channel, and the > Processor must then be configured to use that Webhook. As a result, the > channel cannot be easily configured and cannot be dynamic. This makes it > difficult to use in conjunction with ListenSlack / ConsumeSlack in order to > respond in threads. > We need to consolidate both into a single Processor that is configured and > behaves differently, based on the SDK. > This Processor should be configured with properties that allow specifying: > * Bot Token > * Name of the channel to send to > * How to obtain the message content (FlowFile Content or specified as a > Property that accepts Expression Language) > * If using a Property value, should be configured also with the message to > send, and whether or not to attach the FlowFile content as an attachment to > the message. > * Thread Timestamp (optional to convey which thread the message should be > sent to) - should support Expression Language -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12331) Introduce a PublishSlack processor
[ https://issues.apache.org/jira/browse/NIFI-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12331: Fix Version/s: 1.25.0 2.0.0 Status: Patch Available (was: Open) > Introduce a PublishSlack processor > -- > > Key: NIFI-12331 > URL: https://issues.apache.org/jira/browse/NIFI-12331 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > The Slack API provides multiple different ways to publish messages to a Slack > channel. NiFi already has two Processors for pushing to Slack - PostSlack and > PutSlack. These processors have slightly different nuances, and the > documentation does not articulate when to which one. One of them is oriented > more toward sending FlowFile contents as an attachment while the other is > oriented toward posting a message based on a property value. We should > consolidate both of these Processors into a single Processor that is capable > of sending a message and optionally providing the FlowFile content as an > attachment. > Both PostSlack and PutSlack make use of WebHooks instead of using the > official Slack SDK. This means that rather than simply specifying the name of > the Channel to post to, in order to send a message in Slack, the creator of > the Slack App must explicitly add a Webhook for the desired channel, and the > Processor must then be configured to use that Webhook. As a result, the > channel cannot be easily configured and cannot be dynamic. This makes it > difficult to use in conjunction with ListenSlack / ConsumeSlack in order to > respond in threads. > We need to consolidate both into a single Processor that is configured and > behaves differently, based on the SDK. > This Processor should be configured with properties that allow specifying: > * Bot Token > * Name of the channel to send to > * How to obtain the message content (FlowFile Content or specified as a > Property that accepts Expression Language) > * If using a Property value, should be configured also with the message to > send, and whether or not to attach the FlowFile content as an attachment to > the message. > * Thread Timestamp (optional to convey which thread the message should be > sent to) - should support Expression Language -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12331: Added PublishSlack Processor [nifi]
exceptionfactory commented on PR #8122: URL: https://github.com/apache/nifi/pull/8122#issuecomment-1841627615 Merged in https://github.com/apache/nifi/commit/b27e0f87818af90d745418530c13b11cc510c56e -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12331: Added PublishSlack Processor [nifi]
exceptionfactory closed pull request #8122: NIFI-12331: Added PublishSlack Processor URL: https://github.com/apache/nifi/pull/8122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12331) Introduce a PublishSlack processor
[ https://issues.apache.org/jira/browse/NIFI-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793435#comment-17793435 ] ASF subversion and git services commented on NIFI-12331: Commit b27e0f87818af90d745418530c13b11cc510c56e in nifi's branch refs/heads/support/nifi-1.x from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b27e0f8781 ] NIFI-12331 Added PublishSlack Processor - Deprecated PutSlack and PostSlack in favor of PublishSlack This closes #8122 Signed-off-by: David Handermann > Introduce a PublishSlack processor > -- > > Key: NIFI-12331 > URL: https://issues.apache.org/jira/browse/NIFI-12331 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > The Slack API provides multiple different ways to publish messages to a Slack > channel. NiFi already has two Processors for pushing to Slack - PostSlack and > PutSlack. These processors have slightly different nuances, and the > documentation does not articulate when to which one. One of them is oriented > more toward sending FlowFile contents as an attachment while the other is > oriented toward posting a message based on a property value. We should > consolidate both of these Processors into a single Processor that is capable > of sending a message and optionally providing the FlowFile content as an > attachment. > Both PostSlack and PutSlack make use of WebHooks instead of using the > official Slack SDK. This means that rather than simply specifying the name of > the Channel to post to, in order to send a message in Slack, the creator of > the Slack App must explicitly add a Webhook for the desired channel, and the > Processor must then be configured to use that Webhook. As a result, the > channel cannot be easily configured and cannot be dynamic. This makes it > difficult to use in conjunction with ListenSlack / ConsumeSlack in order to > respond in threads. > We need to consolidate both into a single Processor that is configured and > behaves differently, based on the SDK. > This Processor should be configured with properties that allow specifying: > * Bot Token > * Name of the channel to send to > * How to obtain the message content (FlowFile Content or specified as a > Property that accepts Expression Language) > * If using a Property value, should be configured also with the message to > send, and whether or not to attach the FlowFile content as an attachment to > the message. > * Thread Timestamp (optional to convey which thread the message should be > sent to) - should support Expression Language -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12331) Introduce a PublishSlack processor
[ https://issues.apache.org/jira/browse/NIFI-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793432#comment-17793432 ] ASF subversion and git services commented on NIFI-12331: Commit a21993ef7296113b78697a3fffa8527e343a6583 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a21993ef72 ] NIFI-12331 Added PublishSlack Processor - Removed deprecated PutSlack and PostSlack This closes #8120 Signed-off-by: David Handermann > Introduce a PublishSlack processor > -- > > Key: NIFI-12331 > URL: https://issues.apache.org/jira/browse/NIFI-12331 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > The Slack API provides multiple different ways to publish messages to a Slack > channel. NiFi already has two Processors for pushing to Slack - PostSlack and > PutSlack. These processors have slightly different nuances, and the > documentation does not articulate when to which one. One of them is oriented > more toward sending FlowFile contents as an attachment while the other is > oriented toward posting a message based on a property value. We should > consolidate both of these Processors into a single Processor that is capable > of sending a message and optionally providing the FlowFile content as an > attachment. > Both PostSlack and PutSlack make use of WebHooks instead of using the > official Slack SDK. This means that rather than simply specifying the name of > the Channel to post to, in order to send a message in Slack, the creator of > the Slack App must explicitly add a Webhook for the desired channel, and the > Processor must then be configured to use that Webhook. As a result, the > channel cannot be easily configured and cannot be dynamic. This makes it > difficult to use in conjunction with ListenSlack / ConsumeSlack in order to > respond in threads. > We need to consolidate both into a single Processor that is configured and > behaves differently, based on the SDK. > This Processor should be configured with properties that allow specifying: > * Bot Token > * Name of the channel to send to > * How to obtain the message content (FlowFile Content or specified as a > Property that accepts Expression Language) > * If using a Property value, should be configured also with the message to > send, and whether or not to attach the FlowFile content as an attachment to > the message. > * Thread Timestamp (optional to convey which thread the message should be > sent to) - should support Expression Language -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12331: Added PublishSlack Processor [nifi]
exceptionfactory closed pull request #8120: NIFI-12331: Added PublishSlack Processor URL: https://github.com/apache/nifi/pull/8120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8932: Add capability to skip first N rows in CSVReader [nifi]
dan-s1 commented on code in PR #7952:
URL: https://github.com/apache/nifi/pull/7952#discussion_r1416200289
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/csv/AbstractCSVRecordReader.java:
##
@@ -158,4 +180,46 @@ protected String trim(String value) {
public RecordSchema getSchema() {
return schema;
}
+
+/**
+ * This method searches using the specified Reader character-by-character
until the
+ * record separator is found.
+ * @param reader the Reader providing the input
+ * @param recordSeparator the String specifying the end of a record in the
input
+ * @throws IOException if an error occurs during reading, including not
finding the record separator in the input
+ */
+protected void readNextRecord(Reader reader, String recordSeparator)
throws IOException {
+int indexIntoSeparator = 0;
+int recordSeparatorLength = recordSeparator.length();
+int code = reader.read();
+while (code != -1) {
+char nextChar = (char)code;
+if (recordSeparator.charAt(indexIntoSeparator) == nextChar) {
+if (++indexIntoSeparator == recordSeparatorLength) {
+// We have matched the separator, return the string built
so far
+return;
+}
Review Comment:
@mattyb149 @exceptionfactory A concern I have with this logic is what
happens when the record separator is escaped in the data? How will you
distinguish whether you have an end of record or an escaped record separator.
I did a quick look on line and found that a new line character which usually is
the end of a CSV record can be embedded in the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793412#comment-17793412 ] Joe Witt commented on NIFI-12236: - I should clarify as part of that I think was confusing... Ultimately we should just allow them to have a database instance that they run as per normal. All such desired DBs have installers/tools/to have a healthy instance setup. Then the nifi side would be configured to talk to it for sending/querying. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793409#comment-17793409 ] Joe Witt commented on NIFI-12236: - I went back to quickly review the presumably referenced mailing list thread titled 'NiFi 2.0 - QuestDB'. The concerns I noted then remain for me. It would certainly be nice for the users to have access to persistent and longer duration metric data. That said I'm not sure our user interface for this is very good when it comes to holding this information and making interesting visualizations of it for longer durations. For the type of usage the in memory durations we can reasonably sustain today seem about right. What I am seeing more and more is that users of NiFi want this type of information available in their favorite monitoring or observability tool whether that is Datadog or Prometheus or something else. If our goal is a short term but durable store then perhaps we ought to give them a simple QuestDB based service/process they can run on some node. Then their NiFi nodes are configured to send/query metrics from that service rather than it having to live on every node. This also means it would be better externalized such that maybe they dont even use Quest but instead MySQL or Postgre or whatever they prefer. In a k8s based deployment I can certainly see such a model working well. For users that are looking for more robust data retention and query and analysis we're better off focusing on getting the data to their preferred tools. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
exceptionfactory commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416213317
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
The schemas are persistent in that they will be maintained in the flow
configuration (flow.json.gz), even though temporarily stored in memory.
The `AvroSchemaRegistry` is an interesting example, as it simply describes
what it is, with no particular indication of how it works.
From that angle, I am inclined to just go with `StandardJsonSchemaRegistry`,
and keep the interface name as already named.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
joewitt commented on PR #8123: URL: https://github.com/apache/nifi/pull/8123#issuecomment-1841517110 I have questions about the direction in general so I will take those over to the JIRA. This PR obviously was a ton of work and my questions/thoughts aren't about that right now so I think the JIRA is a better place to discuss/share those. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8932: Add capability to skip first N rows in CSVReader [nifi]
dan-s1 commented on code in PR #7952:
URL: https://github.com/apache/nifi/pull/7952#discussion_r1416200289
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/csv/AbstractCSVRecordReader.java:
##
@@ -158,4 +180,46 @@ protected String trim(String value) {
public RecordSchema getSchema() {
return schema;
}
+
+/**
+ * This method searches using the specified Reader character-by-character
until the
+ * record separator is found.
+ * @param reader the Reader providing the input
+ * @param recordSeparator the String specifying the end of a record in the
input
+ * @throws IOException if an error occurs during reading, including not
finding the record separator in the input
+ */
+protected void readNextRecord(Reader reader, String recordSeparator)
throws IOException {
+int indexIntoSeparator = 0;
+int recordSeparatorLength = recordSeparator.length();
+int code = reader.read();
+while (code != -1) {
+char nextChar = (char)code;
+if (recordSeparator.charAt(indexIntoSeparator) == nextChar) {
+if (++indexIntoSeparator == recordSeparatorLength) {
+// We have matched the separator, return the string built
so far
+return;
+}
Review Comment:
@mattyb149 @exceptionfactory A concern I have with this logic is what
happens when the record separator is escaped in the data? How will you tell
whether you have an end of record or an escaped record separator. I did a
quick look on line and found that a new line character which usually is the end
of a CSV record can be embedded in the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416188685
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
Is `PersistentJsonSchemaRegistry ` accurate? Would the contents persist
after a NIFI shutdown? I thought this would be similar to `AvroSchemaRegistry`
which I thought does not persist. Please correct me if am wrong.
I like `PropertiesJsonSchemaRegistry` better although I am not sure if there
is a consideration to align the naming convention with `AvroSchemaRegistry` or
not. As an end user I would expect similar naming conventions in order to
associate items with. Essentially the `InMemoryJsonSchemaRegistry` conceptually
is the same as `AvroSchemaRegistry` so I would expect a similar name conveying
the point that this stores JSON schemas in NIFI instead of Avro Schemas.
Could we perhaps change the interface name to `JsonSchemaStore` and name
this implementation `JsonSchemaRegistry` so it aligns more with
`AvroSchemaRegistry`? I know it does not align perfectly with the hierarchy for
Avro schemas (as its interface is SchemaRegistry which I would have used for
all type of schemas). Other implementations would use the suffix `Registry`
instead of `Store`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416188685
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
Is `PersistentJsonSchemaRegistry ` accurate? Would the contents persist
after a NIFI shutdown? I thought this would be similar to `AvroSchemaRegistry`
which I thought does not persist. Please correct me if am wrong.
I like `PropertiesJsonSchemaRegistry` better although I am not sure if there
is a consideration to align the naming convention with `AvroSchemaRegistry` or
not. As an end user I would expect similar naming conventions in order to
associate items with. Essentially the `InMemoryJsonSchemaRegistry` conceptually
is the same as `AvroSchemaRegistry` so I would expect a similar name conveying
the point that this stores JSON schemas in NIFI instead of Avro Schemas.
Could we perhaps change the interface name to `JsonSchemaStore` and name
this implementation `JsonSchemaRegistry` so it aligns more with
`AvroSchemaRegistry`? I know it does not align perfectly with the hierarchy for
Avro schemas (as its interface is SchemaRegistry which I would have used for
all type of schemas). Other implementations would use the suffix `Registry
`instead of `Store`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416188685
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
Is `PersistentJsonSchemaRegistry ` accurate? Would the contents persist
after a NIFI shutdown? I thought this would be similar to `AvroSchemaRegistry`
which I thought does not persist. Please correct me if am wrong.
I like `PropertiesJsonSchemaRegistry` better although I am not sure if there
is a consideration to align the naming convention with `AvroSchemaRegistry` or
not. As an end user I would expect similar naming conventions in order to
associate items with. Essentially the `InMemoryJsonSchemaRegistry` conceptually
is the same as `AvroSchemaRegistry` so I would expect a similar name conveying
the point that this stores JSON schemas in NIFI instead of Avro Schemas.
Could we perhaps change the interface name to `JsonSchemaStore `and name
this implementation `JsonSchemaRegistry` so it aligns more with
`AvroSchemaRegistry`? I know it does not align perfectly with the hierarchy for
Avro schemas (as its interface is SchemaRegistry which I would have used for
all type of schemas). Other implementations would use the suffix `Registry
`instead of `Store`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
exceptionfactory commented on code in PR #8123:
URL: https://github.com/apache/nifi/pull/8123#discussion_r1416185237
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml:
##
@@ -108,7 +108,7 @@
10
-
org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
+
org.apache.nifi.controller.status.history.questdb.EmbeddedQuestDbStatusHistoryRepository
Review Comment:
I recommend reverting this particular change for this pull request. There is
a good deal to review, so if we get to the place where this is ready to be the
default setting, we should consider that after reviewing the substantive
changes.
##
nifi-api/src/main/java/org/apache/nifi/controller/status/ConnectionStatus.java:
##
@@ -22,6 +22,7 @@
/**
*/
public class ConnectionStatus implements Cloneable {
+private long createdAtInMs;
Review Comment:
This addition raises several questions and concerns. It does not seem like
the NiFi API definition should include the creation timestamp of components in
all circumstances. It would make much more sense to track this as a feature
internal to the framework.
As far as naming the `InMs` suffix is awkward, and it would be much better
to use `java.time.Instant` for model classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416188685
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
Is `PersistentJsonSchemaRegistry ` accurate? Would the contents persist
after a NIFI shutdown? I thought this would be similar to `AvroSchemaRegistry`
which I thought does not persist. Please correct me if am wrong.
I like `PropertiesJsonSchemaRegistry` better although I am not sure if there
is a consideration to align the naming convention with `AvroSchemaRegistry` or
not. As an end user I would expect similar naming conventions in order to
associate items with. Essentially the `InMemoryJsonSchemaRegistry` conceptually
is the same as `AvroSchemaRegistry` so I would expect a similar name conveying
the point that this stores JSON schemas in NIFI instead of Avro Schemas.
Could we perhaps change the interface name to `JsonSchemaStore `and name
this implementation `JsonSchemaRegistry ` so it aligns more with
`AvroSchemaRegistry`? I know it does not align perfectly with the hierarchy for
Avro schemas (as its interface is SchemaRegistry which I would have used for
all type of schemas). Other implementations would use the suffix `Registry
`instead of `Store`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this
[PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
simonbence opened a new pull request, #8123: URL: https://github.com/apache/nifi/pull/8123 # Summary This is a reworking of the original `QuestDB` based status history based on community feedback. The main intention is to improve the maturity of the feature in order to be a possible default setting for 2.0. This improvement mainly focuses but not limited to: - Centralisation and separation of the `QuestDB` related code in order to better maintenance - Add capabilities to the client and database management which helps us dealing with issues (prioritising the continuity of the `NiFi` core functionality over the availability of the database) - Clarifications on the related repository code including exposing more customisation options in the properties among other things. The review if fairly big and complication. My suggested approach for easier understanding is the following: - The retry capability is something might be tackled separately, it provides a robust solution to retry smaller operations and used by the `quest-db` module. The `SynchronousRetryTemplateTest` is suggested for quicker grasping. - The newly added `quest-db` module is a good starting point for the actual `QuestDB` functionality. The included package description provides some overview of the main areas. The unit tests most helpful for understanding are the `CompositeClientTest` and the `EmbeddedDatabaseManagerTest`. Up to these point the code are considered standalone and however it follows `NiFi` conventions in the most regards, not depending on `NiFi` features. - Last part of the PR is the changes around the application of the code above. The tests were covering the previous implementation (`EmbeddedQuestDbStatusHistoryRepositoryForComponentsTest` and `EmbeddedQuestDbStatusHistoryRepositoryForNodeTest`) are still in place and other than some smaller technical changes they still guard the function the same way. Other then these, the code organisation changed heavily. Most of the business logic around handling the snapshots are the same but as the `QuestDB` specific code are moved out from the core, the structure evolved. I suggest to start with the `EmbeddedQuestDbStatusHistoryRepository` itself and go top-down. Majority of the related code is within the same package, other changes around are to integrate it properly. Please if any question is raising about either the approach or the implementation, do not hesitate to leave a comment about that, some additional context might be helpful to communicate my intent with the details of the change. Thanks! [NIFI-12236](https://issues.apache.org/jira/browse/NIFI-12236) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
exceptionfactory commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416162547
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
A one-word prefix would be ideal, so the concept is good.
`PersistentJsonSchemaRegistry` might work, as it provides a stronger indication
of the lifecycle than `InMemory`. There is always the generic
`StandardJsonSchemaRegistry`, but that doesn't say anything about how it works.
What do you think about `PropertiesJsonSchemaRegistry`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416126018
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
`InternalJsonSchemaRegistry` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12473) Delete Pattern-based Remove Methods from Cache Client Services
David Handermann created NIFI-12473: --- Summary: Delete Pattern-based Remove Methods from Cache Client Services Key: NIFI-12473 URL: https://issues.apache.org/jira/browse/NIFI-12473 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 2.0.0-M1 Reporter: David Handermann Assignee: David Handermann The DistributedMapCacheClient Controller Service interface has methods to remove elements based on cache keys matching a regular expression. Cache Service implementations have implemented this method at different levels, but no Processors or other components use these interface methods. With the only reference to these methods contained in test classes, the removeByPattern methods should be removed from the Controller Service interface and from implementation classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12331: Added PublishSlack Processor [nifi]
markap14 commented on PR #8120: URL: https://github.com/apache/nifi/pull/8120#issuecomment-1841306234 That's a good catch @exceptionfactory . Unfortunately most of the dependencies are still necessary for the Record Sink stuff. Eventually that should be rewritten to use the SDK as well, and then we could eliminate most all of the dependencies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416045115
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
`LocalJsonSchemaRegistry` as opposed to others which are stored elsewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
exceptionfactory commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416049310
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
That's an idea, although it seems to imply that it might not work in a
cluster or network, which is not the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416045115
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
+@Tags({"schema", "registry", "json"})
+@CapabilityDescription("Provides a service for registering and accessing
schemas. One can register a schema "
++ "as a dynamic property where 'name' represents the schema name and
'value' represents the textual "
++ "representation of the actual schema following the syntax and
semantics of JSON's Schema format. "
++ "Empty schemas and schemas only consisting of whitespace are not
acceptable schemas."
++ "The registry is heterogeneous registry as it can store schemas of
multiple schema draft versions. "
++ "By default the registry is configured to store schemas of Draft
2020-12. When a schema is added, the version "
++ "which is currently is set, is what the schema is saved as.")
+@DynamicProperty(name = "Schema name", value = "Schema Content",
+description = "Adds a named schema using the JSON string
representation of a JSON schema",
+expressionLanguageScope = ExpressionLanguageScope.NONE)
+public class InMemoryJsonSchemaRegistry extends AbstractControllerService
implements JsonSchemaRegistry, JsonSchemaRegistryComponent {
Review Comment:
`LocalJsonSchemaRegistry` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
exceptionfactory commented on code in PR #8005:
URL: https://github.com/apache/nifi/pull/8005#discussion_r1416003586
##
nifi-commons/nifi-json-schema-api/src/main/java/org/apache/nifi/schema/access/JsonSchema.java:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schema.access;
+
+import java.util.Objects;
+
+public class JsonSchema {
+private final SchemaVersion schemaDraftVersion;
Review Comment:
Recommend renaming this to just `schemaVersion` removing `draft` from the
name.
```suggestion
private final SchemaVersion schemaVersion;
```
##
nifi-commons/nifi-json-schema-api/src/main/java/org/apache/nifi/schema/access/JsonSchema.java:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schema.access;
+
+import java.util.Objects;
+
+public class JsonSchema {
+private final SchemaVersion schemaDraftVersion;
+private final String schemaText;
+
+public JsonSchema(SchemaVersion schemaDraftVersion, String schemaText) {
+Objects.requireNonNull(schemaDraftVersion, "Schema draft version
cannot be null");
+Objects.requireNonNull(schemaText, "The text of the schema cannot be
null");
+this.schemaDraftVersion = schemaDraftVersion;
+this.schemaText = schemaText;
+}
+
+public SchemaVersion getSchemaDraftVersion() {
Review Comment:
```suggestion
public SchemaVersion getSchemaVersion() {
```
##
nifi-nar-bundles/nifi-registry-bundle/nifi-registry-service/src/main/java/org/apache/nifi/schemaregistry/services/InMemoryJsonSchemaRegistry.java:
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.schemaregistry.services;
+
+import com.networknt.schema.JsonSchemaFactory;
+import com.networknt.schema.SpecVersion;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.JsonSchema;
+import org.apache.nifi.schema.access.JsonSchemaRegistryComponent;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.access.SchemaVersion;
+
+import java.util.Arrays;
+import
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
exceptionfactory commented on PR #8005: URL: https://github.com/apache/nifi/pull/8005#issuecomment-1841235544 > @exceptionfactory This is the exception I see its encountering which has nothing to do with my changes. `Error: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.14.2:install-node-and-npm (install-node-and-npm) on project nifi-web-ui: Could not download Node.js: Could not download https://nodejs.org/dist/v16.13.2/node-v16.13.2-linux-x64.tar.gz: Premature end of Content-Length delimited message body (expected: 32,399,497; received: 1,064,660) -> [Help 1]` > > Does this point to something more problematic with this build or is it just a blip of some sort? That's not a related problem, unfortunately downloading Node.js for front end builds seems unreliable, leading to these intermittent failures. The fact that the other builds completed is sufficient in this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12382: Add DatabaseSchemaRegistry service [nifi]
exceptionfactory commented on code in PR #8042:
URL: https://github.com/apache/nifi/pull/8042#discussion_r1415801708
##
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-db-schema-registry-service/src/main/java/org/apache/nifi/db/schemaregistry/DatabaseTableSchemaRegistry.java:
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.db.schemaregistry;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.dbcp.DBCPService;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+
+@Tags({"schema", "registry", "database", "table"})
+@CapabilityDescription("Provides a service for generating a record schema from
a database table definition. The service is configured "
++ "to use a table name and a database connection fetches the table
metadata (i.e. table definition) such as column names, data types, "
++ "nullability, etc.")
+public class DatabaseTableSchemaRegistry extends AbstractControllerService
implements SchemaRegistry {
+
+private static final Set schemaFields =
EnumSet.of(SchemaField.SCHEMA_NAME);
+
+static final PropertyDescriptor DBCP_SERVICE = new
PropertyDescriptor.Builder()
+.name("Database Connection Pooling Service")
+.displayName("Database Connection Pooling Service")
+.description("The Controller Service that is used to obtain a
connection to the database for retrieving table information.")
+.required(true)
+.identifiesControllerService(DBCPService.class)
+.build();
+
+static final PropertyDescriptor CATALOG_NAME = new
PropertyDescriptor.Builder()
+.name("Catalog Name")
+.displayName("Catalog Name")
+.description("The name of the catalog used to locate the desired
table. This may not apply for the database that you are querying. In this case,
leave the field empty. Note that if the "
++ "property is set and the database is case-sensitive, the
catalog name must match the database's catalog name exactly.")
+.required(false)
+.expressionLanguageSupported(ExpressionLanguageScope.ENVIRONMENT)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor SCHEMA_NAME = new
PropertyDescriptor.Builder()
+.name("Schema Name")
+.displayName("Schema Name")
+.description("The name of the schema that the table belongs to.
This may not apply for the database that you are updating. In this case, leave
the field empty. Note that if the "
++ "property is set and the database is case-sensitive, the
schema name must match the database's schema name exactly. Also notice that if
the same table name exists in multiple "
++ "schemas and Schema Name is not specified, the service
will find those tables and give an error if the different tables have the same
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on PR #8005: URL: https://github.com/apache/nifi/pull/8005#issuecomment-1841173185 @exceptionfactory Sorry to bother you to restart this. This is the exception I see its encountering `Error: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.14.2:install-node-and-npm (install-node-and-npm) on project nifi-web-ui: Could not download Node.js: Could not download https://nodejs.org/dist/v16.13.2/node-v16.13.2-linux-x64.tar.gz: Premature end of Content-Length delimited message body (expected: 32,399,497; received: 1,064,660) -> [Help 1]` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12472) NiFi Registry Client requires that users have view on "Access the controller".
Matthew Clarke created NIFI-12472: - Summary: NiFi Registry Client requires that users have view on "Access the controller". Key: NIFI-12472 URL: https://issues.apache.org/jira/browse/NIFI-12472 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.18.0 Reporter: Matthew Clarke If a user tries to start or change version control of a process group, the UI will show non-stop spinner in toolbar and never display the pop-up Flow Version UI. Through inspect in the browser I can see this exception: nf-canvas-all.js?1.18.0:47 Uncaught TypeError: Cannot read properties of undefined (reading 'name') at Object. (nf-canvas-all.js?1.18.0:47:3629) at Function.each (jquery.min.js:2:3003) at b.each (jquery.each.js:1:96) at Object. (nf-canvas-all.js?1.18.0:47:3562) at c (jquery.min.js:2:28327) at Object.fireWith [as resolveWith] (jquery.min.js:2:29072) at l (jquery.min.js:2:79901) at XMLHttpRequest. (jquery.min.js:2:82355) If I grant the user "view" on the "Access the controller" global policy, this issue goes away. This was not the case prior to NiFi 1.18 I appears that https://issues.apache.org/jira/browse/NIFI-10497 introduced this new policy requirement. The documentation does not reflect this change nor does it make sense to me why a user needs access the controller view access in order to start or change version control for a process group the user has full authorized access to and the user has full authorization within Registry to a bucket. Needing to grant users view on "access the controller" also provides the authorized users with view on other components that that user really has no need to see. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12371 - Support tombstone messages in non-record Kafka processors [nifi]
exceptionfactory commented on code in PR #8076:
URL: https://github.com/apache/nifi/pull/8076#discussion_r1415789879
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-2-6-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublisherLease.java:
##
@@ -159,9 +160,15 @@ void publish(final FlowFile flowFile, final InputStream
flowFileContent, final b
tracker.fail(flowFile, new TokenTooLargeException("A
message in the stream exceeds the maximum allowed message size of " +
maxMessageSize + " bytes."));
return;
}
-// Send FlowFile content as it is, to support sending 0 byte
message.
-messageContent = new byte[(int) flowFile.getSize()];
-StreamUtils.fillBuffer(flowFileContent, messageContent);
+
if(flowFile.getAttribute(KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE) != null
+&&
flowFile.getAttribute(KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE).equals("true")
Review Comment:
Can this `getAttribute()` call return `null`? It looks like it would be
safer to reverse the comparison as follows:
```suggestion
&&
Boolean.TRUE.toString().equals(flowFile.getAttribute(KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE))
```
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-2-6-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublisherLease.java:
##
@@ -159,9 +160,15 @@ void publish(final FlowFile flowFile, final InputStream
flowFileContent, final b
tracker.fail(flowFile, new TokenTooLargeException("A
message in the stream exceeds the maximum allowed message size of " +
maxMessageSize + " bytes."));
return;
}
-// Send FlowFile content as it is, to support sending 0 byte
message.
-messageContent = new byte[(int) flowFile.getSize()];
-StreamUtils.fillBuffer(flowFileContent, messageContent);
+
if(flowFile.getAttribute(KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE) != null
Review Comment:
Spacing:
```suggestion
if
(flowFile.getAttribute(KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE) != null
```
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-2-6-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublishKafka_2_6.java:
##
@@ -87,6 +88,8 @@
+ " In the event a dynamic property represents a property that was
already set, its value will be ignored and WARN message logged."
+ " For the list of available Kafka properties please refer to:
http://kafka.apache.org/documentation.html#configuration. ",
expressionLanguageScope = ExpressionLanguageScope.ENVIRONMENT)
+@ReadsAttribute(attribute = KafkaFlowFileAttribute.KAFKA_IS_TOMBSTONE,
description = "If this attribute is set to 'true', if the processor is not
configured "
++ "with a demarcator and if the FlowFile's content is null, then a
tombtsone message will be sent to Kafka.")
Review Comment:
Recommend noting that tombstone means zero bytes:
```suggestion
+ "with a demarcator and if the FlowFile's content is null, then a
tombstone message with zero bytes will be sent to Kafka.")
```
##
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-shared/src/main/java/org/apache/nifi/kafka/shared/attribute/KafkaFlowFileAttribute.java:
##
@@ -39,4 +39,6 @@ public interface KafkaFlowFileAttribute {
String KAFKA_CONSUMER_GROUP_ID = "kafka.consumer.id";
String KAFKA_CONSUMER_OFFSETS_COMMITTED =
"kafka.consumer.offsets.committed";
+
+String KAFKA_IS_TOMBSTONE = "kafka.isTombstone";
Review Comment:
What do you think about naming this `kafka.tombstone`, dropping the `is`
prefix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12410) ExcelReader - Starting Row should have flowfile attributes scope
[ https://issues.apache.org/jira/browse/NIFI-12410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793320#comment-17793320 ] ASF subversion and git services commented on NIFI-12410: Commit a5de0d2ccb6917817ebbaf4dc3b00094be01e74d in nifi's branch refs/heads/support/nifi-1.x from dan-s1 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a5de0d2ccb ] NIFI-12410 Support EL for Starting Row in ExcelReader This closes #8119 Signed-off-by: David Handermann (cherry picked from commit ab8a82b99747d50e5bdf00e04069bccf8fe82b1e) > ExcelReader - Starting Row should have flowfile attributes scope > > > Key: NIFI-12410 > URL: https://issues.apache.org/jira/browse/NIFI-12410 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.23.2 >Reporter: Philipp Korniets >Assignee: Daniel Stieglitz >Priority: Minor > Labels: backport-needed > Time Spent: 20m > Remaining Estimate: 0h > > New ExcelReader controller should have Starting Row property with flowfile > attribute scope otherwise controller service can not be reused across > different files -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12410) ExcelReader - Starting Row should have flowfile attributes scope
[ https://issues.apache.org/jira/browse/NIFI-12410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12410: Fix Version/s: 1.25.0 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > ExcelReader - Starting Row should have flowfile attributes scope > > > Key: NIFI-12410 > URL: https://issues.apache.org/jira/browse/NIFI-12410 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.23.2 >Reporter: Philipp Korniets >Assignee: Daniel Stieglitz >Priority: Minor > Labels: backport-needed > Fix For: 1.25.0, 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > New ExcelReader controller should have Starting Row property with flowfile > attribute scope otherwise controller service can not be reused across > different files -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12410) ExcelReader - Starting Row should have flowfile attributes scope
[ https://issues.apache.org/jira/browse/NIFI-12410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793319#comment-17793319 ] ASF subversion and git services commented on NIFI-12410: Commit ab8a82b99747d50e5bdf00e04069bccf8fe82b1e in nifi's branch refs/heads/main from dan-s1 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ab8a82b997 ] NIFI-12410 Support EL for Starting Row in ExcelReader This closes #8119 Signed-off-by: David Handermann > ExcelReader - Starting Row should have flowfile attributes scope > > > Key: NIFI-12410 > URL: https://issues.apache.org/jira/browse/NIFI-12410 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.23.2 >Reporter: Philipp Korniets >Assignee: Daniel Stieglitz >Priority: Minor > Labels: backport-needed > Time Spent: 20m > Remaining Estimate: 0h > > New ExcelReader controller should have Starting Row property with flowfile > attribute scope otherwise controller service can not be reused across > different files -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12410 Refactored to allow for 'Starting Row' to have flowfile attributes scope. [nifi]
exceptionfactory closed pull request #8119: NIFI-12410 Refactored to allow for 'Starting Row' to have flowfile attributes scope. URL: https://github.com/apache/nifi/pull/8119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12334: Implement GCS option for FileResourceService [nifi]
exceptionfactory commented on code in PR #7999:
URL: https://github.com/apache/nifi/pull/7999#discussion_r1415731745
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/fileresource/service/GCSFileResourceService.java:
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.fileresource.service;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.ReadChannel;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.cloud.storage.StorageOptions;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.gcp.credentials.service.GCPCredentialsService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.FetchGCSObject;
+import org.apache.nifi.processors.gcp.util.GoogleUtils;
+
+import java.io.IOException;
+import java.nio.channels.Channels;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static
org.apache.nifi.processors.gcp.storage.StorageAttributes.KEY_DESC;
+
+@Tags({"file", "resource", "gcs"})
+@SeeAlso({FetchGCSObject.class})
+@CapabilityDescription("Provides a Google Compute Storage (GCS) file resource
for other components.")
+@UseCase(
+description = "Fetch a specific file from GCS." +
+" The service provides higher performance compared to fetch
processors when the data should be moved between different storages without any
transformation.",
+configuration = """
+"Bucket" = "${gcs.bucket}"
+"Name" = "${filename}"
+
+The "GCP Credentials Provider Service" property should specify
an instance of the GCPCredentialsService in order to provide credentials for
accessing the bucket.
+"""
+)
+public class GCSFileResourceService extends AbstractControllerService
implements FileResourceService {
+
+public static final PropertyDescriptor BUCKET = new PropertyDescriptor
+.Builder().name("gcs-bucket")
+.displayName("Bucket")
+.description(BUCKET_DESC)
+.required(true)
+.defaultValue("${" + BUCKET_ATTR + "}")
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor KEY = new PropertyDescriptor
+.Builder().name("gcs-key")
+.displayName("Name")
Review Comment:
```suggestion
.Builder()
.name("Name")
.displayName("Name")
```
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/fileresource/service/GCSFileResourceService.java:
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for
[jira] [Commented] (NIFI-12412) ConsumeAzureEventHub should support Blob checkpoint store access via proxy
[ https://issues.apache.org/jira/browse/NIFI-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793301#comment-17793301 ] ASF subversion and git services commented on NIFI-12412: Commit 2af5b8607b7655de6470dc9bc7e7da5d621fcec8 in nifi's branch refs/heads/support/nifi-1.x from Peter Turcsanyi [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2af5b8607b ] NIFI-12412 Support Proxies for Blob Checkpoints in ConsumeAzureEventHub This closes #8107 Signed-off-by: David Handermann (cherry picked from commit 8645088e1d5468b4ea21c55d7d634ec94257c8f5) > ConsumeAzureEventHub should support Blob checkpoint store access via proxy > -- > > Key: NIFI-12412 > URL: https://issues.apache.org/jira/browse/NIFI-12412 > Project: Apache NiFi > Issue Type: Bug >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > NIFI-6240 introduced proxy support for AzureEventHub processors and applied > the proxy configuration for the Event Hubs API calls. ConsumeAzureEventHub > also needs access to Blob Storage for storing checkpoints. The proxy > configuration should be used for the Blob API calls too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12412) ConsumeAzureEventHub should support Blob checkpoint store access via proxy
[ https://issues.apache.org/jira/browse/NIFI-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12412: Fix Version/s: 1.25.0 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > ConsumeAzureEventHub should support Blob checkpoint store access via proxy > -- > > Key: NIFI-12412 > URL: https://issues.apache.org/jira/browse/NIFI-12412 > Project: Apache NiFi > Issue Type: Bug >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > NIFI-6240 introduced proxy support for AzureEventHub processors and applied > the proxy configuration for the Event Hubs API calls. ConsumeAzureEventHub > also needs access to Blob Storage for storing checkpoints. The proxy > configuration should be used for the Blob API calls too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12412) ConsumeAzureEventHub should support Blob checkpoint store access via proxy
[ https://issues.apache.org/jira/browse/NIFI-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793299#comment-17793299 ] ASF subversion and git services commented on NIFI-12412: Commit 8645088e1d5468b4ea21c55d7d634ec94257c8f5 in nifi's branch refs/heads/main from Peter Turcsanyi [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8645088e1d ] NIFI-12412 Support Proxies for Blob Checkpoints in ConsumeAzureEventHub This closes #8107 Signed-off-by: David Handermann > ConsumeAzureEventHub should support Blob checkpoint store access via proxy > -- > > Key: NIFI-12412 > URL: https://issues.apache.org/jira/browse/NIFI-12412 > Project: Apache NiFi > Issue Type: Bug >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > NIFI-6240 introduced proxy support for AzureEventHub processors and applied > the proxy configuration for the Event Hubs API calls. ConsumeAzureEventHub > also needs access to Blob Storage for storing checkpoints. The proxy > configuration should be used for the Blob API calls too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12412 Added proxy support for Blob checkpoint store in ConsumeAz… [nifi]
exceptionfactory closed pull request #8107: NIFI-12412 Added proxy support for Blob checkpoint store in ConsumeAz… URL: https://github.com/apache/nifi/pull/8107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-2270) AWS extension does not compile on Visual Studio 17.8 or above
[ https://issues.apache.org/jira/browse/MINIFICPP-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink reassigned MINIFICPP-2270: -- Assignee: (was: Martin Zink) > AWS extension does not compile on Visual Studio 17.8 or above > - > > Key: MINIFICPP-2270 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2270 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Priority: Major > > Due to [https://github.com/aws/aws-sdk-cpp/pull/2726] > > We should upgrade the sdk -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11627 Added an in memory JSON schema registry which ValidationJson can now use to retrieve JSON schemas from for validation. [nifi]
dan-s1 commented on PR #8005: URL: https://github.com/apache/nifi/pull/8005#issuecomment-1840842508 @exceptionfactory Can you please restart the build which failed? It does not seem to have failed as a result of the changes I made. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1415648626 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: I think most (if not all) of PROCESSORS.md is auto generated, so every attribute that has some level of expression langauge support via the `.supportsExpressionLanguage(true)` will be flagged as EL supported. Not sure if we want to deviate from the auto generated docs (since similar properties should be flagged as EL supported (due to auto generation). I think the best solution would be the introduce the concept of different levels of EL support, like NiFi does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12427) Add channel name attribute in ConsumeSlack
[ https://issues.apache.org/jira/browse/NIFI-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12427: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add channel name attribute in ConsumeSlack > -- > > Key: NIFI-12427 > URL: https://issues.apache.org/jira/browse/NIFI-12427 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > If the ConsumeSlack processor is configured with a list of channel names > (example: #general, #user), the generated flow files would only contain the > channel ID information. It makes sense because the channel name is mutable > while the ID is not. However, it is not frequent that channel names would be > changed so it would be useful to have the channel name retained as a FlowFile > attribute in case it needs to be used downstream in the flow. > (in any case if the channel name is changed on the Slack side, a NiFi user > would likely have to update the configuration of the processor if not using > IDs) > Similarly, if the processor is configured with the ID, it would still be > useful to retrieve the name of the channel and have it as a FlowFile > attribute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12427) Add channel name attribute in ConsumeSlack
[ https://issues.apache.org/jira/browse/NIFI-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793288#comment-17793288 ] ASF subversion and git services commented on NIFI-12427: Commit d0dd4e03e0ba5d011aa65dcbc4406cc2cf4df88f in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d0dd4e03e0 ] NIFI-12427 Add channel name attribute to ConsumeSlack This closes #8078 Signed-off-by: David Handermann > Add channel name attribute in ConsumeSlack > -- > > Key: NIFI-12427 > URL: https://issues.apache.org/jira/browse/NIFI-12427 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > If the ConsumeSlack processor is configured with a list of channel names > (example: #general, #user), the generated flow files would only contain the > channel ID information. It makes sense because the channel name is mutable > while the ID is not. However, it is not frequent that channel names would be > changed so it would be useful to have the channel name retained as a FlowFile > attribute in case it needs to be used downstream in the flow. > (in any case if the channel name is changed on the Slack side, a NiFi user > would likely have to update the configuration of the processor if not using > IDs) > Similarly, if the processor is configured with the ID, it would still be > useful to retrieve the name of the channel and have it as a FlowFile > attribute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12427 - Add channel name attribute in ConsumeSlack [nifi]
exceptionfactory closed pull request #8078: NIFI-12427 - Add channel name attribute in ConsumeSlack URL: https://github.com/apache/nifi/pull/8078 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12318) GenerateRecord could not generate bytes data
[ https://issues.apache.org/jira/browse/NIFI-12318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12318: -- Resolution: Fixed Status: Resolved (was: Patch Available) > GenerateRecord could not generate bytes data > > > Key: NIFI-12318 > URL: https://issues.apache.org/jira/browse/NIFI-12318 > Project: Apache NiFi > Issue Type: Bug > Environment: Apache Nifi 1.23.2 > jdk verson:11.0.20 >Reporter: Ashley >Assignee: Matt Burgess >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Attachments: image-2023-11-06-18-49-08-984.png, > image-2023-11-06-18-49-50-676.png, image-2023-11-06-18-50-59-897.png, > image-2023-11-06-18-51-41-326.png, image-2023-11-06-18-52-07-939.png, > image-2023-11-06-18-52-10-969.png > > Time Spent: 20m > Remaining Estimate: 0h > > I tried to generate bytes data using GenerateRecord,but encountered ERROR. > The ERROR: > !image-2023-11-06-18-52-10-969.png! > GenerateRecord properties: > !image-2023-11-06-18-49-50-676.png! > Schema Text: > !image-2023-11-06-18-49-08-984.png! > AvroRecordSetWriter properteis: > !image-2023-11-06-18-50-59-897.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12318) GenerateRecord could not generate bytes data
[ https://issues.apache.org/jira/browse/NIFI-12318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793283#comment-17793283 ] ASF subversion and git services commented on NIFI-12318: Commit 4251da4703db1d48be2976670e2295e5d78f16c8 in nifi's branch refs/heads/support/nifi-1.x from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4251da4703 ] NIFI-12318: Fixed byte array generation in GenerateRecord Signed-off-by: Pierre Villard This closes #8109. > GenerateRecord could not generate bytes data > > > Key: NIFI-12318 > URL: https://issues.apache.org/jira/browse/NIFI-12318 > Project: Apache NiFi > Issue Type: Bug > Environment: Apache Nifi 1.23.2 > jdk verson:11.0.20 >Reporter: Ashley >Assignee: Matt Burgess >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Attachments: image-2023-11-06-18-49-08-984.png, > image-2023-11-06-18-49-50-676.png, image-2023-11-06-18-50-59-897.png, > image-2023-11-06-18-51-41-326.png, image-2023-11-06-18-52-07-939.png, > image-2023-11-06-18-52-10-969.png > > Time Spent: 20m > Remaining Estimate: 0h > > I tried to generate bytes data using GenerateRecord,but encountered ERROR. > The ERROR: > !image-2023-11-06-18-52-10-969.png! > GenerateRecord properties: > !image-2023-11-06-18-49-50-676.png! > Schema Text: > !image-2023-11-06-18-49-08-984.png! > AvroRecordSetWriter properteis: > !image-2023-11-06-18-50-59-897.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12318) GenerateRecord could not generate bytes data
[ https://issues.apache.org/jira/browse/NIFI-12318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793282#comment-17793282 ] ASF subversion and git services commented on NIFI-12318: Commit eeb2b1a64454326aa3b128050165ef001cbfc5f2 in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=eeb2b1a644 ] NIFI-12318: Fixed byte array generation in GenerateRecord Signed-off-by: Pierre Villard This closes #8109. > GenerateRecord could not generate bytes data > > > Key: NIFI-12318 > URL: https://issues.apache.org/jira/browse/NIFI-12318 > Project: Apache NiFi > Issue Type: Bug > Environment: Apache Nifi 1.23.2 > jdk verson:11.0.20 >Reporter: Ashley >Assignee: Matt Burgess >Priority: Major > Fix For: 1.25.0, 2.0.0 > > Attachments: image-2023-11-06-18-49-08-984.png, > image-2023-11-06-18-49-50-676.png, image-2023-11-06-18-50-59-897.png, > image-2023-11-06-18-51-41-326.png, image-2023-11-06-18-52-07-939.png, > image-2023-11-06-18-52-10-969.png > > Time Spent: 10m > Remaining Estimate: 0h > > I tried to generate bytes data using GenerateRecord,but encountered ERROR. > The ERROR: > !image-2023-11-06-18-52-10-969.png! > GenerateRecord properties: > !image-2023-11-06-18-49-50-676.png! > Schema Text: > !image-2023-11-06-18-49-08-984.png! > AvroRecordSetWriter properteis: > !image-2023-11-06-18-50-59-897.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12318: Fixed byte array generation in GenerateRecord [nifi]
asfgit closed pull request #8109: NIFI-12318: Fixed byte array generation in GenerateRecord URL: https://github.com/apache/nifi/pull/8109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1840798813 > @exceptionfactory : Hope you are doing good. Can you please take a look at this PR. This is long pending item, we wanted to close this before the end of this year. We would appreciate your review on this item whenever you have some time. Thanks for the follow-up @tanmaya-panda1. I am in the process of reviewing several pull requests, and I plan to return to this one soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12471) Update bolt-socket-mode to 1.36.1
[ https://issues.apache.org/jira/browse/NIFI-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R reassigned NIFI-12471: - Assignee: Mike R > Update bolt-socket-mode to 1.36.1 > - > > Key: NIFI-12471 > URL: https://issues.apache.org/jira/browse/NIFI-12471 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M1 >Reporter: Mike R >Assignee: Mike R >Priority: Minor > > Update bolt-socket-mode to 1.36.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12471) Update bolt-socket-mode to 1.36.1
Mike R created NIFI-12471: - Summary: Update bolt-socket-mode to 1.36.1 Key: NIFI-12471 URL: https://issues.apache.org/jira/browse/NIFI-12471 Project: Apache NiFi Issue Type: Improvement Affects Versions: 2.0.0-M1 Reporter: Mike R Update bolt-socket-mode to 1.36.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-2244 Update MiNiFi C2 and tests to support NiFi 2.0 [nifi-minifi-cpp]
martinzink commented on code in PR #1702:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1702#discussion_r1415530574
##
docker/test/integration/minifi/flow_serialization/Nifi_flow_json_serializer.py:
##
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import uuid
+import json
+
+from ..core.Processor import Processor
+from ..core.InputPort import InputPort
+
+
+class Nifi_flow_json_serializer:
+def serialize(self, start_nodes, nifi_version=None):
+res = {
+"encodingVersion": {
+"majorVersion": 2,
+"minorVersion": 0
+},
+"maxTimerDrivenThreadCount": 10,
+"maxEventDrivenThreadCount": 1,
+"registries": [],
+"parameterContexts": [],
+"parameterProviders": [],
+"controllerServices": [],
+"reportingTasks": [],
+"templates": [],
+"rootGroup": {
+"identifier": "9802c873-3322-3b60-a71d-732d02bd60f8",
+"instanceIdentifier": str(uuid.uuid4()),
+"name": "NiFi Flow",
+"comments": "",
+"position": {
+"x": 0,
+"y": 0
+},
+"processGroups": [],
+"remoteProcessGroups": [],
+"processors": [],
+"inputPorts": [],
+"outputPorts": [],
+"connections": [],
+"labels": [],
+"funnels": [],
+"controllerServices": [],
+"defaultFlowFileExpiration": "0 sec",
+"defaultBackPressureObjectThreshold": 1,
+"defaultBackPressureDataSizeThreshold": "1 GB",
+"scheduledState": "RUNNING",
+"executionEngine": "INHERITED",
+"maxConcurrentTasks": 1,
+"statelessFlowTimeout": "1 min",
+"flowFileConcurrency": "UNBOUNDED",
+"flowFileOutboundPolicy": "STREAM_WHEN_AVAILABLE",
+"componentType": "PROCESS_GROUP"
+}
+}
+visited = []
+
+for node in start_nodes:
+self.serialize_node(node, nifi_version, res['rootGroup'], visited)
+
+return json.dumps(res)
+
+def serialize_node(self, connectable, nifi_version, root, visited):
+if visited is None:
+visited = []
+
+visited.append(connectable)
+
+if hasattr(connectable, 'name'):
+connectable_name_text = connectable.name
+else:
+connectable_name_text = str(connectable.uuid)
+
+if isinstance(connectable, InputPort):
+root['inputPorts'].append({
+'identifier': str(connectable.uuid),
+'instanceIdentifier': str(connectable.instance_id),
+'name': connectable_name_text,
+"comments": "",
+'position': {
+'x': 0,
+'y': 0
+},
+'type': 'INPUT_PORT',
+'concurrentlySchedulableTaskCount': 1,
+'scheduledState': 'RUNNING',
+'allowRemoteAccess': True,
+'portFunction': 'STANDARD',
+'componentType': 'INPUT_PORT',
+"groupIdentifier": "9802c873-3322-3b60-a71d-732d02bd60f8"
+})
+
+if isinstance(connectable, Processor):
+root['processors'].append({
+"identifier": str(connectable.uuid),
+"instanceIdentifier": str(connectable.instance_id),
+"name": connectable_name_text,
+"comments": "",
+"position": {
+"x": 0,
+"y": 0
+},
+"type": 'org.apache.nifi.processors.standard.' +
connectable.clazz,
+"bundle": {
+"group": "org.apache.nifi",
+"artifact": "nifi-standard-nar",
+"version": nifi_version
+},
+"properties": {key: value for key, value in
connectable.properties.items() if
Re: [PR] MINIFICPP-2265 Implement AttributeRollingWindow and EL nextInt() [nifi-minifi-cpp]
szaszm commented on code in PR #1703:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1703#discussion_r1415251638
##
extensions/standard-processors/RollingWindow.h:
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+#include
+
+namespace org::apache::nifi::minifi::processors::standard::utils {
+
+namespace detail {
+template
+struct priority_queue : std::priority_queue {
+ using std::priority_queue::priority_queue;
+
+ // Expose the underlying container
+ const Container& get_container() const & { return this->c; }
+ Container get_container() && { return std::move(this->c); }
+};
+} // namespace detail
+
+template
+class RollingWindow {
+ public:
+ struct Entry {
+Timestamp timestamp{};
+Value value{};
+ };
+ struct EntryComparator {
+// greater-than, because std::priority_queue order is reversed. This way,
top() is the oldest entry.
+bool operator()(const Entry& lhs, const Entry& rhs) const {
+ return lhs.timestamp > rhs.timestamp;
+}
+ };
+
+ void removeOlderThan(Timestamp timestamp) {
+while (!state_.empty() && state_.top().timestamp < timestamp) {
+ state_.pop();
+}
+ }
+
+ /** Remove the oldest entries until the size is <= size. */
+ void shrinkToSize(size_t size) {
+while (state_.size() > size && !state_.empty()) {
Review Comment:
You're right. When answering last time, I somehow didn't realize that the
lowest size goes is 0, and `>0` can't be 0. I'm changing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2265 Implement AttributeRollingWindow and EL nextInt() [nifi-minifi-cpp]
fgerlits commented on code in PR #1703:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1703#discussion_r1415097648
##
extensions/standard-processors/RollingWindow.h:
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+#include
+
+namespace org::apache::nifi::minifi::processors::standard::utils {
+
+namespace detail {
+template
+struct priority_queue : std::priority_queue {
+ using std::priority_queue::priority_queue;
+
+ // Expose the underlying container
+ const Container& get_container() const & { return this->c; }
+ Container get_container() && { return std::move(this->c); }
+};
+} // namespace detail
+
+template
+class RollingWindow {
+ public:
+ struct Entry {
+Timestamp timestamp{};
+Value value{};
+ };
+ struct EntryComparator {
+// greater-than, because std::priority_queue order is reversed. This way,
top() is the oldest entry.
+bool operator()(const Entry& lhs, const Entry& rhs) const {
+ return lhs.timestamp > rhs.timestamp;
+}
+ };
+
+ void removeOlderThan(Timestamp timestamp) {
+while (!state_.empty() && state_.top().timestamp < timestamp) {
+ state_.pop();
+}
+ }
+
+ /** Remove the oldest entries until the size is <= size. */
+ void shrinkToSize(size_t size) {
+while (state_.size() > size && !state_.empty()) {
Review Comment:
I meant that the first check, `state_.size() > size`, already implies
`state_.size() > 0`, which implies `!state_.empty()`. But yeah, it's just a
few extra (lightweight) operations which the compiler may be able to optimize
out. If you think it's more readable this way, I'm OK with keeping it as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12462) Upgrade Logback to 1.3.14
[ https://issues.apache.org/jira/browse/NIFI-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793179#comment-17793179 ] ASF subversion and git services commented on NIFI-12462: Commit 5e183c583b210dda288feb959c8492034751e38c in nifi's branch refs/heads/support/nifi-1.x from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5e183c583b ] NIFI-12462 Upgraded Logback from 1.3.13 to 1.3.14 Signed-off-by: Pierre Villard This closes #8114. > Upgrade Logback to 1.3.14 > - > > Key: NIFI-12462 > URL: https://issues.apache.org/jira/browse/NIFI-12462 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: backport-needed > Fix For: 1.25.0, 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Logback 1.3.13 resolved a potential denial of service attack for custom > socket-based Logback appenders, which are not used in any default NiFi > component configurations. The vulnerability was published as CVE-2023-6378. > Logback 1.3.14 includes additional improvements to the resolution, so Logback > published the same vulnerability as CVE-2023-648, marking version 1.3.13 as > vulnerable to encourage users to upgrade to the latest version. > Although current NiFi versions do not use socket-based Logback appenders, > Logback should be upgraded to the latest version on current and support > branches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12462) Upgrade Logback to 1.3.14
[ https://issues.apache.org/jira/browse/NIFI-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12462: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Logback to 1.3.14 > - > > Key: NIFI-12462 > URL: https://issues.apache.org/jira/browse/NIFI-12462 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: backport-needed > Fix For: 1.25.0, 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Logback 1.3.13 resolved a potential denial of service attack for custom > socket-based Logback appenders, which are not used in any default NiFi > component configurations. The vulnerability was published as CVE-2023-6378. > Logback 1.3.14 includes additional improvements to the resolution, so Logback > published the same vulnerability as CVE-2023-648, marking version 1.3.13 as > vulnerable to encourage users to upgrade to the latest version. > Although current NiFi versions do not use socket-based Logback appenders, > Logback should be upgraded to the latest version on current and support > branches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12462) Upgrade Logback to 1.3.14
[ https://issues.apache.org/jira/browse/NIFI-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793178#comment-17793178 ] ASF subversion and git services commented on NIFI-12462: Commit 8dc09c8911685ee8b5d2235ba4cd2c552bd7c4ab in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8dc09c8911 ] NIFI-12462 Upgraded Logback from 1.3.13 to 1.3.14 Signed-off-by: Pierre Villard This closes #8114. > Upgrade Logback to 1.3.14 > - > > Key: NIFI-12462 > URL: https://issues.apache.org/jira/browse/NIFI-12462 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: backport-needed > Fix For: 1.25.0, 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Logback 1.3.13 resolved a potential denial of service attack for custom > socket-based Logback appenders, which are not used in any default NiFi > component configurations. The vulnerability was published as CVE-2023-6378. > Logback 1.3.14 includes additional improvements to the resolution, so Logback > published the same vulnerability as CVE-2023-648, marking version 1.3.13 as > vulnerable to encourage users to upgrade to the latest version. > Although current NiFi versions do not use socket-based Logback appenders, > Logback should be upgraded to the latest version on current and support > branches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12462 Upgrade Logback from 1.3.13 to 1.3.14 [nifi]
asfgit closed pull request #8114: NIFI-12462 Upgrade Logback from 1.3.13 to 1.3.14 URL: https://github.com/apache/nifi/pull/8114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
