[jira] [Commented] (NIFI-8436) update the apache parquet library version (to enable zstd compression)
[ https://issues.apache.org/jira/browse/NIFI-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577696#comment-17577696 ] Matthew Hawkins commented on NIFI-8436: --- NIFI-8439 (fixed in 1.14.0) upgraded parquet-avro as desired here. This issue can be closed IMO. > update the apache parquet library version (to enable zstd compression) > -- > > Key: NIFI-8436 > URL: https://issues.apache.org/jira/browse/NIFI-8436 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 1.13.2 >Reporter: Luke Higgins >Priority: Major > > The current apache parquet-avro library version in the below pom is 1.10.0. > Upgrading to 1.12.0 upgrades the library to the latest. One of the upgraded > features is to allow for newer compression algorithms to include zstandard. > This is in the following pom: > https://github.com/apache/nifi/blob/7fe8858d22f580ad9dcd96ebafb59dbe5e73f5cb/nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/pom.xml > Initial testing of making this change and building this nar shows the > functionality desired. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577690#comment-17577690 ] Matthew Hawkins commented on NIFI-2827: --- Re: Brotli. So the current Commons implementation of Brotli is "read-only" (which I take to mean it will decompress but not compress). Whilst there's possibly limited applications where that is desired, personally I'd prefer formats that support both functions. > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-clvl.patch, NIFI-2827-zstd.patch, ZstdTests.ods > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Hawkins updated NIFI-2827: -- Attachment: NIFI-2827-clvl.patch > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-clvl.patch, NIFI-2827-zstd.patch, ZstdTests.ods > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Hawkins updated NIFI-2827: -- Attachment: ZstdTests.ods > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-clvl.patch, NIFI-2827-zstd.patch, ZstdTests.ods > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577689#comment-17577689 ] Matthew Hawkins commented on NIFI-2827: --- Thanks for taking a look [~joewitt]; adding compression levels as described above is 3 lines (see attachment) over the original patch. (patch zstd then patch clvl) This uses the hack as described above as I can't see a clean way to add explicit support for zstd's 22 levels; the property is declared final but would need to be set based on some switch on what the compression format is to overcome the difference between this and other compression methods. Perhaps Commons has a declaration of this, but a generic compression type object could have a property of what compression levels are supported by the underlying format so it's not an explicit 0-9 in that one-liner method conjunction when the property is declared. It's not actually quite as hacky to do what I've done here; the zstd 22 levels are explicitly defined in the standard to be increasing in the same sort of manner to gzip's 0-9, and since they don't recommend using levels over 20 anyway it's not so bad to just double the set value which will achieve the desired effect. It's worthwhile to drop this in as-is IMO because the compression versus gzip is insane (see ods attachment). That's lorem ipsum in batches of 1 into Merge and Compress; so yes easily compressible with a good algorithm. In/Out sizes in bytes, time in seconds (5 min mean) > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-zstd.patch > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] github-actions[bot] commented on pull request #5953: NIFI-34 Enable enforcement of best practices in nifi-mock
github-actions[bot] commented on PR #5953: URL: https://github.com/apache/nifi/pull/5953#issuecomment-1210013146 We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10336) Intermittent Failures in TestMonitorActivity on Windows
[
https://issues.apache.org/jira/browse/NIFI-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-10336:
Status: Patch Available (was: Open)
> Intermittent Failures in TestMonitorActivity on Windows
> ---
>
> Key: NIFI-10336
> URL: https://issues.apache.org/jira/browse/NIFI-10336
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recent automated builds on Windows have failed in TestMonitoryActivity with
> the following error:
> {noformat}
> Error:
> org.apache.nifi.processors.standard.TestMonitorActivity.testFirstMessageWithInherit
> Time elapsed: 0.015 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: not equal but was:
> <1660078247969>
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
> at
> org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:276)
> at
> org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:121)
> at
> org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:113)
> at
> org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2117)
> at
> org.apache.nifi.processors.standard.TestMonitorActivity.testFirstMessageWithInherit(TestMonitorActivity.java:290)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #6285: NIFI-10336 Remove unnecessary lineage checks from TestMonitorActivity
exceptionfactory opened a new pull request, #6285: URL: https://github.com/apache/nifi/pull/6285 # Summary [NIFI-10336](https://issues.apache.org/jira/browse/NIFI-10336) Removes unnecessary FlowFile lineage start date comparisons from `TestMonitorActivity` to avoid intermittent build failures. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-10336) Intermittent Failures in TestMonitorActivity on Windows
David Handermann created NIFI-10336:
---
Summary: Intermittent Failures in TestMonitorActivity on Windows
Key: NIFI-10336
URL: https://issues.apache.org/jira/browse/NIFI-10336
Project: Apache NiFi
Issue Type: Bug
Components: Tools and Build
Reporter: David Handermann
Assignee: David Handermann
Recent automated builds on Windows have failed in TestMonitoryActivity with the
following error:
{noformat}
Error:
org.apache.nifi.processors.standard.TestMonitorActivity.testFirstMessageWithInherit
Time elapsed: 0.015 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: not equal but was:
<1660078247969>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
at
org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:276)
at
org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:121)
at
org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:113)
at
org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2117)
at
org.apache.nifi.processors.standard.TestMonitorActivity.testFirstMessageWithInherit(TestMonitorActivity.java:290)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10183) Update Swagger-Annotations To 1.6.6
[ https://issues.apache.org/jira/browse/NIFI-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577628#comment-17577628 ] David Handermann commented on NIFI-10183: - Thanks for submitting the initial pull request [~msr1716]. After seeing PR 6208 closed, I submitted PR 6284 which replaces separate version references with the single managed dependency version in the root Maven configuration. > Update Swagger-Annotations To 1.6.6 > --- > > Key: NIFI-10183 > URL: https://issues.apache.org/jira/browse/NIFI-10183 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.16.1, 1.16.2, 1.16.3 >Reporter: Mike R >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > NiFi is using a combination of Swagger Annotations version 1.5.16 and 1.6.0. > Although there is a newer version of Swagger Annotations 2+ available, a > start will be to make sure that all of the annotations are at the same > version of 1.6.6 across all of the NiFi instances. A future ticket can > upgrade to 2+ if the need is there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10183) Update Swagger-Annotations To 1.6.6
[ https://issues.apache.org/jira/browse/NIFI-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-10183: --- Assignee: David Handermann > Update Swagger-Annotations To 1.6.6 > --- > > Key: NIFI-10183 > URL: https://issues.apache.org/jira/browse/NIFI-10183 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.16.1, 1.16.2, 1.16.3 >Reporter: Mike R >Assignee: David Handermann >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > NiFi is using a combination of Swagger Annotations version 1.5.16 and 1.6.0. > Although there is a newer version of Swagger Annotations 2+ available, a > start will be to make sure that all of the annotations are at the same > version of 1.6.6 across all of the NiFi instances. A future ticket can > upgrade to 2+ if the need is there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10183) Update Swagger-Annotations To 1.6.6
[ https://issues.apache.org/jira/browse/NIFI-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10183: Status: Patch Available (was: Open) > Update Swagger-Annotations To 1.6.6 > --- > > Key: NIFI-10183 > URL: https://issues.apache.org/jira/browse/NIFI-10183 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.16.3, 1.16.2, 1.16.1 >Reporter: Mike R >Assignee: David Handermann >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > NiFi is using a combination of Swagger Annotations version 1.5.16 and 1.6.0. > Although there is a newer version of Swagger Annotations 2+ available, a > start will be to make sure that all of the annotations are at the same > version of 1.6.6 across all of the NiFi instances. A future ticket can > upgrade to 2+ if the need is there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #6284: NIFI-10183 Upgrade swagger-annotations to 1.6.6
exceptionfactory opened a new pull request, #6284: URL: https://github.com/apache/nifi/pull/6284 # Summary [NIFI-10183](https://issues.apache.org/jira/browse/NIFI-10183) Upgrades different minor versions of Swagger Annotations to 1.6.6. This change removes specific version references in multiple Maven configurations and centralizes the version using a managed dependency in the root Maven configuration. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6283: NIFI-8648 Add Session Affinity section to Admin Guide
exceptionfactory commented on PR #6283: URL: https://github.com/apache/nifi/pull/6283#issuecomment-1209870529 Thanks for the review @ChrisSamo632! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577616#comment-17577616 ] Joe Witt commented on NIFI-2827: Thanks for contributing an example patch. Adding support for these is fine but we'll definitely want the key things related to compression levels included. We use specific libraries for specific compression types (or at least had in the past) because different libraries gave different trade-offs including for things like memory usage/etc.. We'll likely retain that unless research has been done to show it is no longer necessary. In any case adding new compression/decompression types is often painless and certain commons compress helps a lot there. So happy to add support for these but we'll need someone to write up the full patch and someone to review it. Truthfully these days basically all patches that make it in go through Github/Pull Requests because there is some extremely helpful automation that comes along with that to reduce the review timeline. Thanks > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-zstd.patch > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10258) Continuously getting the error "ERROR: Failed to index Provenance Events. See logs for more information. " in the Bulletin Board
[ https://issues.apache.org/jira/browse/NIFI-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577615#comment-17577615 ] Joe Witt commented on NIFI-10258: - That would be a perfectly fine option. It would help you be confident in the root problem. You could then if necessary re-install it and ensure it doesn't impact critical directories NiFi uses to perform its function. > Continuously getting the error "ERROR: Failed to index Provenance Events. See > logs for more information. " in the Bulletin Board > > > Key: NIFI-10258 > URL: https://issues.apache.org/jira/browse/NIFI-10258 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.15.3 > Environment: Operating System: Red Hat Enterprise Linux Server 7.9 > (Maipo) > Architecture: amd 64 > Kernel: Linux 3.10.0-1160.53.1.el7.x86_64 > Java Version:1.8.0_322 >Reporter: Deepak Reddy Chirthani >Priority: Critical > Attachments: Nifi_logs_deepak.txt, Provenance_properties.PNG > > Original Estimate: 336h > Remaining Estimate: 336h > > we have started to use the reporting task *SiteToSiteProvenanceReportingTask > 1.15.3* for auditing purposes on provenance events for a few of our > dataflows. We started to see error quite frequently now in the Bulletin Board. > *ERROR: Failed to index Provenance Events. See logs for more information.* > Also, below are attached snaps of nifi.provenance.properties file and from > logs regarding this error. > Please let me know if you need anything more. > Update: Even though S2SProvenanceReportingTask is stopped, the error still > exisits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10322) invalid_token error after OpenID connect session timeout
[
https://issues.apache.org/jira/browse/NIFI-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough updated NIFI-10322:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> invalid_token error after OpenID connect session timeout
>
>
> Key: NIFI-10322
> URL: https://issues.apache.org/jira/browse/NIFI-10322
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-05-22-48-17-835.png,
> image-2022-08-05-22-48-52-057.png, image-2022-08-07-14-28-09-058.png,
> image-2022-08-07-15-22-36-213.png, image-2022-08-07-15-27-18-902.png,
> image-2022-08-07-15-37-29-739.png, image-2022-08-07-15-43-14-922.png,
> image-2022-08-07-15-47-57-158.png, image-2022-08-07-15-53-47-220.png,
> image-2022-08-07-16-00-11-443.png, image-2022-08-07-16-11-38-180.png,
> image-2022-08-08-23-33-30-220.png, image-2022-08-08-23-35-02-773.png,
> image-2022-08-08-23-59-12-471.png, nginx-access.log.zip,
> nifi-1.16.3-logs.zip, nifi-1.17.0-logs.zip
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I follow
> [https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect] to
> config NIFI 1.16.3 and it is work properly. If the session times out, login
> again and it will work again
> I configured 1.17.0 in the same way. I can login and operate nifi UI. But
> when session times out. I got the following error.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Expired JWT",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-17-835.png|width=758,height=108!
> I try to login again and get a new error, and I cannot enter the NIFI
> interface.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Signed JWT rejected: Another algorithm
> expected, or no matching key(s) found",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-52-057.png|width=594,height=143!
> I did some research, and found
> After the session times out,
> NIFI 1.16.3 leaves 3 cookies in browser:
> * nifi-logout-request-identifier
> * nifi-oidc-request-identifier
> * __Secure-Request-Token
> NIFI 1.17.0 leaves 2 cookies:
> * *__Secure-Authorization-Bearer*
> * __Secure-Request-Token
> __Secure-Authorization-Bearer cookie contains a expired JWT:
> {code:java}
> eyJraWQiOiJhMDlhZDhlMy0xZDkzLTQyZTEtYjg0Ni0xMWU0ODRkODYwYWYiLCJhbGciOiJQUzUxMiJ9.eyJzdWIiOiJhZG1pbi5uaWZpQGd1bWhiMy5jb20iLCJhdWQiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwibmJmIjoxNjU5NjExOTc0LCJpc3MiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW4ubmlmaUBndW1oYjMuY29tIiwiZXhwIjoxNjU5NjEyMjc0LCJpYXQiOjE2NTk2MTE5NzQsImp0aSI6IjFiZTg5MjU4LTliZmYtNDhmOS04OGNmLWU0NDIzMDZjYzg4ZCJ9.Y9yE0hNH_q-W94_cFWOWGc7TPMP2xB9coaSRPT9twYqSyjTtudOiiXGxHEDUWsOvUFf7lT7wNH4RZ_LhOM-5WfTZ3o-DCVFnl0JjeZ-L9d-z3rO4dEspRxXpr46AewEGy_lpstSUFyihr4i8b2VI7IT0aFOCGAIXRWl7gfH75e5La_0tbsu9lgSRdyYBBv8rSjojJC5bBSqxj-BkrfjdMhyMuF9OdMCJNmyh18BrXbavwftNerytkd_Qf9eNLmzsZ3SOdKWpftKt4kClD_KeL0nOglhM-ENyb4QLwxr7l5lhUgQ-2am3x5okbRyYip_WV4YQ6DfmUnLL1FYFATWXa5CUimSRbSZzkqU2JEYerpvKsTf-prdsSNryPbrQdf5HqpwhlGbFrgm4jwtncZHTLEL4ZMciVe0H-zIcQ9vyDqamMpf6fyNWmQN8DdDP9A0Zpo7SL7yhOUjNGsjk1gV4OAHWgp4XQzj4KwoGf7ICjeOrzinECHFZw9Ccyi8KMooRx4u3oAuKPEx3mrZFNFDaiAzWX0kZ31c24-15cno2bLBMGOIx7ipjb6Pv7V6O9S2aA2vC3eVLnfAgHAox3I8_IzWLUKddHCqd6cfA1XW8ckSgg2QddKvgYHiCZpwVV4AMDpK4bI1J0ZbxbgOOke9IMMudNhZUFQdWJIXh-gx1bII{code}
> I manually delete __Secure-Authorization-Bearer cookie, and I can login NIFI
> 1.17.0 again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10322) invalid_token error after OpenID connect session timeout
[
https://issues.apache.org/jira/browse/NIFI-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577614#comment-17577614
]
ASF subversion and git services commented on NIFI-10322:
Commit 77033ec11aa7e738aaa32e80957b4a1538923f18 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=77033ec11a ]
NIFI-10322 Corrected Cookie path when removing Bearer Token
- Appended root path to Cookie path attribute when removing Bearer Tokens as
part of unauthorized response handling
- Updated Saml2AuthenticationSuccessHandler to follow standard Cookie path
building strategy
Signed-off-by: Nathan Gough
This closes #6278.
> invalid_token error after OpenID connect session timeout
>
>
> Key: NIFI-10322
> URL: https://issues.apache.org/jira/browse/NIFI-10322
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-05-22-48-17-835.png,
> image-2022-08-05-22-48-52-057.png, image-2022-08-07-14-28-09-058.png,
> image-2022-08-07-15-22-36-213.png, image-2022-08-07-15-27-18-902.png,
> image-2022-08-07-15-37-29-739.png, image-2022-08-07-15-43-14-922.png,
> image-2022-08-07-15-47-57-158.png, image-2022-08-07-15-53-47-220.png,
> image-2022-08-07-16-00-11-443.png, image-2022-08-07-16-11-38-180.png,
> image-2022-08-08-23-33-30-220.png, image-2022-08-08-23-35-02-773.png,
> image-2022-08-08-23-59-12-471.png, nginx-access.log.zip,
> nifi-1.16.3-logs.zip, nifi-1.17.0-logs.zip
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I follow
> [https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect] to
> config NIFI 1.16.3 and it is work properly. If the session times out, login
> again and it will work again
> I configured 1.17.0 in the same way. I can login and operate nifi UI. But
> when session times out. I got the following error.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Expired JWT",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-17-835.png|width=758,height=108!
> I try to login again and get a new error, and I cannot enter the NIFI
> interface.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Signed JWT rejected: Another algorithm
> expected, or no matching key(s) found",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-52-057.png|width=594,height=143!
> I did some research, and found
> After the session times out,
> NIFI 1.16.3 leaves 3 cookies in browser:
> * nifi-logout-request-identifier
> * nifi-oidc-request-identifier
> * __Secure-Request-Token
> NIFI 1.17.0 leaves 2 cookies:
> * *__Secure-Authorization-Bearer*
> * __Secure-Request-Token
> __Secure-Authorization-Bearer cookie contains a expired JWT:
> {code:java}
> eyJraWQiOiJhMDlhZDhlMy0xZDkzLTQyZTEtYjg0Ni0xMWU0ODRkODYwYWYiLCJhbGciOiJQUzUxMiJ9.eyJzdWIiOiJhZG1pbi5uaWZpQGd1bWhiMy5jb20iLCJhdWQiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwibmJmIjoxNjU5NjExOTc0LCJpc3MiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW4ubmlmaUBndW1oYjMuY29tIiwiZXhwIjoxNjU5NjEyMjc0LCJpYXQiOjE2NTk2MTE5NzQsImp0aSI6IjFiZTg5MjU4LTliZmYtNDhmOS04OGNmLWU0NDIzMDZjYzg4ZCJ9.Y9yE0hNH_q-W94_cFWOWGc7TPMP2xB9coaSRPT9twYqSyjTtudOiiXGxHEDUWsOvUFf7lT7wNH4RZ_LhOM-5WfTZ3o-DCVFnl0JjeZ-L9d-z3rO4dEspRxXpr46AewEGy_lpstSUFyihr4i8b2VI7IT0aFOCGAIXRWl7gfH75e5La_0tbsu9lgSRdyYBBv8rSjojJC5bBSqxj-BkrfjdMhyMuF9OdMCJNmyh18BrXbavwftNerytkd_Qf9eNLmzsZ3SOdKWpftKt4kClD_KeL0nOglhM-ENyb4QLwxr7l5lhUgQ-2am3x5okbRyYip_WV4YQ6DfmUnLL1FYFATWXa5CUimSRbSZzkqU2JEYerpvKsTf-prdsSNryPbrQdf5HqpwhlGbFrgm4jwtncZHTLEL4ZMciVe0H-zIcQ9vyDqamMpf6fyNWmQN8DdDP9A0Zpo7SL7yhOUjNGsjk1gV4OAHWgp4XQzj4KwoGf7ICjeOrzinECHFZw9Ccyi8KMooRx4u3oAuKPEx3mrZFNFDaiAzWX0kZ31c24-15cno2bLBMGOIx7ipjb6Pv7V6O9S2aA2vC3eVLnfAgHAox3I8_IzWLUKddHCqd6cfA1XW8ckSgg2QddKvgYHiCZpwVV4AMDpK4bI1J0ZbxbgOOke9IMMudNhZUFQdWJIXh-gx1bII{code}
> I manually delete __Secure-Authorization-Bearer cookie, and I can login NIFI
> 1.17.0 again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] thenatog closed pull request #6278: NIFI-10322 Correct Cookie path when removing Bearer Token
thenatog closed pull request #6278: NIFI-10322 Correct Cookie path when removing Bearer Token URL: https://github.com/apache/nifi/pull/6278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #6278: NIFI-10322 Correct Cookie path when removing Bearer Token
thenatog commented on PR #6278: URL: https://github.com/apache/nifi/pull/6278#issuecomment-1209822792 Tried setting up a reverse proxy to test this but I had some issues. I did verify that normal bearer token functionality is still working with LDAP for example (checked that when logging out, the cookie removed correctly). I verified the change through your unit test and it looks good to me. +1 will merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8648) Add session affinity info to proxy config section of admin guide
[ https://issues.apache.org/jira/browse/NIFI-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-8648: --- Affects Version/s: (was: 1.13.2) Status: Patch Available (was: Open) > Add session affinity info to proxy config section of admin guide > > > Key: NIFI-8648 > URL: https://issues.apache.org/jira/browse/NIFI-8648 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Joey Frazee >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > When using proxies or load balancers with a cluster the user should use > affinity or stickiness to assure that user sessions are consistent. This > should be noted in the admin guide. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #6283: NIFI-8648 Add Session Affinity section to Admin Guide
exceptionfactory opened a new pull request, #6283: URL: https://github.com/apache/nifi/pull/6283 # Summary [NIFI-8648](https://issues.apache.org/jira/browse/NIFI-8648) Adds a section on `Session Affinity` requirements under the `Proxy Configuration` section of the Administrator's Guide. The section provides some background on the reasons for requiring session affinity, as well as links to configuration details for Apache HTTP Server, Nginx, and the Kubernetes Nginx Ingress Controller. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [X] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-10335) Upgrade Azure-Core to 1.31.0
[ https://issues.apache.org/jira/browse/NIFI-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577577#comment-17577577 ] Mike R commented on NIFI-10335: --- [~exceptionfactory] That makes sense. I'll keep this open since this does resolve a CVE if this can be updated properly. Not sure how to do that for the entire Azure bunch. > Upgrade Azure-Core to 1.31.0 > > > Key: NIFI-10335 > URL: https://issues.apache.org/jira/browse/NIFI-10335 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Azure-Core to 1.31.0 from 1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10335) Upgrade Azure-Core to 1.31.0
[ https://issues.apache.org/jira/browse/NIFI-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577570#comment-17577570 ] David Handermann commented on NIFI-10335: - [~msr1716] Unfortunately unit tests in the standard build are not able to exercise the full range of capabilities for these components. Although code may compile, and unit tests may run to completion, there may be runtime code paths that are not exercised. > Upgrade Azure-Core to 1.31.0 > > > Key: NIFI-10335 > URL: https://issues.apache.org/jira/browse/NIFI-10335 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Azure-Core to 1.31.0 from 1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10335) Upgrade Azure-Core to 1.31.0
[ https://issues.apache.org/jira/browse/NIFI-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577568#comment-17577568 ] Mike R commented on NIFI-10335: --- [~exceptionfactory] Thanks. How would that happen as the upgrade does build successfully > Upgrade Azure-Core to 1.31.0 > > > Key: NIFI-10335 > URL: https://issues.apache.org/jira/browse/NIFI-10335 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Azure-Core to 1.31.0 from 1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mr1716 opened a new pull request, #6282: NIFI-10335 Upgrade azure-core to 1.31.0
mr1716 opened a new pull request, #6282: URL: https://github.com/apache/nifi/pull/6282 # Summary [NIFI-10335](https://issues.apache.org/jira/projects/NIFI/issues/NIFI-10335) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/projects/NIFI/issues/NIFI-10335) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [X] JDK 8 - [X] JDK 11 - [X] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1897) Reenable docker/test/integration/features/https.feature
[ https://issues.apache.org/jira/browse/MINIFICPP-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1897: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1383 > Reenable docker/test/integration/features/https.feature > --- > > Key: MINIFICPP-1897 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1897 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > > Currently docker/test/integration/features/https.feature is disabled, we > should fix the issues and re-enable the test -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1895) InvokeHTTP PropPutOutputAttributes doesnt work
[ https://issues.apache.org/jira/browse/MINIFICPP-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1895: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1383 > InvokeHTTP PropPutOutputAttributes doesnt work > -- > > Key: MINIFICPP-1895 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1895 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > put_attribute_name_ is not used correctly (it is only logged that the > response body is added to as an attribute and the tests only check for this > log message) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1875) HTTPClient should be reusable
[
https://issues.apache.org/jira/browse/MINIFICPP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1875:
---
Status: Patch Available (was: In Progress)
https://github.com/apache/nifi-minifi-cpp/pull/1383
> HTTPClient should be reusable
> -
>
> Key: MINIFICPP-1875
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1875
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
> Fix For: 0.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current form of HTTPCLient is not safe to reuse.
> (e.g. the response headers are appended to the same vector on each submit,
> the request header if not empty gets a {{"Expect:"}} appended on each submit)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-1882) centos build default CA doesnt work on ubuntu
[
https://issues.apache.org/jira/browse/MINIFICPP-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink reassigned MINIFICPP-1882:
--
Assignee: Martin Zink
> centos build default CA doesnt work on ubuntu
> -
>
> Key: MINIFICPP-1882
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1882
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
> Fix For: 0.13.0
>
>
> the default CA path is different on centos and ubuntu, and due to this if we
> use the centos build on ubuntu any libcurl processor trying to verify ssl
> without explicit SSLContextService will fail with
> {code:java}
> curl_easy_perform() failed Problem with the SSL CA cert (path? access
> rights?) on XYZ, error code 77 {code}
> There are workarounds
> * with a symlink to the corrent path
> * explicit SSLContextService with CA set
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1835) InvokeHTTP connection keep-alive
[ https://issues.apache.org/jira/browse/MINIFICPP-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1835: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1383 > InvokeHTTP connection keep-alive > > > Key: MINIFICPP-1835 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1835 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Martin Zink >Priority: Major > > On each trigger, InvokeHTTP set up a brand new connection, possibly with TLS > handshake, and a starting window size. We can improve throughput by reusing > the HTTPClient (and thus the curl session) between triggers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1902) Explore options to stabilize the python dependency of the scripting extension
[ https://issues.apache.org/jira/browse/MINIFICPP-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-1902: Description: libpython changes its SONAME every minor version, making it impossible to distribute binaries that work across multiple python versions. The script extension depends on libpython through pybind11. Python itself provides a stable API since 3.2, which is fairly old at this point: [https://docs.python.org/3/c-api/stable.html#c-api-stability] Pybind11 and boost.python are both using API functions that are not part of the stable API: [https://github.com/pybind/pybind11/issues/1755] There seems to be some activity on swig to support this stable API: [https://github.com/swig/swig/pull/2190] TODO: explore options, so that binary distributions of the scripting extension are possible was: libpython changes its SONAME every minor version, making it impossible to distribute binaries that work across multiple python versions. The script extension depends on libpython through pybind11. Python itself provides a stable API since 3.2, which is fairly old at this point: [https://docs.python.org/3/c-api/stable.html#c-api-stability] Pybind11 and boost.python are both using API functions that are not part of the stable API: [https://github.com/pybind/pybind11/issues/1755] There seems to be some activity on swig to support this stable API: [https://github.com/swig/swig/pull/2190] TODO: explore options, so that binary distributions of the scripting extension is possible > Explore options to stabilize the python dependency of the scripting extension > - > > Key: MINIFICPP-1902 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1902 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > > libpython changes its SONAME every minor version, making it impossible to > distribute binaries that work across multiple python versions. The script > extension depends on libpython through pybind11. > > Python itself provides a stable API since 3.2, which is fairly old at this > point: [https://docs.python.org/3/c-api/stable.html#c-api-stability] > > Pybind11 and boost.python are both using API functions that are not part of > the stable API: [https://github.com/pybind/pybind11/issues/1755] > > There seems to be some activity on swig to support this stable API: > [https://github.com/swig/swig/pull/2190] > > TODO: explore options, so that binary distributions of the scripting > extension are possible -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10335) Upgrade Azure-Core to 1.31.0
[ https://issues.apache.org/jira/browse/NIFI-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577512#comment-17577512 ] David Handermann commented on NIFI-10335: - [~msr1716] This can be a more complicated issue due to the dependence on different versions in different Azure libraries. The dependency versions need to be structured to use the Azure Bill-of-Materials dependency, but that will require some careful alignment in multiple modules. > Upgrade Azure-Core to 1.31.0 > > > Key: NIFI-10335 > URL: https://issues.apache.org/jira/browse/NIFI-10335 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > > Upgrade Azure-Core to 1.31.0 from 1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6159: NIFI-8248 Modify PutAzureDataLakeStorage processor to use temp file i…
exceptionfactory commented on code in PR #6159: URL: https://github.com/apache/nifi/pull/6159#discussion_r941517724 ## nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java: ## @@ -101,6 +101,8 @@ public abstract class AbstractAzureDataLakeStorageProcessor extends AbstractProc REL_FAILURE ))); +public static final String TEMP_FILE_DIRECTORY = "_$azuretempdirectory$"; Review Comment: Having a reusable shared variable is helpful. I may have missed something in previous discussion, but what is the reason for using the `$` characters? Including `azure` in the name does not seem necessary, but providing some association with NiFi might be helpful. ## nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureDataLakeStorage.java: ## @@ -129,13 +129,24 @@ public class ListAzureDataLakeStorage extends AbstractListAzureProcessor
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6159: NIFI-8248 Modify PutAzureDataLakeStorage processor to use temp file i…
exceptionfactory commented on code in PR #6159:
URL: https://github.com/apache/nifi/pull/6159#discussion_r941502092
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java:
##
@@ -20,7 +20,9 @@
import com.azure.storage.file.datalake.DataLakeFileClient;
import com.azure.storage.file.datalake.DataLakeFileSystemClient;
import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.google.common.annotations.VisibleForTesting;
Review Comment:
Use of Google Guava annotations should be avoided.
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java:
##
@@ -190,4 +218,34 @@ static void uploadContent(DataLakeFileClient fileClient,
InputStream in, long le
fileClient.flush(length);
}
+
+@VisibleForTesting
+DataLakeFileClient renameFile(final String fileName, final String
directoryPath, final DataLakeFileClient fileClient, final boolean overwrite) {
+try {
+final DataLakeRequestConditions destinationCondition = new
DataLakeRequestConditions();
+if (!overwrite) {
+destinationCondition.setIfNoneMatch("*");
+}
+final String destinationPath = createPath(directoryPath, fileName);
+return fileClient.renameWithResponse(null, destinationPath, null,
destinationCondition, null, null).getValue();
+} catch (DataLakeStorageException dataLakeStorageException) {
+getLogger().error("Error while renaming temp file " +
fileClient.getFileName() + " on Azure Data Lake Storage",
dataLakeStorageException);
+removeTempFile(fileClient);
+throw dataLakeStorageException;
+}
+}
+
+private String createPath(final String baseDirectory, final String path) {
+return StringUtils.isNotBlank(baseDirectory)
+? baseDirectory + "/" + path
+: path;
+}
+
+private void removeTempFile(final DataLakeFileClient fileClient) {
+try {
+fileClient.delete();
+} catch (Exception e) {
+getLogger().error("Error while removing temp file " +
fileClient.getFileName() + " on Azure Data Lake Storage", e);
Review Comment:
Log messages should use placeholders instead of string concatenation.
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureDataLakeStorage.java:
##
@@ -129,13 +129,24 @@ public class ListAzureDataLakeStorage extends
AbstractListAzureProcessor performListing(final
ProcessContext context, final Lo
options.setRecursive(recurseSubdirectories);
final Pattern baseDirectoryPattern = Pattern.compile("^" +
baseDirectory + "/?");
+final boolean includeTempFiles =
context.getProperty(SHOW_TEMPORARY_FILES).asBoolean();
final long minimumTimestamp = minTimestamp == null ? 0 :
minTimestamp;
final List listing =
fileSystemClient.listPaths(options, null).stream()
.filter(pathItem -> !pathItem.isDirectory())
+.filter(pathItem -> includeTempFiles ||
!pathItem.getName().contains("_$azuretempdirectory$"))
Review Comment:
The `contains()` call should reference the TEMP_FILE_DIRECTORY variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-10312) Broken flowid handling in minifi-c2 service
[ https://issues.apache.org/jira/browse/NIFI-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577509#comment-17577509 ] Robert Liszli commented on NIFI-10312: -- PR: https://github.com/apache/nifi/pull/6281 > Broken flowid handling in minifi-c2 service > --- > > Key: NIFI-10312 > URL: https://issues.apache.org/jira/browse/NIFI-10312 > Project: Apache NiFi > Issue Type: Bug > Components: C2, MiNiFi >Reporter: Csaba Bejan >Assignee: Robert Liszli >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When MiNiFi is communication with minifi-c2 service via the C2 protocol the > flowId parsing seems to be broken so the publishing mechanism is not working. > Pattern needs to be updated to match the expected url structure or flow id > needs to be sent in a more robust way as current approach is error prone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] rliszli opened a new pull request, #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli opened a new pull request, #6281: URL: https://github.com/apache/nifi/pull/6281 # Summary This change fix the communication between the MiNiFi and the C2 Service trough C2 protocol. Also covered the heartbeating and flow update with integration tests. [NIFI-10312](https://issues.apache.org/jira/browse/NIFI-10312) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1902) Explore options to stabilize the python dependency of the scripting extension
Marton Szasz created MINIFICPP-1902: --- Summary: Explore options to stabilize the python dependency of the scripting extension Key: MINIFICPP-1902 URL: https://issues.apache.org/jira/browse/MINIFICPP-1902 Project: Apache NiFi MiNiFi C++ Issue Type: Task Reporter: Marton Szasz libpython changes its SONAME every minor version, making it impossible to distribute binaries that work across multiple python versions. The script extension depends on libpython through pybind11. Python itself provides a stable API since 3.2, which is fairly old at this point: [https://docs.python.org/3/c-api/stable.html#c-api-stability] Pybind11 and boost.python are both using API functions that are not part of the stable API: [https://github.com/pybind/pybind11/issues/1755] There seems to be some activity on swig to support this stable API: [https://github.com/swig/swig/pull/2190] TODO: explore options, so that binary distributions of the scripting extension is possible -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-1902) Explore options to stabilize the python dependency of the scripting extension
[ https://issues.apache.org/jira/browse/MINIFICPP-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz reassigned MINIFICPP-1902: --- Assignee: Marton Szasz > Explore options to stabilize the python dependency of the scripting extension > - > > Key: MINIFICPP-1902 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1902 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > > libpython changes its SONAME every minor version, making it impossible to > distribute binaries that work across multiple python versions. The script > extension depends on libpython through pybind11. > > Python itself provides a stable API since 3.2, which is fairly old at this > point: [https://docs.python.org/3/c-api/stable.html#c-api-stability] > > Pybind11 and boost.python are both using API functions that are not part of > the stable API: [https://github.com/pybind/pybind11/issues/1755] > > There seems to be some activity on swig to support this stable API: > [https://github.com/swig/swig/pull/2190] > > TODO: explore options, so that binary distributions of the scripting > extension is possible -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10335) Upgrade Azure-Core to 1.31.0
[ https://issues.apache.org/jira/browse/NIFI-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R updated NIFI-10335: -- Description: Upgrade Azure-Core to 1.31.0 from 1.26.0 (was: Upgrade Azure-Core to 4.31.0 from 4.24.0) Summary: Upgrade Azure-Core to 1.31.0 (was: Upgrade Azure-Core to 4.31.0) > Upgrade Azure-Core to 1.31.0 > > > Key: NIFI-10335 > URL: https://issues.apache.org/jira/browse/NIFI-10335 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > > Upgrade Azure-Core to 1.31.0 from 1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10335) Upgrade Azure-Core to 4.31.0
Mike R created NIFI-10335: - Summary: Upgrade Azure-Core to 4.31.0 Key: NIFI-10335 URL: https://issues.apache.org/jira/browse/NIFI-10335 Project: Apache NiFi Issue Type: Bug Reporter: Mike R Upgrade Azure-Core to 4.31.0 from 4.24.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] UcanInfosec opened a new pull request, #6280: NIFI-10334 Upgrade AMQP Client
UcanInfosec opened a new pull request, #6280: URL: https://github.com/apache/nifi/pull/6280 # Summary [NIFI-10334](https://issues.apache.org/jira/browse/NIFI-10334) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-10334) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [X] JDK 8 - [X] JDK 11 - [X] JDK 17 ### Licensing - [X] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-10334) Upgrade amqp-client to 5.15.0
Mike R created NIFI-10334: - Summary: Upgrade amqp-client to 5.15.0 Key: NIFI-10334 URL: https://issues.apache.org/jira/browse/NIFI-10334 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.17.0 Reporter: Mike R Upgrade amqp-client to 5.15.0 from 5.8.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-1898) ListenHTTP test fails
[ https://issues.apache.org/jira/browse/MINIFICPP-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Markovics resolved MINIFICPP-1898. --- Resolution: Fixed > ListenHTTP test fails > - > > Key: MINIFICPP-1898 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1898 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Ádám Markovics >Assignee: Ádám Markovics >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > In local builds, the above mentioned test fails with SIGSEGV. But it doesn't > occur in CI. > It turned out that the first commit that made this happen was > 0bf8ce968bdebfa6867eb339049bfabca19e5df0 > Maybe it only happens in debug mode, that could be a difference, because an > assert makes the test fail and they are not compiled in release mode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-1899) Local Docker build fails
[ https://issues.apache.org/jira/browse/MINIFICPP-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Markovics resolved MINIFICPP-1899. --- Resolution: Fixed > Local Docker build fails > > > Key: MINIFICPP-1899 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1899 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Ádám Markovics >Assignee: Ádám Markovics >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Because of unused parameters in BackTrace.cpp. Fix is underway. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6273: NIFI-9953 - The config encryption tool is too complicated to use and can be simplified
exceptionfactory commented on code in PR #6273:
URL: https://github.com/apache/nifi/pull/6273#discussion_r941394985
##
nifi-commons/nifi-property-protection-factory/src/main/java/org/apache/nifi/properties/scheme/PropertyProtectionScheme.java:
##
@@ -19,7 +19,7 @@
/**
* Property Protection Schemes supported as arguments for encryption commands
should not have direct references
*/
-enum PropertyProtectionScheme implements ProtectionScheme {
+public enum PropertyProtectionScheme implements ProtectionScheme {
Review Comment:
This enum was intentionally package-private to encapsulate resolution of the
scheme. As indicated in the enum values, there are different ways of referring
to the scheme. With the introduction of a new command, this seems like a good
opportunity to change the argument contract to use the scheme values, instead
of the enum names.
##
nifi-toolkit/nifi-property-encryptor-tool/pom.xml:
##
@@ -0,0 +1,129 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+nifi
+org.apache.nifi
+1.18.0-SNAPSHOT
+
+4.0.0
+
+nifi-property-encryptor-tool
+
+
+8
+8
+
+
+
+
+info.picocli
+picocli
+4.6.3
+
+
+org.apache.nifi
+nifi-property-protection-factory
+1.18.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-properties-loader
+1.18.0-SNAPSHOT
+compile
+
+
+org.apache.nifi.registry
+nifi-registry-properties
+1.18.0-SNAPSHOT
+compile
+
+
+org.apache.nifi.registry
+nifi-registry-properties-loader
+1.18.0-SNAPSHOT
+compile
+
+
+org.slf4j
+slf4j-api
+${org.slf4j.version}
+
+
+org.slf4j
+slf4j-simple
+${org.slf4j.version}
+
+
+
+
+
+maven-assembly-plugin
+
+
+
+
org.apache.nifi.util.console.PropertyEncryptorCLI
+
+
+
+jar-with-dependencies
+
+
+
+
+make-assembly
+package
+
+single
+
+
+
+
+
+org.codehaus.mojo
+appassembler-maven-plugin
+2.1.0
+
+
+
+
org.apache.nifi.util.console.PropertyEncryptorCLI
+property-encryptor
+
+
+
+
+
+org.apache.maven.plugins
+maven-compiler-plugin
+
Review Comment:
Is this comment necessary? The Maven compiler plugin version should be set
in the root configuration.
##
nifi-toolkit/nifi-property-encryptor-tool/src/main/java/org/apache/nifi/PropertyEncryptorMain.java:
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.properties.AbstractBootstrapPropertiesLoader;
+import org.apache.nifi.properties.BootstrapProperties;
+import org.apache.nifi.properties.MutableBootstrapProperties;
+import org.apache.nifi.properties.SensitivePropertyProviderFactory;
+import org.apache.nifi.properties.StandardSensitivePropertyProviderFactory;
+import org.apache.nifi.properties.scheme.PropertyProtectionScheme;
+import org.apache.nifi.properties.scheme.ProtectionScheme;
+import
[jira] [Commented] (NIFI-10258) Continuously getting the error "ERROR: Failed to index Provenance Events. See logs for more information. " in the Bulletin Board
[ https://issues.apache.org/jira/browse/NIFI-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577470#comment-17577470 ] Deepak Reddy Chirthani commented on NIFI-10258: --- [~joewitt] Removing Symantec entirely not just those folders is another option we can consider. Thoughts? > Continuously getting the error "ERROR: Failed to index Provenance Events. See > logs for more information. " in the Bulletin Board > > > Key: NIFI-10258 > URL: https://issues.apache.org/jira/browse/NIFI-10258 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.15.3 > Environment: Operating System: Red Hat Enterprise Linux Server 7.9 > (Maipo) > Architecture: amd 64 > Kernel: Linux 3.10.0-1160.53.1.el7.x86_64 > Java Version:1.8.0_322 >Reporter: Deepak Reddy Chirthani >Priority: Critical > Attachments: Nifi_logs_deepak.txt, Provenance_properties.PNG > > Original Estimate: 336h > Remaining Estimate: 336h > > we have started to use the reporting task *SiteToSiteProvenanceReportingTask > 1.15.3* for auditing purposes on provenance events for a few of our > dataflows. We started to see error quite frequently now in the Bulletin Board. > *ERROR: Failed to index Provenance Events. See logs for more information.* > Also, below are attached snaps of nifi.provenance.properties file and from > logs regarding this error. > Please let me know if you need anything more. > Update: Even though S2SProvenanceReportingTask is stopped, the error still > exisits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577413#comment-17577413 ] Matthew Hawkins edited comment on NIFI-2827 at 8/9/22 2:38 PM: --- Attached is a quick & dirty zstd implementation; it compiled with JDK11 and passes the test. Key thing is that Apache Commons has zstandard support since 1.16 and 1.21 (latest) is included in NiFi 1.17.0 pom. The same technique could be used to add brotli and others supported by org.apache.commons.compress and in fact you could probably refactor the whole thing to use Apache Commons instead of a gazillion other libraries. That was out of scope for me, I just wanted zstd support and I'm glad I did it because the test file compresses to 80 bytes, which is the winner of all the compression algorithms in the test. Can't wait to use it on real data! Of note, I do NOT add support for compression levels or using the dictionary (read zstandard docs); this is beyond my Java hacking skills tonight. Zstandard currently has 22 compression levels and the property is set around gzip's 0-9. I am experimenting with using property*2 but this is a hack; if serious we should add support for all 22 levels. was (Author: JIRAUSER294164): Attached is a quick & dirty zstd implementation; it compiled with JDK11 and passes the test. Key thing is that Apache Commons has zstandard support since 1.16 and 1.21 (latest) is included in NiFi 1.17.0 pom. The same technique could be used to add brotli and others supported by org.apache.commons.compress and in fact you could probably refactor the whole thing to use Apache Commons instead of a gazillion other libraries. That was out of scope for me, I just wanted zstd support and I'm glad I did it because the test file compresses to 80 bytes, which is the winner of all the compression algorithms in the test. Can't wait to use it on real data! Of note, I do NOT add support for compression levels or using the dictionary (read zstandard docs); this is beyond my Java hacking skills tonight. > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-zstd.patch > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577457#comment-17577457 ] David Handermann commented on NIFI-10313: - Thanks for clarifying, with the issue narrowed down to SPNEGO handling, I will take a closer look at that scenario. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577456#comment-17577456 ] Malthe Borch commented on NIFI-10313: - I wouldn't say it's resolved because this is still a regression since 1.16 in that we were previously fine running this setup on Chrome (which supports SPNEGO) and now the missing cookie seems to cause problems. > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] thenatog commented on pull request #6278: NIFI-10322 Correct Cookie path when removing Bearer Token
thenatog commented on PR #6278: URL: https://github.com/apache/nifi/pull/6278#issuecomment-1209464236 Will review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941410795
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/adapters/HiveMqV5ClientAdapter.java:
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.adapters;
+
+import com.hivemq.client.mqtt.datatypes.MqttQos;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5BlockingClient;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5Client;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5Connect;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5ConnectBuilder;
+import org.apache.nifi.processors.mqtt.common.MqttConnectionProperties;
+import org.apache.nifi.processors.mqtt.common.NifiMqttCallback;
+import org.apache.nifi.processors.mqtt.common.NifiMqttClient;
+import org.apache.nifi.processors.mqtt.common.NifiMqttException;
+import org.apache.nifi.processors.mqtt.common.NifiMqttMessage;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
+import java.util.Arrays;
+import java.util.Objects;
+
+public class HiveMqV5ClientAdapter implements NifiMqttClient {
+
+private final Mqtt5Client mqtt5Client;
+
+private NifiMqttCallback callback;
+
+public HiveMqV5ClientAdapter(Mqtt5BlockingClient mqtt5BlockingClient) {
+this.mqtt5Client = mqtt5BlockingClient;
+}
+
+@Override
+public boolean isConnected() {
+return mqtt5Client.getState().isConnected();
+}
+
+@Override
+public void connect(MqttConnectionProperties connectionProperties) throws
NifiMqttException {
+final Mqtt5ConnectBuilder connectBuilder = Mqtt5Connect.builder()
+.keepAlive(connectionProperties.getKeepAliveInterval());
+
+final boolean cleanSession = connectionProperties.isCleanSession();
+connectBuilder.cleanStart(cleanSession);
+if (!cleanSession) {
+
connectBuilder.sessionExpiryInterval(connectionProperties.getSessionExpiryInterval());
+}
+
+final String lastWillTopic = connectionProperties.getLastWillTopic();
+if (lastWillTopic != null) {
+connectBuilder.willPublish()
+.topic(lastWillTopic)
+
.payload(connectionProperties.getLastWillMessage().getBytes())
+.retain(connectionProperties.getLastWillRetain())
+
.qos(MqttQos.fromCode(connectionProperties.getLastWillQOS()))
+.applyWillPublish();
+}
+
+// checking for presence of password because username can be null
+final char[] password = connectionProperties.getPassword();
+if (password != null) {
+connectBuilder.simpleAuth()
+.username(connectionProperties.getUsername())
+.password(toBytes(password))
+.applySimpleAuth();
+
+clearSensitive(connectionProperties.getPassword());
+clearSensitive(password);
+}
+
+final Mqtt5Connect mqtt5Connect = connectBuilder.build();
+mqtt5Client.toBlocking().connect(mqtt5Connect);
+}
+
+@Override
+public void disconnect(long disconnectTimeout) throws NifiMqttException {
+// Currently it is not possible to set timeout for disconnect with
HiveMQ Client. (Only connect timeout exists.)
+mqtt5Client.toBlocking().disconnect();
+}
+
+@Override
+public void close() throws NifiMqttException {
Review Comment:
Since `NifiMqttException` is a subclass of `RuntimeException`, it does not
need to be declared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941410138
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/adapters/HiveMqV5ClientAdapter.java:
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.adapters;
+
+import com.hivemq.client.mqtt.datatypes.MqttQos;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5BlockingClient;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5Client;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5Connect;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5ConnectBuilder;
+import org.apache.nifi.processors.mqtt.common.MqttConnectionProperties;
+import org.apache.nifi.processors.mqtt.common.NifiMqttCallback;
+import org.apache.nifi.processors.mqtt.common.NifiMqttClient;
+import org.apache.nifi.processors.mqtt.common.NifiMqttException;
+import org.apache.nifi.processors.mqtt.common.NifiMqttMessage;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
+import java.util.Arrays;
+import java.util.Objects;
+
+public class HiveMqV5ClientAdapter implements NifiMqttClient {
+
+private final Mqtt5Client mqtt5Client;
+
+private NifiMqttCallback callback;
+
+public HiveMqV5ClientAdapter(Mqtt5BlockingClient mqtt5BlockingClient) {
+this.mqtt5Client = mqtt5BlockingClient;
+}
+
+@Override
+public boolean isConnected() {
+return mqtt5Client.getState().isConnected();
+}
+
+@Override
+public void connect(MqttConnectionProperties connectionProperties) throws
NifiMqttException {
+final Mqtt5ConnectBuilder connectBuilder = Mqtt5Connect.builder()
+.keepAlive(connectionProperties.getKeepAliveInterval());
+
+final boolean cleanSession = connectionProperties.isCleanSession();
+connectBuilder.cleanStart(cleanSession);
+if (!cleanSession) {
+
connectBuilder.sessionExpiryInterval(connectionProperties.getSessionExpiryInterval());
+}
+
+final String lastWillTopic = connectionProperties.getLastWillTopic();
+if (lastWillTopic != null) {
+connectBuilder.willPublish()
+.topic(lastWillTopic)
+
.payload(connectionProperties.getLastWillMessage().getBytes())
+.retain(connectionProperties.getLastWillRetain())
+
.qos(MqttQos.fromCode(connectionProperties.getLastWillQOS()))
+.applyWillPublish();
+}
+
+// checking for presence of password because username can be null
+final char[] password = connectionProperties.getPassword();
+if (password != null) {
+connectBuilder.simpleAuth()
+.username(connectionProperties.getUsername())
+.password(toBytes(password))
+.applySimpleAuth();
+
+clearSensitive(connectionProperties.getPassword());
+clearSensitive(password);
+}
+
+final Mqtt5Connect mqtt5Connect = connectBuilder.build();
+mqtt5Client.toBlocking().connect(mqtt5Connect);
+}
+
+@Override
+public void disconnect(long disconnectTimeout) throws NifiMqttException {
+// Currently it is not possible to set timeout for disconnect with
HiveMQ Client. (Only connect timeout exists.)
+mqtt5Client.toBlocking().disconnect();
+}
+
+@Override
+public void close() throws NifiMqttException {
+// there is no paho's close equivalent in hivemq client
+}
+
+@Override
+public void publish(String topic, NifiMqttMessage message) throws
NifiMqttException {
+mqtt5Client.toAsync().publishWith()
+.topic(topic)
+
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941407304
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/MqttConstants.java:
##
@@ -67,14 +69,45 @@ public class MqttConstants {
*/
public static final AllowableValue ALLOWABLE_VALUE_MQTT_VERSION_AUTO =
new
AllowableValue(String.valueOf(MqttConnectOptions.MQTT_VERSION_DEFAULT),
-"AUTO",
+"v3 AUTO",
"Start with v3.1.1 and fallback to v3.1.0 if not supported
by a broker");
+public static final AllowableValue ALLOWABLE_VALUE_MQTT_VERSION_500 =
+new
AllowableValue(String.valueOf(MqttVersion.MQTT_VERSION_5_0.getNumericValue()),
+"v5.0");
+
public static final AllowableValue ALLOWABLE_VALUE_MQTT_VERSION_311 =
-new
AllowableValue(String.valueOf(MqttConnectOptions.MQTT_VERSION_3_1_1),
+new
AllowableValue(String.valueOf(MqttVersion.MQTT_VERSION_3_1_1.getNumericValue()),
"v3.1.1");
public static final AllowableValue ALLOWABLE_VALUE_MQTT_VERSION_310 =
-new
AllowableValue(String.valueOf(MqttConnectOptions.MQTT_VERSION_3_1),
+new
AllowableValue(String.valueOf(MqttVersion.MQTT_VERSION_3_1.getNumericValue()),
"v3.1.0");
+
+public enum MqttVersion {
+MQTT_VERSION_3_1(3),
+MQTT_VERSION_3_1_1(4),
+MQTT_VERSION_5_0(5);
+
+private final int numericValue;
+
+MqttVersion(int numericValue) {
+this.numericValue = numericValue;
+}
+
+public int getNumericValue() {
+return numericValue;
+}
+}
+
+public enum SupportedSchemes {
+TCP,
+SSL,
+WS,
+WSS;
+
+public static String getValuesAsString(String delimiter) {
+return String.join(delimiter, Arrays.stream(values()).map(value ->
value.name().toLowerCase()).toArray(String[]::new));
+}
+}
Review Comment:
Recommend promoting this to a separate file instead of embedded within
`MqttConstants`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941406015
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/NifiMqttMessage.java:
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.common;
+
+public class NifiMqttMessage {
Review Comment:
For this class `StandardMqttMessage` seems like a better name to avoid the
`Nifi` prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941405400
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/NifiMqttClient.java:
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.common;
+
+public interface NifiMqttClient {
Review Comment:
In general it is better to avoid prefixing class names with `Nifi` since
that is already implied through the package name. Perhaps something like
`ProcessorMqttClient` or `ComponentMqttClient` would be better, open to other
alternatives.
As an interface, it would also be helpful to include JavaDocs with at least
one sentence describing basic behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941398697
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/adapters/HiveMqV5ClientAdapter.java:
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.adapters;
+
+import com.hivemq.client.mqtt.datatypes.MqttQos;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5BlockingClient;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5Client;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5Connect;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5ConnectBuilder;
+import org.apache.nifi.processors.mqtt.common.MqttConnectionProperties;
+import org.apache.nifi.processors.mqtt.common.NifiMqttCallback;
+import org.apache.nifi.processors.mqtt.common.NifiMqttClient;
+import org.apache.nifi.processors.mqtt.common.NifiMqttException;
+import org.apache.nifi.processors.mqtt.common.NifiMqttMessage;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
+import java.util.Arrays;
+import java.util.Objects;
+
+public class HiveMqV5ClientAdapter implements NifiMqttClient {
+
+private final Mqtt5Client mqtt5Client;
+
+private NifiMqttCallback callback;
+
+public HiveMqV5ClientAdapter(Mqtt5BlockingClient mqtt5BlockingClient) {
+this.mqtt5Client = mqtt5BlockingClient;
+}
+
+@Override
+public boolean isConnected() {
+return mqtt5Client.getState().isConnected();
+}
+
+@Override
+public void connect(MqttConnectionProperties connectionProperties) throws
NifiMqttException {
+final Mqtt5ConnectBuilder connectBuilder = Mqtt5Connect.builder()
+.keepAlive(connectionProperties.getKeepAliveInterval());
+
+final boolean cleanSession = connectionProperties.isCleanSession();
+connectBuilder.cleanStart(cleanSession);
+if (!cleanSession) {
+
connectBuilder.sessionExpiryInterval(connectionProperties.getSessionExpiryInterval());
+}
+
+final String lastWillTopic = connectionProperties.getLastWillTopic();
+if (lastWillTopic != null) {
+connectBuilder.willPublish()
+.topic(lastWillTopic)
+
.payload(connectionProperties.getLastWillMessage().getBytes())
+.retain(connectionProperties.getLastWillRetain())
+
.qos(MqttQos.fromCode(connectionProperties.getLastWillQOS()))
+.applyWillPublish();
+}
+
+// checking for presence of password because username can be null
+final char[] password = connectionProperties.getPassword();
+if (password != null) {
+connectBuilder.simpleAuth()
+.username(connectionProperties.getUsername())
+.password(toBytes(password))
+.applySimpleAuth();
+
+clearSensitive(connectionProperties.getPassword());
+clearSensitive(password);
+}
+
+final Mqtt5Connect mqtt5Connect = connectBuilder.build();
+mqtt5Client.toBlocking().connect(mqtt5Connect);
+}
+
+@Override
+public void disconnect(long disconnectTimeout) throws NifiMqttException {
+// Currently it is not possible to set timeout for disconnect with
HiveMQ Client. (Only connect timeout exists.)
+mqtt5Client.toBlocking().disconnect();
+}
+
+@Override
+public void close() throws NifiMqttException {
+// there is no paho's close equivalent in hivemq client
+}
+
+@Override
+public void publish(String topic, NifiMqttMessage message) throws
NifiMqttException {
+mqtt5Client.toAsync().publishWith()
+.topic(topic)
+
[jira] [Commented] (NIFI-10333) Hikari CP 4.0.3 to 5.0.1
[ https://issues.apache.org/jira/browse/NIFI-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577442#comment-17577442 ] Mike R commented on NIFI-10333: --- [~exceptionfactory] thanks for the information. When NiFi moves from Java 8 to either 11 or 17, this will get updated, so making note of that now > Hikari CP 4.0.3 to 5.0.1 > > > Key: NIFI-10333 > URL: https://issues.apache.org/jira/browse/NIFI-10333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0, 1.16.2, 1.16.3 >Reporter: Mike R >Priority: Major > > The version of HikariCP that NiFi is using is version Hikari CP 4.0.3. It is > vulnerable to the following 8 vulnerabilities due to the dependencies: > [CVE-2022-23221|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23221] > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2021-45105|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105] > [CVE-2021-45046|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046] > [CVE-2021-44832|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832] > [CVE-2021-44228|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228] > [CVE-2021-42392|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] > In version 5.0.1, it is only vulnerable to 2 CVEs. > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[ https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577439#comment-17577439 ] David Handermann commented on NIFI-10313: - Thanks for following up on this issue [~malthe], so aside from the browser support issue, are there any other remaining concerns, or do you consider this issue resolved? > Unexpected "Access Token not found" > --- > > Key: NIFI-10313 > URL: https://issues.apache.org/jira/browse/NIFI-10313 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Malthe Borch >Assignee: David Handermann >Priority: Major > Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log > > > I'm experiencing some unexpected "Access Token not found" errors after > upgrading to 1.17.0. > See attached traceback. > What happens is that the NiFi UI seems to work but after a short while the > view is redirected to a conflict page (Unable to communicate with NiFi). > There are no other problems or evidence of the issue to be found in the logs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10313) Unexpected "Access Token not found"
[
https://issues.apache.org/jira/browse/NIFI-10313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577437#comment-17577437
]
Malthe Borch commented on NIFI-10313:
-
We've figured out why the cookie is missing – it seems that for a browser that
is able to use SPNEGO authentication (such as Chrome in our systems), no
{{__Secure-Authorization-Bearer}} cookie is sent.
Meanwhile, in Firefox (which does not support SPNEGO in our system), we do see
the cookie.
> Unexpected "Access Token not found"
> ---
>
> Key: NIFI-10313
> URL: https://issues.apache.org/jira/browse/NIFI-10313
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Malthe Borch
>Assignee: David Handermann
>Priority: Major
> Attachments: NiFi-Errors.PNG, authorizers.xml, nifi.log
>
>
> I'm experiencing some unexpected "Access Token not found" errors after
> upgrading to 1.17.0.
> See attached traceback.
> What happens is that the NiFi UI seems to work but after a short while the
> view is redirected to a conflict page (Unable to communicate with NiFi).
> There are no other problems or evidence of the issue to be found in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] tpalfy commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
tpalfy commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941386857
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/adapters/HiveMqV5ClientAdapter.java:
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mqtt.adapters;
+
+import com.hivemq.client.mqtt.datatypes.MqttQos;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5BlockingClient;
+import com.hivemq.client.mqtt.mqtt5.Mqtt5Client;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5Connect;
+import com.hivemq.client.mqtt.mqtt5.message.connect.Mqtt5ConnectBuilder;
+import org.apache.nifi.processors.mqtt.common.MqttConnectionProperties;
+import org.apache.nifi.processors.mqtt.common.NifiMqttCallback;
+import org.apache.nifi.processors.mqtt.common.NifiMqttClient;
+import org.apache.nifi.processors.mqtt.common.NifiMqttException;
+import org.apache.nifi.processors.mqtt.common.NifiMqttMessage;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
+import java.util.Arrays;
+import java.util.Objects;
+
+public class HiveMqV5ClientAdapter implements NifiMqttClient {
+
+private final Mqtt5Client mqtt5Client;
+
+private NifiMqttCallback callback;
+
+public HiveMqV5ClientAdapter(Mqtt5BlockingClient mqtt5BlockingClient) {
+this.mqtt5Client = mqtt5BlockingClient;
+}
+
+@Override
+public boolean isConnected() {
+return mqtt5Client.getState().isConnected();
+}
+
+@Override
+public void connect(MqttConnectionProperties connectionProperties) throws
NifiMqttException {
+final Mqtt5ConnectBuilder connectBuilder = Mqtt5Connect.builder()
+.keepAlive(connectionProperties.getKeepAliveInterval());
+
+final boolean cleanSession = connectionProperties.isCleanSession();
+connectBuilder.cleanStart(cleanSession);
+if (!cleanSession) {
+
connectBuilder.sessionExpiryInterval(connectionProperties.getSessionExpiryInterval());
+}
+
+final String lastWillTopic = connectionProperties.getLastWillTopic();
+if (lastWillTopic != null) {
+connectBuilder.willPublish()
+.topic(lastWillTopic)
+
.payload(connectionProperties.getLastWillMessage().getBytes())
+.retain(connectionProperties.getLastWillRetain())
+
.qos(MqttQos.fromCode(connectionProperties.getLastWillQOS()))
+.applyWillPublish();
+}
+
+// checking for presence of password because username can be null
+final char[] password = connectionProperties.getPassword();
+if (password != null) {
+connectBuilder.simpleAuth()
+.username(connectionProperties.getUsername())
+.password(toBytes(password))
+.applySimpleAuth();
+
+clearSensitive(connectionProperties.getPassword());
+clearSensitive(password);
+}
+
+final Mqtt5Connect mqtt5Connect = connectBuilder.build();
+mqtt5Client.toBlocking().connect(mqtt5Connect);
+}
+
+@Override
+public void disconnect(long disconnectTimeout) throws NifiMqttException {
+// Currently it is not possible to set timeout for disconnect with
HiveMQ Client. (Only connect timeout exists.)
+mqtt5Client.toBlocking().disconnect();
+}
+
+@Override
+public void close() throws NifiMqttException {
+// there is no paho's close equivalent in hivemq client
+}
+
+@Override
+public void publish(String topic, NifiMqttMessage message) throws
NifiMqttException {
+mqtt5Client.toAsync().publishWith()
+.topic(topic)
+
[jira] [Commented] (NIFI-10332) Add ID_TOKEN_LOGOUT support for general OpenID connect server, e.g. Keycloak
[
https://issues.apache.org/jira/browse/NIFI-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577427#comment-17577427
]
David Handermann commented on NIFI-10332:
-
Thanks for reporting this issue [~macdoor615], I noticed the same issue while
testing OpenID Connect integration with Keycloak 18. This looks like a helpful
improvement, I will take a closer look at the OpenID Connect specification.
Adhering to the specification should be supported.
> Add ID_TOKEN_LOGOUT support for general OpenID connect server, e.g. Keycloak
>
>
> Key: NIFI-10332
> URL: https://issues.apache.org/jira/browse/NIFI-10332
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Affects Versions: 1.17.0
> Environment: NiFi 1.17.0, Keycloak 18.0.1
>Reporter: macdoor615
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-09-16-56-25-791.png
>
>
> I deploy a NiFi 1.170 and authenticate with OpenID connect. Authentication
> server is Keycloak 18.0.1.
> I can log in and I can use UI properly.
> But when I logout. I get an error, can not redirect to NiFi UI or keycloak
> login UI
> !image-2022-08-09-16-56-25-791.png|width=782,height=347!
> [https://36.133.55.100:8943/realms/zznode/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2F36.138.166.203%3A18089%2Fhb3-dmz-repos-000-nifi%2Fnifi-api%2F..%2Fnifi%2Flogout-complete]
> I made some investigation into source code. I found NiFi only support
> ID_TOKEN_LOGOUT for okta service. Keycloak and other Authentication server
> can not be supported.
> Keycloak say it is compliance OpenID connect spec.
> I modified a few lines of source code. Let it support ID_TOKEN_LOGOUT for
> keycloak. Now I can log out NiFi and redirect to keycloak login UI, and than
> login NiFi again.
> I suggest making nifi to support ID_TOKEN_LOGOUT in later version for general
> OpenID connect server.
> I modified the file,
> [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/OIDCAccessResource.java]
> start from line 403
> {code:java}
> private String determineLogoutMethod(String oidcDiscoveryUrl) {
> Matcher accessTokenMatcher =
> REVOKE_ACCESS_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
> Matcher idTokenMatcher =
> ID_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
>
> if (accessTokenMatcher.find()) {
> return REVOKE_ACCESS_TOKEN_LOGOUT;
> } else {
> return ID_TOKEN_LOGOUT;
> }
> }
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10333) Hikari CP 4.0.3 to 5.0.1
[ https://issues.apache.org/jira/browse/NIFI-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577425#comment-17577425 ] David Handermann commented on NIFI-10333: - [~msr1716], the vulnerabilities listed apply to various database vendor drivers and do not apply to HikariCP. HikariCP 5.0 requires Java 11, and for now NiFi still supports Java 8, so 4.0.3 is the latest version. > Hikari CP 4.0.3 to 5.0.1 > > > Key: NIFI-10333 > URL: https://issues.apache.org/jira/browse/NIFI-10333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0, 1.16.2, 1.16.3 >Reporter: Mike R >Priority: Major > > The version of HikariCP that NiFi is using is version Hikari CP 4.0.3. It is > vulnerable to the following 8 vulnerabilities due to the dependencies: > [CVE-2022-23221|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23221] > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2021-45105|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105] > [CVE-2021-45046|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046] > [CVE-2021-44832|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832] > [CVE-2021-44228|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228] > [CVE-2021-42392|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] > In version 5.0.1, it is only vulnerable to 2 CVEs. > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10322) invalid_token error after OpenID connect session timeout
[
https://issues.apache.org/jira/browse/NIFI-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577423#comment-17577423
]
David Handermann commented on NIFI-10322:
-
That's great [~macdoor615]! I really appreciate your help in tracking down the
problem and verifying the solution!
> invalid_token error after OpenID connect session timeout
>
>
> Key: NIFI-10322
> URL: https://issues.apache.org/jira/browse/NIFI-10322
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-05-22-48-17-835.png,
> image-2022-08-05-22-48-52-057.png, image-2022-08-07-14-28-09-058.png,
> image-2022-08-07-15-22-36-213.png, image-2022-08-07-15-27-18-902.png,
> image-2022-08-07-15-37-29-739.png, image-2022-08-07-15-43-14-922.png,
> image-2022-08-07-15-47-57-158.png, image-2022-08-07-15-53-47-220.png,
> image-2022-08-07-16-00-11-443.png, image-2022-08-07-16-11-38-180.png,
> image-2022-08-08-23-33-30-220.png, image-2022-08-08-23-35-02-773.png,
> image-2022-08-08-23-59-12-471.png, nginx-access.log.zip,
> nifi-1.16.3-logs.zip, nifi-1.17.0-logs.zip
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I follow
> [https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect] to
> config NIFI 1.16.3 and it is work properly. If the session times out, login
> again and it will work again
> I configured 1.17.0 in the same way. I can login and operate nifi UI. But
> when session times out. I got the following error.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Expired JWT",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-17-835.png|width=758,height=108!
> I try to login again and get a new error, and I cannot enter the NIFI
> interface.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Signed JWT rejected: Another algorithm
> expected, or no matching key(s) found",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-52-057.png|width=594,height=143!
> I did some research, and found
> After the session times out,
> NIFI 1.16.3 leaves 3 cookies in browser:
> * nifi-logout-request-identifier
> * nifi-oidc-request-identifier
> * __Secure-Request-Token
> NIFI 1.17.0 leaves 2 cookies:
> * *__Secure-Authorization-Bearer*
> * __Secure-Request-Token
> __Secure-Authorization-Bearer cookie contains a expired JWT:
> {code:java}
> eyJraWQiOiJhMDlhZDhlMy0xZDkzLTQyZTEtYjg0Ni0xMWU0ODRkODYwYWYiLCJhbGciOiJQUzUxMiJ9.eyJzdWIiOiJhZG1pbi5uaWZpQGd1bWhiMy5jb20iLCJhdWQiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwibmJmIjoxNjU5NjExOTc0LCJpc3MiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW4ubmlmaUBndW1oYjMuY29tIiwiZXhwIjoxNjU5NjEyMjc0LCJpYXQiOjE2NTk2MTE5NzQsImp0aSI6IjFiZTg5MjU4LTliZmYtNDhmOS04OGNmLWU0NDIzMDZjYzg4ZCJ9.Y9yE0hNH_q-W94_cFWOWGc7TPMP2xB9coaSRPT9twYqSyjTtudOiiXGxHEDUWsOvUFf7lT7wNH4RZ_LhOM-5WfTZ3o-DCVFnl0JjeZ-L9d-z3rO4dEspRxXpr46AewEGy_lpstSUFyihr4i8b2VI7IT0aFOCGAIXRWl7gfH75e5La_0tbsu9lgSRdyYBBv8rSjojJC5bBSqxj-BkrfjdMhyMuF9OdMCJNmyh18BrXbavwftNerytkd_Qf9eNLmzsZ3SOdKWpftKt4kClD_KeL0nOglhM-ENyb4QLwxr7l5lhUgQ-2am3x5okbRyYip_WV4YQ6DfmUnLL1FYFATWXa5CUimSRbSZzkqU2JEYerpvKsTf-prdsSNryPbrQdf5HqpwhlGbFrgm4jwtncZHTLEL4ZMciVe0H-zIcQ9vyDqamMpf6fyNWmQN8DdDP9A0Zpo7SL7yhOUjNGsjk1gV4OAHWgp4XQzj4KwoGf7ICjeOrzinECHFZw9Ccyi8KMooRx4u3oAuKPEx3mrZFNFDaiAzWX0kZ31c24-15cno2bLBMGOIx7ipjb6Pv7V6O9S2aA2vC3eVLnfAgHAox3I8_IzWLUKddHCqd6cfA1XW8ckSgg2QddKvgYHiCZpwVV4AMDpK4bI1J0ZbxbgOOke9IMMudNhZUFQdWJIXh-gx1bII{code}
> I manually delete __Secure-Authorization-Bearer cookie, and I can login NIFI
> 1.17.0 again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] kulikg opened a new pull request, #6279: NIFI-10230 added FetchSmb
kulikg opened a new pull request, #6279: URL: https://github.com/apache/nifi/pull/6279 # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-10230) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [x] JDK 8 - [x ] JDK 11 - [ ] JDK 17 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-10329) Upgrade jasn1-compiler to 1.11.3
[ https://issues.apache.org/jira/browse/NIFI-10329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-10329. --- Fix Version/s: 1.18.0 Resolution: Fixed > Upgrade jasn1-compiler to 1.11.3 > > > Key: NIFI-10329 > URL: https://issues.apache.org/jira/browse/NIFI-10329 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Fix For: 1.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade jasn1-compiler from 1.11.2 to 1.11.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10329) Upgrade jasn1-compiler to 1.11.3
[ https://issues.apache.org/jira/browse/NIFI-10329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577415#comment-17577415 ] ASF subversion and git services commented on NIFI-10329: Commit 3341e789f02fdc22ad00ef7d3d62d28aa68d13e9 in nifi's branch refs/heads/main from UcanInfosec [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3341e789f0 ] NIFI-10329 Upgrade jasn1-compiler to 1.11.3 Signed-off-by: Pierre Villard This closes #6277. > Upgrade jasn1-compiler to 1.11.3 > > > Key: NIFI-10329 > URL: https://issues.apache.org/jira/browse/NIFI-10329 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade jasn1-compiler from 1.11.2 to 1.11.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] asfgit closed pull request #6277: NIFI-10329 Upgrade jasn1-compiler to 1.11.3
asfgit closed pull request #6277: NIFI-10329 Upgrade jasn1-compiler to 1.11.3 URL: https://github.com/apache/nifi/pull/6277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577413#comment-17577413 ] Matthew Hawkins commented on NIFI-2827: --- Attached is a quick & dirty zstd implementation; it compiled with JDK11 and passes the test. Key thing is that Apache Commons has zstandard support since 1.16 and 1.21 (latest) is included in NiFi 1.17.0 pom. The same technique could be used to add brotli and others supported by org.apache.commons.compress and in fact you could probably refactor the whole thing to use Apache Commons instead of a gazillion other libraries. That was out of scope for me, I just wanted zstd support and I'm glad I did it because the test file compresses to 80 bytes, which is the winner of all the compression algorithms in the test. Can't wait to use it on real data! Of note, I do NOT add support for compression levels or using the dictionary (read zstandard docs); this is beyond my Java hacking skills tonight. > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-zstd.patch > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)
[ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Hawkins updated NIFI-2827: -- Attachment: NIFI-2827-zstd.patch > CompressContent should support brotli (MIT) and zstd (BSD) > -- > > Key: NIFI-2827 > URL: https://issues.apache.org/jira/browse/NIFI-2827 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andre F de Miranda >Priority: Major > Attachments: NIFI-2827-zstd.patch > > > brotli and zstd are gaining lots of momentum across the IoT sphere and I > suspect it would be a nice to have in NiFi to be able to support those two > emerging compression algorithms. > Both brotli and zstd JAVA implementations are JNIs > https://github.com/MeteoGroup/jbrotli (ASL 2) > https://github.com/luben/zstd-jni (3 clause BSD) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941321632
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java:
##
@@ -199,7 +199,7 @@ public abstract class AbstractMQTTProcessor extends
AbstractSessionFactoryProces
.build();
public static final PropertyDescriptor PROP_SESSION_EXPIRY_INTERVAL = new
PropertyDescriptor.Builder()
-.name("Session Expiry Interval")
+.name("Session Expiry Interval (seconds)")
Review Comment:
Reading more closely, I see this is a new property, so disregard the
previous comment. Even though the unit of measure is different, I tend to agree
with @tpalfy that using the Time Unit Validator is better for this new property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
exceptionfactory commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941319081
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java:
##
@@ -199,7 +199,7 @@ public abstract class AbstractMQTTProcessor extends
AbstractSessionFactoryProces
.build();
public static final PropertyDescriptor PROP_SESSION_EXPIRY_INTERVAL = new
PropertyDescriptor.Builder()
-.name("Session Expiry Interval")
+.name("Session Expiry Interval (seconds)")
Review Comment:
Changing the property name breaks backward compatibility, so this change
needs to be reverted. It is possible to change the `displayName`, but in this
case, I think it is better to rely on the description instead of adding hint to
the property name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6265: NIFI-10234 Implement PutIoTDB
exceptionfactory commented on code in PR #6265:
URL: https://github.com/apache/nifi/pull/6265#discussion_r941316372
##
nifi-nar-bundles/nifi-iotdb-bundle/nifi-iotdb-processors/src/main/java/org/apache/nifi/processors/PutIoTDB.java:
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;
+
+import java.io.InputStream;
+import java.text.SimpleDateFormat;
+import java.util.Set;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.processors.model.Schema;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.util.Tuple;
+
+@Tags({"iotdb", "insert", "tablet"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription(
+"This is a record aware processor that reads the content of the
incoming FlowFile as individual records using the "
++ "configured 'Record Reader' and writes them to Apache IoTDB
using native interface.")
+public class PutIoTDB extends AbstractIoTDB {
+
+static final PropertyDescriptor RECORD_READER_FACTORY =
+new PropertyDescriptor.Builder()
+.name("Record Reader")
+.description(
+"Specifies the type of Record Reader controller
service to use for parsing the incoming data "
++ "and determining the schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor TIME_FIELD =
+new PropertyDescriptor.Builder()
+.name("Time Field")
+.description(
+"The field name which represents time. It can be
updated by expression language.")
+.defaultValue("Time")
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.required(false)
+.build();
+
+static final PropertyDescriptor SCHEMA =
+new PropertyDescriptor.Builder()
+.name("Schema")
+.description(
+"The schema that IoTDB needs doesn't support good
by NiFi.\n"
++ "Therefore, you can define the schema
here.\n"
++ "Besides, you can set encoding type and
compression type by this method.\n"
++ "If you don't set this property, the
inferred schema will be used.\n"
++ "It can be updated by expression
language.")
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.required(false)
+.build();
+
+static final PropertyDescriptor ALIGNED =
+new PropertyDescriptor.Builder()
+.name("Aligned")
+
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941304237
##
docker/centos/Dockerfile:
##
@@ -17,33 +17,47 @@
#
# First stage: the build environment
-FROM centos:7 AS build_deps
+FROM centos:7
LABEL maintainer="Apache NiFi "
ARG MINIFI_VERSION
+ARG UID=1000
+ARG GID=1000
+
+# MINIFI_OPTIONS will be passed directly to cmake
+# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
+ARG MINIFI_OPTIONS=""
+ARG CMAKE_BUILD_TYPE=Release
+ARG DOCKER_SKIP_TESTS=ON
# Install the system dependencies needed for a build
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
+ENV USER minificpp
# gpsd-devel is in EPEL
RUN yum -y install epel-release && yum -y install java-1.8.0-openjdk
java-1.8.0-openjdk-devel python36-devel sudo git which maven make libarchive
boost-devel lua-devel libusbx-devel libpng-devel \
-gpsd-devel libpcap-devel
+gpsd-devel libpcap-devel ccache
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
-FROM build_deps AS release
+RUN cd $MINIFI_BASE_DIR && \
+./bootstrap.sh -t && \
+ln -s /usr/bin/ccache /usr/lib64/ccache/c++
-# MINIFI_OPTIONS will be passed directly to cmake
-# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
-ARG MINIFI_OPTIONS=""
-ARG CMAKE_BUILD_TYPE=Release
-# Perform the build
-RUN cd $MINIFI_BASE_DIR \
- && ./bootstrap.sh -t \
- && cd build \
-&& scl enable devtoolset-10 -- cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=true
${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON
-DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. \
-&& scl enable devtoolset-10 -- make -j "$(nproc)" package
+# Setup minificpp user
+RUN groupadd -g ${GID} ${USER} && useradd -g ${GID} ${USER} && \
+chown -R ${USER}:${USER} ${MINIFI_BASE_DIR} && \
+if [ -d "${MINIFI_BASE_DIR}/.ccache" ]; then mv ${MINIFI_BASE_DIR}/.ccache
/home/${USER}/.ccache; fi
+
+USER ${USER}
+# Perform the build
+RUN cd $MINIFI_BASE_DIR && \
+cd build && \
+source /opt/rh/devtoolset-10/enable && \
+export PATH=/usr/lib64/ccache${PATH:+:${PATH}} && \
+cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=${DOCKER_SKIP_TESTS} ${MINIFI_OPTIONS}
-DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON
-DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. && \
+make -j "$(nproc)" package
Review Comment:
I use `source /opt/rh/devtoolset-10/enable` to have it enabled in this
context.
##
docker/DockerBuild.sh:
##
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
-"--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+"--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+"--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target
build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
In the plain docker build we have multiple stages and use the `--target
build` to create the build image then copy the ccache files from that image if
needed, then continue with the second stage for the released image. In the
other distro builds we don't have separate build and release images, because we
don't use those images for running MiNiFi only get the built binaries from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941303138
##
docker/DockerBuild.sh:
##
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
-"--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+"--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+"--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target
build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while
true; do sleep 1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE}
--target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
Review Comment:
DOCKER_BUILDKIT enables additional features like the copying of files
between multiple stages and only building specific stages, that we use in the
alpine based docker image. We don't need that in the other distros. I would use
it with `DOCKER_BUILDKIT=1` by default, but I'm not sure if there was a problem
with that on any build machines previously, as it was changed to 0 in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6159: NIFI-8248 Modify PutAzureDataLakeStorage processor to use temp file i…
exceptionfactory commented on PR #6159: URL: https://github.com/apache/nifi/pull/6159#issuecomment-1209340410 Thanks for the reminder @timeabarna, will plan on reviewing this soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941280092
##
docker/DockerBuild.sh:
##
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
-"--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+"--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+"--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target
build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while
true; do sleep 1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE}
--target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
Review Comment:
What does `DOCKER_BUILDKIT=0/1` do, and why did it change here from 0 to 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941271222
##
.github/workflows/ci.yml:
##
@@ -201,11 +199,19 @@ jobs:
sudo apt install -y ccache
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
-run: mkdir build && cd build && cmake -DSTRICT_GSL_CHECKS=AUDIT
-DENABLE_KUBERNETES=ON .. && make centos
+run: |
+ if [ -d ~/.ccache ]; then mv ~/.ccache .; fi
+ mkdir build && cd build && cmake -DUSE_SHARED_LIBS=ON -DCI_BUILD=ON
-DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DENABLE_AWS=ON
-DENABLE_AZURE=ON -DENABLE_COAP=ON \
+ -DENABLE_ENCRYPT_CONFIG=ON -DENABLE_GPS=ON -DENABLE_JNI=ON
-DENABLE_LIBRDKAFKA=ON -DENABLE_LINTER=ON -DENABLE_MQTT=ON -DENABLE_NANOFI=ON
-DENABLE_OPC=ON -DENABLE_OPENCV=ON \
+ -DENABLE_OPENWSMAN=ON -DENABLE_OPS=ON -DENABLE_PYTHON=ON
-DENABLE_SENSORS=ON -DENABLE_SFTP=ON -DENABLE_SQL=ON -DENABLE_SYSTEMD=ON
-DENABLE_TENSORFLOW=OFF \
+ -DENABLE_USB_CAMERA=ON -DENABLE_SCRIPTING=ON
-DENABLE_LUA_SCRIPTING=ON -DENABLE_KUBERNETES=ON -DENABLE_GCP=ON
-DENABLE_PROCFS=ON -DENABLE_PROMETHEUS=ON -DENABLE_ELASTICSEARCH=ON \
+ -DDOCKER_SKIP_TESTS=OFF -DDOCKER_BUILD_ONLY=ON
-DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache .. && make centos
+ - id: test
+run: docker run --rm apacheminificpp:$(docker images | grep
apacheminificpp | grep centos | awk '{print $2}') bash -c 'cd /opt/minifi/build
&& make test ARGS="--timeout 300 -j8 --output-on-failure"'
docker_integration_tests:
name: "Docker integration tests"
runs-on: ubuntu-20.04
-timeout-minutes: 120
+timeout-minutes: 180
Review Comment:
we could increase the timeout on the MacOS build to 180 minutes, too, as I
have seen timeouts on that job in my fork, too
##
docker/DockerBuild.sh:
##
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
-"--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+"--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+"--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target
build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while
true; do sleep 1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE}
--target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep
1; done")
+mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+docker cp "${container_id}:/home/minificpp/.ccache/."
"${DOCKER_CCACHE_DUMP_LOCATION}"
+docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
Review Comment:
What does `DOCKER_BUILDKIT=0/1` do, and why it change here from 0 to 1?
##
docker/DockerBuild.sh:
##
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
-"--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+"--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+"--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target
build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t
apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build
[jira] [Updated] (NIFI-10333) Hikari CP 4.0.3 to 5.0.1
[ https://issues.apache.org/jira/browse/NIFI-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R updated NIFI-10333: -- Affects Version/s: 1.16.3 1.16.2 1.17.0 > Hikari CP 4.0.3 to 5.0.1 > > > Key: NIFI-10333 > URL: https://issues.apache.org/jira/browse/NIFI-10333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0, 1.16.2, 1.16.3 >Reporter: Mike R >Priority: Major > > The version of HikariCP that NiFi is using is version Hikari CP 4.0.3. It is > vulnerable to the following 8 vulnerabilities due to the dependencies: > [CVE-2022-23221|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23221] > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2021-45105|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105] > [CVE-2021-45046|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046] > [CVE-2021-44832|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832] > [CVE-2021-44228|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228] > [CVE-2021-42392|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] > In version 5.0.1, it is only vulnerable to 2 CVEs. > [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] > [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10333) Hikari CP 4.0.3 to 5.0.1
Mike R created NIFI-10333: - Summary: Hikari CP 4.0.3 to 5.0.1 Key: NIFI-10333 URL: https://issues.apache.org/jira/browse/NIFI-10333 Project: Apache NiFi Issue Type: Bug Reporter: Mike R The version of HikariCP that NiFi is using is version Hikari CP 4.0.3. It is vulnerable to the following 8 vulnerabilities due to the dependencies: [CVE-2022-23221|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23221] [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] [CVE-2021-45105|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105] [CVE-2021-45046|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046] [CVE-2021-44832|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832] [CVE-2021-44228|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228] [CVE-2021-42392|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392] [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] In version 5.0.1, it is only vulnerable to 2 CVEs. [CVE-2022-21724|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21724] [CVE-2020-25638|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25638] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1382: MINIFICPP-1899 - Fix local Docker build failure
fgerlits closed pull request #1382: MINIFICPP-1899 - Fix local Docker build failure URL: https://github.com/apache/nifi-minifi-cpp/pull/1382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1381: MINIFICPP-1898 - Fix ListenHTTP test failure
fgerlits closed pull request #1381: MINIFICPP-1898 - Fix ListenHTTP test failure URL: https://github.com/apache/nifi-minifi-cpp/pull/1381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-10312) Broken flowid handling in minifi-c2 service
[ https://issues.apache.org/jira/browse/NIFI-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Liszli reassigned NIFI-10312: Assignee: Robert Liszli > Broken flowid handling in minifi-c2 service > --- > > Key: NIFI-10312 > URL: https://issues.apache.org/jira/browse/NIFI-10312 > Project: Apache NiFi > Issue Type: Bug > Components: C2, MiNiFi >Reporter: Csaba Bejan >Assignee: Robert Liszli >Priority: Major > > When MiNiFi is communication with minifi-c2 service via the C2 protocol the > flowId parsing seems to be broken so the publishing mechanism is not working. > Pattern needs to be updated to match the expected url structure or flow id > needs to be sent in a more robust way as current approach is error prone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1383: MINIFICPP-1875 HTTPClient should be reusable
martinzink opened a new pull request, #1383: URL: https://github.com/apache/nifi-minifi-cpp/pull/1383 MINIFICPP-1835 InvokeHTTP connection keep-alive MINIFICPP-1895 InvokeHTTP PropPutOutputAttributes fix MINIFICPP-1897 Reenable docker/test/integration/https.feature Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] nandorsoma commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
nandorsoma commented on code in PR #6225: URL: https://github.com/apache/nifi/pull/6225#discussion_r941205695 ## nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-nar/src/main/resources/META-INF/NOTICE: ## @@ -41,6 +41,11 @@ The following binary components are provided under the Apache Software License v in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. + (ASLv2) HiveMQ MQTT Client Review Comment: Thanks for NOTICING that! (pun intended, change added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] nandorsoma commented on a diff in pull request #6225: NIFI-10251 Add v5 protocol support for existing MQTT processors
nandorsoma commented on code in PR #6225:
URL: https://github.com/apache/nifi/pull/6225#discussion_r941155396
##
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java:
##
@@ -199,16 +198,13 @@ public ValidationResult validate(String subject, String
input, ValidationContext
.defaultValue(ALLOWABLE_VALUE_CLEAN_SESSION_TRUE.getValue())
.build();
-public static final PropertyDescriptor PROP_MQTT_VERSION = new
PropertyDescriptor.Builder()
-.name("MQTT Specification Version")
-.description("The MQTT specification version when connecting with
the broker. See the allowable value descriptions for more details.")
-.allowableValues(
-ALLOWABLE_VALUE_MQTT_VERSION_AUTO,
-ALLOWABLE_VALUE_MQTT_VERSION_311,
-ALLOWABLE_VALUE_MQTT_VERSION_310
-)
-.defaultValue(ALLOWABLE_VALUE_MQTT_VERSION_AUTO.getValue())
-.required(true)
+public static final PropertyDescriptor PROP_SESSION_EXPIRY_INTERVAL = new
PropertyDescriptor.Builder()
+.name("Session Expiry Interval")
+.description("After this interval the broker will expire the
client and clear the session state.")
+.addValidator(StandardValidators.NON_NEGATIVE_LONG_VALIDATOR)
+.dependsOn(PROP_MQTT_VERSION, ALLOWABLE_VALUE_MQTT_VERSION_500)
+.dependsOn(PROP_CLEAN_SESSION, ALLOWABLE_VALUE_CLEAN_SESSION_FALSE)
+.defaultValue(Long.toString(SESSION_EXPIRY_INTERVAL_IN_SECONDS))
.build();
Review Comment:
Thanks for the suggestion @tpalfy ! I like this solution however there are
similar properties in the processor where just numbers without units were used.
I think it is better to stick to the original version to avoid mixing the input
types. Nevertheless I've added a hint to the name of the property to match the
other properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-10332) Add ID_TOKEN_LOGOUT support for general OpenID connect server, e.g. Keycloak
[
https://issues.apache.org/jira/browse/NIFI-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
macdoor615 updated NIFI-10332:
--
Summary: Add ID_TOKEN_LOGOUT support for general OpenID connect server,
e.g. Keycloak (was: Add ID_TOKEN_LOGOUT support for general OpenID connect
server, like Keycloak)
> Add ID_TOKEN_LOGOUT support for general OpenID connect server, e.g. Keycloak
>
>
> Key: NIFI-10332
> URL: https://issues.apache.org/jira/browse/NIFI-10332
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Affects Versions: 1.17.0
> Environment: NiFi 1.17.0, Keycloak 18.0.1
>Reporter: macdoor615
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-09-16-56-25-791.png
>
>
> I deploy a NiFi 1.170 and authenticate with OpenID connect. Authentication
> server is Keycloak 18.0.1.
> I can log in and I can use UI properly.
> But when I logout. I get an error, can not redirect to NiFi UI or keycloak
> login UI
> !image-2022-08-09-16-56-25-791.png|width=782,height=347!
> [https://36.133.55.100:8943/realms/zznode/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2F36.138.166.203%3A18089%2Fhb3-dmz-repos-000-nifi%2Fnifi-api%2F..%2Fnifi%2Flogout-complete]
> I made some investigation into source code. I found NiFi only support
> ID_TOKEN_LOGOUT for okta service. Keycloak and other Authentication server
> can not be supported.
> Keycloak say it is compliance OpenID connect spec.
> I modified a few lines of source code. Let it support ID_TOKEN_LOGOUT for
> keycloak. Now I can log out NiFi and redirect to keycloak login UI, and than
> login NiFi again.
> I suggest making nifi to support ID_TOKEN_LOGOUT in later version for general
> OpenID connect server.
> I modified the file,
> [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/OIDCAccessResource.java]
> start from line 403
> {code:java}
> private String determineLogoutMethod(String oidcDiscoveryUrl) {
> Matcher accessTokenMatcher =
> REVOKE_ACCESS_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
> Matcher idTokenMatcher =
> ID_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
>
> if (accessTokenMatcher.find()) {
> return REVOKE_ACCESS_TOKEN_LOGOUT;
> } else {
> return ID_TOKEN_LOGOUT;
> }
> }
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-10332) Add ID_TOKEN_LOGOUT support for general OpenID connect server, like Keycloak
macdoor615 created NIFI-10332:
-
Summary: Add ID_TOKEN_LOGOUT support for general OpenID connect
server, like Keycloak
Key: NIFI-10332
URL: https://issues.apache.org/jira/browse/NIFI-10332
Project: Apache NiFi
Issue Type: Improvement
Components: Core UI
Affects Versions: 1.17.0
Environment: NiFi 1.17.0, Keycloak 18.0.1
Reporter: macdoor615
Fix For: 1.18.0
Attachments: image-2022-08-09-16-56-25-791.png
I deploy a NiFi 1.170 and authenticate with OpenID connect. Authentication
server is Keycloak 18.0.1.
I can log in and I can use UI properly.
But when I logout. I get an error, can not redirect to NiFi UI or keycloak
login UI
!image-2022-08-09-16-56-25-791.png|width=782,height=347!
[https://36.133.55.100:8943/realms/zznode/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2F36.138.166.203%3A18089%2Fhb3-dmz-repos-000-nifi%2Fnifi-api%2F..%2Fnifi%2Flogout-complete]
I made some investigation into source code. I found NiFi only support
ID_TOKEN_LOGOUT for okta service. Keycloak and other Authentication server can
not be supported.
Keycloak say it is compliance OpenID connect spec.
I modified a few lines of source code. Let it support ID_TOKEN_LOGOUT for
keycloak. Now I can log out NiFi and redirect to keycloak login UI, and than
login NiFi again.
I suggest making nifi to support ID_TOKEN_LOGOUT in later version for general
OpenID connect server.
I modified the file,
[https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/OIDCAccessResource.java]
start from line 403
{code:java}
private String determineLogoutMethod(String oidcDiscoveryUrl) {
Matcher accessTokenMatcher =
REVOKE_ACCESS_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
Matcher idTokenMatcher =
ID_TOKEN_LOGOUT_FORMAT.matcher(oidcDiscoveryUrl);
if (accessTokenMatcher.find()) {
return REVOKE_ACCESS_TOKEN_LOGOUT;
} else {
return ID_TOKEN_LOGOUT;
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10331) Bootstraping NiFI with the java arguments that contains space not possible
[ https://issues.apache.org/jira/browse/NIFI-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lopata updated NIFI-10331: -- Issue Type: Bug (was: New Feature) > Bootstraping NiFI with the java arguments that contains space not possible > -- > > Key: NIFI-10331 > URL: https://issues.apache.org/jira/browse/NIFI-10331 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.17.0 > Environment: all >Reporter: Lopata >Priority: Major > > We are running NiFi on Java 17 and we are not able to add _*--add-opens > java.base/java.lang.reflect=ALL-UNNAMED*_ java argument when bootstrapping > the NiFi. RunNifi.java is using ProcessBuilder that expects command without > spaces, so we have to defined --add-opens argument in the bootstrap.conf file > with two _*java.arg.N*_ properties. RunNifi.java when creates the java > command doesn't respect the order of _*java.arg.N*_ properties and randomly > places the --add-opens argument parts into java command which is than not > valid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10331) Bootstraping NiFI with the java arguments that contains space not possible
[ https://issues.apache.org/jira/browse/NIFI-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lopata updated NIFI-10331: -- Summary: Bootstraping NiFI with the java arguments that contains space not possible (was: Bootstraping NiFI with the java arguments with space not possible) > Bootstraping NiFI with the java arguments that contains space not possible > -- > > Key: NIFI-10331 > URL: https://issues.apache.org/jira/browse/NIFI-10331 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.17.0 > Environment: all >Reporter: Lopata >Priority: Major > > We are running NiFi on Java 17 and we are not able to add _*--add-opens > java.base/java.lang.reflect=ALL-UNNAMED*_ java argument when bootstrapping > the NiFi. RunNifi.java is using ProcessBuilder that expects command without > spaces, so we have to defined --add-opens argument in the bootstrap.conf file > with two _*java.arg.N*_ properties. RunNifi.java when creates the java > command doesn't respect the order of _*java.arg.N*_ properties and randomly > places the --add-opens argument parts into java command which is than not > valid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10331) Bootstraping NiFI with java java arguments with space not possible
[ https://issues.apache.org/jira/browse/NIFI-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lopata updated NIFI-10331: -- Summary: Bootstraping NiFI with java java arguments with space not possible (was: Bootstrap NiFI with java java arguments with space not possible) > Bootstraping NiFI with java java arguments with space not possible > -- > > Key: NIFI-10331 > URL: https://issues.apache.org/jira/browse/NIFI-10331 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.17.0 > Environment: all >Reporter: Lopata >Priority: Major > > We are running NiFi on Java 17 and we are not able to add _*--add-opens > java.base/java.lang.reflect=ALL-UNNAMED*_ java argument when bootstrapping > the NiFi. RunNifi.java is using ProcessBuilder that expects command without > spaces, so we have to defined --add-opens argument in the bootstrap.conf file > with two _*java.arg.N*_ properties. RunNifi.java when creates the java > command doesn't respect the order of _*java.arg.N*_ properties and randomly > places the --add-opens argument parts into java command which is than not > valid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10331) Bootstraping NiFI with the java arguments with space not possible
[ https://issues.apache.org/jira/browse/NIFI-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lopata updated NIFI-10331: -- Summary: Bootstraping NiFI with the java arguments with space not possible (was: Bootstraping NiFI with java java arguments with space not possible) > Bootstraping NiFI with the java arguments with space not possible > - > > Key: NIFI-10331 > URL: https://issues.apache.org/jira/browse/NIFI-10331 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.17.0 > Environment: all >Reporter: Lopata >Priority: Major > > We are running NiFi on Java 17 and we are not able to add _*--add-opens > java.base/java.lang.reflect=ALL-UNNAMED*_ java argument when bootstrapping > the NiFi. RunNifi.java is using ProcessBuilder that expects command without > spaces, so we have to defined --add-opens argument in the bootstrap.conf file > with two _*java.arg.N*_ properties. RunNifi.java when creates the java > command doesn't respect the order of _*java.arg.N*_ properties and randomly > places the --add-opens argument parts into java command which is than not > valid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10331) Bootstrap NiFI with java java arguments with space not possible
Lopata created NIFI-10331: - Summary: Bootstrap NiFI with java java arguments with space not possible Key: NIFI-10331 URL: https://issues.apache.org/jira/browse/NIFI-10331 Project: Apache NiFi Issue Type: New Feature Affects Versions: 1.17.0 Environment: all Reporter: Lopata We are running NiFi on Java 17 and we are not able to add _*--add-opens java.base/java.lang.reflect=ALL-UNNAMED*_ java argument when bootstrapping the NiFi. RunNifi.java is using ProcessBuilder that expects command without spaces, so we have to defined --add-opens argument in the bootstrap.conf file with two _*java.arg.N*_ properties. RunNifi.java when creates the java command doesn't respect the order of _*java.arg.N*_ properties and randomly places the --add-opens argument parts into java command which is than not valid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10322) invalid_token error after OpenID connect session timeout
[
https://issues.apache.org/jira/browse/NIFI-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577256#comment-17577256
]
macdoor615 commented on NIFI-10322:
---
[~exceptionfactory] Thank you!
I have tried both option.
* rebuild NiFi 1.17.0 from source with [GitHub Pull Request
#6278|https://github.com/apache/nifi/pull/6278]
* add proxy_cookie_path directive in nginx conf
Both options can be successful.
> invalid_token error after OpenID connect session timeout
>
>
> Key: NIFI-10322
> URL: https://issues.apache.org/jira/browse/NIFI-10322
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: image-2022-08-05-22-48-17-835.png,
> image-2022-08-05-22-48-52-057.png, image-2022-08-07-14-28-09-058.png,
> image-2022-08-07-15-22-36-213.png, image-2022-08-07-15-27-18-902.png,
> image-2022-08-07-15-37-29-739.png, image-2022-08-07-15-43-14-922.png,
> image-2022-08-07-15-47-57-158.png, image-2022-08-07-15-53-47-220.png,
> image-2022-08-07-16-00-11-443.png, image-2022-08-07-16-11-38-180.png,
> image-2022-08-08-23-33-30-220.png, image-2022-08-08-23-35-02-773.png,
> image-2022-08-08-23-59-12-471.png, nginx-access.log.zip,
> nifi-1.16.3-logs.zip, nifi-1.17.0-logs.zip
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I follow
> [https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect] to
> config NIFI 1.16.3 and it is work properly. If the session times out, login
> again and it will work again
> I configured 1.17.0 in the same way. I can login and operate nifi UI. But
> when session times out. I got the following error.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Expired JWT",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-17-835.png|width=758,height=108!
> I try to login again and get a new error, and I cannot enter the NIFI
> interface.
>
> {code:java}
> Unauthorized error="invalid_token", error_description="An error occurred
> while attempting to decode the Jwt: Signed JWT rejected: Another algorithm
> expected, or no matching key(s) found",
> error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"{code}
>
> !image-2022-08-05-22-48-52-057.png|width=594,height=143!
> I did some research, and found
> After the session times out,
> NIFI 1.16.3 leaves 3 cookies in browser:
> * nifi-logout-request-identifier
> * nifi-oidc-request-identifier
> * __Secure-Request-Token
> NIFI 1.17.0 leaves 2 cookies:
> * *__Secure-Authorization-Bearer*
> * __Secure-Request-Token
> __Secure-Authorization-Bearer cookie contains a expired JWT:
> {code:java}
> eyJraWQiOiJhMDlhZDhlMy0xZDkzLTQyZTEtYjg0Ni0xMWU0ODRkODYwYWYiLCJhbGciOiJQUzUxMiJ9.eyJzdWIiOiJhZG1pbi5uaWZpQGd1bWhiMy5jb20iLCJhdWQiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwibmJmIjoxNjU5NjExOTc0LCJpc3MiOiJodHRwcyUzQSUyRiUyRjM2LjEzMy41NS4xMDAlM0E4OTQzJTJGcmVhbG1zJTJGenpub2RlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW4ubmlmaUBndW1oYjMuY29tIiwiZXhwIjoxNjU5NjEyMjc0LCJpYXQiOjE2NTk2MTE5NzQsImp0aSI6IjFiZTg5MjU4LTliZmYtNDhmOS04OGNmLWU0NDIzMDZjYzg4ZCJ9.Y9yE0hNH_q-W94_cFWOWGc7TPMP2xB9coaSRPT9twYqSyjTtudOiiXGxHEDUWsOvUFf7lT7wNH4RZ_LhOM-5WfTZ3o-DCVFnl0JjeZ-L9d-z3rO4dEspRxXpr46AewEGy_lpstSUFyihr4i8b2VI7IT0aFOCGAIXRWl7gfH75e5La_0tbsu9lgSRdyYBBv8rSjojJC5bBSqxj-BkrfjdMhyMuF9OdMCJNmyh18BrXbavwftNerytkd_Qf9eNLmzsZ3SOdKWpftKt4kClD_KeL0nOglhM-ENyb4QLwxr7l5lhUgQ-2am3x5okbRyYip_WV4YQ6DfmUnLL1FYFATWXa5CUimSRbSZzkqU2JEYerpvKsTf-prdsSNryPbrQdf5HqpwhlGbFrgm4jwtncZHTLEL4ZMciVe0H-zIcQ9vyDqamMpf6fyNWmQN8DdDP9A0Zpo7SL7yhOUjNGsjk1gV4OAHWgp4XQzj4KwoGf7ICjeOrzinECHFZw9Ccyi8KMooRx4u3oAuKPEx3mrZFNFDaiAzWX0kZ31c24-15cno2bLBMGOIx7ipjb6Pv7V6O9S2aA2vC3eVLnfAgHAox3I8_IzWLUKddHCqd6cfA1XW8ckSgg2QddKvgYHiCZpwVV4AMDpK4bI1J0ZbxbgOOke9IMMudNhZUFQdWJIXh-gx1bII{code}
> I manually delete __Secure-Authorization-Bearer cookie, and I can login NIFI
> 1.17.0 again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-1901) Add agent identity to Prometheus metrics
[ https://issues.apache.org/jira/browse/MINIFICPP-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi reassigned MINIFICPP-1901: Assignee: Gábor Gyimesi > Add agent identity to Prometheus metrics > > > Key: MINIFICPP-1901 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1901 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Minor > > We should be able to monitor multiple agents from a single Prometheus > instance. It is possible to add multiple agent addresses to a Prometheus > config, but it is not possible to distinguish which metric belongs to which > agent. We should add a label with the agent's identity to be able to filter > metrics for a single agent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-1877) Windows cache corrupted in CI
[ https://issues.apache.org/jira/browse/MINIFICPP-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi resolved MINIFICPP-1877. -- Resolution: Fixed Cache cleared through github's new clear feature > Windows cache corrupted in CI > - > > Key: MINIFICPP-1877 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1877 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Trivial > > Cache object is corrupted in CI for windows build. Cache key should be > updated to avoid this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1785) InvokeHTTP request fails when sending windows event logs
[ https://issues.apache.org/jira/browse/MINIFICPP-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi updated MINIFICPP-1785: - Fix Version/s: 0.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > InvokeHTTP request fails when sending windows event logs > > > Key: MINIFICPP-1785 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1785 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Minor > Fix For: 0.12.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In a simple flow scenario where ConsumeWindowsEventLogs processor forwards > logs to NiFi with InvokeHTTP and NiFi consumes these logs, the request is > always answered with error code 400. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1665) Create ListAzureBlobStorage processor
[ https://issues.apache.org/jira/browse/MINIFICPP-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi updated MINIFICPP-1665: - Fix Version/s: 0.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Create ListAzureBlobStorage processor > - > > Key: MINIFICPP-1665 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1665 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Major > Fix For: 0.12.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Create a processor similar to NiFi's ListAzureBlobStorage: > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.14.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/index.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1901) Add agent identity to Prometheus metrics
Gábor Gyimesi created MINIFICPP-1901: Summary: Add agent identity to Prometheus metrics Key: MINIFICPP-1901 URL: https://issues.apache.org/jira/browse/MINIFICPP-1901 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Gábor Gyimesi We should be able to monitor multiple agents from a single Prometheus instance. It is possible to add multiple agent addresses to a Prometheus config, but it is not possible to distinguish which metric belongs to which agent. We should add a label with the agent's identity to be able to filter metrics for a single agent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] xuanronaldo commented on a diff in pull request #6265: NIFI-10234 Implement PutIoTDB
xuanronaldo commented on code in PR #6265:
URL: https://github.com/apache/nifi/pull/6265#discussion_r940939151
##
nifi-nar-bundles/nifi-iotdb-bundle/nifi-iotdb-processors/src/main/java/org/apache/nifi/processors/PutIoTDB.java:
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;
+
+import java.io.InputStream;
+import java.text.SimpleDateFormat;
+import java.util.Set;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.processors.model.Schema;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.util.Tuple;
+
+@Tags({"iotdb", "insert", "tablet"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription(
+"This is a record aware processor that reads the content of the
incoming FlowFile as individual records using the "
++ "configured 'Record Reader' and writes them to Apache IoTDB
using native interface.")
+public class PutIoTDB extends AbstractIoTDB {
+
+static final PropertyDescriptor RECORD_READER_FACTORY =
+new PropertyDescriptor.Builder()
+.name("Record Reader")
+.description(
+"Specifies the type of Record Reader controller
service to use for parsing the incoming data "
++ "and determining the schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor TIME_FIELD =
+new PropertyDescriptor.Builder()
+.name("Time Field")
+.description(
+"The field name which represents time. It can be
updated by expression language.")
+.defaultValue("Time")
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.required(false)
+.build();
+
+static final PropertyDescriptor SCHEMA =
+new PropertyDescriptor.Builder()
+.name("Schema")
+.description(
+"The schema that IoTDB needs doesn't support good
by NiFi.\n"
++ "Therefore, you can define the schema
here.\n"
++ "Besides, you can set encoding type and
compression type by this method.\n"
++ "If you don't set this property, the
inferred schema will be used.\n"
++ "It can be updated by expression
language.")
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.required(false)
+.build();
+
+static final PropertyDescriptor ALIGNED =
+new PropertyDescriptor.Builder()
+.name("Aligned")
+
