[jira] [Updated] (NIFI-8248) Modify PutAzureDataLakeStorage processor to use temp file instead of inline replacement
[ https://issues.apache.org/jira/browse/NIFI-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8248: -- Fix Version/s: 1.18.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Modify PutAzureDataLakeStorage processor to use temp file instead of inline > replacement > --- > > Key: NIFI-8248 > URL: https://issues.apache.org/jira/browse/NIFI-8248 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Fix For: 1.18.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Currently PutAzureDataLakeStorage processor overwrites the existing file with > an empty file before uploading the content. In case of some reason upload > fails the original content of the file is lost. Recommended solution would be > to create a temporary file first and rename it after a successful upload. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10219) jna-platform missing from ./lib/properties
[ https://issues.apache.org/jira/browse/NIFI-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565250#comment-17565250 ] Joey Frazee commented on NIFI-10219: [~exceptionfactory] I have a fix that works just adding jna-platform as an include directly in the assembly common.xml. But I have no clue whether that’s the right solution and it seems like the intent was to have the dependencies come in through nifi-property-protection-factory. I messed about with adding the jna-platform dependency to nifi-property-protection-azure but it never ended up in ./lib/properties. > jna-platform missing from ./lib/properties > -- > > Key: NIFI-10219 > URL: https://issues.apache.org/jira/browse/NIFI-10219 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.16.1, 1.16.2, 1.16.3 >Reporter: Joey Frazee >Assignee: David Handermann >Priority: Major > > NIFI-9883 introduced classloader isolation for sensitive property providers > and updated the assembly config to output dependencies in ./lib/properties. > jna-platform, required by azure-identity, isn't getting pulled through so > when using an Azure sensitive properties provider startup fails with a > NoClassDefFoundError. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10219) jna-platform missing from ./lib/properties
[ https://issues.apache.org/jira/browse/NIFI-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-10219: --- Description: NIFI-9883 introduced classloader isolation for sensitive property providers and updated the assembly config to output dependencies in ./lib/properties. jna-platform, required by azure-identity, isn't getting pulled through so when using an Azure sensitive properties provider startup fails with a NoClassDefFoundError. (was: NIFI-9883 introduced classloader isolation for sensitive property providers and updated the assembly config to output dependencies in ./lib/properties. jna-platform, required by azure-identity, isn't getting pulled through so when using an Azure sensitive properties provider startup fails with a NoCalssDefFoundError.) > jna-platform missing from ./lib/properties > -- > > Key: NIFI-10219 > URL: https://issues.apache.org/jira/browse/NIFI-10219 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.16.1, 1.16.2, 1.16.3 >Reporter: Joey Frazee >Priority: Major > > NIFI-9883 introduced classloader isolation for sensitive property providers > and updated the assembly config to output dependencies in ./lib/properties. > jna-platform, required by azure-identity, isn't getting pulled through so > when using an Azure sensitive properties provider startup fails with a > NoClassDefFoundError. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10219) jna-platform missing from ./lib/properties
Joey Frazee created NIFI-10219: -- Summary: jna-platform missing from ./lib/properties Key: NIFI-10219 URL: https://issues.apache.org/jira/browse/NIFI-10219 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.16.3, 1.16.2, 1.16.1 Reporter: Joey Frazee NIFI-9883 introduced classloader isolation for sensitive property providers and updated the assembly config to output dependencies in ./lib/properties. jna-platform, required by azure-identity, isn't getting pulled through so when using an Azure sensitive properties provider startup fails with a NoCalssDefFoundError. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-8677) Add endpoint suffix for ConsumeAzureEventHub Processor
[ https://issues.apache.org/jira/browse/NIFI-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8677. --- Fix Version/s: 1.17.0 Resolution: Fixed > Add endpoint suffix for ConsumeAzureEventHub Processor > -- > > Key: NIFI-8677 > URL: https://issues.apache.org/jira/browse/NIFI-8677 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Fix For: 1.17.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > ConsumeAzureEventHub Processor does not work for any special regions > including US Government, China, and Germany. need to find a way to add > endpoint suffix support. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-7192) nifi.sh install not working on systemd servers
[
https://issues.apache.org/jira/browse/NIFI-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee resolved NIFI-7192.
---
Fix Version/s: 1.16.0
Resolution: Fixed
> nifi.sh install not working on systemd servers
> --
>
> Key: NIFI-7192
> URL: https://issues.apache.org/jira/browse/NIFI-7192
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Roberto Marturano
>Assignee: Roberto Marturano
>Priority: Major
> Fix For: 1.16.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Systemd is still honouring init.d services, but we need to trigger the
> {{systemd-sysv-generator}} in order to create a .service wrapper.
> Right now works like this:
>
> {code:java}
> $ ./bin/nifi.sh install nifi
> Service nifi installed
> $ service nifi start
> Failed to start nifi.service: Unit nifi.service not found.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-7192) nifi.sh install not working on systemd servers
[
https://issues.apache.org/jira/browse/NIFI-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486141#comment-17486141
]
Joey Frazee commented on NIFI-7192:
---
[~17reasons] Hey, I'd be happy to see this go in and wrap this up. Does this
patch work for you? I really just want some signal of what it's been tested
against.
> nifi.sh install not working on systemd servers
> --
>
> Key: NIFI-7192
> URL: https://issues.apache.org/jira/browse/NIFI-7192
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Roberto Marturano
>Assignee: Roberto Marturano
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Systemd is still honouring init.d services, but we need to trigger the
> {{systemd-sysv-generator}} in order to create a .service wrapper.
> Right now works like this:
>
> {code:java}
> $ ./bin/nifi.sh install nifi
> Service nifi installed
> $ service nifi start
> Failed to start nifi.service: Unit nifi.service not found.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9188) Upgrade docker base image to 11-jre
[ https://issues.apache.org/jira/browse/NIFI-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408934#comment-17408934 ] Joey Frazee commented on NIFI-9188: --- [~cemremengu] This was a conscious decision because so many users still have to be on 1.8; see some of the discussion in NIFI-7717 and a mailing list thread on 2.0 goals [1, 2]. As you've noticed you can build a 11-based container image using official NiFi releases which is the middle-ground we went for in NIFI-7717. A PR like you submitted will almost certainly be needed for 2.0, so there's no reason to close it, but it might not be merged for a while. 1. https://lists.apache.org/thread.html/rd681f24ef919ed8705e4b4642cc0f69f3f69acbee415e9604250cc80%40%3Cdev.nifi.apache.org%3E 2. https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals > Upgrade docker base image to 11-jre > --- > > Key: NIFI-9188 > URL: https://issues.apache.org/jira/browse/NIFI-9188 > Project: Apache NiFi > Issue Type: Improvement > Components: Docker >Reporter: Cemre Mengu >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Since container uses binaries, and java 11 is backwards compatible is there > any reason why the base image is `8-jre` ? > I think it will be great to bump the java version so that we can both use > better features and java 11 based custom processors. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-9055) FetchS3Object fails to fetch small files
[ https://issues.apache.org/jira/browse/NIFI-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401488#comment-17401488 ] Joey Frazee commented on NIFI-9055: --- [~joewitt] Will do/have done. > FetchS3Object fails to fetch small files > > > Key: NIFI-9055 > URL: https://issues.apache.org/jira/browse/NIFI-9055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Joey Frazee >Assignee: Joey Frazee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > FetchS3Object fails to retrieve small or zero byte files returning: The > requested range is not satisfiable, Status Code: 416, Error Code: > InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-9055) FetchS3Object fails to fetch small files
[ https://issues.apache.org/jira/browse/NIFI-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee reassigned NIFI-9055: - Assignee: Joey Frazee > FetchS3Object fails to fetch small files > > > Key: NIFI-9055 > URL: https://issues.apache.org/jira/browse/NIFI-9055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Joey Frazee >Assignee: Joey Frazee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > FetchS3Object fails to retrieve small or zero byte files returning: The > requested range is not satisfiable, Status Code: 416, Error Code: > InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9063) FetchS3Object doesn't handle invalid data size in Range properties
Joey Frazee created NIFI-9063: - Summary: FetchS3Object doesn't handle invalid data size in Range properties Key: NIFI-9063 URL: https://issues.apache.org/jira/browse/NIFI-9063 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.14.0 Reporter: Joey Frazee FetchS3Object supports FlowFile-scoped EL for both the Range Start and Range Length properties but doesn't check if the values are non-empty or handle the IllegalArgumentException for invalid data sizes. These should be checked before converting to a data size and routed to failure if invalid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9062) Add endpoint override property to S3 processor integration tests
Joey Frazee created NIFI-9062: - Summary: Add endpoint override property to S3 processor integration tests Key: NIFI-9062 URL: https://issues.apache.org/jira/browse/NIFI-9062 Project: Apache NiFi Issue Type: Test Components: Extensions Affects Versions: 1.14.0 Reporter: Joey Frazee The S3 processor integration tests don't expose the endpoint override so they currently only work with AWS vs. any S3-compatible service. Allowing the endpoint override would make it possible to test against other S3 API implementations as well as make it easier to do more rigorous testing locally, e.g., using something like Minio. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-9055) FetchS3Object fails to fetch small files
[ https://issues.apache.org/jira/browse/NIFI-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400628#comment-17400628 ] Joey Frazee edited comment on NIFI-9055 at 8/17/21, 9:07 PM: - This can be repro'd on Minio too. The fix [1] is pretty simple, but we ought to update the tests and integration tests. I'll probably have to test against Minio too and the ITs aren't setup to use the override so it'll take me some time to get to it. 1. https://github.com/apache/nifi/compare/main...jfrazee:NIFI-9055 was (Author: jfrazee): This can be repro'd on Minio too. The fix [1] is pretty simple, but we ought to update the tests and integration tests. I'll probably have to test against Minio too and the ITs aren't setup to use the override so it'll take me some time to get to it. 1. https://github.com/jfrazee/nifi/commit/ca13e2ea2ca2b2b8dde37beb6488f6f530336b9e > FetchS3Object fails to fetch small files > > > Key: NIFI-9055 > URL: https://issues.apache.org/jira/browse/NIFI-9055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Joey Frazee >Priority: Major > > FetchS3Object fails to retrieve small or zero byte files returning: The > requested range is not satisfiable, Status Code: 416, Error Code: > InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-9055) FetchS3Object fails to fetch small files
[ https://issues.apache.org/jira/browse/NIFI-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400628#comment-17400628 ] Joey Frazee commented on NIFI-9055: --- This can be repro'd on Minio too. The fix [1] is pretty simple, but we ought to update the tests and integration tests. I'll probably have to test against Minio too and the ITs aren't setup to use the override so it'll take me some time to get to it. 1. https://github.com/jfrazee/nifi/commit/ca13e2ea2ca2b2b8dde37beb6488f6f530336b9e > FetchS3Object fails to fetch small files > > > Key: NIFI-9055 > URL: https://issues.apache.org/jira/browse/NIFI-9055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Joey Frazee >Priority: Major > > FetchS3Object fails to retrieve small or zero byte files returning: The > requested range is not satisfiable, Status Code: 416, Error Code: > InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-9055) FetchS3Object fails to fetch small files
[ https://issues.apache.org/jira/browse/NIFI-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400572#comment-17400572 ] Joey Frazee commented on NIFI-9055: --- S3 makes use of a range header following RFC 2616 [1] for doing a range request. AFAICT, specifying a range for a 0 byte file is invalid -- it does not "include[s] at least one byte-range-spec whose first-byte-pos is less than the current length of the entity-body" since 0 is not less than 0. So, a range shouldn't be specified at all and we need a third branch in the code where we just skip the setRange() when the intent is 0-. 1. https://datatracker.ietf.org/doc/html/rfc2616#section-14.35.1 > FetchS3Object fails to fetch small files > > > Key: NIFI-9055 > URL: https://issues.apache.org/jira/browse/NIFI-9055 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Joey Frazee >Priority: Major > > FetchS3Object fails to retrieve small or zero byte files returning: The > requested range is not satisfiable, Status Code: 416, Error Code: > InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9055) FetchS3Object fails to fetch small files
Joey Frazee created NIFI-9055: - Summary: FetchS3Object fails to fetch small files Key: NIFI-9055 URL: https://issues.apache.org/jira/browse/NIFI-9055 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.14.0 Reporter: Joey Frazee FetchS3Object fails to retrieve small or zero byte files returning: The requested range is not satisfiable, Status Code: 416, Error Code: InvalidRange. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9045) GetAzureEventHub doesn't handle network timeouts correctly
Joey Frazee created NIFI-9045: - Summary: GetAzureEventHub doesn't handle network timeouts correctly Key: NIFI-9045 URL: https://issues.apache.org/jira/browse/NIFI-9045 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Joey Frazee Over high latency networks GetAzureEventHub is prone to timeout and timeouts and retries aren't properly handled resulting in significant performance degradation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9044) Support multiple hubs in ConsumeAzureEventHub
Joey Frazee created NIFI-9044: - Summary: Support multiple hubs in ConsumeAzureEventHub Key: NIFI-9044 URL: https://issues.apache.org/jira/browse/NIFI-9044 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Joey Frazee ConsumeAzureEventHub currently only supports reading from a single hub at a time. It's not uncommon to create data flows that mux together data from multiple sources so it'd be convenient to be able to specify multiple hubs the way that Kafaka supports reading from multiple topics. AFAIK, this isn't a built in feature of the Azure SDK so it will introduce some complexity that will need to be considered. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9043) Add alternative checkpoint storage for ConsumeAzureEventHub
Joey Frazee created NIFI-9043: - Summary: Add alternative checkpoint storage for ConsumeAzureEventHub Key: NIFI-9043 URL: https://issues.apache.org/jira/browse/NIFI-9043 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Joey Frazee ConsumeAzureEventHub currently only supports Azure Storage-based checkpoint storage. This can be a performance bottleneck when the workload requires frequent checkpointing. It would be nice if ConsumeAzureEventHub supported alternative checkpointing mechanisms such as a DistributedMapCache controller service or ZooKeeper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9042) ConsumeAzureEventHub batch size has no effect on frequency of storage requests
Joey Frazee created NIFI-9042: - Summary: ConsumeAzureEventHub batch size has no effect on frequency of storage requests Key: NIFI-9042 URL: https://issues.apache.org/jira/browse/NIFI-9042 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.14.0, 1.15.0 Reporter: Joey Frazee ConsumeAzureEventHub uses Azure Storage to store checkpoints. The checkpoint frequency should align with the Batch Size and Message Receive Timeout but neither have any effect. This can be seen in the Azure portal in the storage requests metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-9025) Build is failing because TestTailFile is references the wrong property
[ https://issues.apache.org/jira/browse/NIFI-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-9025. --- Fix Version/s: 1.15.0 Resolution: Fixed > Build is failing because TestTailFile is references the wrong property > -- > > Key: NIFI-9025 > URL: https://issues.apache.org/jira/browse/NIFI-9025 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.15.0 >Reporter: Joey Frazee >Priority: Minor > Fix For: 1.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > TestTailFile references TailFile.LINE_START_REGEX which does not exist as of > NIFI-8773. The reference should be TailFile.LINE_START_PATTERN. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-9025) Build is failing because TestTailFile is references the wrong property
Joey Frazee created NIFI-9025: - Summary: Build is failing because TestTailFile is references the wrong property Key: NIFI-9025 URL: https://issues.apache.org/jira/browse/NIFI-9025 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.15.0 Reporter: Joey Frazee TestTailFile references TailFile.LINE_START_REGEX which does not exist as of NIFI-8773. The reference should be TailFile.LINE_START_PATTERN. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8920) Dockerhub build should use mirrors
[
https://issues.apache.org/jira/browse/NIFI-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8920:
--
Description:
Currently the Dockerhub Dockerfile will pull binaries from
https://archive.apache.org by default. It's preferable to pull these from a
mirror when available.
The Dockerfile should use
https://www.apache.org/dyn/closer.lua/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}?as_json
and the returned 'in_dist' and 'preferred' properties to determine the best
mirror for the MIRROR_BASE_URL.
was:
Currently the Dockerhub Dockerfile will pull binaries from
https://archive.apache.org by default. It's preferable to pull these from a
mirror when available.
The Dockerfile should use
https://www.apache.org/dyn/closer.lua/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}?as_json
and the returned 'in_dist' and 'preferred' properties to determine the best
mirror for the BASE_URL.
> Dockerhub build should use mirrors
> --
>
> Key: NIFI-8920
> URL: https://issues.apache.org/jira/browse/NIFI-8920
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.14.0
>Reporter: Joey Frazee
>Priority: Minor
>
> Currently the Dockerhub Dockerfile will pull binaries from
> https://archive.apache.org by default. It's preferable to pull these from a
> mirror when available.
> The Dockerfile should use
> https://www.apache.org/dyn/closer.lua/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}?as_json
> and the returned 'in_dist' and 'preferred' properties to determine the best
> mirror for the MIRROR_BASE_URL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8920) Dockerhub build should use mirrors
[
https://issues.apache.org/jira/browse/NIFI-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8920:
--
Summary: Dockerhub build should use mirrors (was: Dockerhub build should
use Apache mirrors)
> Dockerhub build should use mirrors
> --
>
> Key: NIFI-8920
> URL: https://issues.apache.org/jira/browse/NIFI-8920
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.14.0
>Reporter: Joey Frazee
>Priority: Minor
>
> Currently the Dockerhub Dockerfile will pull binaries from
> https://archive.apache.org by default. It's preferable to pull these from a
> mirror when available.
> The Dockerfile should use
> https://www.apache.org/dyn/closer.lua/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}?as_json
> and the returned 'in_dist' and 'preferred' properties to determine the best
> mirror for the BASE_URL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8920) Dockerhub build should use Apache mirrors
Joey Frazee created NIFI-8920:
-
Summary: Dockerhub build should use Apache mirrors
Key: NIFI-8920
URL: https://issues.apache.org/jira/browse/NIFI-8920
Project: Apache NiFi
Issue Type: Improvement
Components: Docker
Affects Versions: 1.14.0
Reporter: Joey Frazee
Currently the Dockerhub Dockerfile will pull binaries from
https://archive.apache.org by default. It's preferable to pull these from a
mirror when available.
The Dockerfile should use
https://www.apache.org/dyn/closer.lua/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}?as_json
and the returned 'in_dist' and 'preferred' properties to determine the best
mirror for the BASE_URL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-4913) Specifying "run.as" in bootstrap.properties causes environment variables to not be passed to NiFi
[ https://issues.apache.org/jira/browse/NIFI-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-4913. --- Fix Version/s: 1.14.0 Resolution: Fixed This was merged during 1.14.0 but not closed. > Specifying "run.as" in bootstrap.properties causes environment variables to > not be passed to NiFi > - > > Key: NIFI-4913 > URL: https://issues.apache.org/jira/browse/NIFI-4913 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.5.0 >Reporter: Jonathan Peterson >Assignee: Sushil Kumar >Priority: Minor > Labels: beginner > Fix For: 1.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > > I solved this by including "-E" after "sudo" in nifi.sh [1]. > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.5.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh#L312] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8784) The Docker start script variable NIFI_WEB_PROXY_HOST is ignored when using single user authorization
Joey Frazee created NIFI-8784:
-
Summary: The Docker start script variable NIFI_WEB_PROXY_HOST is
ignored when using single user authorization
Key: NIFI-8784
URL: https://issues.apache.org/jira/browse/NIFI-8784
Project: Apache NiFi
Issue Type: Bug
Components: Docker
Affects Versions: 1.14.0
Reporter: Joey Frazee
Assignee: Joey Frazee
In the Docker start scripts nifi.web.proxy.host is only set when sourcing
secure.sh. This only happens when AUTH=tls or AUTH=ldap so the exposed
environment variable NIFI_WEB_PROXY_HOST is ignored for the default single user
authorization and echoes the following:
{{NIFI_WEB_PROXY_HOST was set but NiFi is not configured to run in a secure
mode. Will not update nifi.web.proxy.host.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8778) README maven version requirements out of date
[ https://issues.apache.org/jira/browse/NIFI-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379370#comment-17379370 ] Joey Frazee commented on NIFI-8778: --- BTW, this isn't just a README problem but also specified as 3.1.1 in the enforcer in the root pom.xml > README maven version requirements out of date > - > > Key: NIFI-8778 > URL: https://issues.apache.org/jira/browse/NIFI-8778 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Otto Fowler >Priority: Major > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > (install-node-and-npm) on project nifi-web-ui: The plugin > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version 3.6.0 > -> [Help 1] > I have maven version: > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T14:33:14-04:00) > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > What is the minimum version of maven required to build nifi? is it now 3.6.0? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8501) Add support for Azure Storage Client-Side Encryption
[ https://issues.apache.org/jira/browse/NIFI-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8501: -- Fix Version/s: 1.14.0 > Add support for Azure Storage Client-Side Encryption > > > Key: NIFI-8501 > URL: https://issues.apache.org/jira/browse/NIFI-8501 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Major > Labels: AZURE > Fix For: 1.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Microsoft allows for Blob stored on Azure to be encrypted client-side using > key wrapping algorithm. > Implementation details can be found here: > [https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption-java?tabs=java] > Adding support for such encryption method would offer more compatibility with > the Azure ecosystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8501) Add support for Azure Storage Client-Side Encryption
[ https://issues.apache.org/jira/browse/NIFI-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8501. --- Resolution: Fixed > Add support for Azure Storage Client-Side Encryption > > > Key: NIFI-8501 > URL: https://issues.apache.org/jira/browse/NIFI-8501 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Major > Labels: AZURE > Fix For: 1.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Microsoft allows for Blob stored on Azure to be encrypted client-side using > key wrapping algorithm. > Implementation details can be found here: > [https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption-java?tabs=java] > Adding support for such encryption method would offer more compatibility with > the Azure ecosystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8501) Add support for Azure Storage Client-Side Encryption
[ https://issues.apache.org/jira/browse/NIFI-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8501: -- Affects Version/s: 1.13.2 > Add support for Azure Storage Client-Side Encryption > > > Key: NIFI-8501 > URL: https://issues.apache.org/jira/browse/NIFI-8501 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.13.2 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Major > Labels: AZURE > Time Spent: 2h > Remaining Estimate: 0h > > Microsoft allows for Blob stored on Azure to be encrypted client-side using > key wrapping algorithm. > Implementation details can be found here: > [https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption-java?tabs=java] > Adding support for such encryption method would offer more compatibility with > the Azure ecosystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8501) Add support for Azure Storage Client-Side Encryption
[ https://issues.apache.org/jira/browse/NIFI-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8501: -- Component/s: Extensions > Add support for Azure Storage Client-Side Encryption > > > Key: NIFI-8501 > URL: https://issues.apache.org/jira/browse/NIFI-8501 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Major > Labels: AZURE > Time Spent: 2h > Remaining Estimate: 0h > > Microsoft allows for Blob stored on Azure to be encrypted client-side using > key wrapping algorithm. > Implementation details can be found here: > [https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption-java?tabs=java] > Adding support for such encryption method would offer more compatibility with > the Azure ecosystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8656) Support expression language for SAS Token in the ADLS Gen2 processors
[ https://issues.apache.org/jira/browse/NIFI-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8656: -- Fix Version/s: 1.14.0 > Support expression language for SAS Token in the ADLS Gen2 processors > - > > Key: NIFI-8656 > URL: https://issues.apache.org/jira/browse/NIFI-8656 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Fix For: 1.14.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > In use cases where the SAS Token acquired at runtime (e.g. via InvokeHttp), > it would be useful to support EL for SAS Token property to pass the token to > the processor dynamically. > The Blob counterpart already have this feature and the common description of > the property already mentions the security considerations of this scenario. > The same can be used for the ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8656) Support expression language for SAS Token in the ADLS Gen2 processors
[ https://issues.apache.org/jira/browse/NIFI-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8656: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Support expression language for SAS Token in the ADLS Gen2 processors > - > > Key: NIFI-8656 > URL: https://issues.apache.org/jira/browse/NIFI-8656 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > In use cases where the SAS Token acquired at runtime (e.g. via InvokeHttp), > it would be useful to support EL for SAS Token property to pass the token to > the processor dynamically. > The Blob counterpart already have this feature and the common description of > the property already mentions the security considerations of this scenario. > The same can be used for the ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8656) Support expression language for SAS Token in the ADLS Gen2 processors
[ https://issues.apache.org/jira/browse/NIFI-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362072#comment-17362072 ] Joey Frazee commented on NIFI-8656: --- [~joewitt] Not sure where you are now on building the release so I've deliberately left the Fix Version unset. LMK which way it goes and I'll set it. > Support expression language for SAS Token in the ADLS Gen2 processors > - > > Key: NIFI-8656 > URL: https://issues.apache.org/jira/browse/NIFI-8656 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > In use cases where the SAS Token acquired at runtime (e.g. via InvokeHttp), > it would be useful to support EL for SAS Token property to pass the token to > the processor dynamically. > The Blob counterpart already have this feature and the common description of > the property already mentions the security considerations of this scenario. > The same can be used for the ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8656) Support expression language for SAS Token in the ADLS Gen2 processors
[ https://issues.apache.org/jira/browse/NIFI-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8656: -- Component/s: Extensions > Support expression language for SAS Token in the ADLS Gen2 processors > - > > Key: NIFI-8656 > URL: https://issues.apache.org/jira/browse/NIFI-8656 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > In use cases where the SAS Token acquired at runtime (e.g. via InvokeHttp), > it would be useful to support EL for SAS Token property to pass the token to > the processor dynamically. > The Blob counterpart already have this feature and the common description of > the property already mentions the security considerations of this scenario. > The same can be used for the ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8656) Support expression language for SAS Token in the ADLS Gen2 processors
[ https://issues.apache.org/jira/browse/NIFI-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8656: -- Affects Version/s: 1.13.2 > Support expression language for SAS Token in the ADLS Gen2 processors > - > > Key: NIFI-8656 > URL: https://issues.apache.org/jira/browse/NIFI-8656 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.13.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > In use cases where the SAS Token acquired at runtime (e.g. via InvokeHttp), > it would be useful to support EL for SAS Token property to pass the token to > the processor dynamically. > The Blob counterpart already have this feature and the common description of > the property already mentions the security considerations of this scenario. > The same can be used for the ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8449) Add support for transactional batch operations to PutAzureCosmosDBRecord
[ https://issues.apache.org/jira/browse/NIFI-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8449: -- Issue Type: Improvement (was: Bug) > Add support for transactional batch operations to PutAzureCosmosDBRecord > > > Key: NIFI-8449 > URL: https://issues.apache.org/jira/browse/NIFI-8449 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Joey Frazee >Priority: Minor > > Right now PutAzureCosmosDBRecord only batches records for processing, not for > writing data to Cosmos DB, so inserting and updating data happens serially. > The Azure Java SDK supports using transactional batch operations, however, > which if used should improve performance as well as adding more > transactionality. > Unfortunately, the API for this only allows batching by partition id so this > will introduce some complexity since the processor or flow will have to > "manually" handle grouping records to create the transactional batch calls. > While not ideal it should be ok -- best case we could have 1 call for N > records, worst case we could have the status quo of N calls for N records. So > it's probably worth doing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8669) Invalid UUIDs generated
[
https://issues.apache.org/jira/browse/NIFI-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359472#comment-17359472
]
Joey Frazee commented on NIFI-8669:
---
[~aprakash] What JVMs does this happen on? AFAICT,
java.util.UUID#nameUUIDFromBytes gets used where this matters which suggests
it's not a NiFi problem.
> Invalid UUIDs generated
> ---
>
> Key: NIFI-8669
> URL: https://issues.apache.org/jira/browse/NIFI-8669
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: Arun Prakash
>Priority: Major
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The UUIDs generated (type 3) are not fully compliant to the RFC.
> For instance:
> {code:java}
> 9adabada-3b4b-32eb-fd7e-c41191eefa2a
> 642ff728-2fb3-3e6b-f48f-0cfb94b772e2 {code}
> {{generated by NiFi as _id_ for components are based on type 3. For type 3,
> 4, and 5, the *UUID variant* has to be one of (8, 9, a, b)}}
> *Example A:*
> 9adabada-3b4b-*3*2eb-*f*d7e-c41191eefa2a
> This is a *uuid v3*, and variant character is *{{f}}*, and *{{f}}* is not a
> valid uuid variant (just 8, 9, a and b are valid). *NOT VALID*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8648) Add session affinity info to proxy config section of admin guide
[ https://issues.apache.org/jira/browse/NIFI-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355356#comment-17355356 ] Joey Frazee commented on NIFI-8648: --- [~Chris S] Do you think it'd be useful to add configuration snippets to the walkthroughs for the common solutions (NGINX, HAproxy, cloud, K8s LB, ingress)? Individuals don't have an easy route to getting blog posts and vendor docs updated when things change. I also don't think I see people using or asking about using Apache httpd for this so I sort of wonder if it should be modified to reflect one of the more common choices. Thoughts? > Add session affinity info to proxy config section of admin guide > > > Key: NIFI-8648 > URL: https://issues.apache.org/jira/browse/NIFI-8648 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Affects Versions: 1.13.2 >Reporter: Joey Frazee >Priority: Minor > > When using proxies or load balancers with a cluster the user should use > affinity or stickiness to assure that user sessions are consistent. This > should be noted in the admin guide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8648) Add session affinity info to proxy config section of admin guide
Joey Frazee created NIFI-8648: - Summary: Add session affinity info to proxy config section of admin guide Key: NIFI-8648 URL: https://issues.apache.org/jira/browse/NIFI-8648 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.13.2 Reporter: Joey Frazee When using proxies or load balancers with a cluster the user should use affinity or stickiness to assure that user sessions are consistent. This should be noted in the admin guide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-4360) Add support for Azure Data Lake Store (ADLS)
[ https://issues.apache.org/jira/browse/NIFI-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-4360. --- Resolution: Duplicate > Add support for Azure Data Lake Store (ADLS) > > > Key: NIFI-4360 > URL: https://issues.apache.org/jira/browse/NIFI-4360 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Milan Chandna >Assignee: Milan Chandna >Priority: Major > Labels: adls, azure, hdfs > Original Estimate: 336h > Time Spent: 10m > Remaining Estimate: 335h 50m > > Currently ingress and egress in ADLS account is possible only using HDFS > processors. > Opening this feature to support separate processors for interaction with ADLS > accounts directly. > Benefits are many like > - simple configuration. > - Helping users not familiar with HDFS > - Helping users who currently are accessing ADLS accounts directly. > - using the ADLS SDK rather than HDFS client, one lesser layer to go through. > Can be achieved by adding separate ADLS processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8506) Azure Integration tests fail
[ https://issues.apache.org/jira/browse/NIFI-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8506. --- Fix Version/s: 1.14.0 Resolution: Fixed > Azure Integration tests fail > - > > Key: NIFI-8506 > URL: https://issues.apache.org/jira/browse/NIFI-8506 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Minor > Fix For: 1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > NIFI-7886 introduced some changes in the integration-tests of the > nifi-azure-bundle. > integration-tests of this component don't pass after PR #4576 > More specifically: > # *org.apache.nifi.processors.azure.storage.ITListAzureBlobStorage* tries to > upload a blob, retrieve it and then expects its lengths to be 10. But > AbstractAzureBlobStorageIT was changed to upload a blob of length 36 instead > of 10. > # Two tests in > *org.apache.nifi.processors.azure.storage.ITPutAzureDataLakeStorage* and > *org.apache.nifi.processors.azure.storage.ITFetchAzureDataLakeStorage* are > set to Ignore. By default a skipped test, makes the verify phase to fail. > # -I also saw an intermittent failure on > *org.apache.nifi.processors.azure.storage.ITListAzureDataLakeStorage* in > testListRootRecursive but I could not reproduce it systematically.- This was > moved to NIFI-8508 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8478) Unable to build the project (XmlUtilsTest.testShouldHandleXXEInDocumentBuilder:97 assert msg =~ "SAXParseException.* DOCTYPE is disallowed when the feature")
[
https://issues.apache.org/jira/browse/NIFI-8478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338665#comment-17338665
]
Joey Frazee commented on NIFI-8478:
---
Was just in the middle of reporting this. The TestJdbcClobReadable.testClobWith
tests fail when user.language and user.timezone are empty.
This seems to happen when MAVEN_OPTS or maven.surefire.arguments is not set
because the root pom.xml will set user.language and user.timezone to empty
properties.
> Unable to build the project
> (XmlUtilsTest.testShouldHandleXXEInDocumentBuilder:97 assert msg =~
> "SAXParseException.* DOCTYPE is disallowed when the feature")
> -
>
> Key: NIFI-8478
> URL: https://issues.apache.org/jira/browse/NIFI-8478
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.2
> Environment: $ java -version
> openjdk version "11.0.11" 2021-04-20 LTS
> OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
> OpenJDK 64-Bit Server VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)
> $ mvn -version
> Apache Maven 3.6.3
> Maven home: /usr/share/maven
> Java version: 11.0.11, vendor: Azul Systems, Inc., runtime:
> /usr/lib/jvm/zulu11-ca-amd64
> Default locale: fr_FR, platform encoding: UTF-8
> OS name: "linux", version: "5.8.0-50-generic", arch: "amd64", family: "unix"
> $ git status
> Sur la branche main
> Votre branche est à jour avec 'origin/main'.
> rien à valider, la copie de travail est propre
>Reporter: Julien G.
>Assignee: Paul Grey
>Priority: Blocker
> Fix For: 1.14.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Unable to build the projet with the command:
> {noformat}
> mvn clean install{noformat}
> I got the error:
> {code:java}
> [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @
> nifi-security-utils ---
> [INFO] Surefire report directory:
> /home/juldrixx/Documents/tmp/nifi/nifi-commons/nifi-security-utils/target/surefire-reports
> [INFO] Using configured provider
> org.apache.maven.surefire.junit4.JUnit4Provider
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.nifi.security.XXEValidatorTest
> [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.4 s
> - in org.apache.nifi.security.XXEValidatorTest
> [INFO] Running org.apache.nifi.security.xml.XmlUtilsTest
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.347
> s <<< FAILURE! - in org.apache.nifi.security.xml.XmlUtilsTest
> [ERROR]
> testShouldHandleXXEInDocumentBuilder(org.apache.nifi.security.xml.XmlUtilsTest)
> Time elapsed: 0.017 s <<< FAILURE!
> org.codehaus.groovy.runtime.powerassert.PowerAssertionError:
> assert msg =~ "SAXParseException.* DOCTYPE is disallowed when the feature"
>| |
>| java.util.regex.Matcher[pattern=SAXParseException.* DOCTYPE is
> disallowed when the feature region=0,314 lastmatch=]
>org.xml.sax.SAXParseException; systemId:
> file:/home/juldrixx/Documents/tmp/nifi/nifi-commons/nifi-security-utils/src/test/resources/local_xxe_file.xml;
> lineNumber: 1; columnNumber: 65; DOCTYPE n'est pas autorisé lorsque la
> fonctionnalité "http://apache.org/xml/features/disallow-doctype-decl; est
> définie sur True.
> at
> org.apache.nifi.security.xml.XmlUtilsTest.testShouldHandleXXEInDocumentBuilder(XmlUtilsTest.groovy:97)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8433) Add decommission command to nifi.sh
[
https://issues.apache.org/jira/browse/NIFI-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8433:
--
Fix Version/s: 1.14.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add decommission command to nifi.sh
> ---
>
> Key: NIFI-8433
> URL: https://issues.apache.org/jira/browse/NIFI-8433
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Add the ability to run {{bin/nifi.sh decommission}} and have that take care
> of disconnecting node from the cluster, offloading, removing the node, and
> shutting it down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8507) Update documentation for nifi.sh in Admin Guide
Joey Frazee created NIFI-8507: - Summary: Update documentation for nifi.sh in Admin Guide Key: NIFI-8507 URL: https://issues.apache.org/jira/browse/NIFI-8507 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.14.0 Reporter: Joey Frazee Some of the nifi.sh commands were never documented in the admin guide and some of the recent additions in 1.12.x and 1.13.x didn't come along with docs either. It'd be useful to highlight some of the following so users know they exist and how to use them: * dump * diagnostics * start --wait-for-init * decommission * stateless (?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8506) Azure Integration tests fail
[ https://issues.apache.org/jira/browse/NIFI-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337558#comment-17337558 ] Joey Frazee commented on NIFI-8506: --- [~gschaer] Thanks for this. The first I probably should have caught. We use @Ignore fairly liberally so I wouldn't worry about that per se, but I also think it'd make sense to un-ignore those tests as well. If it's unreasonably long, then making the size is a little smaller (or configurable) probably makes more sense than ignoring altogether. If the last isn't reproducible yet, I think it'd be good to address the first two and handle the other separately? > Azure Integration tests fail > - > > Key: NIFI-8506 > URL: https://issues.apache.org/jira/browse/NIFI-8506 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.14.0 >Reporter: Guillaume Schaer >Assignee: Guillaume Schaer >Priority: Minor > > NIFI-7886 introduced some changes in the integration-tests of the > nifi-azure-bundle. > integration-tests of this component don't pass after PR #4576 > More specifically: > # *org.apache.nifi.processors.azure.storage.ITListAzureBlobStorage* tries to > upload a blob, retrieve it and then expects its lengths to be 10. But > AbstractAzureBlobStorageIT was changed to upload a blob of length 36 instead > of 10. > # Two tests in > *org.apache.nifi.processors.azure.storage.ITPutAzureDataLakeStorage* and > *org.apache.nifi.processors.azure.storage.ITFetchAzureDataLakeStorage* are > set to Ignore. By default a skipped test, makes the verify phase to fail. > # I also saw an intermittent failure on > *org.apache.nifi.processors.azure.storage.ITListAzureDataLakeStorage* in > testListRootRecursive but I could not reproduce it systematically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8505) RunNiFi is_loaded returns 0 exit code for all statuses
Joey Frazee created NIFI-8505: - Summary: RunNiFi is_loaded returns 0 exit code for all statuses Key: NIFI-8505 URL: https://issues.apache.org/jira/browse/NIFI-8505 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.13.2, 1.13.1, 1.13.0 Reporter: Joey Frazee The is_loaded command in RunNiFi should return a non-zero exit code when NiFi is not running. RunNiFi [1] handles the is_loaded not running state by catching a NiFiNotRunningException and then exiting cleanly. This is inconsistent with error handling in the rest of the commands, particularly compared to status() which returns a non-zero exit code for not running. Additionally, isNiFiFullyRunning() is unique in handling its error states via an exception. It'd be good if there was consistency across all the commands, either managing this via exception or not. 1. https://github.com/apache/nifi/blob/3963f66dffcdc0ca8c3c5a35844007c077d77f41/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7593) Azure Blob Storage uses sensitive property for account name
[ https://issues.apache.org/jira/browse/NIFI-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7593. --- Resolution: Not A Problem [~malthe] This is intentional as account names can correspond to publicly accessible resources and storing this in plain text or visible to a variety of users presents a security risk if the flow.xml.gz is compromised or a template is unintentionally shared, etc. I agree that it's very unintuitive but it's important to do. I'm going to close this, but I want to ask, do you think it'd be helpful to have this rationale documented somewhere in the user guide? > Azure Blob Storage uses sensitive property for account name > --- > > Key: NIFI-7593 > URL: https://issues.apache.org/jira/browse/NIFI-7593 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Minor > Labels: easyfix > > Processors such as > [AzureListBlobStorage|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/] > has account name as a sensitive property. > This seems wrong and also causes some usability issues with parameter context > parameters (which are either sensitive or non-sensitive). > Note that while the fix for this is trivial, it could have real implications > for existing flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7593) Azure Blob Storage uses sensitive property for account name
[ https://issues.apache.org/jira/browse/NIFI-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7593: -- Labels: easyfix (was: easyfix newbie) > Azure Blob Storage uses sensitive property for account name > --- > > Key: NIFI-7593 > URL: https://issues.apache.org/jira/browse/NIFI-7593 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Minor > Labels: easyfix > > Processors such as > [AzureListBlobStorage|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/] > has account name as a sensitive property. > This seems wrong and also causes some usability issues with parameter context > parameters (which are either sensitive or non-sensitive). > Note that while the fix for this is trivial, it could have real implications > for existing flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7548) Nifi embedded secure zookeeper
[ https://issues.apache.org/jira/browse/NIFI-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7548. --- Resolution: Duplicate Hi [~santosh239]. Closing this because it's a duplicate of NIFI-7203 which was completed in 1.13.x. > Nifi embedded secure zookeeper > -- > > Key: NIFI-7548 > URL: https://issues.apache.org/jira/browse/NIFI-7548 > Project: Apache NiFi > Issue Type: Bug > Components: Security >Affects Versions: 1.11.4 > Environment: DEV >Reporter: Santosh >Priority: Major > > Trying to use embedded zookeeper. Want to secure the zookeper either through > SSL or kerberos. Want to know if SSL for zookeeper is supported and can be > integrated with Nifi 1.11.4 > > Thanks !! > Santosh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7593) Azure Blob Storage uses sensitive property for account name
[ https://issues.apache.org/jira/browse/NIFI-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7593: -- Labels: easyfix newbie (was: easyfix) > Azure Blob Storage uses sensitive property for account name > --- > > Key: NIFI-7593 > URL: https://issues.apache.org/jira/browse/NIFI-7593 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Malthe Borch >Priority: Minor > Labels: easyfix, newbie > > Processors such as > [AzureListBlobStorage|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/] > has account name as a sensitive property. > This seems wrong and also causes some usability issues with parameter context > parameters (which are either sensitive or non-sensitive). > Note that while the fix for this is trivial, it could have real implications > for existing flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7598) Add instructions for how to use named volumes to Docker README.md
[ https://issues.apache.org/jira/browse/NIFI-7598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee reassigned NIFI-7598: - Assignee: (was: Joey Frazee) > Add instructions for how to use named volumes to Docker README.md > - > > Key: NIFI-7598 > URL: https://issues.apache.org/jira/browse/NIFI-7598 > Project: Apache NiFi > Issue Type: Improvement > Components: Docker >Reporter: Joey Frazee >Priority: Minor > > Using named volumes provides an easy way to persist data when running the > Docker container. This has been possible since NIFI-5438 but it isn't > currently documented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7886) FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be able to fetch ranges
[ https://issues.apache.org/jira/browse/NIFI-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7886: -- Resolution: Fixed Status: Resolved (was: Patch Available) > FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be > able to fetch ranges > -- > > Key: NIFI-7886 > URL: https://issues.apache.org/jira/browse/NIFI-7886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.12.0, 1.13.0 >Reporter: Paul Kelly >Assignee: Paul Kelly >Priority: Minor > Labels: azureblob, gcs, s3 > Fix For: 1.14.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Azure Blob Storage, AWS S3, and Google Cloud Storage all support retrieving > byte ranges of stored objects. Current versions of NiFi processors for these > services do not support fetching by byte range. > Allowing to fetch by range would allow multiple enhancements: > * Parallelized downloads > ** Faster speeds if the bandwidth delay product of the connection is lower > than the available bandwidth > ** Load distribution over a cluster > * Cost savings > ** If the file is large and only part of the file is needed, the desired > part of the file can be downloaded, saving bandwidth costs by not retrieving > unnecessary bytes > ** Download failures would only need to retry the failed segment, rather > than the full file > * Download extremely large files > ** Ability to download files that are larger than the available content repo > by downloading a segment and moving it off to a system with more capacity > before downloading another segment > > Some of these enhancements would require an upstream processor to generate > multiple flow files, each covering a different part of the overall range. > Something like this: > ListS3 -> ExecuteGroovyScript (to split into multiple flow files with > different range attributes) -> FetchS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7886) FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be able to fetch ranges
[ https://issues.apache.org/jira/browse/NIFI-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7886: -- Affects Version/s: (was: 1.13.2) (was: 1.13.1) > FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be > able to fetch ranges > -- > > Key: NIFI-7886 > URL: https://issues.apache.org/jira/browse/NIFI-7886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.12.0, 1.13.0 >Reporter: Paul Kelly >Assignee: Paul Kelly >Priority: Minor > Labels: azureblob, gcs, s3 > Fix For: 1.14.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Azure Blob Storage, AWS S3, and Google Cloud Storage all support retrieving > byte ranges of stored objects. Current versions of NiFi processors for these > services do not support fetching by byte range. > Allowing to fetch by range would allow multiple enhancements: > * Parallelized downloads > ** Faster speeds if the bandwidth delay product of the connection is lower > than the available bandwidth > ** Load distribution over a cluster > * Cost savings > ** If the file is large and only part of the file is needed, the desired > part of the file can be downloaded, saving bandwidth costs by not retrieving > unnecessary bytes > ** Download failures would only need to retry the failed segment, rather > than the full file > * Download extremely large files > ** Ability to download files that are larger than the available content repo > by downloading a segment and moving it off to a system with more capacity > before downloading another segment > > Some of these enhancements would require an upstream processor to generate > multiple flow files, each covering a different part of the overall range. > Something like this: > ListS3 -> ExecuteGroovyScript (to split into multiple flow files with > different range attributes) -> FetchS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7886) FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be able to fetch ranges
[ https://issues.apache.org/jira/browse/NIFI-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7886: -- Fix Version/s: 1.14.0 > FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be > able to fetch ranges > -- > > Key: NIFI-7886 > URL: https://issues.apache.org/jira/browse/NIFI-7886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.12.0, 1.13.0, 1.13.1, 1.13.2 >Reporter: Paul Kelly >Assignee: Paul Kelly >Priority: Minor > Labels: azureblob, gcs, s3 > Fix For: 1.14.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Azure Blob Storage, AWS S3, and Google Cloud Storage all support retrieving > byte ranges of stored objects. Current versions of NiFi processors for these > services do not support fetching by byte range. > Allowing to fetch by range would allow multiple enhancements: > * Parallelized downloads > ** Faster speeds if the bandwidth delay product of the connection is lower > than the available bandwidth > ** Load distribution over a cluster > * Cost savings > ** If the file is large and only part of the file is needed, the desired > part of the file can be downloaded, saving bandwidth costs by not retrieving > unnecessary bytes > ** Download failures would only need to retry the failed segment, rather > than the full file > * Download extremely large files > ** Ability to download files that are larger than the available content repo > by downloading a segment and moving it off to a system with more capacity > before downloading another segment > > Some of these enhancements would require an upstream processor to generate > multiple flow files, each covering a different part of the overall range. > Something like this: > ListS3 -> ExecuteGroovyScript (to split into multiple flow files with > different range attributes) -> FetchS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7886) FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be able to fetch ranges
[ https://issues.apache.org/jira/browse/NIFI-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7886: -- Affects Version/s: 1.13.0 1.13.1 1.13.2 > FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be > able to fetch ranges > -- > > Key: NIFI-7886 > URL: https://issues.apache.org/jira/browse/NIFI-7886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.12.0, 1.13.0, 1.13.1, 1.13.2 >Reporter: Paul Kelly >Assignee: Paul Kelly >Priority: Minor > Labels: azureblob, gcs, s3 > Time Spent: 5h > Remaining Estimate: 0h > > Azure Blob Storage, AWS S3, and Google Cloud Storage all support retrieving > byte ranges of stored objects. Current versions of NiFi processors for these > services do not support fetching by byte range. > Allowing to fetch by range would allow multiple enhancements: > * Parallelized downloads > ** Faster speeds if the bandwidth delay product of the connection is lower > than the available bandwidth > ** Load distribution over a cluster > * Cost savings > ** If the file is large and only part of the file is needed, the desired > part of the file can be downloaded, saving bandwidth costs by not retrieving > unnecessary bytes > ** Download failures would only need to retry the failed segment, rather > than the full file > * Download extremely large files > ** Ability to download files that are larger than the available content repo > by downloading a segment and moving it off to a system with more capacity > before downloading another segment > > Some of these enhancements would require an upstream processor to generate > multiple flow files, each covering a different part of the overall range. > Something like this: > ListS3 -> ExecuteGroovyScript (to split into multiple flow files with > different range attributes) -> FetchS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8447) Add Vault encryption as an option in the Encrypt Tool
[
https://issues.apache.org/jira/browse/NIFI-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333443#comment-17333443
]
Joey Frazee commented on NIFI-8447:
---
[~jgresock] I have a minor worry that the top-level properties of uri and
transit key are implementation specific and the naming could be confusing.
Sure most of it will be externalized in another properties file but at some
point this should work with AWS KMS, Azure KeyVault. GCP CMEK (?), or event a
custom provider so I think it'd be useful to be use provider-specific naming
where applicable.
I also think it's important to be thinking towards having a
nifi.sensitive.props.vault.implementation=MyVaultImplementation as custom
vaults are not uncommon. (Also should help with testing.)
Thoughts?
> Add Vault encryption as an option in the Encrypt Tool
> -
>
> Key: NIFI-8447
> URL: https://issues.apache.org/jira/browse/NIFI-8447
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: Joseph Gresock
>Priority: Minor
>
> Using the StandardVaultCommunicationService, add options to the Encrypt Tool
> in nifi-toolkit for the following:
> # Select encryption method (aes/gcm vs. vault)
> # Select vault configuration (recommended as a
> vault-configuration.properties file, since there are so many configuration
> properties). Vault configuration properties include:
> {code}
> nifi.sensitive.props.vault.uri=
> nifi.sensitive.props.vault.transit.key=
> nifi.sensitive.props.vault.auth.properties.file=
> # Optional TLS options if addr is https
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorPasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=
> {code}
> Selecting vault encryption method should set the encryption value in XML
> files or the *.protected property in properties files to "vault/[transitKey]"
> A transitKey represents a distinct Vault configuration of encryption settings.
> Additionally, the corresponding nifi.sensitive.props.vault.* properties
> should be configured in the resulting nifi.properties file so that the NiFi
> instance can use the same Vault configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8449) Add support for transactional batch operations to PutAzureCosmosDBRecord
Joey Frazee created NIFI-8449: - Summary: Add support for transactional batch operations to PutAzureCosmosDBRecord Key: NIFI-8449 URL: https://issues.apache.org/jira/browse/NIFI-8449 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Joey Frazee Right now PutAzureCosmosDBRecord only batches records for processing, not for writing data to Cosmos DB, so inserting and updating data happens serially. The Azure Java SDK supports using transactional batch operations, however, which if used should improve performance as well as adding more transactionality. Unfortunately, the API for this only allows batching by partition id so this will introduce some complexity since the processor or flow will have to "manually" handle grouping records to create the transactional batch calls. While not ideal it should be ok -- best case we could have 1 call for N records, worst case we could have the status quo of N calls for N records. So it's probably worth doing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7573) Add a Azure Active Directory (AAD) User Group Provider
[ https://issues.apache.org/jira/browse/NIFI-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7573. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add a Azure Active Directory (AAD) User Group Provider > --- > > Key: NIFI-7573 > URL: https://issues.apache.org/jira/browse/NIFI-7573 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Fix For: 1.14.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > NiFi on Azure Cloud environment may be integrated with AAD and define the > group based authorization policies. Combined with OIDC based user > authentication, this will improve user/group authorization policy management > experience. > Reference: > [https://docs.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0] > [https://docs.microsoft.com/en-us/graph/api/group-list-members?view=graph-rest-1.0=http] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8394) Fix NPE error when NiFi connects to external zookeeper insecurely
[
https://issues.apache.org/jira/browse/NIFI-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8394:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix NPE error when NiFi connects to external zookeeper insecurely
> -
>
> Key: NIFI-8394
> URL: https://issues.apache.org/jira/browse/NIFI-8394
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0, 1.13.1, 1.13.2
>Reporter: Nadeem
>Assignee: Nadeem
>Priority: Minor
> Labels: zookeeper
> Fix For: 1.14.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When NiFi connects to external zookeeper insecurely, then following error is
> logged due to null watcher passed in zookeeper constructor. This error is
> introduced due to changes made for adding zookeeper client with TLS NIFI-7819
> {noformat}
> ERROR [Framework Task Thread Thread-3-EventThread]
> org.apache.zookeeper.ClientCnxn Error while calling watcher
> java.lang.NullPointerException: null
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8394) Fix NPE error when NiFi connects to external zookeeper insecurely
[
https://issues.apache.org/jira/browse/NIFI-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8394:
--
Priority: Minor (was: Major)
> Fix NPE error when NiFi connects to external zookeeper insecurely
> -
>
> Key: NIFI-8394
> URL: https://issues.apache.org/jira/browse/NIFI-8394
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0, 1.13.1, 1.13.2
>Reporter: Nadeem
>Assignee: Nadeem
>Priority: Minor
> Labels: zookeeper
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When NiFi connects to external zookeeper insecurely, then following error is
> logged due to null watcher passed in zookeeper constructor. This error is
> introduced due to changes made for adding zookeeper client with TLS NIFI-7819
> {noformat}
> ERROR [Framework Task Thread Thread-3-EventThread]
> org.apache.zookeeper.ClientCnxn Error while calling watcher
> java.lang.NullPointerException: null
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8394) Fix NPE error when NiFi connects to external zookeeper insecurely
[
https://issues.apache.org/jira/browse/NIFI-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8394:
--
Fix Version/s: 1.14.0
> Fix NPE error when NiFi connects to external zookeeper insecurely
> -
>
> Key: NIFI-8394
> URL: https://issues.apache.org/jira/browse/NIFI-8394
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0, 1.13.1, 1.13.2
>Reporter: Nadeem
>Assignee: Nadeem
>Priority: Minor
> Labels: zookeeper
> Fix For: 1.14.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When NiFi connects to external zookeeper insecurely, then following error is
> logged due to null watcher passed in zookeeper constructor. This error is
> introduced due to changes made for adding zookeeper client with TLS NIFI-7819
> {noformat}
> ERROR [Framework Task Thread Thread-3-EventThread]
> org.apache.zookeeper.ClientCnxn Error while calling watcher
> java.lang.NullPointerException: null
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-8346) PutAzureBlobStorage doesn't route to failure despite the exception during upload
[ https://issues.apache.org/jira/browse/NIFI-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8346. --- Resolution: Fixed > PutAzureBlobStorage doesn't route to failure despite the exception during > upload > > > Key: NIFI-8346 > URL: https://issues.apache.org/jira/browse/NIFI-8346 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0, 1.13.1, 1.13.2 >Reporter: Denes Arvay >Assignee: Denes Arvay >Priority: Major > Fix For: 1.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > If PutAzureBlobStorage tries to upload to an archived blob an IOException is > thrown that's not properly handled thus the affected flowfiles are directed > back to the incoming connection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8346) PutAzureBlobStorage doesn't route to failure despite the exception during upload
[ https://issues.apache.org/jira/browse/NIFI-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8346: -- Fix Version/s: 1.14.0 > PutAzureBlobStorage doesn't route to failure despite the exception during > upload > > > Key: NIFI-8346 > URL: https://issues.apache.org/jira/browse/NIFI-8346 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0, 1.13.1, 1.13.2 >Reporter: Denes Arvay >Assignee: Denes Arvay >Priority: Major > Fix For: 1.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > If PutAzureBlobStorage tries to upload to an archived blob an IOException is > thrown that's not properly handled thus the affected flowfiles are directed > back to the incoming connection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7850) NIFI dockerhub image does not start when configured with LDAP authentication
[
https://issues.apache.org/jira/browse/NIFI-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee resolved NIFI-7850.
---
Fix Version/s: 1.14.0
Resolution: Fixed
> NIFI dockerhub image does not start when configured with LDAP authentication
>
>
> Key: NIFI-7850
> URL: https://issues.apache.org/jira/browse/NIFI-7850
> Project: Apache NiFi
> Issue Type: Bug
> Components: Docker
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.13.1, 1.13.2
> Environment: Docker on Kubernetes
>Reporter: Ney Walens De Mesquita
>Priority: Major
> Fix For: 1.14.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When starting NIFI docks image configured to use LDAP authentication, the
> process exists with the error:
> {quote}The nifi.web.http.port must be specified if running in a cluster with
> nifi.cluster.protocol.is.secure set to false.
> {quote}
>
> When checking the start.sh and secure.sh scripts, I have noticed that some
> standard parameters are set but not the one referred in the error.
>
> I managed to fix the issue on my local environment by adding the following to
> secure.sh:
> {quote}prop_replace 'nifi.cluser.protocol.is.secure' 'true'
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7850) NIFI dockerhub image does not start when configured with LDAP authentication
[
https://issues.apache.org/jira/browse/NIFI-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-7850:
--
Affects Version/s: 1.14.0
1.13.0
1.13.1
1.13.2
> NIFI dockerhub image does not start when configured with LDAP authentication
>
>
> Key: NIFI-7850
> URL: https://issues.apache.org/jira/browse/NIFI-7850
> Project: Apache NiFi
> Issue Type: Bug
> Components: Docker
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.13.1, 1.13.2
> Environment: Docker on Kubernetes
>Reporter: Ney Walens De Mesquita
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When starting NIFI docks image configured to use LDAP authentication, the
> process exists with the error:
> {quote}The nifi.web.http.port must be specified if running in a cluster with
> nifi.cluster.protocol.is.secure set to false.
> {quote}
>
> When checking the start.sh and secure.sh scripts, I have noticed that some
> standard parameters are set but not the one referred in the error.
>
> I managed to fix the issue on my local environment by adding the following to
> secure.sh:
> {quote}prop_replace 'nifi.cluser.protocol.is.secure' 'true'
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8346) PutAzureBlobStorage doesn't route to failure despite the exception during upload
[ https://issues.apache.org/jira/browse/NIFI-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8346: -- Affects Version/s: 1.13.2 1.14.0 > PutAzureBlobStorage doesn't route to failure despite the exception during > upload > > > Key: NIFI-8346 > URL: https://issues.apache.org/jira/browse/NIFI-8346 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0, 1.13.1, 1.13.2 >Reporter: Denes Arvay >Assignee: Denes Arvay >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > If PutAzureBlobStorage tries to upload to an archived blob an IOException is > thrown that's not properly handled thus the affected flowfiles are directed > back to the incoming connection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8329) Upgrade NiFi direct dependencies
[ https://issues.apache.org/jira/browse/NIFI-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304423#comment-17304423 ] Joey Frazee commented on NIFI-8329: --- [~joewitt] [~thenatog] FYI, I added a note for the Storm spout removal under "1.14.0 (unreleased)" migration guidance so it's not forgotten. LMK if you'd rather not have it listed since it's not imminent and I'll remove it. > Upgrade NiFi direct dependencies > > > Key: NIFI-8329 > URL: https://issues.apache.org/jira/browse/NIFI-8329 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.13.1 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 1.14.0 > > Time Spent: 10m > Remaining Estimate: 0h > > |org.apache.derby:derby| > |org.apache.ignite:ignite| > |com.fasterxml.jackson.core:jackson-databind| > |org.apache.lucene:lucene-core| > |spring-integration| > |xerces:xerces or xercesimpl| > |org.apache.activemq:activemq| > |Jetty (already upgraded)| > |org.yaml:snakeyaml| > |org.apache.storm:storm-core| > |com.ibm.icu:icu4j| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8335) Remove nifi-storm-spout module
[ https://issues.apache.org/jira/browse/NIFI-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8335. --- Resolution: Fixed > Remove nifi-storm-spout module > -- > > Key: NIFI-8335 > URL: https://issues.apache.org/jira/browse/NIFI-8335 > Project: Apache NiFi > Issue Type: Sub-task >Affects Versions: 1.13.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 1.14.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Remove the nifi-external/nifi-storm-spout module as it is not regularly > maintained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8324) Fix X-ProxiedEntitiesChain header examples in Admin Guide
Joey Frazee created NIFI-8324: - Summary: Fix X-ProxiedEntitiesChain header examples in Admin Guide Key: NIFI-8324 URL: https://issues.apache.org/jira/browse/NIFI-8324 Project: Apache NiFi Issue Type: Bug Components: Documentation Website Affects Versions: 1.13.0, 1.14.0, 1.13.1 Reporter: Joey Frazee Assignee: Joey Frazee The handling of the X-ProxiedEntitiesChain header for secure proxied requests became more strict in NIFI-7744 and now requires the outermost <, >. The NGINX examples in the Admin Guide use $ssl_client_s_dn which results in: ERROR [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Unable to authenticate: Proxy chain format is not recognized and can not safely be converted to a list. java.lang.IllegalArgumentException: Proxy chain format is not recognized and can not safely be converted to a list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8323) Azure Managed Identity Support for AzureBlobStorage Processors
[ https://issues.apache.org/jira/browse/NIFI-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-8323. --- Resolution: Duplicate > Azure Managed Identity Support for AzureBlobStorage Processors > -- > > Key: NIFI-8323 > URL: https://issues.apache.org/jira/browse/NIFI-8323 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.0 >Reporter: Michael Morrone >Priority: Major > > The DeleteAzureBlobStorage, FetchAzureBlobStorage, ListAzureBlobStorage, and > PutAzureBlobStorage require SAS Tokens or Storage Account Keys. We would > like support added for Managed Identities, similar to what was done for > NIFI-6149 and NIFI-7409. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8323) Azure Managed Identity Support for AzureBlobStorage Processors
[ https://issues.apache.org/jira/browse/NIFI-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301778#comment-17301778 ] Joey Frazee commented on NIFI-8323: --- [~mmorrone] This is a duplicate of the work that's happened over in NIFI-7677 and https://github.com/apache/nifi/pull/4430 so I'm going to close this. It's lagged a bit because it involved some breaking changes, but it may be time to say that the benefits outweigh that. > Azure Managed Identity Support for AzureBlobStorage Processors > -- > > Key: NIFI-8323 > URL: https://issues.apache.org/jira/browse/NIFI-8323 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.13.0 >Reporter: Michael Morrone >Priority: Major > > The DeleteAzureBlobStorage, FetchAzureBlobStorage, ListAzureBlobStorage, and > PutAzureBlobStorage require SAS Tokens or Storage Account Keys. We would > like support added for Managed Identities, similar to what was done for > NIFI-6149 and NIFI-7409. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8289) EmbeddedQuestDbRolloverHandlerTest tests fail when local date and UTC date differ
[
https://issues.apache.org/jira/browse/NIFI-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee resolved NIFI-8289.
---
Fix Version/s: 1.14.0
Resolution: Fixed
> EmbeddedQuestDbRolloverHandlerTest tests fail when local date and UTC date
> differ
> -
>
> Key: NIFI-8289
> URL: https://issues.apache.org/jira/browse/NIFI-8289
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.14.0
>Reporter: Joey Frazee
>Assignee: Simon Bence
>Priority: Minor
> Fix For: 1.14.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I'm seeing test failures with EmbeddedQuestDbRolloverHandlerTest on Linux and
> macOS when the local date and UTC date differ (e.g., in US time zones near
> the end of the day).
> {code}
> Failures:
>
> EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenLessPartitionThanNeeded:99->thenRemainingPartitionsAre:204
> Partition 2021-03-01 is expected
>
> EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenNoPartitionToDrop:112->thenRemainingPartitionsAre:204
> Partition 2021-02-28 is expected
>
> EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenNonconsecutivePartitionsPresent:139->thenRemainingPartitionsAre:204
> Partition 2021-03-01 is expected
>
> EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenOldPartitionsPresent:125->thenRemainingPartitionsAre:201
> expected:<3> but was:<4>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8305) Update nifi-docker Docker builds for consistency and best practices
Joey Frazee created NIFI-8305: - Summary: Update nifi-docker Docker builds for consistency and best practices Key: NIFI-8305 URL: https://issues.apache.org/jira/browse/NIFI-8305 Project: Apache NiFi Issue Type: Improvement Components: Docker Reporter: Joey Frazee Assignee: Joey Frazee The nifi-docker Docker builds need a little bit of clean up: - There are some inconsistencies w.r.t. installed packages and labels between nifi-docker/dockerhub and nifi-docker/dockermaven - The default NIFI_VERSION for nifi-docker/dockerhub is 1.7.0 - The apt cache isn't cleaned up after installing runtime dependencies - unzip is assumed to be installed - The notes at the bottom of nifi-docker/dockerhub/README.md don't render correctly on Dockerhub - The pom's sometimes use project.version and sometimes nifi.version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7717) Add ARGs to Docker build for parent image and tag
[ https://issues.apache.org/jira/browse/NIFI-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7717. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add ARGs to Docker build for parent image and tag > - > > Key: NIFI-7717 > URL: https://issues.apache.org/jira/browse/NIFI-7717 > Project: Apache NiFi > Issue Type: Task >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7717) Add ARGs to Docker build for parent image and tag
[ https://issues.apache.org/jira/browse/NIFI-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7717: -- Summary: Add ARGs to Docker build for parent image and tag (was: Update Docker images to use JRE 11) > Add ARGs to Docker build for parent image and tag > - > > Key: NIFI-7717 > URL: https://issues.apache.org/jira/browse/NIFI-7717 > Project: Apache NiFi > Issue Type: Task >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8289) EmbeddedQuestDbRolloverHandlerTest tests fail when local date and UTC date differ
Joey Frazee created NIFI-8289:
-
Summary: EmbeddedQuestDbRolloverHandlerTest tests fail when local
date and UTC date differ
Key: NIFI-8289
URL: https://issues.apache.org/jira/browse/NIFI-8289
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.14.0
Reporter: Joey Frazee
I'm seeing test failures with EmbeddedQuestDbRolloverHandlerTest on Linux and
macOS when the local date and UTC date differ (e.g., in US time zones near the
end of the day).
{code}
Failures:
EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenLessPartitionThanNeeded:99->thenRemainingPartitionsAre:204
Partition 2021-03-01 is expected
EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenNoPartitionToDrop:112->thenRemainingPartitionsAre:204
Partition 2021-02-28 is expected
EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenNonconsecutivePartitionsPresent:139->thenRemainingPartitionsAre:204
Partition 2021-03-01 is expected
EmbeddedQuestDbRolloverHandlerTest.testRollOverWhenOldPartitionsPresent:125->thenRemainingPartitionsAre:201
expected:<3> but was:<4>
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8200) PutAzureDataLakeStorage processor leaves behind a 0B file if upload fails
[ https://issues.apache.org/jira/browse/NIFI-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8200: -- Fix Version/s: 1.14.0 Resolution: Fixed Status: Resolved (was: Patch Available) > PutAzureDataLakeStorage processor leaves behind a 0B file if upload fails > - > > Key: NIFI-8200 > URL: https://issues.apache.org/jira/browse/NIFI-8200 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Peter Gyori >Assignee: Timea Barna >Priority: Minor > Labels: azure > Fix For: 1.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > > The PutAzureDataLakeStorage processor works in a way that first it creates an > empty file with the given name, and then uploads the content to this file. > However, if the upload fails, the empty file does not get removed. > The processor needs to be modified to remove the file if the upload is not > successful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8200) PutAzureDataLakeStorage processor leaves behind a 0B file if upload fails
[ https://issues.apache.org/jira/browse/NIFI-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8200: -- Affects Version/s: 1.13.0 > PutAzureDataLakeStorage processor leaves behind a 0B file if upload fails > - > > Key: NIFI-8200 > URL: https://issues.apache.org/jira/browse/NIFI-8200 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.13.0 >Reporter: Peter Gyori >Assignee: Timea Barna >Priority: Minor > Labels: azure > Fix For: 1.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > > The PutAzureDataLakeStorage processor works in a way that first it creates an > empty file with the given name, and then uploads the content to this file. > However, if the upload fails, the empty file does not get removed. > The processor needs to be modified to remove the file if the upload is not > successful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8220) Establish a secure by default configuration for NiFi
[ https://issues.apache.org/jira/browse/NIFI-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287405#comment-17287405 ] Joey Frazee commented on NIFI-8220: --- I don't have anything to show for it, but prior to this I started looking around at tools for what would ostensibly be a deployment linter -- even if more secure by default happens, I was thinking about how to identify when a best practice was ignored or overridden. Is this in the domain of of the toolkit? Would a documentation walkthrough using existing server admin tools be more sensible? > Establish a secure by default configuration for NiFi > > > Key: NIFI-8220 > URL: https://issues.apache.org/jira/browse/NIFI-8220 > Project: Apache NiFi > Issue Type: Epic > Components: Tools and Build >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Blocker > Fix For: 1.14.0 > > > Inspired by this tweet > https://twitter.com/_escctrl_/status/1359280656174510081?s=21 and the > resulting discussion here > https://lists.apache.org/thread.html/rc590f21807192a0dce18293c2d5b47392a6fd8a1ef26d77fbd6ee695%40%3Cdev.nifi.apache.org%3E > It is time to change our config model. It was also setup to be easy to use. > We've seen these silly setups on the Internet before but has gotten > ridiculous. We need to take action. > Will create a set of one or more JIRAs to roughly do the following. > 1. Disable HTTP by default. If a user wants to enable to it for whatever > reason then also make them enable a new property which says something to the > effect of 'allow completely non secure access to the entire nifi instance - > not recommended' > 2. Enable HTTPS with one way authentication by default which would be the > client authenticating the server whereby the server has a server cert. We > could either make that cert a self-signed (and thus not trusted by client's > by default) cert or give a way for the user to run through command line > process to make a legit cert. > 3. If not already configured with an authorization provider supply and out of > the box provider which supports only a single auto generated at first startup > user/password enabling access to the NiFi system. > 4. Disable all restricted processors by default. Require the user to > explicitly enable them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8231) Clarify documentation for MonitorMemory Memory Pool allowable values
Joey Frazee created NIFI-8231: - Summary: Clarify documentation for MonitorMemory Memory Pool allowable values Key: NIFI-8231 URL: https://issues.apache.org/jira/browse/NIFI-8231 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Affects Versions: 1.14.0 Reporter: Joey Frazee The allowable values for the Memory Pools isn't static but varies by platform and JVM. The docs don't indicate that so there are a few things that need to be clarified in the processor documentation since the published docs may vary: * The allowable values for Memory Pools is just an example. * Moving a flow between different JVMs or platforms might invalidate the reporting service if you configure a memory pool that exists on one and not the other. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-3383) MonitorMemory produces UnsupportedOperationException
[
https://issues.apache.org/jira/browse/NIFI-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-3383:
--
Fix Version/s: 1.14.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> MonitorMemory produces UnsupportedOperationException
>
>
> Key: NIFI-3383
> URL: https://issues.apache.org/jira/browse/NIFI-3383
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Bryan Bende
>Assignee: Matt Burgess
>Priority: Minor
> Fix For: 1.14.0
>
> Attachments: MonitorMemory Error.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {code}
> [na:1.8.0_111] Caused by: java.lang.UnsupportedOperationException: Usage
> threshold is not supported at
> sun.management.MemoryPoolImpl.setUsageThreshold(MemoryPoolImpl.java:114)
> ~[na:1.8.0_111] at
> org.apache.nifi.controller.MonitorMemory.onConfigured(MonitorMemory.java:178)
> ~[na:na] ... 16 common frames omitted
> {code}
> MonitoryMemory checks if
> memoryPoolBean.isCollectionUsageThresholdSupported()) and if so it eventually
> calls monitoredBean.setUsageThreshold(calculatedThreshold), but
> setUsageThreshold then checks isUsageThresholdSupported() so it can throw
> UnsupportedOperationException.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8229) Add support for --wait-for-init to nifi.sh restart
Joey Frazee created NIFI-8229:
-
Summary: Add support for --wait-for-init to nifi.sh restart
Key: NIFI-8229
URL: https://issues.apache.org/jira/browse/NIFI-8229
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.13.0
Reporter: Joey Frazee
Copying from https://github.com/apache/nifi/pull/4814#issuecomment-776138163
--wait-for-init doesn't apply to nifi.sh restart because the $@ isn't provided
to the start. The naive change would be this:
{code}
restart)
init
shift
run "stop"
run "start" "$@"
;;
{code}
I'm not 100% sure whether that's correct though because in run() there's some
use of $1 and the shift changes $@. Should we change this now, or do it later?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8229) Add support for --wait-for-init to nifi.sh restart
[
https://issues.apache.org/jira/browse/NIFI-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-8229:
--
Description:
Copying from https://github.com/apache/nifi/pull/4814#issuecomment-776138163
--wait-for-init doesn't apply to nifi.sh restart because the $@ isn't provided
to the start. The naive change would be this:
{code}
restart)
init
shift
run "stop"
run "start" "$@"
;;
{code}
I'm not 100% sure whether that's correct though because in run() there's some
use of $1 and the shift changes $@.
was:
Copying from https://github.com/apache/nifi/pull/4814#issuecomment-776138163
--wait-for-init doesn't apply to nifi.sh restart because the $@ isn't provided
to the start. The naive change would be this:
{code}
restart)
init
shift
run "stop"
run "start" "$@"
;;
{code}
I'm not 100% sure whether that's correct though because in run() there's some
use of $1 and the shift changes $@. Should we change this now, or do it later?
> Add support for --wait-for-init to nifi.sh restart
> --
>
> Key: NIFI-8229
> URL: https://issues.apache.org/jira/browse/NIFI-8229
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.13.0
>Reporter: Joey Frazee
>Priority: Major
>
> Copying from https://github.com/apache/nifi/pull/4814#issuecomment-776138163
> --wait-for-init doesn't apply to nifi.sh restart because the $@ isn't
> provided to the start. The naive change would be this:
> {code}
> restart)
> init
>
> shift
> run "stop"
> run "start" "$@"
> ;;
> {code}
> I'm not 100% sure whether that's correct though because in run() there's some
> use of $1 and the shift changes $@.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8215) NiFi startup: --wait-for-init feature does not work on Ubuntu
[ https://issues.apache.org/jira/browse/NIFI-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8215: -- Resolution: Fixed Status: Resolved (was: Patch Available) > NiFi startup: --wait-for-init feature does not work on Ubuntu > - > > Key: NIFI-8215 > URL: https://issues.apache.org/jira/browse/NIFI-8215 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Blocker > Fix For: 1.13.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The nifi.sh script contains features that the sh on Ubuntu does not > understand. It needs to be refactored to be compatible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8213) Add shellcheck to CI builds to ensure POSIX shell compatibility
[ https://issues.apache.org/jira/browse/NIFI-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281474#comment-17281474 ] Joey Frazee commented on NIFI-8213: --- [~kdoran] FWIW I took a quick look at using this on the ./bin scripts as well as the Docker ./sh scripts. At the error severity level it misses issues. At warning, it seems to be capturing most of the relevant stuff, but there's stuff to remediate that probably doesn't actually matter. > Add shellcheck to CI builds to ensure POSIX shell compatibility > --- > > Key: NIFI-8213 > URL: https://issues.apache.org/jira/browse/NIFI-8213 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Kevin Doran >Priority: Minor > Fix For: 1.13.0 > > > In response to NIFI-8211, I would suggest we look into adding a call to a > tool such as shellcheck during our CI build to verify nifi.sh remains POSIX > shell portable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8211) nifi.sh doesn't work as expected when /bin/sh isn't bash
[
https://issues.apache.org/jira/browse/NIFI-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281410#comment-17281410
]
Joey Frazee commented on NIFI-8211:
---
AFAICT nothing destructive happens, but the behavior is a little wonky:
- It's very noisy -- see below.
- If for some reason NiFi can't start then the --wait-for-init will get stuck
in a loop and never exit. You can test this by killing NiFi while it's coming
up or trying to start it with a missing nifi.properties.
- It also creates a 0-byte file '=' in $NIFI_HOME/bin whether or not it
successfully starts.
{code}
./bin/nifi.sh: 366: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 253: ./bin/nifi.sh: [[: not found
./bin/nifi.sh: 373: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 374: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 375: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 376: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 377: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 378: ./bin/nifi.sh: declare: not found
./bin/nifi.sh: 385: ./bin/nifi.sh: current_time: not found
./bin/nifi.sh: 390: ./bin/nifi.sh: time_since_feedback: not found
./bin/nifi.sh: 391: ./bin/nifi.sh: time_since_feedback: not found
./bin/nifi.sh: 399: ./bin/nifi.sh: not_running_counter++: not found
./bin/nifi.sh: 400: ./bin/nifi.sh: not_running_counter: not found
sleep: missing operand
Try 'sleep --help' for more information.
./bin/nifi.sh: 385: ./bin/nifi.sh: current_time: not found
./bin/nifi.sh: 390: ./bin/nifi.sh: time_since_feedback: not found
./bin/nifi.sh: 391: ./bin/nifi.sh: time_since_feedback: not found
sleep: missing operand
Try 'sleep --help' for more information.
./bin/nifi.sh: 385: ./bin/nifi.sh: current_time: not found
./bin/nifi.sh: 390: ./bin/nifi.sh: time_since_feedback: not found
./bin/nifi.sh: 391: ./bin/nifi.sh: time_since_feedback: not found
{code}
> nifi.sh doesn't work as expected when /bin/sh isn't bash
>
>
> Key: NIFI-8211
> URL: https://issues.apache.org/jira/browse/NIFI-8211
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0
>Reporter: Joey Frazee
>Priority: Blocker
>
> NIFI-8123 introduced a --wait-for-init option to nifi.sh, making use of some
> Bash and Zsh-only builtins such as declare. Currently nifi.sh points to
> /bin/sh though and even when bash is available it can point to other shells
> such as ash and dash.
> This creates some noise during a regular startup with `start` and the new
> wait functionality doesn't work with a `start --wait-for-init` with other
> shells. Startup still succeeds however.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8211) nifi.sh doesn't work as expected when /bin/sh isn't bash
[ https://issues.apache.org/jira/browse/NIFI-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8211: -- Priority: Blocker (was: Major) > nifi.sh doesn't work as expected when /bin/sh isn't bash > > > Key: NIFI-8211 > URL: https://issues.apache.org/jira/browse/NIFI-8211 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Joey Frazee >Priority: Blocker > > NIFI-8123 introduced a --wait-for-init option to nifi.sh, making use of some > Bash and Zsh-only builtins such as declare. Currently nifi.sh points to > /bin/sh though and even when bash is available it can point to other shells > such as ash and dash. > This creates some noise during a regular startup with `start` and the new > wait functionality doesn't work with a `start --wait-for-init` with other > shells. Startup still succeeds however. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8211) nifi.sh doesn't work as expected when /bin/sh isn't bash
Joey Frazee created NIFI-8211: - Summary: nifi.sh doesn't work as expected when /bin/sh isn't bash Key: NIFI-8211 URL: https://issues.apache.org/jira/browse/NIFI-8211 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.13.0 Reporter: Joey Frazee NIFI-8123 introduced a --wait-for-init option to nifi.sh, making use of some Bash and Zsh-only builtins such as declare. Currently nifi.sh points to /bin/sh though and even when bash is available it can point to other shells such as ash and dash. This creates some noise during a regular startup with `start` and the new wait functionality doesn't work with a `start --wait-for-init` with other shells. Startup still succeeds however. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8208) Upgrade MongoDB Driver to 4.2 series
[ https://issues.apache.org/jira/browse/NIFI-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280310#comment-17280310 ] Joey Frazee commented on NIFI-8208: --- I took a little look at this and if we use mongodb-driver-legacy, most of the changes seem to be of two sorts: * Two of the WriteConcern modes have been removed after being deprecated for a while. These are FSYNCED and REPLICA_ACKNOWLEDGED. The recommended replacements are JOURNALED and W2. This change would be just an update to the property descriptor description and probably a warning. * Use of the Bson library APIs. This requires some more changes, but still straightforward. These changes make it all incompatible with the 3.x driver versions so this begs the question about whether we should just jump to mongo-driver-sync or mongo-driver-reactivestreams. 4.1 and 4.2 are advertised to be compatible with all recent Mongo versions, so this should be ok. > Upgrade MongoDB Driver to 4.2 series > > > Key: NIFI-8208 > URL: https://issues.apache.org/jira/browse/NIFI-8208 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.12.1 > Environment: Linux / Kerberos >Reporter: Nick Lange >Priority: Minor > Labels: MongoDB > > MongoDBs 3.12.X series driver tries to create a Kerberos Subject from a > String which fails. The driver was reworked for version 4.2 and may just do > the right thing. Without this fix, clients are forced to turn off subject > matching in Java which seems to defeat some of the compartmentalization the > nifi is looking to achieve. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7357) Make Zookeeper TLS Properties Available via nifi.properties
[ https://issues.apache.org/jira/browse/NIFI-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7357: -- Fix Version/s: 1.13.0 > Make Zookeeper TLS Properties Available via nifi.properties > --- > > Key: NIFI-7357 > URL: https://issues.apache.org/jira/browse/NIFI-7357 > Project: Apache NiFi > Issue Type: Sub-task > Components: Configuration, Configuration Management, Security >Reporter: Troy Melhase >Assignee: Troy Melhase >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7124) Improve experience configuring Zookeeper to support TLS
[ https://issues.apache.org/jira/browse/NIFI-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7124. --- Fix Version/s: 1.13.0 Resolution: Fixed > Improve experience configuring Zookeeper to support TLS > --- > > Key: NIFI-7124 > URL: https://issues.apache.org/jira/browse/NIFI-7124 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration Management, Security >Affects Versions: 1.11.1 >Reporter: M Tien >Assignee: Troy Melhase >Priority: Major > Labels: TLS, properties, security, zookeeper > Fix For: 1.13.0 > > > Improve experience around configuring ZK (embedded and external) to support > TLS now that the ZK version we include (3.5.x) supports TLS. > * Embedded ZK - allow for ZK to start with TLS, and NiFi/ZK should mutually > trust each other (ie, certificates are in each other's truststore). > * External ZK - allow for `nifi.properties` to contain properties that allow > NiFi to connect to it. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7357) Make Zookeeper TLS Properties Available via nifi.properties
[ https://issues.apache.org/jira/browse/NIFI-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7357. --- Resolution: Duplicate > Make Zookeeper TLS Properties Available via nifi.properties > --- > > Key: NIFI-7357 > URL: https://issues.apache.org/jira/browse/NIFI-7357 > Project: Apache NiFi > Issue Type: Sub-task > Components: Configuration, Configuration Management, Security >Reporter: Troy Melhase >Assignee: Troy Melhase >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7203) Add support for ZooKeeper TLS
[ https://issues.apache.org/jira/browse/NIFI-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7203. --- Fix Version/s: 1.13.0 Resolution: Fixed > Add support for ZooKeeper TLS > - > > Key: NIFI-7203 > URL: https://issues.apache.org/jira/browse/NIFI-7203 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Security >Reporter: Joey Frazee >Assignee: Joey Frazee >Priority: Major > Fix For: 1.13.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Currently the embedded ZooKeeper cannot be configured to use secure client > access or quorum TLS because the > [ZooKeeperStateServer|https://github.com/apache/nifi/blob/5f4ce8f431d1cb90451102efb516140d87610c52/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java#L41] > only sets a subset of properties, ignoring valid properties whether set in > zookeeper.properties or elsewhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7115) Add Zookeeper TLS configuration to Administration Guide
[ https://issues.apache.org/jira/browse/NIFI-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7115. --- Resolution: Duplicate > Add Zookeeper TLS configuration to Administration Guide > --- > > Key: NIFI-7115 > URL: https://issues.apache.org/jira/browse/NIFI-7115 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Joey Frazee >Assignee: Joey Frazee >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7924) Fallback claim(s) support in OIDC based authentication
[ https://issues.apache.org/jira/browse/NIFI-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee resolved NIFI-7924. --- Resolution: Fixed > Fallback claim(s) support in OIDC based authentication > -- > > Key: NIFI-7924 > URL: https://issues.apache.org/jira/browse/NIFI-7924 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.12.1 >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Minor > Fix For: 1.13.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently, 'nifi.security.user.oidc.claim.identifying.user' NiFi > configuration sets only one claim to bind ID token to username. There are > corner-case where fallback claim should search in case the configured claim > is not found in ID token. > For example, not all user directory objects has email address in Azure > Activity Directory > ([https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent#email]). > We need a fallback claim support so that when there is no email address > claim available for a user, the OIDC identity provider should pick up > fallback claim(s) for the user name. For other users with emails, it should > continue to use the configured claim to set user name. > > I will introduce 'nifi.security.user.oidc.fallback.claims.identifying.user' > in NiFi properties and implement the fallback logic . > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7924) Fallback claim(s) support in OIDC based authentication
[ https://issues.apache.org/jira/browse/NIFI-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-7924: -- Fix Version/s: 1.13.0 > Fallback claim(s) support in OIDC based authentication > -- > > Key: NIFI-7924 > URL: https://issues.apache.org/jira/browse/NIFI-7924 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.12.1 >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Minor > Fix For: 1.13.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently, 'nifi.security.user.oidc.claim.identifying.user' NiFi > configuration sets only one claim to bind ID token to username. There are > corner-case where fallback claim should search in case the configured claim > is not found in ID token. > For example, not all user directory objects has email address in Azure > Activity Directory > ([https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent#email]). > We need a fallback claim support so that when there is no email address > claim available for a user, the OIDC identity provider should pick up > fallback claim(s) for the user name. For other users with emails, it should > continue to use the configured claim to set user name. > > I will introduce 'nifi.security.user.oidc.fallback.claims.identifying.user' > in NiFi properties and implement the fallback logic . > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7964) PutAzureBlobStorage OutOfMemory Exception
[
https://issues.apache.org/jira/browse/NIFI-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Frazee updated NIFI-7964:
--
Priority: Blocker (was: Major)
> PutAzureBlobStorage OutOfMemory Exception
> -
>
> Key: NIFI-7964
> URL: https://issues.apache.org/jira/browse/NIFI-7964
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.1
>Reporter: Eric Secules
>Assignee: Joey Frazee
>Priority: Blocker
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of my flow I upload files to azure blob storage. They can be several
> hundred MB in size. I putting a 300 MB file into my flow and it choked on the
> PutAzureBlobStorage processor with the following log message.
> {code:java}
> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6]
> o.a.n.p.a.storage.PutAzureBlobStorage
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5]
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to
> process session due to java.lang.OutOfMemoryError: Java heap space; Processor
> Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap
> space
> java.lang.OutOfMemoryError: Java heap space
> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6]
> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to uncaught
> Exception: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> {code}
> I did not expect this to happen because I think the PutAzureBlob processor
> should be streaming the flowfile from disk directly to blob. But this
> behaviour suggests to me that it's getting read into memory in its entirety.
> My JVM heap size is set to 512 MB, which shouldn't be a problem if streaming
> was used to upload to blob storage in chunks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8097) Add support for connection strings to Azure credentials services
[ https://issues.apache.org/jira/browse/NIFI-8097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Frazee updated NIFI-8097: -- Description: The Azure credentials services currently require specifying accounts, keys, etc. as individual values but don't support using [connection strings|https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string] which would provide more flexibility. In order to support this the existing credentials services should be enhanced to support connection strings or a new controller service should be created for this. was: The Azure credentials services currently require specifying accounts, keys, etc. as individual values but don't support using [connection strings|[https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string]] which would provide more flexibility. In order to support this the existing credentials services should be enhanced to support connection strings or a new controller service should be created for this. > Add support for connection strings to Azure credentials services > > > Key: NIFI-8097 > URL: https://issues.apache.org/jira/browse/NIFI-8097 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Priority: Major > > The Azure credentials services currently require specifying accounts, keys, > etc. as individual values but don't support using [connection > strings|https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string] > which would provide more flexibility. > In order to support this the existing credentials services should be enhanced > to support connection strings or a new controller service should be created > for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
