[GitHub] [nifi] AnujJain7 commented on pull request #2724: NIFI-5133: Implemented Google Cloud PubSub Processors
AnujJain7 commented on pull request #2724:
URL: https://github.com/apache/nifi/pull/2724#issuecomment-631241948
We are facing issue in PublishGCPubSub operator , some threads are going in
hang state
checking on thread dump get following error for those thread
`{"stackTrace":"\"Timer-Driven Process Thread-50\" Id=8466 TIMED_WAITING on
null\n\tat java.lang.Thread.sleep(Native Method)\n\tat
org.apache.nifi.processors.gcp.pubsub.PublishGCPubSub.onTrigger(PublishGCPubSub.java:166)\n\tat
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)\n\tat
`
Looking into code of PublishGCPubSub.java at line number 166
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L166
there is use of Thread.sleep with condition that messageIdFuture.isDone,
seems like an issue
why we need to sleep the thread when the task is completed ?
_```
while (messageIdFuture.isDone()) {
Thread.sleep(500L);
}
```_
Looking forward to get response on this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] sjyang18 commented on a change in pull request #4265: NIFI-7434: Endpoint suffix property in AzureStorageAccount NIFI processors
sjyang18 commented on a change in pull request #4265:
URL: https://github.com/apache/nifi/pull/4265#discussion_r427714506
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java
##
@@ -85,6 +85,22 @@
.sensitive(true)
.build();
+public static final PropertyDescriptor ENDPOINT_SUFFIX = new
PropertyDescriptor.Builder()
+.name("storage-endpoint-suffix")
+.displayName("Storage Endpoint Suffix")
+.description(
+"Storage accounts in public Azure always use a common FQDN
suffix. " +
+"Override this endpoint suffix with a different suffix in
certain circumsances (like Azure Stack or non-public Azure regions). " +
+"The preferred way is to configure them through a
controller service specified in the Storage Credentials property. " +
+"The controller service can provide a common/shared
configuration for multiple/all Azure processors. Furthermore, the credentials "
+
+"can also be looked up dynamically with the 'Lookup'
version of the service.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
I opened a new PR for NIFI-7386.
https://github.com/apache/nifi/pull/4286.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] sjyang18 commented on pull request #4286: NIFI-7386: Azurite emulator support
sjyang18 commented on pull request #4286: URL: https://github.com/apache/nifi/pull/4286#issuecomment-631205319 @esecules, @jfrazee This is the approach I suggest we go to support Azurite emulator. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] sjyang18 opened a new pull request #4286: NIFI-7386: Azurite emulator support
sjyang18 opened a new pull request #4286:
URL: https://github.com/apache/nifi/pull/4286
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [X] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7463) Write empty flowfile for RunMongoAggregation empty results
[ https://issues.apache.org/jira/browse/NIFI-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduardo Mota Fontes updated NIFI-7463: -- Summary: Write empty flowfile for RunMongoAggregation empty results (was: Create empty relationship for RunMongoAggregation) > Write empty flowfile for RunMongoAggregation empty results > -- > > Key: NIFI-7463 > URL: https://issues.apache.org/jira/browse/NIFI-7463 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Eduardo Mota Fontes >Assignee: Eduardo Mota Fontes >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > In RunMongoAggregation processor, when the aggregation returns no value the > processor only returns the original flowfile in its relationship. In this > case we have to create a test auxiliary flow to handle each situation. This > improvement will save some processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] eduardofontes commented on a change in pull request #4281: NIFI-7463 - Write empty flowfile for RunMongoAggregation empty results
eduardofontes commented on a change in pull request #4281: URL: https://github.com/apache/nifi/pull/4281#discussion_r427710719 ## File path: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/RunMongoAggregation.java ## @@ -121,6 +126,7 @@ final Set _relationships = new HashSet<>(); _relationships.add(REL_RESULTS); +_relationships.add(REL_EMPTY); Review comment: Ok. I've changed to write an empty flowfile instead a new relationship. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-registry] apiri closed pull request #279: [NIFIREG-389] remove npx and use npm-force-resolutions
apiri closed pull request #279: URL: https://github.com/apache/nifi-registry/pull/279 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFIREG-389) libnpx not found
[ https://issues.apache.org/jira/browse/NIFIREG-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved NIFIREG-389. - Resolution: Fixed > libnpx not found > > > Key: NIFIREG-389 > URL: https://issues.apache.org/jira/browse/NIFIREG-389 > Project: NiFi Registry > Issue Type: Bug >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Users who do not have npx installed globally can run into a build issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-registry] apiri commented on pull request #279: [NIFIREG-389] remove npx and use npm-force-resolutions
apiri commented on pull request #279: URL: https://github.com/apache/nifi-registry/pull/279#issuecomment-631107081 Ran into some test issues (https://issues.apache.org/jira/browse/NIFIREG-391) but verified this corrected build issues existing on master that now work in my Ubuntu environment. Changes look good here! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFIREG-391) Master (2881e2) currently failing on tests
[ https://issues.apache.org/jira/browse/NIFIREG-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111571#comment-17111571 ] Aldrin Piri commented on NIFIREG-391: - Did not dig into the specific tests, but at a cursory glance appears it may be timestamp related. This was done on an Ubuntu based system. $ mvn -version Apache Maven 3.6.3 Maven home: /usr/share/maven Java version: 1.8.0_252, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.4.0-29-generic", arch: "amd64", family: "unix" > Master (2881e2) currently failing on tests > -- > > Key: NIFIREG-391 > URL: https://issues.apache.org/jira/browse/NIFIREG-391 > Project: NiFi Registry > Issue Type: Bug >Reporter: Aldrin Piri >Priority: Major > > Master (hash: 2881e29dce3a179f3e56069b82ef8cbb7bd8d85c) is currently failing > on ITs. > Failed tests: > BucketsIT.testCreateBucketGetBucket:148 expected:<1589924875387> but > was:<1589910475387> > BucketsIT.testDeleteBucket:268 expected:<1589924875360> but > was:<1589910475360> > BucketsIT.testUpdateBucket:199 expected:<1589924875324> but > was:<1589910475324> > FlowsIT.testCreateFlowGetFlow:157 expected:<1589924880021> but > was:<1589910480021> > FlowsIT.testCreateFlowVersionGetFlowVersion:466 expected:<1589924880090> > but was:<1589910480090> > FlowsIT.testDeleteFlow:292 expected:<1589924880124> but was:<1589910480124> > FlowsIT.testUpdateFlow:217 expected:<1589924879924> but was:<1589910479924> -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFIREG-391) Master (2881e2) currently failing on tests
Aldrin Piri created NIFIREG-391: --- Summary: Master (2881e2) currently failing on tests Key: NIFIREG-391 URL: https://issues.apache.org/jira/browse/NIFIREG-391 Project: NiFi Registry Issue Type: Bug Reporter: Aldrin Piri Master (hash: 2881e29dce3a179f3e56069b82ef8cbb7bd8d85c) is currently failing on ITs. Failed tests: BucketsIT.testCreateBucketGetBucket:148 expected:<1589924875387> but was:<1589910475387> BucketsIT.testDeleteBucket:268 expected:<1589924875360> but was:<1589910475360> BucketsIT.testUpdateBucket:199 expected:<1589924875324> but was:<1589910475324> FlowsIT.testCreateFlowGetFlow:157 expected:<1589924880021> but was:<1589910480021> FlowsIT.testCreateFlowVersionGetFlowVersion:466 expected:<1589924880090> but was:<1589910480090> FlowsIT.testDeleteFlow:292 expected:<1589924880124> but was:<1589910480124> FlowsIT.testUpdateFlow:217 expected:<1589924879924> but was:<1589910479924> -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-registry] apiri commented on pull request #279: [NIFIREG-389] remove npx and use npm-force-resolutions
apiri commented on pull request #279: URL: https://github.com/apache/nifi-registry/pull/279#issuecomment-631098719 reviewing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7468) Improve internal handling of SSL channels
[
https://issues.apache.org/jira/browse/NIFI-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111485#comment-17111485
]
ASF subversion and git services commented on NIFI-7468:
---
Commit 441781cec50f77d9f1e65093f55bbd614b8c5ec6 in nifi's branch
refs/heads/master from Andy LoPresto
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=441781c ]
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in
shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special
cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific
exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant
NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory
(Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to
nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining
SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all
references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket
listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values
explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and
StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't
support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes #4263.
Signed-off-by: Nathan Gough
Signed-off-by: Mark Payne
> Improve internal handling of SSL channels
> -
>
> Key: NIFI-7468
> URL: https://issues.apache.org/jira/browse/NIFI-7468
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: security, ssl, tcp, tls, tlsv1.3, udp
>
> While refactoring the TLS protocol version issue in NIFI-7407, I discovered
> that some processors make use of NiFi custom implementations of
> {{SSLSocketChannel}}, {{SSLCommsSession}}, and
> {{SSLSocketChannelInputStream}}. These implementations break on TLSv1.3.
> Further investigation is needed to determine why these custom implementations
> were provided originally, whether they are still required, and why they do
> not handle TLSv1.3 successfully.
> Diagnostic error:
> {code}
> Error reading from channel due to Tag mismatch!: javax.net.ssl.SSLException:
> Tag mismatch!
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7407) Change cluster communication protocol listener
[ https://issues.apache.org/jira/browse/NIFI-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7407: Resolution: Fixed Status: Resolved (was: Patch Available) > Change cluster communication protocol listener > -- > > Key: NIFI-7407 > URL: https://issues.apache.org/jira/browse/NIFI-7407 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > Labels: cluster, listener, protocol, socket > Fix For: 1.12.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Modify how the cluster communications for heartbeat, request replication, > site to site, and load balancing operate, specifically protocol listener and > socket listener. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4263: NIFI-7407 Refactored SSL context generation throughout framework and extensions.
asfgit closed pull request #4263: URL: https://github.com/apache/nifi/pull/4263 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7407) Change cluster communication protocol listener
[ https://issues.apache.org/jira/browse/NIFI-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111484#comment-17111484 ] ASF subversion and git services commented on NIFI-7407: --- Commit 441781cec50f77d9f1e65093f55bbd614b8c5ec6 in nifi's branch refs/heads/master from Andy LoPresto [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=441781c ] NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant). Changed JettyServer default SSL initialization and updated unit test. Removed SecurityStoreTypes (unused). Added StringUtils inverted blank and empty checks. Added TlsConfiguration container object. Enhanced KeystoreType enum. Added clean #createSSLContext() method to serve as base method for special cases/other method signatures. Added utility methods in KeyStoreUtils. Added generic TlsException for callers that cannot resolve TLS-specific exceptions. Added utility methods for component object debugging. Enforced TLS protocol version on cluster comms socket creation. Added utility method for SSL server socket creation. Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version). Removed duplicate SSLContextFactory. Switched duplicate SslContextFactory to wrap shared SSLContextFactory. Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test). Added further validation & boundary checking in uses of TlsConfiguration. Provided SSLSocketFactory accessor in SslContextFactory. Refactored OkHttpReplicationClient tuple method. Refactored OcspCertificateValidator TLS logic. Added utility method to apply TLS configs to OkHttpClientBuilder. Removed references to duplicate SslContextFactory. Removed unnecessary SslContextFactory. Moved OkHttpClientUtils to nifi-web-util module. Updated module dependencies. Removed now empty nifi-security module. Enforced TLS protocol selection on LB server socket. Enforced TLS protocol selection on S2S server socket. Applied specified TLS protocol versions to S2S socket creation. Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory. Replaced references to creation methods throughout codebase. Replaced references to unnecessary NiFiProperties file reads throughout tests. Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth. Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners. Cleaned up legacy code. Added external timing check to timing test assertion. Made RestrictedSSLContextService TLS protocol versions allowable values explicit. Enabled TLSv1.3 on Java 11. Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService. Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task. This closes #4263. Signed-off-by: Nathan Gough Signed-off-by: Mark Payne > Change cluster communication protocol listener > -- > > Key: NIFI-7407 > URL: https://issues.apache.org/jira/browse/NIFI-7407 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > Labels: cluster, listener, protocol, socket > Time Spent: 3h 20m > Remaining Estimate: 0h > > Modify how the cluster communications for heartbeat, request replication, > site to site, and load balancing operate, specifically protocol listener and > socket listener. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7407) Change cluster communication protocol listener
[ https://issues.apache.org/jira/browse/NIFI-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7407: Fix Version/s: 1.12.0 Status: Patch Available (was: Open) > Change cluster communication protocol listener > -- > > Key: NIFI-7407 > URL: https://issues.apache.org/jira/browse/NIFI-7407 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > Labels: cluster, listener, protocol, socket > Fix For: 1.12.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Modify how the cluster communications for heartbeat, request replication, > site to site, and load balancing operate, specifically protocol listener and > socket listener. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-registry] joewitt commented on pull request #280: NIFIREG-390: Add .asf.yaml file to GitHub repo
joewitt commented on pull request #280: URL: https://github.com/apache/nifi-registry/pull/280#issuecomment-631045493 +1 thanks mattyb. Feel free to merge from here This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] sjyang18 commented on a change in pull request #4265: NIFI-7434: Endpoint suffix property in AzureStorageAccount NIFI processors
sjyang18 commented on a change in pull request #4265:
URL: https://github.com/apache/nifi/pull/4265#discussion_r427556095
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java
##
@@ -85,6 +85,22 @@
.sensitive(true)
.build();
+public static final PropertyDescriptor ENDPOINT_SUFFIX = new
PropertyDescriptor.Builder()
+.name("storage-endpoint-suffix")
+.displayName("Storage Endpoint Suffix")
+.description(
+"Storage accounts in public Azure always use a common FQDN
suffix. " +
+"Override this endpoint suffix with a different suffix in
certain circumsances (like Azure Stack or non-public Azure regions). " +
+"The preferred way is to configure them through a
controller service specified in the Storage Credentials property. " +
+"The controller service can provide a common/shared
configuration for multiple/all Azure processors. Furthermore, the credentials "
+
+"can also be looked up dynamically with the 'Lookup'
version of the service.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
@esecules, @jfrazee , let me work on a patch for NIFI-7386 and let's
review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] jfrazee closed pull request #3906: NIFI-6911 Removed the default value for blob property for PutAzureBlobStorage
jfrazee closed pull request #3906: URL: https://github.com/apache/nifi/pull/3906 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-6911) PutAzureBlobStorage processor uses an undefined default value for 'Blob' property
[
https://issues.apache.org/jira/browse/NIFI-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111464#comment-17111464
]
ASF subversion and git services commented on NIFI-6911:
---
Commit 5826a09a8316b2e0fc57ed89646c05b1977ab297 in nifi's branch
refs/heads/master from Sushil Kumar
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5826a09 ]
NIFI-6911 Removed default Blob value for PutAzureBlobStorage
This closes #3906
Signed-off-by: Joey Frazee
> PutAzureBlobStorage processor uses an undefined default value for 'Blob'
> property
> -
>
> Key: NIFI-6911
> URL: https://issues.apache.org/jira/browse/NIFI-6911
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.10.0
>Reporter: Sushil Kumar
>Assignee: Sushil Kumar
>Priority: Minor
> Time Spent: 5h
> Remaining Estimate: 0h
>
> PutAzureBlobStorage uses an undefined default value ${azure.blobname} for
> Blob property.
> This processor error-out when started without updating the value of the
> 'Blob' property.
> And a user has to almost always update that value to something.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (NIFI-7386) AzureStorageCredentialsControllerService should also connect to storage emulator
[ https://issues.apache.org/jira/browse/NIFI-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seokwon Yang reassigned NIFI-7386: -- Assignee: Seokwon Yang > AzureStorageCredentialsControllerService should also connect to storage > emulator > > > Key: NIFI-7386 > URL: https://issues.apache.org/jira/browse/NIFI-7386 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Eric Secules >Assignee: Seokwon Yang >Priority: Major > > The controller service "AzureStorageCredentialsControllerService" should be > able to take an optional parameter to connect to another azure storage > provider, like the [Azurite > emulator|https://hub.docker.com/_/microsoft-azure-storage-azurite]. This will > likely mean taking additional parameters for a base URL and possibly > switching between http and https depending on Azurite's capabilities. > I am currently setting up an isolated test environment for my application and > the only piece that I cannot effectively isolate is our azure storage > connection because NiFi doesn't support connecting to anything but the > official azure storage service. > *Acceptance Criteria:* > * AzureStorageCredentialsControllerService can connect to an alternative > azure storage provider > * Individual azure storage processors can connect to an alternative azure > storage provider > * New parameters will be optional and default to existing behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-registry] mattyb149 opened a new pull request #280: NIFIREG-390: Add .asf.yaml file to GitHub repo
mattyb149 opened a new pull request #280: URL: https://github.com/apache/nifi-registry/pull/280 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFIREG-390) Add .asf.yaml file to GitHub repo
[ https://issues.apache.org/jira/browse/NIFIREG-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFIREG-390: Assignee: Matt Burgess > Add .asf.yaml file to GitHub repo > - > > Key: NIFIREG-390 > URL: https://issues.apache.org/jira/browse/NIFIREG-390 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > The ASF has a new way of configuring integrations and such for the Apache > GitHub repositories. It involves putting a .asf.yaml file at the root of the > project with various settings, such as email addresses for notifications, > which buttons/facets should be available on the UI, etc. > This case is to cover adding a .asf.yaml file to the nifi-registry repo. The > other repos will be covered in their respective Jira projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFIREG-390) Add .asf.yaml file to GitHub repo
Matt Burgess created NIFIREG-390: Summary: Add .asf.yaml file to GitHub repo Key: NIFIREG-390 URL: https://issues.apache.org/jira/browse/NIFIREG-390 Project: NiFi Registry Issue Type: Improvement Reporter: Matt Burgess The ASF has a new way of configuring integrations and such for the Apache GitHub repositories. It involves putting a .asf.yaml file at the root of the project with various settings, such as email addresses for notifications, which buttons/facets should be available on the UI, etc. This case is to cover adding a .asf.yaml file to the nifi-registry repo. The other repos will be covered in their respective Jira projects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pcgrenier commented on a change in pull request #4282: NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types
pcgrenier commented on a change in pull request #4282:
URL: https://github.com/apache/nifi/pull/4282#discussion_r427542274
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileTable.java
##
@@ -223,12 +225,69 @@ private RelDataType getRelDataType(final DataType
fieldType, final JavaTypeFacto
case BIGINT:
return typeFactory.createJavaType(BigInteger.class);
case CHOICE:
+final ChoiceDataType choiceDataType = (ChoiceDataType)
fieldType;
+DataType widestDataType =
choiceDataType.getPossibleSubTypes().get(0);
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (possibleType == widestDataType) {
+continue;
+}
+if
(possibleType.getFieldType().isWiderThan(widestDataType.getFieldType())) {
+widestDataType = possibleType;
+continue;
+}
+if
(widestDataType.getFieldType().isWiderThan(possibleType.getFieldType())) {
+continue;
+}
+
+// Neither is wider than the other.
+widestDataType = null;
+break;
+}
+
+// If one of the CHOICE data types is the widest, use it.
+if (widestDataType != null) {
+return getRelDataType(widestDataType, typeFactory);
+}
+
+// None of the data types is strictly the widest. Check if all
data types are numeric.
+// This would happen, for instance, if the data type is a
choice between float and integer.
+// If that is the case, we can use a String type for the table
schema because all values will fit
+// into a String. This will still allow for casting, etc. if
the query requires it.
+boolean allNumeric = true;
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (!isNumeric(possibleType)) {
+allNumeric = false;
+break;
+}
+}
+
+if (allNumeric) {
Review comment:
While I do agree, but given the example if you have split the records to
sort out all the "number" values, the schema and the data could disagree. So
logically you could know the data is all numbers but the schema disagrees. So
if we are hacking to a string anyway, I don't think it would hurt to just allow
it to pass on to let Calcite try and process it. This is just allowing a "best
case" situation where Calcite isn't passed something it has no idea what to do
with. So I do see a benefit in allowing the string to pass through, since if it
is not a number it will still fail at query time but will at least attempt it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] IlyaKovalev commented on pull request #4285: NIFI-7471 fix bug with property validation
IlyaKovalev commented on pull request #4285: URL: https://github.com/apache/nifi/pull/4285#issuecomment-631023807 In createStateProvider() ValidationContext always try to validate default values not actual. so if property is required and does not have default value error occurs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] IlyaKovalev opened a new pull request #4285: NIFI-7471 fix bug with property validation
IlyaKovalev opened a new pull request #4285:
URL: https://github.com/apache/nifi/pull/4285
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
fix bug NIFI-7471
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #4282: NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types
markap14 commented on a change in pull request #4282:
URL: https://github.com/apache/nifi/pull/4282#discussion_r427525012
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileTable.java
##
@@ -223,12 +225,69 @@ private RelDataType getRelDataType(final DataType
fieldType, final JavaTypeFacto
case BIGINT:
return typeFactory.createJavaType(BigInteger.class);
case CHOICE:
+final ChoiceDataType choiceDataType = (ChoiceDataType)
fieldType;
+DataType widestDataType =
choiceDataType.getPossibleSubTypes().get(0);
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (possibleType == widestDataType) {
+continue;
+}
+if
(possibleType.getFieldType().isWiderThan(widestDataType.getFieldType())) {
+widestDataType = possibleType;
+continue;
+}
+if
(widestDataType.getFieldType().isWiderThan(possibleType.getFieldType())) {
+continue;
+}
+
+// Neither is wider than the other.
+widestDataType = null;
+break;
+}
+
+// If one of the CHOICE data types is the widest, use it.
+if (widestDataType != null) {
+return getRelDataType(widestDataType, typeFactory);
+}
+
+// None of the data types is strictly the widest. Check if all
data types are numeric.
+// This would happen, for instance, if the data type is a
choice between float and integer.
+// If that is the case, we can use a String type for the table
schema because all values will fit
+// into a String. This will still allow for casting, etc. if
the query requires it.
+boolean allNumeric = true;
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (!isNumeric(possibleType)) {
+allNumeric = false;
+break;
+}
+}
+
+if (allNumeric) {
Review comment:
As an example, consider a csv like:
```
name, other
markap14, 48
pcgrenier, computer
```
In this case, the schema has a field with name 'name' and a type String. But
'other' field is a CHOICE[INT, STRING]. So what should best represent that
field in terms of java objects? I'd say `Object.class`. The `String.class`
being used here is honestly a bit of a hack because Calcite doesn't give us a
better way to represent Number - or, at least, not to my knowledge :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7471) impossible run nifi with redis cluster state manager
[
https://issues.apache.org/jira/browse/NIFI-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Kovalev updated NIFI-7471:
---
Description:
below error occurs due to start nifi with redis cluster state manager
{code:java}
Caused by: java.lang.IllegalStateException: Could not initialize State
Providers because the Cluster State Provider is not valid. The following 1
Valid ation Errors occurred:'Connection String' is invalid because
Connection String is required Please check the configuration of the Cluster
State Provider with ID [redis-provider] in the file
/opt/nifi/nifi-current/./conf/state-management.xml
{code}
attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
{color:#1d1c1d}How to recreate{color}
{color:#1d1c1d}1 docker-compose up containers{color}
{color:#1d1c1d}2 copy configurations into nifi node{color}
{color:#1d1c1d}3 restart node{color}
was:
below error occurs due to start nifi with redis cluster state manager
{code:java}
Caused by: java.lang.IllegalStateException: Could not initialize State
Providers because the Cluster State Provider is not valid. The following 1
Valid ation Errors occurred:'Connection String' is invalid because
Connection String is required Please check the configuration of the Cluster
State Provider with ID [redis-provider] in the file
/opt/nifi/nifi-current/./conf/state-management.xml
{code}
attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
> impossible run nifi with redis cluster state manager
>
>
> Key: NIFI-7471
> URL: https://issues.apache.org/jira/browse/NIFI-7471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Ilya Kovalev
>Priority: Critical
> Attachments: docker-compose.yml, nifi.properties, state-management.xml
>
>
> below error occurs due to start nifi with redis cluster state manager
> {code:java}
> Caused by: java.lang.IllegalStateException: Could not initialize State
> Providers because the Cluster State Provider is not valid. The following 1
> Valid ation Errors occurred:'Connection String' is invalid
> because Connection String is required Please check the configuration of the
> Cluster State Provider with ID [redis-provider] in the file
> /opt/nifi/nifi-current/./conf/state-management.xml
> {code}
> attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
> {color:#1d1c1d}How to recreate{color}
> {color:#1d1c1d}1 docker-compose up containers{color}
> {color:#1d1c1d}2 copy configurations into nifi node{color}
> {color:#1d1c1d}3 restart node{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7471) impossible run nifi with redis cluster state manager
[
https://issues.apache.org/jira/browse/NIFI-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Kovalev updated NIFI-7471:
---
Attachment: nifi.properties
docker-compose.yml
> impossible run nifi with redis cluster state manager
>
>
> Key: NIFI-7471
> URL: https://issues.apache.org/jira/browse/NIFI-7471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Ilya Kovalev
>Priority: Critical
> Attachments: docker-compose.yml, nifi.properties, state-management.xml
>
>
> below error occurs due to start nifi with redis cluster state manager
> {code:java}
> Caused by: java.lang.IllegalStateException: Could not initialize State
> Providers because the Cluster State Provider is not valid. The following 1
> Valid ation Errors occurred:'Connection String' is invalid
> because Connection String is required Please check the configuration of the
> Cluster State Provider with ID [redis-provider] in the file
> /opt/nifi/nifi-current/./conf/state-management.xml
> {code}
> attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-7471) impossible run nifi with redis cluster state manager
Ilya Kovalev created NIFI-7471:
--
Summary: impossible run nifi with redis cluster state manager
Key: NIFI-7471
URL: https://issues.apache.org/jira/browse/NIFI-7471
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Ilya Kovalev
Attachments: state-management.xml
below error occurs due to start nifi with redis cluster state manager
{code:java}
Caused by: java.lang.IllegalStateException: Could not initialize State
Providers because the Cluster State Provider is not valid. The following 1
Valid ation Errors occurred:'Connection String' is invalid because
Connection String is required Please check the configuration of the Cluster
State Provider with ID [redis-provider] in the file
/opt/nifi/nifi-current/./conf/state-management.xml
{code}
attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7471) impossible run nifi with redis cluster state manager
[
https://issues.apache.org/jira/browse/NIFI-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Kovalev updated NIFI-7471:
---
Attachment: state-management.xml
> impossible run nifi with redis cluster state manager
>
>
> Key: NIFI-7471
> URL: https://issues.apache.org/jira/browse/NIFI-7471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Ilya Kovalev
>Priority: Critical
> Attachments: state-management.xml
>
>
> below error occurs due to start nifi with redis cluster state manager
> {code:java}
> Caused by: java.lang.IllegalStateException: Could not initialize State
> Providers because the Cluster State Provider is not valid. The following 1
> Valid ation Errors occurred:'Connection String' is invalid
> because Connection String is required Please check the configuration of the
> Cluster State Provider with ID [redis-provider] in the file
> /opt/nifi/nifi-current/./conf/state-management.xml
> {code}
> attach my {color:#1d1c1d}state-management.xml and nifi.properties{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] markap14 commented on a change in pull request #4282: NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types
markap14 commented on a change in pull request #4282:
URL: https://github.com/apache/nifi/pull/4282#discussion_r427522041
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileTable.java
##
@@ -223,12 +225,69 @@ private RelDataType getRelDataType(final DataType
fieldType, final JavaTypeFacto
case BIGINT:
return typeFactory.createJavaType(BigInteger.class);
case CHOICE:
+final ChoiceDataType choiceDataType = (ChoiceDataType)
fieldType;
+DataType widestDataType =
choiceDataType.getPossibleSubTypes().get(0);
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (possibleType == widestDataType) {
+continue;
+}
+if
(possibleType.getFieldType().isWiderThan(widestDataType.getFieldType())) {
+widestDataType = possibleType;
+continue;
+}
+if
(widestDataType.getFieldType().isWiderThan(possibleType.getFieldType())) {
+continue;
+}
+
+// Neither is wider than the other.
+widestDataType = null;
+break;
+}
+
+// If one of the CHOICE data types is the widest, use it.
+if (widestDataType != null) {
+return getRelDataType(widestDataType, typeFactory);
+}
+
+// None of the data types is strictly the widest. Check if all
data types are numeric.
+// This would happen, for instance, if the data type is a
choice between float and integer.
+// If that is the case, we can use a String type for the table
schema because all values will fit
+// into a String. This will still allow for casting, etc. if
the query requires it.
+boolean allNumeric = true;
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (!isNumeric(possibleType)) {
+allNumeric = false;
+break;
+}
+}
+
+if (allNumeric) {
Review comment:
I don't think we want to combine String types here. The idea here is to
have a "parent" type that can be used for all numbers. Unfortunately, if the
class is `Number.class` calcite does not recognize this, so it translates it
into a SQL "OTHER" type. That prevents us still from doing things like SUM() on
that column. By using a String type here, we can use SUM and other functions
because all numbers can be converted to Strings and those particular Strings
can also be converted to numbers under the hood.
However, if we allowed STRINGs to be included, it means we could have a
schema whose field is a CHOICE between STRING and INT. It doesn't really make
sense to perform aggregate functions such as SUM() across those values.
Or, said another way, String is the internal Java representation that we
need to use in order for Calcite to allow both INT and FLOAT to be okay in the
same column. But if our schema says that a value should be a CHOICE of STRING
or INT, we shouldn't allow things like SUM() over that because you can't sum
together STRINGs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-registry] thenatog edited a comment on pull request #276: NIFIREG-385 Make revision feature configurable
thenatog edited a comment on pull request #276: URL: https://github.com/apache/nifi-registry/pull/276#issuecomment-631004233 Looks good to me, thanks Bryan! +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-registry] thenatog commented on pull request #276: NIFIREG-385 Make revision feature configurable
thenatog commented on pull request #276: URL: https://github.com/apache/nifi-registry/pull/276#issuecomment-631004233 Looks good to be, thanks Bryan! +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4263: NIFI-7407 Refactored SSL context generation throughout framework and extensions.
thenatog commented on pull request #4263: URL: https://github.com/apache/nifi/pull/4263#issuecomment-630997583 +1, looks good. Thanks Andy! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (MINIFICPP-889) Create MiNiFI NAR equivalent
[ https://issues.apache.org/jira/browse/MINIFICPP-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111364#comment-17111364 ] Marc Parisi commented on MINIFICPP-889: --- For reference and openness: [https://github.com/apache/nifi-registry/pull/191/files] In retrospect that can be made simpler, but the idea was to use the current binary with minor changes to support that PR and that feature set. Happy to answer any questions. I'll keep watch of this closely today if there are any additional questions. > Create MiNiFI NAR equivalent > - > > Key: MINIFICPP-889 > URL: https://issues.apache.org/jira/browse/MINIFICPP-889 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Marc Parisi >Priority: Major > Fix For: 0.8.0 > > > Create MiNiFI NAR equivalent so that we can integrate with Apache NiFi > registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (MINIFICPP-889) Create MiNiFI NAR equivalent
[ https://issues.apache.org/jira/browse/MINIFICPP-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111362#comment-17111362 ] Marc Parisi edited comment on MINIFICPP-889 at 5/19/20, 5:10 PM: - Hi [~pvillard] The last update to this branch was about a year ago. I don't believe it was defined as a set of breaking changes. as it was expecting to leverage a PR on Registry + this change set. Frankly it was experimental ( [https://github.com/phrocker/nifi-minifi-cpp/compare/master...phrocker:bootstrap] ). I built the changeset and tested it with Apache NiFi MiNiFi + Registry changeset and it worked as expected. With that said, I do think a 1.0 is a better place for this when considering all objectives. >From an API perspective it can probably fit in a 0.x as nothing *should* change, but there are a lot of maintenance/improvements that can be made and we've been towing the 0.x line a while. Adding more to it just means it last longer. I more than anyone would like 1.x to come as soon as possible so changes can happen. I think converting this to an Epic and defining tractable user stories is spot on. I would love to partake, but the last two months I've had to clear my plate for a hot button issue and won't be able to return to something like this until mid June. I've had to admit defeat until then :( I think [~aboda] , [~bakaid] , and [~szaszm] have been doing fantastic work and have the insight to be able to do the work if they have time. Happy to help in ideas ( bad and good :) ), until then. I think fresh eyes on this would also Thanks for uncovering this . I think this would be super useful. was (Author: phrocker): Hi [~pvillard] The last update to this branch was about a year ago. I don't believe it was defined as a set of breaking changes. as it was expecting to leverage a PR on Registry + this change set. Frankly it was experimental ( [https://github.com/phrocker/nifi-minifi-cpp/compare/master...phrocker:bootstrap] ). I built the changeset and tested it with Apache NiFi MiNiFi + Registry changeset and it worked as expected. With that said, I do think a 1.0 is a better place for this when considering all objectives. >From an API perspective it can probably fit in a 0.x as nothing *should* change, but there are a lot of maintenance/improvements that can be made and we've been towing the 0.x line a while. Adding more to it just means it last longer. I more than anyone would like 1.x to come as soon as possible. I think converting this to an Epic and defining tractable user stories is spot on. I would love to partake, but the last two months I've had to clear my plate for a hot button issue and won't be able to return to something like this until mid June. I've had to admit defeat until then :( I think [~aboda] , [~bakaid] , and [~szaszm] have been doing fantastic work and have the insight to be able to do the work if they have time. Happy to help in ideas ( bad and good :) ), until then. I think fresh eyes on this would also Thanks for uncovering this . I think this would be super useful. > Create MiNiFI NAR equivalent > - > > Key: MINIFICPP-889 > URL: https://issues.apache.org/jira/browse/MINIFICPP-889 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Marc Parisi >Priority: Major > Fix For: 0.8.0 > > > Create MiNiFI NAR equivalent so that we can integrate with Apache NiFi > registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFICPP-889) Create MiNiFI NAR equivalent
[ https://issues.apache.org/jira/browse/MINIFICPP-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111362#comment-17111362 ] Marc Parisi commented on MINIFICPP-889: --- Hi [~pvillard] The last update to this branch was about a year ago. I don't believe it was defined as a set of breaking changes. as it was expecting to leverage a PR on Registry + this change set. Frankly it was experimental ( [https://github.com/phrocker/nifi-minifi-cpp/compare/master...phrocker:bootstrap] ). I built the changeset and tested it with Apache NiFi MiNiFi + Registry changeset and it worked as expected. With that said, I do think a 1.0 is a better place for this when considering all objectives. >From an API perspective it can probably fit in a 0.x as nothing *should* change, but there are a lot of maintenance/improvements that can be made and we've been towing the 0.x line a while. Adding more to it just means it last longer. I more than anyone would like 1.x to come as soon as possible. I think converting this to an Epic and defining tractable user stories is spot on. I would love to partake, but the last two months I've had to clear my plate for a hot button issue and won't be able to return to something like this until mid June. I've had to admit defeat until then :( I think [~aboda] , [~bakaid] , and [~szaszm] have been doing fantastic work and have the insight to be able to do the work if they have time. Happy to help in ideas ( bad and good :) ), until then. I think fresh eyes on this would also Thanks for uncovering this . I think this would be super useful. > Create MiNiFI NAR equivalent > - > > Key: MINIFICPP-889 > URL: https://issues.apache.org/jira/browse/MINIFICPP-889 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Marc Parisi >Priority: Major > Fix For: 0.8.0 > > > Create MiNiFI NAR equivalent so that we can integrate with Apache NiFi > registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFICPP-889) Create MiNiFI NAR equivalent
[ https://issues.apache.org/jira/browse/MINIFICPP-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marc Parisi reassigned MINIFICPP-889: - Assignee: (was: Marc Parisi) > Create MiNiFI NAR equivalent > - > > Key: MINIFICPP-889 > URL: https://issues.apache.org/jira/browse/MINIFICPP-889 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Marc Parisi >Priority: Major > Fix For: 0.8.0 > > > Create MiNiFI NAR equivalent so that we can integrate with Apache NiFi > registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7336) Add tests for DeleteAzureDataLakeStorage
[ https://issues.apache.org/jira/browse/NIFI-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-7336: -- Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add tests for DeleteAzureDataLakeStorage > > > Key: NIFI-7336 > URL: https://issues.apache.org/jira/browse/NIFI-7336 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Gyori >Priority: Major > Labels: azure > Fix For: 1.12.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7336) Add tests for DeleteAzureDataLakeStorage
[ https://issues.apache.org/jira/browse/NIFI-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111335#comment-17111335 ] ASF subversion and git services commented on NIFI-7336: --- Commit d6240a1074aa33d650cb91dec3e69b7d5e3da9b5 in nifi's branch refs/heads/master from Peter Gyori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d6240a1 ] NIFI-7336: Add tests for DeleteAzureDataLakeStorage DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case This closes #4272. Signed-off-by: Peter Turcsanyi > Add tests for DeleteAzureDataLakeStorage > > > Key: NIFI-7336 > URL: https://issues.apache.org/jira/browse/NIFI-7336 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Gyori >Priority: Major > Labels: azure > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7336) Add tests for DeleteAzureDataLakeStorage
[ https://issues.apache.org/jira/browse/NIFI-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111336#comment-17111336 ] ASF subversion and git services commented on NIFI-7336: --- Commit d6240a1074aa33d650cb91dec3e69b7d5e3da9b5 in nifi's branch refs/heads/master from Peter Gyori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d6240a1 ] NIFI-7336: Add tests for DeleteAzureDataLakeStorage DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case This closes #4272. Signed-off-by: Peter Turcsanyi > Add tests for DeleteAzureDataLakeStorage > > > Key: NIFI-7336 > URL: https://issues.apache.org/jira/browse/NIFI-7336 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Gyori >Priority: Major > Labels: azure > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7336) Add tests for DeleteAzureDataLakeStorage
[ https://issues.apache.org/jira/browse/NIFI-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111334#comment-17111334 ] ASF subversion and git services commented on NIFI-7336: --- Commit d6240a1074aa33d650cb91dec3e69b7d5e3da9b5 in nifi's branch refs/heads/master from Peter Gyori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d6240a1 ] NIFI-7336: Add tests for DeleteAzureDataLakeStorage DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case This closes #4272. Signed-off-by: Peter Turcsanyi > Add tests for DeleteAzureDataLakeStorage > > > Key: NIFI-7336 > URL: https://issues.apache.org/jira/browse/NIFI-7336 > Project: Apache NiFi > Issue Type: Test > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Gyori >Priority: Major > Labels: azure > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4272: NIFI-7336: Add tests for DeleteAzureDataLakeStorage
asfgit closed pull request #4272: URL: https://github.com/apache/nifi/pull/4272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] esecules commented on a change in pull request #4281: NIFI-7463 - Create empty relationship for RunMongoAggregation
esecules commented on a change in pull request #4281: URL: https://github.com/apache/nifi/pull/4281#discussion_r427438496 ## File path: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/RunMongoAggregation.java ## @@ -121,6 +126,7 @@ final Set _relationships = new HashSet<>(); _relationships.add(REL_RESULTS); +_relationships.add(REL_EMPTY); Review comment: Could you instead add a property like GetMongo has "Send Empty Result"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] esecules commented on a change in pull request #4281: NIFI-7463 - Create empty relationship for RunMongoAggregation
esecules commented on a change in pull request #4281: URL: https://github.com/apache/nifi/pull/4281#discussion_r427439135 ## File path: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/RunMongoAggregation.java ## @@ -121,6 +126,7 @@ final Set _relationships = new HashSet<>(); _relationships.add(REL_RESULTS); +_relationships.add(REL_EMPTY); Review comment: In the GetMongo processor there's an option to "Send Empty Result" instead of creating another relationship, what about adding this option instead for symmetry. This option sends an empty JSON object out the success relationship if there's no result. I had the same issue with GetMongo and the "Send Empty Result" fixed my issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] esecules removed a comment on pull request #4281: NIFI-7463 - Create empty relationship for RunMongoAggregation
esecules removed a comment on pull request #4281: URL: https://github.com/apache/nifi/pull/4281#issuecomment-630932760 In the GetMongo processor there's an option to "Send Empty Result" instead of creating another relationship, what about adding this option instead for symmetry. This option sends an empty JSON object out the success relationship if there's no result. I had the same issue with GetMongo and the "Send Empty Result" fixed my issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] esecules commented on pull request #4281: NIFI-7463 - Create empty relationship for RunMongoAggregation
esecules commented on pull request #4281: URL: https://github.com/apache/nifi/pull/4281#issuecomment-630932760 In the GetMongo processor there's an option to "Send Empty Result" instead of creating another relationship, what about adding this option instead for symmetry. This option sends an empty JSON object out the success relationship if there's no result. I had the same issue with GetMongo and the "Send Empty Result" fixed my issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] esecules commented on a change in pull request #4265: NIFI-7434: Endpoint suffix property in AzureStorageAccount NIFI processors
esecules commented on a change in pull request #4265:
URL: https://github.com/apache/nifi/pull/4265#discussion_r427431756
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java
##
@@ -85,6 +85,22 @@
.sensitive(true)
.build();
+public static final PropertyDescriptor ENDPOINT_SUFFIX = new
PropertyDescriptor.Builder()
+.name("storage-endpoint-suffix")
+.displayName("Storage Endpoint Suffix")
+.description(
+"Storage accounts in public Azure always use a common FQDN
suffix. " +
+"Override this endpoint suffix with a different suffix in
certain circumsances (like Azure Stack or non-public Azure regions). " +
+"The preferred way is to configure them through a
controller service specified in the Storage Credentials property. " +
+"The controller service can provide a common/shared
configuration for multiple/all Azure processors. Furthermore, the credentials "
+
+"can also be looked up dynamically with the 'Lookup'
version of the service.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
@sjyang18 I would love to have NIFI-7386 addressed here but I will defer
to someone affiliated with Apache ( @jfrazee ) as I am the user who logged the
bug in the first place.
I wonder, is there a finite list of possible endpoints? If so the endpoint
suffix could maybe be a dropdown list. There'd still need to be a text field
for the storage emulator though so maybe it'll not be intuitive.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #786: MINIFICPP-1225 - Fix flaky HTTP tests.
arpadboda closed pull request #786: URL: https://github.com/apache/nifi-minifi-cpp/pull/786 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on pull request #786: MINIFICPP-1225 - Fix flaky HTTP tests.
arpadboda commented on pull request #786: URL: https://github.com/apache/nifi-minifi-cpp/pull/786#issuecomment-630924894 CI jobs green, executed the tests parallel on my server 3 times, still all green. Nice work! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7454) Directory handle leak in ListFile
[ https://issues.apache.org/jira/browse/NIFI-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-7454: - Fix Version/s: 1.12.0 > Directory handle leak in ListFile > - > > Key: NIFI-7454 > URL: https://issues.apache.org/jira/browse/NIFI-7454 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 >Reporter: Paul Kelly >Assignee: Simon Bence >Priority: Major > Fix For: 1.12.0 > > Attachments: flow.xml.gz > > > ListFile does not seem to be closing directory handles if an exception is > raised during the directory walk. For example, if ListFile is recursively > polling a folder and the NiFi user does not have permission to list the > contents of one of the subfolders, an "Access Denied" bulletin is raised and > the number of DIR handles opened by the NiFi process as shown by lsof > increases. Eventually, NiFi stops working due to "Too many open files". > This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11. I'm not > sure in which version this behavior started. > Attached is a sample flow.xml.gz that demonstrates this behavior. A clean > copy of NiFi 1.11.4 was installed with no modifications to any configuration > file. NiFi is running as the user "xubuntu". The permissions on the folders > are as follows: > drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test > drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest > NiFi has permission to list the contents of "/home/xubuntu/test", but not > "/home/xubuntu/test/subtest". > The number of open DIR handles is being monitored with the command: > lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c > where 347813 is the pid of the NiFi process. > DIR handle counts start at 4 immediately after NiFi is launched and steadily > grow to tens of thousands, until file handles are exhausted. Additionally, > these handles do not seem to be released even if the ListFile processor is > stopped. > Please let me know if there is any additional information I can provide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pgyori commented on a change in pull request #4272: NIFI-7336: Add tests for DeleteAzureDataLakeStorage
pgyori commented on a change in pull request #4272:
URL: https://github.com/apache/nifi/pull/4272#discussion_r427399115
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java
##
@@ -16,44 +16,388 @@
*/
package org.apache.nifi.processors.azure.storage;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.provenance.ProvenanceEventType;
import org.apache.nifi.util.MockFlowFile;
-import org.junit.Before;
-import org.junit.Ignore;
import org.junit.Test;
-import java.util.List;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
-public class ITDeleteAzureDataLakeStorage extends AbstractAzureBlobStorageIT {
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class ITDeleteAzureDataLakeStorage extends
AbstractAzureDataLakeStorageIT {
@Override
protected Class getProcessorClass() {
return DeleteAzureDataLakeStorage.class;
}
-@Before
-public void setUp() {
-runner.setProperty(DeleteAzureDataLakeStorage.FILE, TEST_FILE_NAME);
+@Test
+public void testDeleteFileFromRoot() {
+// GIVEN
+String directory= "";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+uploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileFromDirectory() {
+// GIVEN
+String directory = "TestDirectory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileFromDeepDirectory() {
+// GIVEN
+String directory=
"Directory01/Directory02/Directory03/Directory04/Directory05/Directory06/Directory07/"
++
"Directory08/Directory09/Directory10/Directory11/Directory12/Directory13/Directory14/Directory15/"
++
"Directory16/Directory17/Directory18/Directory19/Directory20/TestDirectory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileWithWhitespaceInFilename() {
+// GIVEN
+String directory= "TestDirectory";
+String filename = "A test file.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileWithWhitespaceInDirectoryName() {
+// GIVEN
+String directory= "A Test Directory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteEmptyDirectory() {
+// GIVEN
+String parentDirectory = "ParentDirectory";
+String childDirectory = "ChildDirectory";
+String inputFlowFileContent = "InputFlowFileContent";
+
+fileSystemClient.createDirectory(parentDirectory + "/" +
childDirectory);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, parentDirectory, childDirectory,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileCaseSensitiveFilename() {
Review comment:
Thank you for the info! I rebased the branch, and
[jira] [Commented] (MINIFICPP-889) Create MiNiFI NAR equivalent
[ https://issues.apache.org/jira/browse/MINIFICPP-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111246#comment-17111246 ] Pierre Villard commented on MINIFICPP-889: -- Hey [~phrocker] - is it still something you're looking at? I think it'd make sense to convert this into an epic, split it in multiple subtasks and have this targeting a 1.0.0 version of the agent (as it'd probably need breaking changes). Thoughts? > Create MiNiFI NAR equivalent > - > > Key: MINIFICPP-889 > URL: https://issues.apache.org/jira/browse/MINIFICPP-889 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > Fix For: 0.8.0 > > > Create MiNiFI NAR equivalent so that we can integrate with Apache NiFi > registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7454) Directory handle leak in ListFile
[ https://issues.apache.org/jira/browse/NIFI-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111212#comment-17111212 ] Paul Kelly commented on NIFI-7454: -- Thanks for looking into this and the information, [~simonbence]. That's great that it will be fixed in the next version. > Directory handle leak in ListFile > - > > Key: NIFI-7454 > URL: https://issues.apache.org/jira/browse/NIFI-7454 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.4 >Reporter: Paul Kelly >Assignee: Simon Bence >Priority: Major > Attachments: flow.xml.gz > > > ListFile does not seem to be closing directory handles if an exception is > raised during the directory walk. For example, if ListFile is recursively > polling a folder and the NiFi user does not have permission to list the > contents of one of the subfolders, an "Access Denied" bulletin is raised and > the number of DIR handles opened by the NiFi process as shown by lsof > increases. Eventually, NiFi stops working due to "Too many open files". > This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11. I'm not > sure in which version this behavior started. > Attached is a sample flow.xml.gz that demonstrates this behavior. A clean > copy of NiFi 1.11.4 was installed with no modifications to any configuration > file. NiFi is running as the user "xubuntu". The permissions on the folders > are as follows: > drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test > drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest > NiFi has permission to list the contents of "/home/xubuntu/test", but not > "/home/xubuntu/test/subtest". > The number of open DIR handles is being monitored with the command: > lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c > where 347813 is the pid of the NiFi process. > DIR handle counts start at 4 immediately after NiFi is launched and steadily > grow to tens of thousands, until file handles are exhausted. Additionally, > these handles do not seem to be released even if the ListFile processor is > stopped. > Please let me know if there is any additional information I can provide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7148) Invalid config event received - zookeeper
[
https://issues.apache.org/jira/browse/NIFI-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1784#comment-1784
]
dan young commented on NIFI-7148:
-
Hello Joshua,
We upgraded to Nifi 1.11.3, and also changed the server.X config in the zoo.cfg
to this:
server.1= xx.xxx.x.xx:2888:3888;2181
server.2= xx.xxx.x.xx:2888:3888;2181
server.3= xx.xxx.x.xx:2888:3888;2181
That seemed to be the winning ticket for us.
> Invalid config event received - zookeeper
> --
>
> Key: NIFI-7148
> URL: https://issues.apache.org/jira/browse/NIFI-7148
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.11.1
>Reporter: dan young
>Priority: Minor
>
> We're seeing the following messages in nifi logs on our cluster nodes. We
> have 3 nodes running Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
> Functionality seems not to be impacted, but wondering if there's something
> else
> going on or the version of zookeeper we're using is causing this.
> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
> {server.1=10.190.3.170:2888:3888:participant, version=0,
> server.3=10.190.3.91:2888:3888:participant,
> server.2=10.190.3.172:2888:3888:participant}
> Regards,
> Dano
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] pcgrenier commented on pull request #4280: NIFI-7462: QueryRecord cast functions for Choice datatypes
pcgrenier commented on pull request #4280: URL: https://github.com/apache/nifi/pull/4280#issuecomment-630830638 Closing in favor of #4282 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pcgrenier closed pull request #4280: NIFI-7462: QueryRecord cast functions for Choice datatypes
pcgrenier closed pull request #4280: URL: https://github.com/apache/nifi/pull/4280 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pcgrenier commented on a change in pull request #4282: NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types
pcgrenier commented on a change in pull request #4282:
URL: https://github.com/apache/nifi/pull/4282#discussion_r427311324
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileTable.java
##
@@ -223,12 +225,69 @@ private RelDataType getRelDataType(final DataType
fieldType, final JavaTypeFacto
case BIGINT:
return typeFactory.createJavaType(BigInteger.class);
case CHOICE:
+final ChoiceDataType choiceDataType = (ChoiceDataType)
fieldType;
+DataType widestDataType =
choiceDataType.getPossibleSubTypes().get(0);
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (possibleType == widestDataType) {
+continue;
+}
+if
(possibleType.getFieldType().isWiderThan(widestDataType.getFieldType())) {
+widestDataType = possibleType;
+continue;
+}
+if
(widestDataType.getFieldType().isWiderThan(possibleType.getFieldType())) {
+continue;
+}
+
+// Neither is wider than the other.
+widestDataType = null;
+break;
+}
+
+// If one of the CHOICE data types is the widest, use it.
+if (widestDataType != null) {
+return getRelDataType(widestDataType, typeFactory);
+}
+
+// None of the data types is strictly the widest. Check if all
data types are numeric.
+// This would happen, for instance, if the data type is a
choice between float and integer.
+// If that is the case, we can use a String type for the table
schema because all values will fit
+// into a String. This will still allow for casting, etc. if
the query requires it.
+boolean allNumeric = true;
+for (final DataType possibleType :
choiceDataType.getPossibleSubTypes()) {
+if (!isNumeric(possibleType)) {
+allNumeric = false;
+break;
+}
+}
+
+if (allNumeric) {
Review comment:
Maybe allow String types too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mabr3 opened a new pull request #4284: [NIFI-7470] - ElasticsearchHttp malformed query when using Fields
mabr3 opened a new pull request #4284: URL: https://github.com/apache/nifi/pull/4284 Change "_source_include" to "_source_includes" in response to [NIFI-7470], so that queries work. Enables using the Fields functionality in elasticsearch queries. fixes bug NIFI-7470. Adds an 's' character to the String FIELD_INCLUDE_QUERY_PARAM so that it won't break the queries when using Fields in ElasticSearch HTTP processors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-7470) QueryElasticsearchHttp malformed query when using Fields
Miguel Rodrigues created NIFI-7470:
--
Summary: QueryElasticsearchHttp malformed query when using Fields
Key: NIFI-7470
URL: https://issues.apache.org/jira/browse/NIFI-7470
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.11.4
Reporter: Miguel Rodrigues
Fix For: 1.12.0
The query used in this processor is malformed when using the Fields section, as
it writes _source_include=xxx instead of _source_includes=xxx. The missing 's'
at the end makes the query fail.
E.g.:
{code:java}
"request":"GET
/files-metadata-2020/_search?q=state%3A%22started%22=200=0&_source_include=ip%2Cpath%2Csize
HTTP/1.1"
{code}
and it should be:
{code:java}
"request":"GET
/files-metadata-2020/_search?q=state%3A%22started%22=200=0&_source_includes=ip%2Cpath%2Csize
HTTP/1.1"
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7454) Directory handle leak in ListFile
[
https://issues.apache.org/jira/browse/NIFI-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Bence resolved NIFI-7454.
---
Resolution: Duplicate
*Summary:*
This issue is fixed by [#NIFI-7292]. There is an ongoing discussion about a
1.12.0 release or a 1.11.5 release to happen soon.
*In details:*
I could reproduce this easily (thank you for the flow!) using MacOS. It turned
out that the stream usage keeps a reference alive after the exception (based on
what I did see, most probably it's a _sun.nio.fs.UnixPath_, but I did not
follow up with this). The fix with the current version of _ListFile_ looks
fairly simple:
{code:java}
finally {
performanceTracker.completeActiveDirectory();
listing.close();
}
{code}
If the listing is closed
[here|https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java#L579]
the handler will be released. This issue does not effect the master branch
however, as the listing of the files was replaced but a different
implementation, not using the stream in this way. It is also tested that, with
the current head, no similar issue appears.
> Directory handle leak in ListFile
> -
>
> Key: NIFI-7454
> URL: https://issues.apache.org/jira/browse/NIFI-7454
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.11.4
>Reporter: Paul Kelly
>Assignee: Simon Bence
>Priority: Major
> Attachments: flow.xml.gz
>
>
> ListFile does not seem to be closing directory handles if an exception is
> raised during the directory walk. For example, if ListFile is recursively
> polling a folder and the NiFi user does not have permission to list the
> contents of one of the subfolders, an "Access Denied" bulletin is raised and
> the number of DIR handles opened by the NiFi process as shown by lsof
> increases. Eventually, NiFi stops working due to "Too many open files".
> This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11. I'm not
> sure in which version this behavior started.
> Attached is a sample flow.xml.gz that demonstrates this behavior. A clean
> copy of NiFi 1.11.4 was installed with no modifications to any configuration
> file. NiFi is running as the user "xubuntu". The permissions on the folders
> are as follows:
> drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test
> drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest
> NiFi has permission to list the contents of "/home/xubuntu/test", but not
> "/home/xubuntu/test/subtest".
> The number of open DIR handles is being monitored with the command:
> lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c
> where 347813 is the pid of the NiFi process.
> DIR handle counts start at 4 immediately after NiFi is launched and steadily
> grow to tens of thousands, until file handles are exhausted. Additionally,
> these handles do not seem to be released even if the ListFile processor is
> stopped.
> Please let me know if there is any additional information I can provide.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (MINIFICPP-1229) Clean up CompressContentTests
Adam Debreceni created MINIFICPP-1229: - Summary: Clean up CompressContentTests Key: MINIFICPP-1229 URL: https://issues.apache.org/jira/browse/MINIFICPP-1229 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Adam Debreceni Assignee: Adam Debreceni Most tests in CompressContentTests were wrapped in a try-catch masking all errors. We should fix this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #786: MINIFICPP-1225 - Fix flaky HTTP tests.
szaszm commented on a change in pull request #786:
URL: https://github.com/apache/nifi-minifi-cpp/pull/786#discussion_r427205135
##
File path: extensions/civetweb/processors/ListenHTTP.h
##
@@ -201,6 +201,8 @@ class ListenHTTP : public core::Processor {
}
return 0;
}
+ protected:
+ void notifyStop() override;
Review comment:
(Correctly) adding `override` to this virtual function triggered clang's
`-Winconsistent-missing-override` warnings for all other member functions that
override virtual functions. Could you add `override` to at least `onTrigger`,
`initialize` and `onSchedule` as well (but preferably to all such functions in
the file)?
https://travis-ci.org/github/apache/nifi-minifi-cpp/jobs/688702969#L12034
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rh-override
https://google.github.io/styleguide/cppguide.html#Inheritance
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4272: NIFI-7336: Add tests for DeleteAzureDataLakeStorage
turcsanyip commented on a change in pull request #4272:
URL: https://github.com/apache/nifi/pull/4272#discussion_r427214122
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java
##
@@ -16,44 +16,388 @@
*/
package org.apache.nifi.processors.azure.storage;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.provenance.ProvenanceEventType;
import org.apache.nifi.util.MockFlowFile;
-import org.junit.Before;
-import org.junit.Ignore;
import org.junit.Test;
-import java.util.List;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
-public class ITDeleteAzureDataLakeStorage extends AbstractAzureBlobStorageIT {
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class ITDeleteAzureDataLakeStorage extends
AbstractAzureDataLakeStorageIT {
@Override
protected Class getProcessorClass() {
return DeleteAzureDataLakeStorage.class;
}
-@Before
-public void setUp() {
-runner.setProperty(DeleteAzureDataLakeStorage.FILE, TEST_FILE_NAME);
+@Test
+public void testDeleteFileFromRoot() {
+// GIVEN
+String directory= "";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+uploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileFromDirectory() {
+// GIVEN
+String directory = "TestDirectory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileFromDeepDirectory() {
+// GIVEN
+String directory=
"Directory01/Directory02/Directory03/Directory04/Directory05/Directory06/Directory07/"
++
"Directory08/Directory09/Directory10/Directory11/Directory12/Directory13/Directory14/Directory15/"
++
"Directory16/Directory17/Directory18/Directory19/Directory20/TestDirectory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileWithWhitespaceInFilename() {
+// GIVEN
+String directory= "TestDirectory";
+String filename = "A test file.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileWithWhitespaceInDirectoryName() {
+// GIVEN
+String directory= "A Test Directory";
+String filename = "testFile.txt";
+String fileContent = "AzureFileContent";
+String inputFlowFileContent = "InputFlowFileContent";
+
+createDirectoryAndUploadFile(directory, filename, fileContent);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, directory, filename,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteEmptyDirectory() {
+// GIVEN
+String parentDirectory = "ParentDirectory";
+String childDirectory = "ChildDirectory";
+String inputFlowFileContent = "InputFlowFileContent";
+
+fileSystemClient.createDirectory(parentDirectory + "/" +
childDirectory);
+
+// WHEN
+// THEN
+testSuccessfulDelete(fileSystemName, parentDirectory, childDirectory,
inputFlowFileContent, inputFlowFileContent);
+}
+
+@Test
+public void testDeleteFileCaseSensitiveFilename() {
Review comment:
This test fails on the current master due to the
[jira] [Created] (NIFI-7469) ReplaceText Processor May cause Content Repository Cleanup to fail
Stefano Piana created NIFI-7469: --- Summary: ReplaceText Processor May cause Content Repository Cleanup to fail Key: NIFI-7469 URL: https://issues.apache.org/jira/browse/NIFI-7469 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.11.4, 1.11.1, 1.11.0, 1.10.0 Environment: Windows 10, ubuntu 18.04 Reporter: Stefano Piana Attachments: ReplaceTextBug.xml Trying to replace an empty flowfile with and empty value may cause the content repository to accumulate and not being correctly cleaned up until the next NiFi restart. this behaviour has been observed on nifi 1.10+ to reproduce the problem just generate a flowfile that has some content, then replace the content with the ReplaceText processor setting it to always replace and set the replacement to empty string. Copy the ReplaceText processor and try replacing the content with an empty string again, then just use any processor to drain the flowfiles. This will cause the content repository to accumulate and stop being correctly cleaned up. here attached a simple flow that reproduces the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1163) Modify installer to include VS redistributable Dlls in bin directory.
[ https://issues.apache.org/jira/browse/MINIFICPP-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda resolved MINIFICPP-1163. --- Fix Version/s: 0.8.0 Resolution: Fixed > Modify installer to include VS redistributable Dlls in bin directory. > - > > Key: MINIFICPP-1163 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1163 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Reporter: Alex Marmer >Assignee: Alex Marmer >Priority: Major > Fix For: 0.8.0 > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #786: MINIFICPP-1225 - Fix flaky HTTP tests.
szaszm commented on a change in pull request #786:
URL: https://github.com/apache/nifi-minifi-cpp/pull/786#discussion_r426653075
##
File path: libminifi/include/io/CRCStream.h
##
@@ -145,17 +145,15 @@ class CRCStream : public BaseStream {
protected:
/**
- * Creates a vector and returns the vector using the provided
- * type name.
+ * Populates the vector using the provided type name.
+ * @param buf output buffer
* @param t incoming object
- * @returns vector.
+ * @returns number of bytes read.
*/
template
- std::vector readBuffer(const K& t) {
-std::vector buf;
+ int readBuffer(std::vector& buf, const K& t) {
Review comment:
This leaves the option of a new overload with proper error handling. If
existing 3rd party extensions happened to use the old overload, it should work
similarly, even if without error checking.
If you have a better idea, feel free to propose it. I have a few more as
well, but they also highly deviate from the style of the codebase with limited
benefit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (MINIFICPP-1218) C2 metrics simplification introduced an undefined and unused member function
[
https://issues.apache.org/jira/browse/MINIFICPP-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz closed MINIFICPP-1218.
---
> C2 metrics simplification introduced an undefined and unused member function
>
>
> Key: MINIFICPP-1218
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1218
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Marton Szasz
>Assignee: Marton Szasz
>Priority: Trivial
> Fix For: 0.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{FlowController::getAgentInformation()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-7409: -- Component/s: Extensions > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Fix For: 1.12.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-7409: -- Fix Version/s: 1.12.0 > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Fix For: 1.12.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi resolved NIFI-7409. --- Resolution: Fixed > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111028#comment-17111028 ] ASF subversion and git services commented on NIFI-7409: --- Commit 852715aadd9e989d37aeadecfe92093e212a5ad1 in nifi's branch refs/heads/master from sjyang18 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=852715a ] NIFI-7409: Azure managed identity support to Azure Datalake processors NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4249: NIFI-7409: Azure managed identity support to Azure Datalake processors
asfgit closed pull request #4249: URL: https://github.com/apache/nifi/pull/4249 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111029#comment-17111029 ] ASF subversion and git services commented on NIFI-7409: --- Commit 852715aadd9e989d37aeadecfe92093e212a5ad1 in nifi's branch refs/heads/master from sjyang18 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=852715a ] NIFI-7409: Azure managed identity support to Azure Datalake processors NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111031#comment-17111031 ] ASF subversion and git services commented on NIFI-7409: --- Commit 852715aadd9e989d37aeadecfe92093e212a5ad1 in nifi's branch refs/heads/master from sjyang18 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=852715a ] NIFI-7409: Azure managed identity support to Azure Datalake processors NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7409) Azure Managed Identity support to Azure DataLake Storage Processors
[ https://issues.apache.org/jira/browse/NIFI-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111030#comment-17111030 ] ASF subversion and git services commented on NIFI-7409: --- Commit 852715aadd9e989d37aeadecfe92093e212a5ad1 in nifi's branch refs/heads/master from sjyang18 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=852715a ] NIFI-7409: Azure managed identity support to Azure Datalake processors NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi > Azure Managed Identity support to Azure DataLake Storage Processors > --- > > Key: NIFI-7409 > URL: https://issues.apache.org/jira/browse/NIFI-7409 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Seokwon Yang >Assignee: Seokwon Yang >Priority: Major > Labels: AZURE, azure > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently AzureDatalake Storage processor supports either account key or SAS > (Shared Access Signature) token based authentication. Add support for > managed identities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-6112) Add some useful commands to NiFi Toolkit for automating NiFi cluster construction.
[
https://issues.apache.org/jira/browse/NIFI-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110985#comment-17110985
]
Mayki edited comment on NIFI-6112 at 5/19/20, 9:04 AM:
---
Hi
I used NIFI-TOOLKIT with version 1.10 and 1.11.4 and NIFI version 1.11.4 also
But both I got this error :
{code:java}
bin/cli.sh nifi create-service --input conf/ssl_service.json
--baseUrl=http://x:19091{code}
{code:java}
ERROR: Error executing command 'create-service' : Error creating controller
service:
Error 404 Not Found
HTTP ERROR 404
Problem accessing /nifi-api/controller/controller-services. Reason:
Not Foundhttp://eclipse.org/jetty;>Powered by
Jetty:// 9.4.19.v20190610
{code}
{code:java}
{code}
Could you explain why I got this error ?
Thanks
was (Author: wogno):
Hi
I used NIFI-TOOLKIT with version 1.10 and 1.11.4 and NIFI version 1.11.4 also
But both I got this error :
{code:java}
// code placeholder
{code}
bin/cli.sh nifi create-service --input conf/ssl_service.json
--baseUrl=[http://x:19091|http://x:19091/]
{code:java}
// code placeholder
{code}
ERROR: Error executing command 'create-service' : Error creating controller
service:Error 404 Not Found
HTTP ERROR 404 Problem accessing
/nifi-api/controller/controller-services. Reason: Not
Foundhttp://eclipse.org/jetty;>Powered by Jetty://
9.4.19.v20190610
Could you explain why I got this error ?
Thanks
> Add some useful commands to NiFi Toolkit for automating NiFi cluster
> construction.
> --
>
> Key: NIFI-6112
> URL: https://issues.apache.org/jira/browse/NIFI-6112
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Affects Versions: 1.9.0
>Reporter: Yoshiaki Takahashi
>Assignee: Yoshiaki Takahashi
>Priority: Major
> Labels: SDLC
> Fix For: 1.10.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Add toolkit commands useful for automating NiFi cluster construction.
> h2. Commands for "User"
> * nifi create-user
> ** Add a user
> {code}
> % ./cli.sh nifi create-user --userName user_1
> 6ba52e46-0169-1000--49c07246
> % ./cli.sh nifi create-user --userName user_2
> 6ba54b58-0169-1000--69e8abcc
> % ./cli.sh nifi create-user --userName user_3
> 6bd400c3-0169-1000--c605c27f
> {code}
> * nifi list-users
> ** Retrieve the user list
> {code}
> % ./cli.sh nifi list-users
> # Name Member of
> - -
> 1 user_1
> 2 user_2
> 3 user_3
> {code}
> h2. Commands for "User group"
> * nifi create-user-group
> ** Add a user group
> {code}
> % ./cli.sh nifi create-user-group --userGroupName admin --userList
> 6ba52e46-0169-1000--49c07246
> 6bd06533-0169-1000--ac5cfbcf
> % ./cli.sh nifi create-user-group --userGroupName users
> 6bd1b4b2-0169-1000--dd0ff820
> {code}
> * nifi list-user-groups
> ** Retrieve the user group list
> {code}
> % ./cli.sh nifi list-user-groups
> # Name Members
> - --
> 1 adminuser_1
> 2 users
> {code}
> * nifi update-user-group
> ** Update users belonging the user group
> {code}
> % ./cli.sh nifi update-user-group --userGroupId
> 6bd1b4b2-0169-1000--dd0ff820 --userList
> 6ba54b58-0169-1000--69e8abcc,6bd400c3-0169-1000--c605c27f
> % ./cli.sh nifi list-user-groups
> # Name Members
> - --
> 1 adminuser_1
> 2 usersuser_2, user_3
> {code}
> h2. Commands for "Access policy"
> * nifi get-policy
> ** Retrieve the access policy
> {code}
> % ./cli.sh nifi get-policy --accessPolicyResource /tenants
> --accessPolicyAction write
> Resource: /tenants
> Action : write
> Users :
> Groups : admin
> {code}
> * nifi update-policy
> ** Update users authorized for the resource
> {code}
> % ./cli.sh nifi update-policy --accessPolicyResource /tenants
> --accessPolicyAction write --userList
> 6ba52e46-0169-1000--49c07246,6ba54b58-0169-1000--69e8abcc
> --groupList
> 6bd06533-0169-1000--ac5cfbcf,6bd1b4b2-0169-1000--dd0ff820
> User "user_2" (id 6ba54b58-0169-1000--69e8abcc) added
> User
[jira] [Commented] (NIFI-6112) Add some useful commands to NiFi Toolkit for automating NiFi cluster construction.
[
https://issues.apache.org/jira/browse/NIFI-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110985#comment-17110985
]
Mayki commented on NIFI-6112:
-
Hi
I used NIFI-TOOLKIT with version 1.10 and 1.11.4 and NIFI version 1.11.4 also
But both I got this error :
{code:java}
// code placeholder
{code}
bin/cli.sh nifi create-service --input conf/ssl_service.json
--baseUrl=[http://x:19091|http://x:19091/]
{code:java}
// code placeholder
{code}
ERROR: Error executing command 'create-service' : Error creating controller
service:Error 404 Not Found
HTTP ERROR 404 Problem accessing
/nifi-api/controller/controller-services. Reason: Not
Foundhttp://eclipse.org/jetty;>Powered by Jetty://
9.4.19.v20190610
Could you explain why I got this error ?
Thanks
> Add some useful commands to NiFi Toolkit for automating NiFi cluster
> construction.
> --
>
> Key: NIFI-6112
> URL: https://issues.apache.org/jira/browse/NIFI-6112
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Affects Versions: 1.9.0
>Reporter: Yoshiaki Takahashi
>Assignee: Yoshiaki Takahashi
>Priority: Major
> Labels: SDLC
> Fix For: 1.10.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Add toolkit commands useful for automating NiFi cluster construction.
> h2. Commands for "User"
> * nifi create-user
> ** Add a user
> {code}
> % ./cli.sh nifi create-user --userName user_1
> 6ba52e46-0169-1000--49c07246
> % ./cli.sh nifi create-user --userName user_2
> 6ba54b58-0169-1000--69e8abcc
> % ./cli.sh nifi create-user --userName user_3
> 6bd400c3-0169-1000--c605c27f
> {code}
> * nifi list-users
> ** Retrieve the user list
> {code}
> % ./cli.sh nifi list-users
> # Name Member of
> - -
> 1 user_1
> 2 user_2
> 3 user_3
> {code}
> h2. Commands for "User group"
> * nifi create-user-group
> ** Add a user group
> {code}
> % ./cli.sh nifi create-user-group --userGroupName admin --userList
> 6ba52e46-0169-1000--49c07246
> 6bd06533-0169-1000--ac5cfbcf
> % ./cli.sh nifi create-user-group --userGroupName users
> 6bd1b4b2-0169-1000--dd0ff820
> {code}
> * nifi list-user-groups
> ** Retrieve the user group list
> {code}
> % ./cli.sh nifi list-user-groups
> # Name Members
> - --
> 1 adminuser_1
> 2 users
> {code}
> * nifi update-user-group
> ** Update users belonging the user group
> {code}
> % ./cli.sh nifi update-user-group --userGroupId
> 6bd1b4b2-0169-1000--dd0ff820 --userList
> 6ba54b58-0169-1000--69e8abcc,6bd400c3-0169-1000--c605c27f
> % ./cli.sh nifi list-user-groups
> # Name Members
> - --
> 1 adminuser_1
> 2 usersuser_2, user_3
> {code}
> h2. Commands for "Access policy"
> * nifi get-policy
> ** Retrieve the access policy
> {code}
> % ./cli.sh nifi get-policy --accessPolicyResource /tenants
> --accessPolicyAction write
> Resource: /tenants
> Action : write
> Users :
> Groups : admin
> {code}
> * nifi update-policy
> ** Update users authorized for the resource
> {code}
> % ./cli.sh nifi update-policy --accessPolicyResource /tenants
> --accessPolicyAction write --userList
> 6ba52e46-0169-1000--49c07246,6ba54b58-0169-1000--69e8abcc
> --groupList
> 6bd06533-0169-1000--ac5cfbcf,6bd1b4b2-0169-1000--dd0ff820
> User "user_2" (id 6ba54b58-0169-1000--69e8abcc) added
> User "user_1" (id 6ba52e46-0169-1000--49c07246) added
> User group id 6bd06533-0169-1000--ac5cfbcf already included
> User group "users" (id 6bd1b4b2-0169-1000--dd0ff820) added
> Access policy was updated
> id: 15e4e0bd-cb28-34fd-8587-f8d15162cba5
> % ./cli.sh nifi get-policy --accessPolicyResource /tenants
> --accessPolicyAction write
> Resource: /tenants
> Action : write
> Users : user_2, user_1
> Groups : admin, users
> {code}
> h2. Commands for "Controller service"
> * nifi create-service
> ** Create a controller service
> {code}
> % cat ./ssl_service.json
> {
> "component": {
> "name": "Sample SSL context service",
>
[GitHub] [nifi] pvillard31 commented on pull request #4283: NIFI-7331 Fixed grammatical errors in log output.
pvillard31 commented on pull request #4283: URL: https://github.com/apache/nifi/pull/4283#issuecomment-630675991 Merged to master, thanks @mtien-apache This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7331) Grammatical and syntactic errors in log and error messages
[ https://issues.apache.org/jira/browse/NIFI-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-7331: - Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Grammatical and syntactic errors in log and error messages > -- > > Key: NIFI-7331 > URL: https://issues.apache.org/jira/browse/NIFI-7331 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: M Tien >Priority: Trivial > Labels: logging, messaging > Fix For: 1.12.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Improve the quality of the log and error messages in the OIDC identity > provider. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4283: NIFI-7331 Fixed grammatical errors in log output.
asfgit closed pull request #4283: URL: https://github.com/apache/nifi/pull/4283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7331) Grammatical and syntactic errors in log and error messages
[ https://issues.apache.org/jira/browse/NIFI-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110970#comment-17110970 ] ASF subversion and git services commented on NIFI-7331: --- Commit 3fec4d8c270385e10fab843254ee851614fa4738 in nifi's branch refs/heads/master from mtien [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3fec4d8 ] NIFI-7331 Fixed grammatical errors in log output. Signed-off-by: Pierre Villard This closes #4283. > Grammatical and syntactic errors in log and error messages > -- > > Key: NIFI-7331 > URL: https://issues.apache.org/jira/browse/NIFI-7331 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: M Tien >Priority: Trivial > Labels: logging, messaging > Time Spent: 10m > Remaining Estimate: 0h > > Improve the quality of the log and error messages in the OIDC identity > provider. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pgyori commented on pull request #4272: NIFI-7336: Add tests for DeleteAzureDataLakeStorage
pgyori commented on pull request #4272: URL: https://github.com/apache/nifi/pull/4272#issuecomment-630666271 Thank you @turcsanyip! Fixed them in the next commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #786: MINIFICPP-1225 - Fix flaky HTTP tests.
adamdebreceni commented on a change in pull request #786:
URL: https://github.com/apache/nifi-minifi-cpp/pull/786#discussion_r427085519
##
File path: libminifi/include/io/CRCStream.h
##
@@ -145,17 +145,15 @@ class CRCStream : public BaseStream {
protected:
/**
- * Creates a vector and returns the vector using the provided
- * type name.
+ * Populates the vector using the provided type name.
+ * @param buf output buffer
* @param t incoming object
- * @returns vector.
+ * @returns number of bytes read.
*/
template
- std::vector readBuffer(const K& t) {
-std::vector buf;
+ int readBuffer(std::vector& buf, const K& t) {
Review comment:
readded the previous versions to libminifi
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (MINIFICPP-1226) Improve C2 heartbeat performance
[ https://issues.apache.org/jira/browse/MINIFICPP-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz closed MINIFICPP-1226. --- > Improve C2 heartbeat performance > > > Key: MINIFICPP-1226 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1226 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > During a profiling session I noticed that heartbeat serialization took about > 30% of the used CPU time in a simple flow, with 1 sec heartbeat interval. > This is way too long. > In C2Agent::serializeMetrics, while constructing the payload, we used > C2Payload's move facilities incorrectly resulting in a silent fallback to > copy every time. This seems to have caused the high CPU usage. After the > submitted change, heartbeat generation was no longer showing up in the my > profiler view, so it should be significantly faster. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFICPP-1226) Improve C2 heartbeat performance
[ https://issues.apache.org/jira/browse/MINIFICPP-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-1226. - Resolution: Fixed > Improve C2 heartbeat performance > > > Key: MINIFICPP-1226 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1226 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > During a profiling session I noticed that heartbeat serialization took about > 30% of the used CPU time in a simple flow, with 1 sec heartbeat interval. > This is way too long. > In C2Agent::serializeMetrics, while constructing the payload, we used > C2Payload's move facilities incorrectly resulting in a silent fallback to > copy every time. This seems to have caused the high CPU usage. After the > submitted change, heartbeat generation was no longer showing up in the my > profiler view, so it should be significantly faster. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
