[jira] [Created] (NIFI-8355) JVM metrics testing fails a valid state
Noble Numbat created NIFI-8355: -- Summary: JVM metrics testing fails a valid state Key: NIFI-8355 URL: https://issues.apache.org/jira/browse/NIFI-8355 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.14.0 Reporter: Noble Numbat Testing of the jvm metrics in the 'def "Get numeric metric for #metricName via method #methodName"()' method in JmxJvmMetricsSpec.groovy includes an explicit test that the metrics values should not equal -1. This test should be removed as -1 is a valid state for the "init" and "max" metrics when they are undefined. The test for exceptions being thrown will catch the other metrics if they return a -1 value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4917: NIFI-8346 PutAzureBlobStorage doesn't route to failure despite the exception during upload
exceptionfactory commented on a change in pull request #4917:
URL: https://github.com/apache/nifi/pull/4917#discussion_r599236166
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java
##
@@ -141,14 +142,14 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
}
try {
-blob.upload(in, -1, null, null, operationContext);
+uploadBlob(blob, operationContext, in);
BlobProperties properties = blob.getProperties();
attributes.put("azure.container", containerName);
attributes.put("azure.primaryUri",
blob.getSnapshotQualifiedUri().toString());
attributes.put("azure.etag", properties.getEtag());
attributes.put("azure.length", String.valueOf(length));
attributes.put("azure.timestamp",
String.valueOf(properties.getLastModified()));
-} catch (StorageException | URISyntaxException e) {
+} catch (StorageException | URISyntaxException | IOException
e) {
storedException.set(e);
Review comment:
Reviewing the larger context of this block, is there a reason to
continue maintaining this reference to the `storageException`? The checked
`StorageException` and `URISyntaxException` could be wrapped in the existing
`throw new IOException()`. If there is a particular reason to evaluate or log
the `StorageException`, that could be handled here, or in the outer catch
block. Either way, it seems like it would be worth untangling the exception
handling to avoid this kind of scenario down the road.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] s9514171 opened a new pull request #4923: NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file…
s9514171 opened a new pull request #4923:
URL: https://github.com/apache/nifi/pull/4923
… if the process start failed
Description of PR
Delete the temp file to avoid the i-node of /tmp full
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on JDK 8?
- [X] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8354) ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
Hsin-Ying Lee created NIFI-8354: --- Summary: ExecuteStreamCommand processor doesn't delete the temp file if the process start failed Key: NIFI-8354 URL: https://issues.apache.org/jira/browse/NIFI-8354 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Hsin-Ying Lee Assignee: Hsin-Ying Lee ExecuteStreamCommand processor doesn't delete the temp file if the process start failed The i-node of /tmp will be exhausted, and the service will run into an unexpected situation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
exceptionfactory commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804519573 @joewitt Minor note, it looks like the PR title has the incorrect Jira issue number, it should be NIFI-8350 in order for the PR to be linked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jfrazee commented on a change in pull request #4917: NIFI-8346 PutAzureBlobStorage doesn't route to failure despite the exception during upload
jfrazee commented on a change in pull request #4917:
URL: https://github.com/apache/nifi/pull/4917#discussion_r599122961
##
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java
##
@@ -141,14 +142,14 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
}
try {
-blob.upload(in, -1, null, null, operationContext);
+uploadBlob(blob, operationContext, in);
BlobProperties properties = blob.getProperties();
attributes.put("azure.container", containerName);
attributes.put("azure.primaryUri",
blob.getSnapshotQualifiedUri().toString());
attributes.put("azure.etag", properties.getEtag());
attributes.put("azure.length", String.valueOf(length));
attributes.put("azure.timestamp",
String.valueOf(properties.getLastModified()));
-} catch (StorageException | URISyntaxException e) {
+} catch (StorageException | URISyntaxException | IOException
e) {
storedException.set(e);
throw new IOException(e);
Review comment:
I should have caught this before but we probably shouldn't double-wrap
this.
```suggestion
throw (e instanceof IOException) ? (IOException) e : new
IOException(e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4922: NIFI-8352 Changed TestPutTCP.assertServerConnections to loop and sleep
exceptionfactory commented on pull request #4922: URL: https://github.com/apache/nifi/pull/4922#issuecomment-804405865 > +1, the Rule stuff is very nice, I did not know about that. Thanks for the feedback @ottobackwards! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4922: NIFI-8352 Changed TestPutTCP.assertServerConnections to loop and sleep
ottobackwards commented on pull request #4922: URL: https://github.com/apache/nifi/pull/4922#issuecomment-804401351 +1, the Rule stuff is very nice, I did not know about that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
ottobackwards commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804398296 +1, the failure is not due to these settings. The changes look good and we should get them running -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-8353) When node is offloaded, it may still receive data from load-balanced connections
Mark Payne created NIFI-8353: Summary: When node is offloaded, it may still receive data from load-balanced connections Key: NIFI-8353 URL: https://issues.apache.org/jira/browse/NIFI-8353 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.14.0 Consider a simple flow: GenerateFlowFile (9 FlowFiles per batch, each 10 KB, scheduled once per second) -> UpdateAttribute. The connection between the two processors is configured with Round Robin load balancing. Start GenerateFlowFile. Wait a few seconds for some data to queue up. Disconnect Node 2. Immediately offload Node 2. Refresh cluster page to see that the node is offloaded. Navigate to Node 2's UI. Sometimes it will show data queued up. This is because after the node was offloaded, the connection from Node 1 was still transferring data to Node 2. Node 1 won't start any additional transactions to send data to Node 2, but it will complete the already-started transactions. So the node can be offloaded and end up with data queued. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8352) Improve reliability of TestPutTCP
[
https://issues.apache.org/jira/browse/NIFI-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-8352:
---
Affects Version/s: 1.14.0
Status: Patch Available (was: In Progress)
> Improve reliability of TestPutTCP
> -
>
> Key: NIFI-8352
> URL: https://issues.apache.org/jira/browse/NIFI-8352
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.14.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{TestPutTCP}} class has intermittent failures when running through
> GitHub Continuous Integration, particularly when checking the number of
> server connections after sending an empty message string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] exceptionfactory opened a new pull request #4922: NIFI-8352 Changed TestPutTCP.assertServerConnections to loop and sleep
exceptionfactory opened a new pull request #4922:
URL: https://github.com/apache/nifi/pull/4922
Description of PR
NIFI-8352 Changes `TestPutTCP.assertServerConnections` to loop and sleep
while evaluating the number expected server connections. Replace individual
test method annotation timeouts with JUnit Timeout Rule applicable to all
methods.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8352) Improve reliability of TestPutTCP
David Handermann created NIFI-8352:
--
Summary: Improve reliability of TestPutTCP
Key: NIFI-8352
URL: https://issues.apache.org/jira/browse/NIFI-8352
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: David Handermann
Assignee: David Handermann
The {{TestPutTCP}} class has intermittent failures when running through GitHub
Continuous Integration, particularly when checking the number of server
connections after sending an empty message string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8351) Investigate and remove build warnings on documentation generation
Otto Fowler created NIFI-8351: - Summary: Investigate and remove build warnings on documentation generation Key: NIFI-8351 URL: https://issues.apache.org/jira/browse/NIFI-8351 Project: Apache NiFi Issue Type: Improvement Reporter: Otto Fowler Warning: Could not generate extensions' documentation 2237 org.apache.maven.plugin.MojoExecutionException: Failed to create Extension Documentation 2238 at org.apache.nifi.NarMojo.generateDocumentation (NarMojo.java:596) 2239 at org.apache.nifi.NarMojo.execute (NarMojo.java:499) snip. These are seen in the GitHub action builds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] ottobackwards commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
ottobackwards commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804287722 I'll try to create a jira on it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
joewitt commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804278806 we should kill all warnings. but that isn't part of this JIRA/PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ottobackwards commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
ottobackwards commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804276106 Fair enough, waiting for builds to finish. Wondering if we should see ```bash Warning: Could not generate extensions' documentation 2237 org.apache.maven.plugin.MojoExecutionException: Failed to create Extension Documentation 2238 at org.apache.nifi.NarMojo.generateDocumentation (NarMojo.java:596) 2239 at org.apache.nifi.NarMojo.execute (NarMojo.java:499) ``` at all. They *are* warnings not errors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (MINIFICPP-1532) Create a processor to monitor consumption of resources in Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306455#comment-17306455 ] Arpad Boda commented on MINIFICPP-1532: --- [~joewitt]: thanks, bad wording of mine. Fixed. :) > Create a processor to monitor consumption of resources in Windows > - > > Key: MINIFICPP-1532 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1532 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Martin Zink >Priority: Major > Fix For: 0.10.0 > > > Phase 1: > Windows only. > Only monitor the consumption of the following (across the system, no per > process data): > +CPU+: > * Idle (%) > * Stolen > * System > * User > +Memory:+ > * Total > * Used (%) > * Free (%) > * Swap Free > * Swap Used > +I/O+: > * Read > * Write > * Avg Utilization > +Disk+: > * Total > * Free > * Used > * Volumes/Partitions > +Network+: > * Avg Utilization (%) > * In Errors > * Out Errors > Phase 2: > Add processor and memory consumption of each process running on the system > Phase 3: > Linux? (Not sure we will ever need this) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1532) Create a processor to capture resource consumption metrics on Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda updated MINIFICPP-1532: -- Summary: Create a processor to capture resource consumption metrics on Windows (was: Create a processor to monitor consumption of resources in Windows) > Create a processor to capture resource consumption metrics on Windows > - > > Key: MINIFICPP-1532 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1532 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Martin Zink >Priority: Major > Fix For: 0.10.0 > > > Phase 1: > Windows only. > Only monitor the consumption of the following (across the system, no per > process data): > +CPU+: > * Idle (%) > * Stolen > * System > * User > +Memory:+ > * Total > * Used (%) > * Free (%) > * Swap Free > * Swap Used > +I/O+: > * Read > * Write > * Avg Utilization > +Disk+: > * Total > * Free > * Used > * Volumes/Partitions > +Network+: > * Avg Utilization (%) > * In Errors > * Out Errors > Phase 2: > Add processor and memory consumption of each process running on the system > Phase 3: > Linux? (Not sure we will ever need this) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] joewitt commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
joewitt commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804261089 The commentary in the link I provided suggests the TTL might not be that valuable as i read it. So I'll keep it out for now. Updated JIRA to reference it for future purposes in case this change doesn't do enough. I'm also a little concerned about build performance getting even worse by adding anything else. WE'll see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8350) Improve reliability of artifact retrieval by Maven in Github CI
[ https://issues.apache.org/jira/browse/NIFI-8350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306451#comment-17306451 ] Joe Witt commented on NIFI-8350: adding ttl may be worth doing in a later PR if this one doesn't make things more stable https://github.com/actions/virtual-environments/issues/1499#issuecomment-689467080 > Improve reliability of artifact retrieval by Maven in Github CI > --- > > Key: NIFI-8350 > URL: https://issues.apache.org/jira/browse/NIFI-8350 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.14.0 > > > Quite often our builds fail in Github CI due to issues retrieving artifacts > by Maven. This appears to be not all that uncommon as shown in > https://github.com/apache/pulsar/issues/6231 > Going to try the advice in that thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFICPP-1532) Create a processor to monitor consumption of resources in Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306449#comment-17306449 ] Joe Witt commented on MINIFICPP-1532: - I dont think we're creating processors to do monitoring themselves. Rather the purpose of a processor would be to capture 'monitoring data/metrics' from the target system to then share that collected information in a flow that send it to some other system/collection point. > Create a processor to monitor consumption of resources in Windows > - > > Key: MINIFICPP-1532 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1532 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Martin Zink >Priority: Major > Fix For: 0.10.0 > > > Phase 1: > Windows only. > Only monitor the consumption of the following (across the system, no per > process data): > +CPU+: > * Idle (%) > * Stolen > * System > * User > +Memory:+ > * Total > * Used (%) > * Free (%) > * Swap Free > * Swap Used > +I/O+: > * Read > * Write > * Avg Utilization > +Disk+: > * Total > * Free > * Used > * Volumes/Partitions > +Network+: > * Avg Utilization (%) > * In Errors > * Out Errors > Phase 2: > Add processor and memory consumption of each process running on the system > Phase 3: > Linux? (Not sure we will ever need this) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] ottobackwards commented on pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
ottobackwards commented on pull request #4921: URL: https://github.com/apache/nifi/pull/4921#issuecomment-804258714 I found another issue similar to this, with _almost_ the same fix, but they also set the ttl, do we want to try that? https://github.com/actions/virtual-environments/issues/1499#issuecomment-689467080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-1532) Create a processor to monitor consumption of resources in Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink reassigned MINIFICPP-1532: -- Assignee: Martin Zink > Create a processor to monitor consumption of resources in Windows > - > > Key: MINIFICPP-1532 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1532 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Martin Zink >Priority: Major > Fix For: 0.10.0 > > > Phase 1: > Windows only. > Only monitor the consumption of the following (across the system, no per > process data): > +CPU+: > * Idle (%) > * Stolen > * System > * User > +Memory:+ > * Total > * Used (%) > * Free (%) > * Swap Free > * Swap Used > +I/O+: > * Read > * Write > * Avg Utilization > +Disk+: > * Total > * Free > * Used > * Volumes/Partitions > +Network+: > * Avg Utilization (%) > * In Errors > * Out Errors > Phase 2: > Add processor and memory consumption of each process running on the system > Phase 3: > Linux? (Not sure we will ever need this) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] joewitt opened a new pull request #4921: NIFI-8530 updated worfklow to hopefully better handle artifact retrieval
joewitt opened a new pull request #4921:
URL: https://github.com/apache/nifi/pull/4921
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-8344) Allow TailFile to continue tailing a file for some time after it has been rolled over
[ https://issues.apache.org/jira/browse/NIFI-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-8344. --- Fix Version/s: 1.14.0 Resolution: Fixed > Allow TailFile to continue tailing a file for some time after it has been > rolled over > - > > Key: NIFI-8344 > URL: https://issues.apache.org/jira/browse/NIFI-8344 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0 > > Time Spent: 40m > Remaining Estimate: 0h > > TailFile makes the assumption that once a file has been rolled over, it will > never be appended to. If the file's Last Modified timestamp changes, the > processor assumes that it's a new file and imports the entire contents of the > file again. > However, one practice that I've encountered is that users have a syslog > server that rotates periodically. To rotate, they rename the existing file, > and then restart the server. When that happens, the server will flush out any > data that it has buffered to the file that was just rolled over, and then > begin writing to the new file. > This results in the TailFile processor ingesting the entire file that has > been rolled over. Because we can't keep state about every file that is rolled > over, we should introduce a property that allows the user to indicate that > upon rollover they want to continue tailing that rolled over file until it is > no longer being written to, and then begin tailing the new file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8344) Allow TailFile to continue tailing a file for some time after it has been rolled over
[ https://issues.apache.org/jira/browse/NIFI-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306432#comment-17306432 ] ASF subversion and git services commented on NIFI-8344: --- Commit e1b9548ab68ece3a9904f50cd42898e0d41ce166 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e1b9548 ] NIFI-8344: Introduced new Rollover Tail Period property This closes #4916. Signed-off-by: Tamas Palfy > Allow TailFile to continue tailing a file for some time after it has been > rolled over > - > > Key: NIFI-8344 > URL: https://issues.apache.org/jira/browse/NIFI-8344 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > TailFile makes the assumption that once a file has been rolled over, it will > never be appended to. If the file's Last Modified timestamp changes, the > processor assumes that it's a new file and imports the entire contents of the > file again. > However, one practice that I've encountered is that users have a syslog > server that rotates periodically. To rotate, they rename the existing file, > and then restart the server. When that happens, the server will flush out any > data that it has buffered to the file that was just rolled over, and then > begin writing to the new file. > This results in the TailFile processor ingesting the entire file that has > been rolled over. Because we can't keep state about every file that is rolled > over, we should introduce a property that allows the user to indicate that > upon rollover they want to continue tailing that rolled over file until it is > no longer being written to, and then begin tailing the new file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4916: NIFI-8344: Introduced new Rollover Tail Period property
asfgit closed pull request #4916: URL: https://github.com/apache/nifi/pull/4916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1532) Create a processor to monitor consumption of resources in Windows
Arpad Boda created MINIFICPP-1532: - Summary: Create a processor to monitor consumption of resources in Windows Key: MINIFICPP-1532 URL: https://issues.apache.org/jira/browse/MINIFICPP-1532 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Arpad Boda Fix For: 0.10.0 Phase 1: Windows only. Only monitor the consumption of the following (across the system, no per process data): +CPU+: * Idle (%) * Stolen * System * User +Memory:+ * Total * Used (%) * Free (%) * Swap Free * Swap Used +I/O+: * Read * Write * Avg Utilization +Disk+: * Total * Free * Used * Volumes/Partitions +Network+: * Avg Utilization (%) * In Errors * Out Errors Phase 2: Add processor and memory consumption of each process running on the system Phase 3: Linux? (Not sure we will ever need this) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8350) Improve reliability of artifact retrieval by Maven in Github CI
[ https://issues.apache.org/jira/browse/NIFI-8350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306334#comment-17306334 ] Joe Witt commented on NIFI-8350: here is a recent failure Error: Failed to execute goal org.eclipse.jetty:jetty-jspc-maven-plugin:9.4.38.v20210224:jspc (default) on project nifi-web-ui: Execution default of goal org.eclipse.jetty:jetty-jspc-maven-plugin:9.4.38.v20210224:jspc failed: Plugin org.eclipse.jetty:jetty-jspc-maven-plugin:9.4.38.v20210224 or one of its dependencies could not be resolved: Failed to collect dependencies at org.eclipse.jetty:jetty-jspc-maven-plugin:jar:9.4.38.v20210224 -> org.apache.ant:ant:jar:1.10.9: Failed to read artifact descriptor for org.apache.ant:ant:jar:1.10.9: Could not transfer artifact org.apache.ant:ant:pom:1.10.9 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.10.9/ant-1.10.9.pom: Connection reset -> [Help 1] Error: Error: To see the full stack trace of the errors, re-run Maven with the -e switch. Error: Re-run Maven using the -X switch to enable full debug logging. Error: Error: For more information about the errors and possible solutions, please read the following articles: Error: [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException Error: Error: After correcting the problems, you can resume the build with the command Error:mvn -rf :nifi-web-ui Error: Process completed with exit code 1. https://github.com/apache/nifi/runs/2166405104?check_suite_focus=true > Improve reliability of artifact retrieval by Maven in Github CI > --- > > Key: NIFI-8350 > URL: https://issues.apache.org/jira/browse/NIFI-8350 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 1.14.0 > > > Quite often our builds fail in Github CI due to issues retrieving artifacts > by Maven. This appears to be not all that uncommon as shown in > https://github.com/apache/pulsar/issues/6231 > Going to try the advice in that thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8350) Improve reliability of artifact retrieval by Maven in Github CI
Joe Witt created NIFI-8350: -- Summary: Improve reliability of artifact retrieval by Maven in Github CI Key: NIFI-8350 URL: https://issues.apache.org/jira/browse/NIFI-8350 Project: Apache NiFi Issue Type: Task Components: Tools and Build Reporter: Joe Witt Assignee: Joe Witt Fix For: 1.14.0 Quite often our builds fail in Github CI due to issues retrieving artifacts by Maven. This appears to be not all that uncommon as shown in https://github.com/apache/pulsar/issues/6231 Going to try the advice in that thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #4916: NIFI-8344: Introduced new Rollover Tail Period property
markap14 commented on pull request #4916: URL: https://github.com/apache/nifi/pull/4916#issuecomment-804192256 Thanks for the review @tpalfy. After reading through your suggestions, I agree with all recommendations. Will put up a non-draft PR shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4898: NIFI-8325: Improve SNMP processors
exceptionfactory commented on a change in pull request #4898:
URL: https://github.com/apache/nifi/pull/4898#discussion_r598682741
##
File path:
nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/context/SNMPContext.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.snmp.context;
+
+import org.apache.nifi.snmp.configuration.BasicConfiguration;
+import org.apache.nifi.snmp.configuration.SecurityConfiguration;
+import org.apache.nifi.snmp.exception.AgentSecurityConfigurationException;
+import org.apache.nifi.snmp.utils.SNMPUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.snmp4j.AbstractTarget;
+import org.snmp4j.CommunityTarget;
+import org.snmp4j.Snmp;
+import org.snmp4j.UserTarget;
+import org.snmp4j.mp.SnmpConstants;
+import org.snmp4j.security.SecurityLevel;
+import org.snmp4j.security.UsmUser;
+import org.snmp4j.smi.OctetString;
+import org.snmp4j.smi.UdpAddress;
+import org.snmp4j.transport.DefaultUdpTransportMapping;
+
+import java.io.IOException;
+
+public class SNMPContext {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(SNMPContext.class);
+
+private Snmp snmp;
+private AbstractTarget target;
+
+public static SNMPContext newInstance() {
+return new SNMPContext();
+}
+
+public void init(final BasicConfiguration basicConfiguration, final
SecurityConfiguration securityConfiguration) {
+initSnmp(basicConfiguration);
+
+final String snmpVersion = securityConfiguration.getVersion();
+
+final int version = SNMPUtils.getSnmpVersion(snmpVersion);
+
+if (version == SnmpConstants.version3) {
+createUserTarget(basicConfiguration, securityConfiguration, snmp,
version);
+} else {
+target = createCommunityTarget(basicConfiguration,
securityConfiguration, version);
+}
+}
+
+public void close() {
+try {
+snmp.close();
+} catch (IOException e) {
+LOGGER.error("Could not close SNMP session.");
+}
+
+}
+
+private CommunityTarget createCommunityTarget(BasicConfiguration
basicConfiguration, SecurityConfiguration securityConfiguration, int version) {
+CommunityTarget communityTarget = new CommunityTarget();
+setupTargetBasicProperties(communityTarget, basicConfiguration,
version);
+String community = securityConfiguration.getCommunityString();
+if (community != null) {
+communityTarget.setCommunity(new OctetString(community));
+}
+return communityTarget;
+}
+
+private void createUserTarget(BasicConfiguration basicConfiguration,
SecurityConfiguration securityConfiguration, Snmp snmp, int version) {
+final String username = securityConfiguration.getSecurityName();
+final String authProtocol = securityConfiguration.getAuthProtocol();
+final String authPassword = securityConfiguration.getAuthPassword();
+final String privacyProtocol =
securityConfiguration.getPrivacyProtocol();
+final String privacyPassword =
securityConfiguration.getPrivacyPassword();
+final OctetString authPasswordOctet = authPassword != null ? new
OctetString(authPassword) : null;
+final OctetString privacyPasswordOctet = privacyPassword != null ? new
OctetString(privacyPassword) : null;
+
+if (snmp.getUSM() == null) {
+throw new AgentSecurityConfigurationException("No security model
has been configured in agent.");
+}
+
+// Add user information.
+snmp.getUSM().addUser(
+new OctetString(username),
+new UsmUser(new OctetString(username),
SNMPUtils.getAuth(authProtocol), authPasswordOctet,
+SNMPUtils.getPriv(privacyProtocol),
privacyPasswordOctet));
+
+target = new UserTarget();
+setupTargetBasicProperties(target, basicConfiguration, version);
+int securityLevel =
SecurityLevel.valueOf(securityConfiguration.getSecurityLevel()).getSnmpValue();
+target.setSecurityLevel(securityLevel);
+
+final String securityName
[GitHub] [nifi] adenes commented on pull request #4917: NIFI-8346 PutAzureBlobStorage doesn't route to failure despite the exception during upload
adenes commented on pull request #4917: URL: https://github.com/apache/nifi/pull/4917#issuecomment-804006166 @jfrazee , I have added a new test case to test the invalid credentials scenario. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-8182) Nested version control broken by "Single FlowFile Per Node"
[ https://issues.apache.org/jira/browse/NIFI-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens M Kofoed resolved NIFI-8182. - Fix Version/s: 1.13.1 Resolution: Fixed > Nested version control broken by "Single FlowFile Per Node" > --- > > Key: NIFI-8182 > URL: https://issues.apache.org/jira/browse/NIFI-8182 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.12.1 >Reporter: Jens M Kofoed >Priority: Major > Fix For: 1.13.1 > > > I have an issue with nested PG where the root PG continuously is showing it > has local changes but it has not. > I have now found why, and how you can reproduce the bug > 1. Create an empty Process Group (PG-Root) > 2. Add version control to the PG-Root > 3. Add an empty PG (PG-Sub) in PG-Root > 4. Add version control to PG-Sub > 5. Commit changes to PG-Root > 6. Check that both PG-Root and PG-Sub are up-to-date with the green check > mark > 7. Change "Process Group FlowFile Concurrency" to "Single FlowFile Per > Node" for the PG-Sub > 8. Commit changes to PR-Sub > 9. Refresh page > 10. PG-Sub should have a green check mark, and PG-Root still has local > changes > 11. Commit changes to PG-Root > 12. PG-Root is still showing it has local changes, but it has not. > Bug is allso created in Nifi registry, as: > https://issues.apache.org/jira/browse/NIFIREG-437 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1504) Add Resource consumption data to heartbeats
[
https://issues.apache.org/jira/browse/MINIFICPP-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1504:
---
Status: Patch Available (was: In Progress)
https://github.com/apache/nifi-minifi-cpp/pull/1032
> Add Resource consumption data to heartbeats
> ---
>
> Key: MINIFICPP-1504
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1504
> Project: Apache NiFi MiNiFi C++
> Issue Type: New Feature
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
> Attachments: desktop.png, laptop.png, macos.png, ubuntu.png,
> windows.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> It would be beneficial to add CPU utilization and RAM consumption to the
> heartbeat. So the C2 server can monitor how much resources does the agent,
> and the agent's host(overall) consume.
> Currently the heartbeat {color:#172b4d}already{color} contains the number of
> vCores, and the total RAM of the system.
> Here are the proposed changes to the heartbeat JSON in the C2 Protocol:
>
> {code:java}
> ...
> "deviceInfo": {
> "identifier": "",
> "systemInfo": {
>"physicalMem": 8360783872,
>"vCores": 4,
>// New fields to be added below.
>// cpuUtilization
> // - a normalized (max == 100%) percentage value
> // - passed as floating point in the range [0.0, 1.0]
> // - the agent would average a sliding window of system values taken
> at some interval
>// memoryUtilization
> // - bytes in use, whatever the system reports.
> // could exceed max physical memory due to virtual mem / swap on
> host.
> "cpuUtilization": 0.50,
> "memoryUtilization": 7360783872
> },
> "networkInfo": {
> ...
> }
> },
> "agentInfo": {
> ...
> "status": {
> ...
> // New nested object to be added below.
> // Same as values in systemInfo, but for the minifi agent process only
> // cpuUtilization
> // - a normalized (max == 100%) percentage value
> // - passed as floating point in the range [0.0, 1.0]
> // - agent will average a sliding window of system values taken at
> some interval
> // memoryUtilization
> // - bytes in use, whatever the system reports.
> // could exceed max physical memory due to virtual mem / swap on host.
> "resourceConsumption": {
>"cpuUtilization": 0.10,
>"memoryUtilization": 1360783872
> },
>}
> ...{code}
> These new field would be optional (maybe configurable?)
> This should work on all supported platforms and return platform-independent
> information to the C2.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1493) GPS extension incompatible with newer gpsd versions (3.21 or higher)
[
https://issues.apache.org/jira/browse/MINIFICPP-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1493:
---
Status: Patch Available (was: In Progress)
https://github.com/apache/nifi-minifi-cpp/pull/1033
> GPS extension incompatible with newer gpsd versions (3.21 or higher)
>
>
> Key: MINIFICPP-1493
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1493
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> GPS extension is incompatible with the latest gpsd library releases.
> Causing build error
> {code:java}
> [ 47%] Building CXX object
> extensions/gps/CMakeFiles/minifi-gps.dir/GetGPS.cpp.o
> ./nifi-minifi-cpp/extensions/gps/GetGPS.cpp: In member function ‘virtual void
> org::apache::nifi::minifi::processors::GetGPS::onTrigger(const
> std::shared_ptr&, const
> std::shared_ptr&)’:
> ./nifi-minifi-cpp/extensions/gps/GetGPS.cpp:109:22: error: ‘struct
> gps_data_t’ has no member named ‘status’{code}
> This is due to that gpsd 3.21 (released in 2020-08-04) changed gps_data_t
> struct.
> https://gitlab.com/gpsd/gpsd/commit/29991d6ffeb41ecfc8297db68bb68be0128c8514
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1354) Fix minor errors found by the address sanitizer
[ https://issues.apache.org/jira/browse/MINIFICPP-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1354: --- Status: Patch Available (was: Open) > Fix minor errors found by the address sanitizer > --- > > Key: MINIFICPP-1354 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1354 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Ferenc Gerlits >Assignee: Martin Zink >Priority: Minor > Labels: MiNiFi-CPP-Hygiene > Fix For: 0.10.0 > > > If we run gcc's (version 8.4.0) address sanitizer on the unit tests, it finds > one "mismatched malloc vs operator delete" error in nanofi (triggered from 4 > unit tests), and 49 memory leaks, which all seem to be in the unit tests. > I don't think any of these errors affect production minifi, but still we > should fix them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFICPP-1354) Fix minor errors found by the address sanitizer
[ https://issues.apache.org/jira/browse/MINIFICPP-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306073#comment-17306073 ] Martin Zink commented on MINIFICPP-1354: Provided fix for most of these errors in [https://github.com/apache/nifi-minifi-cpp/pull/1035] (mainly they were just missing deallocations in the tests, but there were a couple of issues which were present in the production code as well) The ones remaining come from https://issues.apache.org/jira/browse/MINIFICPP-839 which is a separate issue which needs more substantial changes. > Fix minor errors found by the address sanitizer > --- > > Key: MINIFICPP-1354 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1354 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Ferenc Gerlits >Assignee: Martin Zink >Priority: Minor > Labels: MiNiFi-CPP-Hygiene > Fix For: 0.10.0 > > > If we run gcc's (version 8.4.0) address sanitizer on the unit tests, it finds > one "mismatched malloc vs operator delete" error in nanofi (triggered from 4 > unit tests), and 49 memory leaks, which all seem to be in the unit tests. > I don't think any of these errors affect production minifi, but still we > should fix them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1036: MINIFICPP-1352 Fix warnings in JNI and Sensors extensions
lordgamez opened a new pull request #1036: URL: https://github.com/apache/nifi-minifi-cpp/pull/1036 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1479) Rework integration tests for HashContent
[
https://issues.apache.org/jira/browse/MINIFICPP-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amina Dinari updated MINIFICPP-1479:
Description:
*Background:*
HashContent is expected to add the hash value of the content of a flowfile to a
flowfile attribute. Currently, neither the unit tests tests for the presence of
such an attribute. The unit tests seem to test for logging and the docker based
integration test only test for the flowfile content being unmodified. The
actual expected behaviour of appending attributes currently seems untested.
*Proposal:*
We should create proper integration tests for HashContent:
{code:python|title=Example feature definition}
Scenario: HashContent adds hash attribute to flowfiles
Given a GetFile processor with the "Input Directory" property set to
"/tmp/input"
And a file with the content is present in "/tmp/input"
And a HashContent processor with the "Hash Attribute" property set to "hash"
And the "Hash Algorithm" of the HashContent processor is set to
""
And a PutFile processor with the "Directory" property set to "/tmp/output"
And the "success" relationship of the GetFile processor is connected to the
HashContent
And the "success" relationship of the HashContent processor is connected to
the PutFile
When the MiNiFi instance starts up
Then a flowfile with the content is placed in the monitored
directory in less than 10 seconds
And the flowfile has an attribute called "hash" set to
Examples:
| content | hash_algorithm | hash_value
|
| "test" | MD5| 098f6bcd4621d373cade4e832627b4f6
|
| "test" | SHA1 | a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
|
| "test" | SHA256 |
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 |
| "coffee" | MD5| 24eb05d18318ac2db8b2b959315d10f2
|
| "coffee" | SHA1 | 44213f9f4d59b557314fadcd233232eebcac8012
|
| "coffee" | SHA256 |
37290d74ac4d186e3a8e5785d259d2ec04fac91ae28092e7620ec8bc99e830aa |
{code}
Details:
1. In HashContentTest.cpp, HashAlgorithm property was being checked at a point
where FailOnEmpty had to be supposedly checked; thereby giving incorrect
results for the test.
was:
*Background:*
HashContent is expected to add the hash value of the content of a flowfile to a
flowfile attribute. Currently, neither the unit tests tests for the presence of
such an attribute. The unit tests seem to test for logging and the docker based
integration test only test for the flowfile content being unmodified. The
actual expected behaviour of appending attributes currently seems untested.
*Proposal:*
We should create proper integration tests for HashContent:
{code:python|title=Example feature definition}
Scenario: HashContent adds hash attribute to flowfiles
Given a GetFile processor with the "Input Directory" property set to
"/tmp/input"
And a file with the content is present in "/tmp/input"
And a HashContent processor with the "Hash Attribute" property set to "hash"
And the "Hash Algorithm" of the HashContent processor is set to
""
And a PutFile processor with the "Directory" property set to "/tmp/output"
And the "success" relationship of the GetFile processor is connected to the
HashContent
And the "success" relationship of the HashContent processor is connected to
the PutFile
When the MiNiFi instance starts up
Then a flowfile with the content is placed in the monitored
directory in less than 10 seconds
And the flowfile has an attribute called "hash" set to
Examples:
| content | hash_algorithm | hash_value
|
| "test" | MD5| 098f6bcd4621d373cade4e832627b4f6
|
| "test" | SHA1 | a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
|
| "test" | SHA256 |
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 |
| "coffee" | MD5| 24eb05d18318ac2db8b2b959315d10f2
|
| "coffee" | SHA1 | 44213f9f4d59b557314fadcd233232eebcac8012
|
| "coffee" | SHA256 |
37290d74ac4d186e3a8e5785d259d2ec04fac91ae28092e7620ec8bc99e830aa |
{code}
> Rework integration tests for HashContent
>
>
> Key: MINIFICPP-1479
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1479
> Project: Apache NiFi MiNiFi C++
> Issue Type: Task
>Affects Versions: 0.7.0
>Reporter: Adam Hunyadi
>Assignee: Amina Dinari
>Priority: Minor
>
[jira] [Assigned] (MINIFICPP-1479) Rework integration tests for HashContent
[
https://issues.apache.org/jira/browse/MINIFICPP-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amina Dinari reassigned MINIFICPP-1479:
---
Assignee: Amina Dinari
> Rework integration tests for HashContent
>
>
> Key: MINIFICPP-1479
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1479
> Project: Apache NiFi MiNiFi C++
> Issue Type: Task
>Affects Versions: 0.7.0
>Reporter: Adam Hunyadi
>Assignee: Amina Dinari
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
> Fix For: 1.0.0
>
>
> *Background:*
> HashContent is expected to add the hash value of the content of a flowfile to
> a flowfile attribute. Currently, neither the unit tests tests for the
> presence of such an attribute. The unit tests seem to test for logging and
> the docker based integration test only test for the flowfile content being
> unmodified. The actual expected behaviour of appending attributes currently
> seems untested.
> *Proposal:*
> We should create proper integration tests for HashContent:
> {code:python|title=Example feature definition}
> Scenario: HashContent adds hash attribute to flowfiles
> Given a GetFile processor with the "Input Directory" property set to
> "/tmp/input"
> And a file with the content is present in "/tmp/input"
> And a HashContent processor with the "Hash Attribute" property set to
> "hash"
> And the "Hash Algorithm" of the HashContent processor is set to
> ""
> And a PutFile processor with the "Directory" property set to "/tmp/output"
> And the "success" relationship of the GetFile processor is connected to
> the HashContent
> And the "success" relationship of the HashContent processor is connected
> to the PutFile
> When the MiNiFi instance starts up
> Then a flowfile with the content is placed in the monitored
> directory in less than 10 seconds
> And the flowfile has an attribute called "hash" set to
> Examples:
>| content | hash_algorithm | hash_value
> |
>| "test" | MD5| 098f6bcd4621d373cade4e832627b4f6
> |
>| "test" | SHA1 | a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
> |
>| "test" | SHA256 |
> 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 |
>| "coffee" | MD5| 24eb05d18318ac2db8b2b959315d10f2
> |
>| "coffee" | SHA1 | 44213f9f4d59b557314fadcd233232eebcac8012
> |
>| "coffee" | SHA256 |
> 37290d74ac4d186e3a8e5785d259d2ec04fac91ae28092e7620ec8bc99e830aa |
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8339) Input Threads get Interrupted and stuck indefinitely
[
https://issues.apache.org/jira/browse/NIFI-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305944#comment-17305944
]
Rene Weidlinger commented on NIFI-8339:
---
Hey Mark, we see it permanently now. I've added the Threaddump.
df -h outputs 21% used on repo disk now.
i disabled repo archive for now, but didn't helped.
> Input Threads get Interrupted and stuck indefinitely
>
>
> Key: NIFI-8339
> URL: https://issues.apache.org/jira/browse/NIFI-8339
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Rene Weidlinger
>Priority: Major
> Attachments: firefox_Yf6NUeQe5X.png, td1.txt
>
>
> After some seconds we see this stack trace in nifi on one of our inputs:
> {noformat}
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18]
> o.a.nifi.web.api.ApplicationResource Unexpected exception occurred.
> portId=c4d93fb6-5e5b-1382-b39b-66fbc04660f0
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18]
> o.a.nifi.web.api.ApplicationResource Exception detail:
> org.apache.nifi.processor.exception.ProcessException:
> org.apache.nifi.processor.exception.ProcessException: Interrupted while
> waiting for site-to-site request to be serviced
> at
> org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:588)
> at
> org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277)
> at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> at
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
[jira] [Updated] (NIFI-8339) Input Threads get Interrupted and stuck indefinitely
[
https://issues.apache.org/jira/browse/NIFI-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rene Weidlinger updated NIFI-8339:
--
Attachment: td1.txt
> Input Threads get Interrupted and stuck indefinitely
>
>
> Key: NIFI-8339
> URL: https://issues.apache.org/jira/browse/NIFI-8339
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Rene Weidlinger
>Priority: Major
> Attachments: firefox_Yf6NUeQe5X.png, td1.txt
>
>
> After some seconds we see this stack trace in nifi on one of our inputs:
> {noformat}
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18]
> o.a.nifi.web.api.ApplicationResource Unexpected exception occurred.
> portId=c4d93fb6-5e5b-1382-b39b-66fbc04660f0
> 2021-03-18 07:33:34,703 ERROR [NiFi Web Server-18]
> o.a.nifi.web.api.ApplicationResource Exception detail:
> org.apache.nifi.processor.exception.ProcessException:
> org.apache.nifi.processor.exception.ProcessException: Interrupted while
> waiting for site-to-site request to be serviced
> at
> org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:588)
> at
> org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277)
> at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> at
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
>
