[jira] [Updated] (NIFI-5912) Add a GCP User Group Provider for GSuite
[ https://issues.apache.org/jira/browse/NIFI-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5912: - Status: Patch Available (was: Open) > Add a GCP User Group Provider for GSuite > > > Key: NIFI-5912 > URL: https://issues.apache.org/jira/browse/NIFI-5912 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In order to better integrate NiFi on the Google Cloud Platform, it'd be > useful to have a GCP User Group Provider using the Google Admin SDK and the > Directory API: it would allow syncing the users and groups of a GSuite > organization. In combination with the OIDC authentication mechanism, it > improves the user experience when running NiFi on GCP (Compute Engine and/or > GKE). > For reference - > https://developers.google.com/admin-sdk/directory/v1/get-start/getting-started -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pvillard31 opened a new pull request #3230: NIFI-5912 - Add a GCP User Group Provider for GSuite
pvillard31 opened a new pull request #3230: NIFI-5912 - Add a GCP User Group Provider for GSuite URL: https://github.com/apache/nifi/pull/3230 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-5912) Add a GCP User Group Provider for GSuite
Pierre Villard created NIFI-5912: Summary: Add a GCP User Group Provider for GSuite Key: NIFI-5912 URL: https://issues.apache.org/jira/browse/NIFI-5912 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Pierre Villard Assignee: Pierre Villard In order to better integrate NiFi on the Google Cloud Platform, it'd be useful to have a GCP User Group Provider using the Google Admin SDK and the Directory API: it would allow syncing the users and groups of a GSuite organization. In combination with the OIDC authentication mechanism, it improves the user experience when running NiFi on GCP (Compute Engine and/or GKE). For reference - https://developers.google.com/admin-sdk/directory/v1/get-start/getting-started -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] SavtechSolutions commented on a change in pull request #3203: NIFI-5871 ignore UUID attribute when copying flow file attributes
SavtechSolutions commented on a change in pull request #3203: NIFI-5871 ignore
UUID attribute when copying flow file attributes
URL: https://github.com/apache/nifi/pull/3203#discussion_r243079211
##
File path:
nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
##
@@ -268,7 +268,7 @@ public void testInputFlowFileUuid() throws Exception {
ArgumentCaptor capture =
ArgumentCaptor.forClass(RawJsonDocument.class);
verify(bucket, times(1)).upsert(capture.capture(), eq(PersistTo.NONE),
eq(ReplicateTo.NONE));
-assertEquals(uuid, capture.getValue().id());
+// assertEquals(uuid, capture.getValue().id());
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] patricker commented on a change in pull request #3203: NIFI-5871 ignore UUID attribute when copying flow file attributes
patricker commented on a change in pull request #3203: NIFI-5871 ignore UUID
attribute when copying flow file attributes
URL: https://github.com/apache/nifi/pull/3203#discussion_r243071420
##
File path:
nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
##
@@ -268,7 +268,7 @@ public void testInputFlowFileUuid() throws Exception {
ArgumentCaptor capture =
ArgumentCaptor.forClass(RawJsonDocument.class);
verify(bucket, times(1)).upsert(capture.capture(), eq(PersistTo.NONE),
eq(ReplicateTo.NONE));
-assertEquals(uuid, capture.getValue().id());
+// assertEquals(uuid, capture.getValue().id());
Review comment:
No, I want you to completely delete the line of code instead of commenting
it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SavtechSolutions commented on a change in pull request #3203: NIFI-5871 ignore UUID attribute when copying flow file attributes
SavtechSolutions commented on a change in pull request #3203: NIFI-5871 ignore
UUID attribute when copying flow file attributes
URL: https://github.com/apache/nifi/pull/3203#discussion_r243066679
##
File path:
nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
##
@@ -268,7 +268,7 @@ public void testInputFlowFileUuid() throws Exception {
ArgumentCaptor capture =
ArgumentCaptor.forClass(RawJsonDocument.class);
verify(bucket, times(1)).upsert(capture.capture(), eq(PersistTo.NONE),
eq(ReplicateTo.NONE));
-assertEquals(uuid, capture.getValue().id());
+// assertEquals(uuid, capture.getValue().id());
Review comment:
Do you want me to revert the changes made to the unit tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-5894) FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null
[ https://issues.apache.org/jira/browse/NIFI-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Ulicny updated NIFI-5894: -- Status: Patch Available (was: In Progress) > FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null > -- > > Key: NIFI-5894 > URL: https://issues.apache.org/jira/browse/NIFI-5894 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 >Reporter: henning ottesen >Assignee: Eric Ulicny >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > 2018-12-13 10:34:07,405 WARN [Timer-Driven Process Thread-2] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > ListFTP[id=a6cee9c8-0167-1000-2bc2-5e69d2e33af5] due to uncaught Exception: > java.lang.NullPointerException java.lang.NullPointerException: null > at > org.apache.nifi.processors.standard.util.FTPTransfer.newFileInfo(FTPTransfer.java:305) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:270) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:260) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:471) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:413) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] patricker commented on a change in pull request #3203: NIFI-5871 ignore UUID attribute when copying flow file attributes
patricker commented on a change in pull request #3203: NIFI-5871 ignore UUID
attribute when copying flow file attributes
URL: https://github.com/apache/nifi/pull/3203#discussion_r243059052
##
File path:
nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
##
@@ -268,7 +268,7 @@ public void testInputFlowFileUuid() throws Exception {
ArgumentCaptor capture =
ArgumentCaptor.forClass(RawJsonDocument.class);
verify(bucket, times(1)).upsert(capture.capture(), eq(PersistTo.NONE),
eq(ReplicateTo.NONE));
-assertEquals(uuid, capture.getValue().id());
+// assertEquals(uuid, capture.getValue().id());
Review comment:
If you drop all of these commented lines, I'll go ahead and merge this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] patricker commented on issue #3091: NIFI-5722 Expose Penalty Remaining Duration
patricker commented on issue #3091: NIFI-5722 Expose Penalty Remaining Duration URL: https://github.com/apache/nifi/pull/3091#issuecomment-448727060 Rebased to latest, ready for review anytime. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-5911) Create Integration Test Framework for Flow Developers
Kyle Sykes created NIFI-5911: Summary: Create Integration Test Framework for Flow Developers Key: NIFI-5911 URL: https://issues.apache.org/jira/browse/NIFI-5911 Project: Apache NiFi Issue Type: Wish Components: Tools and Build Reporter: Kyle Sykes Attachments: nifi_integrated_testing_sample.png (Encouraged to post this from the Nifi Slack) As a flow developer, i would like to be able to test my flows using a integrated testing approach, rather than unit testing each individual processor. I currently can guarantee a single processor will work as intended, but making a change early on in a flow can lead to unanticipated consequences later on as the flow file gets passed off to other flows for processing. The basic functionality I'm looking for is generating a series of test flow files to insert into my pipeline at some step and be able to insert test checks (akin to assert statements) throughout my flow that are only executed with the test flow files. Currently, I can implement this using the following workflow (see attached file for something similar I made a while back to proof of concept the idea): # Generate a flow file with a boolean attribute `testFlowFile`. # Connect the `GenerateFlowFile` processor into a flow where I want to run some test cases on # Insert `RouteOnAttribute` FlowFiles that route all test cases to a group of processors that holds a set of processors that asserts things about the attributes or content, adding an `testFailed` attribute if any of the test cases fail # If `testFailed` is true, then route the FlowFile to a "Failed" location for further examination by me # If `testFailed` is false, then route the FlowFile to an Output Port which inserts it back into the flow to continue down the line. # Insert `RouteOnAttribute` processors to prevent the test FlowFiles from inserting the test cases into databases or stop the testing for that particular flow. The above causes a large number of extra processors to be created and generally makes the flows more difficult to understand. My work has me handing of flows to a client and I like to keep things as understandable as possible and it can get a little messy. I envision several ways this could work as an integrated part of Nifi, so I'll give some thoughts and leave it to the contributors to determine what's actually feasible/useful. # A `GenerateTestFile` processor that generates a flow file that internally is marked as a test FlowFile (probably by an attribute), allowing someone to set both the attributes and content. The content could be either text (in the case of JSON) or an external file to be loaded (in cases of binary formats, image, video, etc). ## This could also extend to a processor that loads many test cases at once to test specific scenarios. Giving a useful name to the test case would also be nice for debugging. ## Allow for the possibility that you anticipate that a flowfile is expected to fail the test case (xfail type situations)? I don't have an immediate use case, but it's worth considering if it's something that needs to be implemented. # Processors specifically designed to create a wide variety of test cases and store information as to why it failed somewhere on the flow file (a stack trace of errors if applicable, or just pointing at which test case failed). Could be something like `TestAttribute` or `TestContent` or `TestExecuteScript`. Some thought could be given to how to best setup test cases in general. Perhaps the work around I use above isn't the best approach but I'm sure people more familiar with testing in general might have better approaches. # The ability to mark processors as "Non Testable", specifically things like database connections, adding to queues, some output ports to other flows you don't necessarily want to pass a test FlowFile onto (maybe set some/all as Non Testable by default?). Similar to how you can right click a processor and begin tracking it with the registry, I was envisioning something along the lines of "Allow Processing of Test FlowFiles" # Perhaps have some way to automate runs of a test suite and monitor the results visually, with an overview of what test cases failed (by name) ## Allow for the situation where a developer might desire a CI workflow to develop and promote their flows, and have the ability to support the workflow where someone can create an isolated testing environment (using Docker to load nifi), load the flow to be tested, then run the test suite and report back the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] patricker commented on issue #3188: NIFI-5829 Create Lookup Controller Services for RecordSetWriter and R…
patricker commented on issue #3188: NIFI-5829 Create Lookup Controller Services for RecordSetWriter and R… URL: https://github.com/apache/nifi/pull/3188#issuecomment-448714216 @mattyb149 @ijokarumawak, fixed a conflict that had showed up since my initial PR creation. Happy to make any further updates as needed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5894) FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null
[ https://issues.apache.org/jira/browse/NIFI-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725288#comment-16725288 ] Eric Ulicny commented on NIFI-5894: --- Set LastModifiedTime == System time if the circumstance occurs where it is null from the FtpFile. > FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null > -- > > Key: NIFI-5894 > URL: https://issues.apache.org/jira/browse/NIFI-5894 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 >Reporter: henning ottesen >Assignee: Eric Ulicny >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > 2018-12-13 10:34:07,405 WARN [Timer-Driven Process Thread-2] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > ListFTP[id=a6cee9c8-0167-1000-2bc2-5e69d2e33af5] due to uncaught Exception: > java.lang.NullPointerException java.lang.NullPointerException: null > at > org.apache.nifi.processors.standard.util.FTPTransfer.newFileInfo(FTPTransfer.java:305) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:270) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:260) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:471) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:413) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Xsixteen opened a new pull request #3229: NIFI-5894 FTPTransfer NullPointerException
Xsixteen opened a new pull request #3229: NIFI-5894 FTPTransfer NullPointerException URL: https://github.com/apache/nifi/pull/3229 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] patricker opened a new pull request #3228: NIFI-5897 Update GetFile and ListFile Min Age to handle future dated …
patricker opened a new pull request #3228: NIFI-5897 Update GetFile and ListFile Min Age to handle future dated … URL: https://github.com/apache/nifi/pull/3228 …files ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (NIFI-5894) FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null
[ https://issues.apache.org/jira/browse/NIFI-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Ulicny reassigned NIFI-5894: - Assignee: Eric Ulicny > FTPTransfer NullPointerException - FTPFile.getTimestamp() returns null > -- > > Key: NIFI-5894 > URL: https://issues.apache.org/jira/browse/NIFI-5894 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 >Reporter: henning ottesen >Assignee: Eric Ulicny >Priority: Major > > 2018-12-13 10:34:07,405 WARN [Timer-Driven Process Thread-2] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > ListFTP[id=a6cee9c8-0167-1000-2bc2-5e69d2e33af5] due to uncaught Exception: > java.lang.NullPointerException java.lang.NullPointerException: null > at > org.apache.nifi.processors.standard.util.FTPTransfer.newFileInfo(FTPTransfer.java:305) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:270) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:260) > at > org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:471) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:413) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725153#comment-16725153
]
Kevin Hill commented on NIFI-1893:
--
Great [~Honda] I'm looking forward to it, as I my next todo involves validating
some json-schema speced inputs.
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
> Attachments: image-2018-12-07-18-02-52-813.png
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] SavtechSolutions commented on issue #3227: NIFI-5909
SavtechSolutions commented on issue #3227: NIFI-5909 URL: https://github.com/apache/nifi/pull/3227#issuecomment-448653658 No unit tests have been added/modified, as existing tests verify only the URL used to call ES, but not the content. I'm thinking of adding a callback hook to the realRequest inside PutElasticsearchHttpRecordTestProcessor.createElasticsearchClient() to do any custom content assertions This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] SavtechSolutions opened a new pull request #3227: NIFI-5909
SavtechSolutions opened a new pull request #3227: NIFI-5909 URL: https://github.com/apache/nifi/pull/3227 added optional settings for date, time, and timestamp formats used to write Records to Elasticsearch Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725131#comment-16725131 ] Otto Fowler commented on NIFI-5892: --- Ok, just to sum up: removing the timestamp alone is what is required, everything else can work as documented? > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5910) Upgrading versioned process group overwrites external service reference
Bryan Bende created NIFI-5910: - Summary: Upgrading versioned process group overwrites external service reference Key: NIFI-5910 URL: https://issues.apache.org/jira/browse/NIFI-5910 Project: Apache NiFi Issue Type: Bug Reporter: Bryan Bende Using two NiFi instances and one NiFi registry... 1) In NiFi 1: * Create a controller service at the root level * Create a process group with a processor that uses the service from the root level * Start version control 2) In NiFi 2: * Import the flow and see the processor has an invalid service reference * Create a service at root level and change processor to use that service * Notice no local changes to commit, this is correct 3) In NiFi 1: * Make a change to the processor, just move its position * Save version 2 of the flow to registry 4) In NiFi 2: * Upgrade the process group to version 2 * Notice the service reference of the processor is reset back to the invalid reference If you did the same steps, but in step 3 you modify something else besides that processor, then when upgrading in NiFi 2 it retains the correct service reference. So it is only when the upgrade has changes to the processor with the external service reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] MikeThomsen commented on issue #3226: NIFI-5908 Inject locale info into JsonInjector object to make sure we…
MikeThomsen commented on issue #3226: NIFI-5908 Inject locale info into JsonInjector object to make sure we… URL: https://github.com/apache/nifi/pull/3226#issuecomment-448608638 @pvillard31 @markap14 @trixpan how did y'all test the ISPEnrich processor? Did someone buy access to the maxmind database for that data? Asking because I just did GeoEnrichIPRecord and am thinking about doing a record enrichment option for ISP data. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-5909) PutElasticsearchHttpRecord doesn't allow to customize the timestamp format
Alex Savitsky created NIFI-5909: --- Summary: PutElasticsearchHttpRecord doesn't allow to customize the timestamp format Key: NIFI-5909 URL: https://issues.apache.org/jira/browse/NIFI-5909 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.8.0 Reporter: Alex Savitsky All timestamps are sent to Elasticsearch in the "-MM-dd HH:mm:ss" format, coming from the RecordFieldType.TIMESTAMP.getDefaultFormat(). There's plenty of use cases that call for Elasticsearch data to be presented differently, and the format should be customizable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pvillard31 commented on issue #3226: NIFI-5908 Inject locale info into JsonInjector object to make sure we…
pvillard31 commented on issue #3226: NIFI-5908 Inject locale info into JsonInjector object to make sure we… URL: https://github.com/apache/nifi/pull/3226#issuecomment-448594952 I'm a +1, giving a chance to @markap14 to also have a look, otherwise will merge later This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-5893) Wrong region id is being used for custom region in AbstractAWSProcessor
[ https://issues.apache.org/jira/browse/NIFI-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724844#comment-16724844 ] ASF subversion and git services commented on NIFI-5893: --- Commit f22a6c46ad05289f4211186a8f5854b5413da2d3 in nifi's branch refs/heads/master from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f22a6c4 ] Merge pull request #3219 from zenfenan/NIFI-5893 NIFI-5893: AWS Endpoint Overriding now functions properly > Wrong region id is being used for custom region in AbstractAWSProcessor > --- > > Key: NIFI-5893 > URL: https://issues.apache.org/jira/browse/NIFI-5893 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 > Environment: RHEL >Reporter: Andrew Mcdonald >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.9.0 > > Attachments: > 0001-NIFI-5893-AWS-Endpoint-Overriding-now-functions-prop.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The Jira https://issues.apache.org/jira/browse/NIFI-5456 fixed the vpc but > for the simpler case of a custom region it is using the default region id and > fails with a similar error message. > Failed to receive messages from Amazon SQS due to > com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be > scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: > 403; Error Code: SignatureDoesNotMatch; Request ID: > x----). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5893) Wrong region id is being used for custom region in AbstractAWSProcessor
[ https://issues.apache.org/jira/browse/NIFI-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5893: - Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Wrong region id is being used for custom region in AbstractAWSProcessor > --- > > Key: NIFI-5893 > URL: https://issues.apache.org/jira/browse/NIFI-5893 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 > Environment: RHEL >Reporter: Andrew Mcdonald >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.9.0 > > Attachments: > 0001-NIFI-5893-AWS-Endpoint-Overriding-now-functions-prop.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The Jira https://issues.apache.org/jira/browse/NIFI-5456 fixed the vpc but > for the simpler case of a custom region it is using the default region id and > fails with a similar error message. > Failed to receive messages from Amazon SQS due to > com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be > scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: > 403; Error Code: SignatureDoesNotMatch; Request ID: > x----). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5893) Wrong region id is being used for custom region in AbstractAWSProcessor
[ https://issues.apache.org/jira/browse/NIFI-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724843#comment-16724843 ] ASF subversion and git services commented on NIFI-5893: --- Commit f22a6c46ad05289f4211186a8f5854b5413da2d3 in nifi's branch refs/heads/master from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f22a6c4 ] Merge pull request #3219 from zenfenan/NIFI-5893 NIFI-5893: AWS Endpoint Overriding now functions properly > Wrong region id is being used for custom region in AbstractAWSProcessor > --- > > Key: NIFI-5893 > URL: https://issues.apache.org/jira/browse/NIFI-5893 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 > Environment: RHEL >Reporter: Andrew Mcdonald >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.9.0 > > Attachments: > 0001-NIFI-5893-AWS-Endpoint-Overriding-now-functions-prop.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The Jira https://issues.apache.org/jira/browse/NIFI-5456 fixed the vpc but > for the simpler case of a custom region it is using the default region id and > fails with a similar error message. > Failed to receive messages from Amazon SQS due to > com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be > scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: > 403; Error Code: SignatureDoesNotMatch; Request ID: > x----). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5893) Wrong region id is being used for custom region in AbstractAWSProcessor
[ https://issues.apache.org/jira/browse/NIFI-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724842#comment-16724842 ] ASF subversion and git services commented on NIFI-5893: --- Commit ee24a593e984eabafdeb0b27c03367e67a4f26f1 in nifi's branch refs/heads/master from zenfenan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ee24a59 ] NIFI-5893: AWS Endpoint Overriding now functions properly > Wrong region id is being used for custom region in AbstractAWSProcessor > --- > > Key: NIFI-5893 > URL: https://issues.apache.org/jira/browse/NIFI-5893 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.8.0 > Environment: RHEL >Reporter: Andrew Mcdonald >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.9.0 > > Attachments: > 0001-NIFI-5893-AWS-Endpoint-Overriding-now-functions-prop.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The Jira https://issues.apache.org/jira/browse/NIFI-5456 fixed the vpc but > for the simpler case of a custom region it is using the default region id and > fails with a similar error message. > Failed to receive messages from Amazon SQS due to > com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be > scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: > 403; Error Code: SignatureDoesNotMatch; Request ID: > x----). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pvillard31 closed pull request #3219: NIFI-5893: AWS Endpoint Overriding now functions properly
pvillard31 closed pull request #3219: NIFI-5893: AWS Endpoint Overriding now
functions properly
URL: https://github.com/apache/nifi/pull/3219
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
index fe5ce62e60..06f4a16590 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
@@ -37,6 +37,8 @@
import java.util.List;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import javax.net.ssl.SSLContext;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.conn.ssl.DefaultHostnameVerifier;
@@ -283,13 +285,43 @@ protected void initializeRegionAndEndpoint(ProcessContext
context) {
// (per Amazon docs this should only be configured at client creation)
if (getSupportedPropertyDescriptors().contains(ENDPOINT_OVERRIDE)) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
+
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr, this.client.getServiceName(),
this.region.getName());
+
+if (urlstr.endsWith(".vpce.amazonaws.com")) {
+String region = parseRegionForVPCE(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), region);
+} else {
+this.client.setEndpoint(urlstr);
+}
}
}
}
+/*
+Note to developer(s):
+When setting an endpoint for an AWS Client i.e.
client.setEndpoint(endpointUrl),
+AWS Java SDK fails to parse the region correctly when the provided
endpoint
+is an AWS PrivateLink so this method does the job of parsing the
region name and
+returning it.
+
+Refer NIFI-5456 & NIFI-5893
+ */
+private String parseRegionForVPCE(String url) {
+int index = url.length() - ".vpce.amazonaws.com".length();
+
+Pattern VPCE_ENDPOINT_PATTERN =
Pattern.compile("^(?:.+[vpce-][a-z0-9-]+\\.)?([a-z0-9-]+)$");
+Matcher matcher = VPCE_ENDPOINT_PATTERN.matcher(url.substring(0,
index));
+
+if (matcher.matches()) {
+return matcher.group(1);
+} else {
+getLogger().warn("Unable to get a match with the VPCE endpoint
pattern; defaulting the region to us-east-1...");
+return "us-east-1";
+}
+}
+
/**
* Create client from the arguments
* @param context process context
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
