[jira] [Resolved] (NIFI-4628) Update ListS3 proccesor to support List Objects Version 2

[James Wing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Wing resolved NIFI-4628.
--
   Resolution: Fixed
Fix Version/s: 1.5.0

> Update ListS3 proccesor to support List Objects Version 2
> -
>
> Key: NIFI-4628
> URL: https://issues.apache.org/jira/browse/NIFI-4628
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Andrew Burkard
>Priority: Minor
> Fix For: 1.5.0
>
>
> AWS has revised the List Objects API: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> The main difference is the new API passes a continuation token in the 
> response for multi-page listings rather than requiring the client to keep 
> track of the lexicographically last key listed. The new version also appears 
> to be the only API supported going forward for new AWS services (e.g. 
> Snowball). It would be nice if the ListS3 processor supported either via an 
> attribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2300: NIFI-4628 Add support for ListS3Version2 API

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/2300
  
@aburkard Thanks again for putting this together.


---

[jira] [Commented] (NIFI-4628) Update ListS3 proccesor to support List Objects Version 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273984#comment-16273984
 ] 

ASF GitHub Bot commented on NIFI-4628:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/2300
  
@aburkard Thanks again for putting this together.


> Update ListS3 proccesor to support List Objects Version 2
> -
>
> Key: NIFI-4628
> URL: https://issues.apache.org/jira/browse/NIFI-4628
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Andrew Burkard
>Priority: Minor
>
> AWS has revised the List Objects API: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> The main difference is the new API passes a continuation token in the 
> response for multi-page listings rather than requiring the client to keep 
> track of the lexicographically last key listed. The new version also appears 
> to be the only API supported going forward for new AWS services (e.g. 
> Snowball). It would be nice if the ListS3 processor supported either via an 
> attribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4628) Update ListS3 proccesor to support List Objects Version 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273981#comment-16273981
 ] 

ASF GitHub Bot commented on NIFI-4628:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2300


> Update ListS3 proccesor to support List Objects Version 2
> -
>
> Key: NIFI-4628
> URL: https://issues.apache.org/jira/browse/NIFI-4628
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Andrew Burkard
>Priority: Minor
>
> AWS has revised the List Objects API: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> The main difference is the new API passes a continuation token in the 
> response for multi-page listings rather than requiring the client to keep 
> track of the lexicographically last key listed. The new version also appears 
> to be the only API supported going forward for new AWS services (e.g. 
> Snowball). It would be nice if the ListS3 processor supported either via an 
> attribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2300: NIFI-4628 Add support for ListS3Version2 API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2300


---

[jira] [Commented] (NIFI-4628) Update ListS3 proccesor to support List Objects Version 2

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273980#comment-16273980
 ] 

ASF subversion and git services commented on NIFI-4628:
---

Commit 103a345e859ef16088fd8a92ba40422d25213c56 in nifi's branch 
refs/heads/master from [~aburkard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=103a345 ]

NIFI-4628 Add support for ListS3Version2 API

Signed-off-by: James Wing 

This closes #2300.


> Update ListS3 proccesor to support List Objects Version 2
> -
>
> Key: NIFI-4628
> URL: https://issues.apache.org/jira/browse/NIFI-4628
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Andrew Burkard
>Priority: Minor
>
> AWS has revised the List Objects API: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> The main difference is the new API passes a continuation token in the 
> response for multi-page listings rather than requiring the client to keep 
> track of the lexicographically last key listed. The new version also appears 
> to be the only API supported going forward for new AWS services (e.g. 
> Snowball). It would be nice if the ListS3 processor supported either via an 
> attribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4256) Add support for all AWS S3 Encryption Options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273941#comment-16273941
 ] 

ASF GitHub Bot commented on NIFI-4256:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/2291
  
@baank - closed?


> Add support for all AWS S3 Encryption Options
> -
>
> Key: NIFI-4256
> URL: https://issues.apache.org/jira/browse/NIFI-4256
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Franco
>  Labels: aws, aws-s3, security
>
> NiFi currently only supports SSE-S3 encryption (AES256).
> Support needs to be added for:
> * SSE-S3
> * SSE-KMS
> * SSE-C
> * CSE-KMS CMK
> * CSE-Master Key
> With all of the appropriate configuration options and such that SSE is 
> available only for PutS3Object whilst CSE is available also for FetchS3Object.
> Given that this will add another 20 or so UI properties the intention is to 
> split it into a Client Side Encryption Service and Server Side Encryption 
> Service. This will allow users to reuse "encryption" across different 
> workflows.
> Existing flows using the Server Side Encryption option will still work as is 
> but will be overridden if a service is added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2291: NIFI-4256 - Add support for all AWS S3 Encryption Options

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/2291
  
@baank - closed?


---

[jira] [Commented] (NIFI-4650) Add Hadoop HttpCore property independent of the HttpClient property

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273557#comment-16273557
 ] 

Joseph Witt commented on NIFI-4650:
---

+1 will merge to master shortly - waiting for contrib check build to complete.

> Add Hadoop HttpCore property independent of the HttpClient property
> ---
>
> Key: NIFI-4650
> URL: https://issues.apache.org/jira/browse/NIFI-4650
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>Priority: Minor
> Fix For: 1.5.0
>
>
> Apache HttpComponents versions HttpClient and HttpCore independently (e.g., 
> 4.5.2 and 4.4.4 used in Hadoop 2.8.x) so overriding 
> hadoop.http.client.version with certain versions of HttpClient or HttpCore 
> isn't currently possible. We should add a separate hadoop.http.core.version 
> to pom.xml and nifi-nar-bundles/nifi-hadoop-libraries-bundle/pom.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4650) Add Hadoop HttpCore property independent of the HttpClient property

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4650:
--
Fix Version/s: 1.5.0

> Add Hadoop HttpCore property independent of the HttpClient property
> ---
>
> Key: NIFI-4650
> URL: https://issues.apache.org/jira/browse/NIFI-4650
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>Priority: Minor
> Fix For: 1.5.0
>
>
> Apache HttpComponents versions HttpClient and HttpCore independently (e.g., 
> 4.5.2 and 4.4.4 used in Hadoop 2.8.x) so overriding 
> hadoop.http.client.version with certain versions of HttpClient or HttpCore 
> isn't currently possible. We should add a separate hadoop.http.core.version 
> to pom.xml and nifi-nar-bundles/nifi-hadoop-libraries-bundle/pom.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4645) NiFi build on powerpc fails because the google/rpc nar cannot be built/used there

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273540#comment-16273540
 ] 

Joseph Witt commented on NIFI-4645:
---

[~mcgilman] can you give it another look?

https://github.com/apache/nifi/pull/2309

I made it more explicit and corrected the logic.  It now builds on non ppc 
systems so I think it is more in the right direction.  Will advise results in 
the ppc environment too.

> NiFi build on powerpc fails because the google/rpc nar cannot be built/used 
> there
> -
>
> Key: NIFI-4645
> URL: https://issues.apache.org/jira/browse/NIFI-4645
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
> Environment: ibm/power pc
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
>
> Going to update the poms to detect the os family and if not powerpc will 
> activate module builds and assembly usage of nars that are not capable in 
> that environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4645) NiFi build on powerpc fails because the google/rpc nar cannot be built/used there

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273536#comment-16273536
 ] 

ASF GitHub Bot commented on NIFI-4645:
--

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/2309

NIFI-4645 now being very specific to include if not ppc

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-4645-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2309.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2309


commit c9e4c3b630326b419dd02d62793a2d0adb29c90d
Author: joewitt 
Date:   2017-11-30T18:37:01Z

NIFI-4645 now being very specific to include if not ppc




> NiFi build on powerpc fails because the google/rpc nar cannot be built/used 
> there
> -
>
> Key: NIFI-4645
> URL: https://issues.apache.org/jira/browse/NIFI-4645
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
> Environment: ibm/power pc
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
>
> Going to update the poms to detect the os family and if not powerpc will 
> activate module builds and assembly usage of nars that are not capable in 
> that environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2309: NIFI-4645 now being very specific to include if not...

[joewitt]

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/2309

NIFI-4645 now being very specific to include if not ppc

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-4645-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2309.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2309


commit c9e4c3b630326b419dd02d62793a2d0adb29c90d
Author: joewitt 
Date:   2017-11-30T18:37:01Z

NIFI-4645 now being very specific to include if not ppc




---

[jira] [Assigned] (NIFI-4477) Create a Docker Image quickstart

[Aldrin Piri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aldrin Piri reassigned NIFI-4477:
-

Assignee: Aldrin Piri

> Create a Docker Image quickstart
> 
>
> Key: NIFI-4477
> URL: https://issues.apache.org/jira/browse/NIFI-4477
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Docker, Documentation & Website
>Reporter: Aldrin Piri
>Assignee: Aldrin Piri
>
> Let's put the doc in docker for NiFi.  The idea would be to talk about 
> running the Docker image.  This may be best targeted toward a wiki page.  
> This is minimal for the current state, but with additional changes to provide 
> greater configurability, will have greater details on configuring key 
> scenarios.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4642) Unstable nifi core build due to validate process scheduler test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273309#comment-16273309
 ] 

ASF GitHub Bot commented on NIFI-4642:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2303
  
All looks good. +1 merged to master.


> Unstable nifi core build due to validate process scheduler test
> ---
>
> Key: NIFI-4642
> URL: https://issues.apache.org/jira/browse/NIFI-4642
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
> Attachments: 
> 0001-NIFI-4642-updated-tests-to-be-more-flexible-in-timin.patch
>
>
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 34.747 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 3.147 sec  <<< FAILURE!
> java.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(TestProcessorLifecycle.java:372)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4642) Unstable nifi core build due to validate process scheduler test

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4642:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Unstable nifi core build due to validate process scheduler test
> ---
>
> Key: NIFI-4642
> URL: https://issues.apache.org/jira/browse/NIFI-4642
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
> Attachments: 
> 0001-NIFI-4642-updated-tests-to-be-more-flexible-in-timin.patch
>
>
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 34.747 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 3.147 sec  <<< FAILURE!
> java.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(TestProcessorLifecycle.java:372)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4642) Unstable nifi core build due to validate process scheduler test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273310#comment-16273310
 ] 

ASF GitHub Bot commented on NIFI-4642:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2303


> Unstable nifi core build due to validate process scheduler test
> ---
>
> Key: NIFI-4642
> URL: https://issues.apache.org/jira/browse/NIFI-4642
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
> Attachments: 
> 0001-NIFI-4642-updated-tests-to-be-more-flexible-in-timin.patch
>
>
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 34.747 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 3.147 sec  <<< FAILURE!
> java.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(TestProcessorLifecycle.java:372)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4642) Unstable nifi core build due to validate process scheduler test

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273308#comment-16273308
 ] 

ASF subversion and git services commented on NIFI-4642:
---

Commit dd981e87ddadc907590a1d89ef3925f668cb00c1 in nifi's branch 
refs/heads/master from [~joewitt]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=dd981e8 ]

NIFI-4642 updated tests to be more tolerant/variable to different system 
speeds.  Many of these should be integration tests and not unit tests. This 
closes #2303.


> Unstable nifi core build due to validate process scheduler test
> ---
>
> Key: NIFI-4642
> URL: https://issues.apache.org/jira/browse/NIFI-4642
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
> Attachments: 
> 0001-NIFI-4642-updated-tests-to-be-more-flexible-in-timin.patch
>
>
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 34.747 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 3.147 sec  <<< FAILURE!
> java.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateProcessScheduledAfterAdministrativeDelayDueToTheOnScheduledException(TestProcessorLifecycle.java:372)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2303: NIFI-4642 updated tests to be more tolerant/variable to di...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2303
  
All looks good. +1 merged to master.


---

[GitHub] nifi pull request #2303: NIFI-4642 updated tests to be more tolerant/variabl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2303


---

[jira] [Commented] (NIFI-4650) Add Hadoop HttpCore property independent of the HttpClient property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273289#comment-16273289
 ] 

ASF GitHub Bot commented on NIFI-4650:
--

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2308
  
FWIW, the tests pass on the following build as well:

```sh
$ mvn clean install -Dhadoop.version=2.8.2 -Dhadoop.guava.version=12.0.1 
-Dhadoop.http.client.version=4.5.2 -Dhadoop.http.core.version=4.4.4 
-Pcontrib-check
```


> Add Hadoop HttpCore property independent of the HttpClient property
> ---
>
> Key: NIFI-4650
> URL: https://issues.apache.org/jira/browse/NIFI-4650
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>Priority: Minor
>
> Apache HttpComponents versions HttpClient and HttpCore independently (e.g., 
> 4.5.2 and 4.4.4 used in Hadoop 2.8.x) so overriding 
> hadoop.http.client.version with certain versions of HttpClient or HttpCore 
> isn't currently possible. We should add a separate hadoop.http.core.version 
> to pom.xml and nifi-nar-bundles/nifi-hadoop-libraries-bundle/pom.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2308: NIFI-4650 Added independent hadoop.http.core.version prope...

[jfrazee]

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2308
  
FWIW, the tests pass on the following build as well:

```sh
$ mvn clean install -Dhadoop.version=2.8.2 -Dhadoop.guava.version=12.0.1 
-Dhadoop.http.client.version=4.5.2 -Dhadoop.http.core.version=4.4.4 
-Pcontrib-check
```


---

[jira] [Created] (NIFIREG-62) Minor updates to flow diff module

[Mark Payne (JIRA)]

Mark Payne created NIFIREG-62:
-

 Summary: Minor updates to flow diff module
 Key: NIFIREG-62
 URL: https://issues.apache.org/jira/browse/NIFIREG-62
 Project: NiFi Registry
  Issue Type: Bug
Reporter: Mark Payne
Assignee: Mark Payne






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFIREG-61) Add support for encrypted properties in configuration files

[Kevin Doran (JIRA)]

Kevin Doran created NIFIREG-61:
--

 Summary: Add support for encrypted properties in configuration 
files
 Key: NIFIREG-61
 URL: https://issues.apache.org/jira/browse/NIFIREG-61
 Project: NiFi Registry
  Issue Type: New Feature
Reporter: Kevin Doran
Assignee: Kevin Doran


The NiFi Registry server is configured by files on disk, e.g., 
nifi-registry.properties, bootstrap.conf, and XML files for loading extensions. 
Sometimes these files contain properties with sensitive values, such as 
credentials.

We want to be able to support encrypting property values in NiFi Registry 
configuration files on disk that get decrypted in memory at runtime. As an 
initial step, the decryption key will be specified in the bootstrap.conf file. 
In the future, it might be input to the bootstrap processes via other means.

For the design of this feature, the NiFi implementation of this capability 
should be used as a guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (MINIFICPP-328) Add user docs for MiNiFi - C++ processors

[marco polo (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273112#comment-16273112
 ] 

marco polo edited comment on MINIFICPP-328 at 11/30/17 6:23 PM:


[~achristianson] Thanks for adding/doing this. Let me know if you want to tag 
team this. I'm adding some docs to the wiki and improving some extension 
documentation. Would be happy to carve some time if needed in the name of docs. 


was (Author: phrocker):
[~achristianson] Thanks for adding this. Let me know if you want to tag team 
this. I'm adding some docs to the wiki and improving some extension 
documentation. Would be happy to carve some time if needed in the name of docs. 

> Add user docs for MiNiFi - C++ processors
> -
>
> Key: MINIFICPP-328
> URL: https://issues.apache.org/jira/browse/MINIFICPP-328
> Project: NiFi MiNiFi C++
>  Issue Type: Documentation
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>
> Because MiNiFi - C++ processors are currently different from what's offered 
> in NiFi, and will likely remain different to some degree, we can't continue 
> to rely solely on NiFi's documentation. We need to improve MiNiFi's 
> documentation to cover all processors included in MiNiFi and what 
> features/options they support, how they work, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-328) Add user docs for MiNiFi - C++ processors

[marco polo (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273112#comment-16273112
 ] 

marco polo commented on MINIFICPP-328:
--

[~achristianson] Thanks for adding this. Let me know if you want to tag team 
this. I'm adding some docs to the wiki and improving some extension 
documentation. Would be happy to carve some time if needed in the name of docs. 

> Add user docs for MiNiFi - C++ processors
> -
>
> Key: MINIFICPP-328
> URL: https://issues.apache.org/jira/browse/MINIFICPP-328
> Project: NiFi MiNiFi C++
>  Issue Type: Documentation
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>
> Because MiNiFi - C++ processors are currently different from what's offered 
> in NiFi, and will likely remain different to some degree, we can't continue 
> to rely solely on NiFi's documentation. We need to improve MiNiFi's 
> documentation to cover all processors included in MiNiFi and what 
> features/options they support, how they work, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (MINIFICPP-304) Release MiNiFI C++ Version 0.3.0

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo resolved MINIFICPP-304.
--
Resolution: Fixed

> Release MiNiFI C++ Version 0.3.0
> 
>
> Key: MINIFICPP-304
> URL: https://issues.apache.org/jira/browse/MINIFICPP-304
> Project: NiFi MiNiFi C++
>  Issue Type: Task
>Reporter: marco polo
>Assignee: marco polo
> Fix For: 0.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (MINIFICPP-329) Kafka Refactor

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo updated MINIFICPP-329:
-
Fix Version/s: (was: 0.3.0)
   0.4.0

> Kafka Refactor
> --
>
> Key: MINIFICPP-329
> URL: https://issues.apache.org/jira/browse/MINIFICPP-329
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: bqiu
>Assignee: bqiu
>Priority: Minor
> Fix For: 0.4.0
>
>
> refactor PutKafka to PublishKafka
> add kafka.key as key attribute if flow has the same in flow attribute



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4650) Add Hadoop HttpCore property independent of the HttpClient property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273100#comment-16273100
 ] 

ASF GitHub Bot commented on NIFI-4650:
--

GitHub user jfrazee opened a pull request:

https://github.com/apache/nifi/pull/2308

NIFI-4650 Added independent hadoop.http.core.version property

For later releases of HttpComponents HttpClient and HttpCore versioning 
diverges so these need to be separate properties to support overriding them, 
e.g., for builds against Hadoop 2.8.x, etc.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jfrazee/nifi NIFI-4650

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2308


commit cd5dd2d18af1e70b2ceca34035eccc6533fbd21d
Author: Joey Frazee 
Date:   2017-11-30T16:59:19Z

NIFI-4650 Added independent hadoop.http.core.version property




> Add Hadoop HttpCore property independent of the HttpClient property
> ---
>
> Key: NIFI-4650
> URL: https://issues.apache.org/jira/browse/NIFI-4650
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>Priority: Minor
>
> Apache HttpComponents versions HttpClient and HttpCore independently (e.g., 
> 4.5.2 and 4.4.4 used in Hadoop 2.8.x) so overriding 
> hadoop.http.client.version with certain versions of HttpClient or HttpCore 
> isn't currently possible. We should add a separate hadoop.http.core.version 
> to pom.xml and nifi-nar-bundles/nifi-hadoop-libraries-bundle/pom.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2308: NIFI-4650 Added independent hadoop.http.core.versio...

[jfrazee]

GitHub user jfrazee opened a pull request:

https://github.com/apache/nifi/pull/2308

NIFI-4650 Added independent hadoop.http.core.version property

For later releases of HttpComponents HttpClient and HttpCore versioning 
diverges so these need to be separate properties to support overriding them, 
e.g., for builds against Hadoop 2.8.x, etc.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jfrazee/nifi NIFI-4650

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2308


commit cd5dd2d18af1e70b2ceca34035eccc6533fbd21d
Author: Joey Frazee 
Date:   2017-11-30T16:59:19Z

NIFI-4650 Added independent hadoop.http.core.version property




---

[jira] [Commented] (NIFI-4333) Dockerize NiFi Toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273067#comment-16273067
 ] 

ASF GitHub Bot commented on NIFI-4333:
--

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2161#discussion_r154153740
  
--- Diff: nifi-toolkit/nifi-toolkit-assembly/docker/Dockerfile ---
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM openjdk:8-jre-alpine
+LABEL maintainer "Apache NiFi "
+
+ARG UID=1000
+ARG GID=1000
+ARG NIFI_TOOLKIT_VERSION=1.4.0-SNAPSHOT
--- End diff --

Updated with a new commit that does resource filtering as linked above.  


> Dockerize NiFi Toolkit
> --
>
> Key: NIFI-4333
> URL: https://issues.apache.org/jira/browse/NIFI-4333
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Docker
>Reporter: Aldrin Piri
>Assignee: Aldrin Piri
>
> It would be helpful to have a TLS Toolkit image to work in conjunction with 
> NiFI and would help in orchestration of clustered instances as well as 
> provisioning security items.
> As this is one assembly, we should support all tooling provided through one 
> image.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2161: NIFI-4333: Providing Docker support of the NiFi Too...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2161#discussion_r154153740
  
--- Diff: nifi-toolkit/nifi-toolkit-assembly/docker/Dockerfile ---
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM openjdk:8-jre-alpine
+LABEL maintainer "Apache NiFi "
+
+ARG UID=1000
+ARG GID=1000
+ARG NIFI_TOOLKIT_VERSION=1.4.0-SNAPSHOT
--- End diff --

Updated with a new commit that does resource filtering as linked above.  


---

[jira] [Reopened] (NIFI-4645) NiFi build on powerpc fails because the google/rpc nar cannot be built/used there

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reopened NIFI-4645:
---

> NiFi build on powerpc fails because the google/rpc nar cannot be built/used 
> there
> -
>
> Key: NIFI-4645
> URL: https://issues.apache.org/jira/browse/NIFI-4645
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
> Environment: ibm/power pc
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
>
> Going to update the poms to detect the os family and if not powerpc will 
> activate module builds and assembly usage of nars that are not capable in 
> that environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4645) NiFi build on powerpc fails because the google/rpc nar cannot be built/used there

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273058#comment-16273058
 ] 

Joseph Witt commented on NIFI-4645:
---

gosh darn it - pretty sure it is now excluding it always.  Will dig further...

> NiFi build on powerpc fails because the google/rpc nar cannot be built/used 
> there
> -
>
> Key: NIFI-4645
> URL: https://issues.apache.org/jira/browse/NIFI-4645
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
> Environment: ibm/power pc
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.5.0
>
>
> Going to update the poms to detect the os family and if not powerpc will 
> activate module builds and assembly usage of nars that are not capable in 
> that environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4628) Update ListS3 proccesor to support List Objects Version 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273031#comment-16273031
 ] 

ASF GitHub Bot commented on NIFI-4628:
--

Github user aburkard commented on the issue:

https://github.com/apache/nifi/pull/2300
  
@jvwing I've added descriptive names for the List Type options.


> Update ListS3 proccesor to support List Objects Version 2
> -
>
> Key: NIFI-4628
> URL: https://issues.apache.org/jira/browse/NIFI-4628
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Andrew Burkard
>Priority: Minor
>
> AWS has revised the List Objects API: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> The main difference is the new API passes a continuation token in the 
> response for multi-page listings rather than requiring the client to keep 
> track of the lexicographically last key listed. The new version also appears 
> to be the only API supported going forward for new AWS services (e.g. 
> Snowball). It would be nice if the ListS3 processor supported either via an 
> attribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2300: NIFI-4628 Add support for ListS3Version2 API

[aburkard]

Github user aburkard commented on the issue:

https://github.com/apache/nifi/pull/2300
  
@jvwing I've added descriptive names for the List Type options.


---

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank so given joey's comments/JIRA/example do you feel this gets you to a 
good state?


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273024#comment-16273024
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank so given joey's comments/JIRA/example do you feel this gets you to a 
good state?


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273010#comment-16273010
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@joewitt I don't think so. If we get NIFI-4650 in, then we can probably 
support builds against 2.8.x, 2.9.x, etc. on official releases without 
introducing more deps to PutParquet.

@baank I know the suggested change would allow a lot more convenience and 
maybe it's a little opaque to know that we intend to use the additional 
classpath. What might be super helpful in these cases in "Additional Details" 
documentation on the processor to lay out some known jar configurations and 
build scenarios.


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[jfrazee]

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@joewitt I don't think so. If we get NIFI-4650 in, then we can probably 
support builds against 2.8.x, 2.9.x, etc. on official releases without 
introducing more deps to PutParquet.

@baank I know the suggested change would allow a lot more convenience and 
maybe it's a little opaque to know that we intend to use the additional 
classpath. What might be super helpful in these cases in "Additional Details" 
documentation on the processor to lay out some known jar configurations and 
build scenarios.


---

[jira] [Commented] (NIFI-4256) Add support for all AWS S3 Encryption Options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272998#comment-16272998
 ] 

ASF GitHub Bot commented on NIFI-4256:
--

Github user baank closed the pull request at:

https://github.com/apache/nifi/pull/2291


> Add support for all AWS S3 Encryption Options
> -
>
> Key: NIFI-4256
> URL: https://issues.apache.org/jira/browse/NIFI-4256
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Franco
>  Labels: aws, aws-s3, security
>
> NiFi currently only supports SSE-S3 encryption (AES256).
> Support needs to be added for:
> * SSE-S3
> * SSE-KMS
> * SSE-C
> * CSE-KMS CMK
> * CSE-Master Key
> With all of the appropriate configuration options and such that SSE is 
> available only for PutS3Object whilst CSE is available also for FetchS3Object.
> Given that this will add another 20 or so UI properties the intention is to 
> split it into a Client Side Encryption Service and Server Side Encryption 
> Service. This will allow users to reuse "encryption" across different 
> workflows.
> Existing flows using the Server Side Encryption option will still work as is 
> but will be overridden if a service is added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2291: NIFI-4256 - Add support for all AWS S3 Encryption O...

[baank]

Github user baank closed the pull request at:

https://github.com/apache/nifi/pull/2291


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272996#comment-16272996
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user baank closed the pull request at:

https://github.com/apache/nifi/pull/2293


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[baank]

Github user baank commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@joewitt .. I appreciate the situation and what I meant was that HDF is 
likely to adopt whatever changes are part of the main line branch. But I 
appreciate it's an open community.

Unfortunately for us building our own custom NiFi is not acceptable unless 
we take on the fully own the support risk. 

Happy to close this.


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272995#comment-16272995
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user baank commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@joewitt .. I appreciate the situation and what I meant was that HDF is 
likely to adopt whatever changes are part of the main line branch. But I 
appreciate it's an open community.

Unfortunately for us building our own custom NiFi is not acceptable unless 
we take on the fully own the support risk. 

Happy to close this.


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2293: NIFI-4565 - PutParquet doesn't support S3

[baank]

Github user baank closed the pull request at:

https://github.com/apache/nifi/pull/2293


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272987#comment-16272987
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@jfrazee did I misunderstand your suggestion?


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@jfrazee did I misunderstand your suggestion?


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272976#comment-16272976
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank To be clear this is the Apache NiFi open source community.  There 
are no customers here - just an open community.  

Regarding this contribution: The contribution as provided thus far creates 
an alternative problem, for which, a couple of committers have said we should 
find a better way.  What Joey is showing/suggesting above, specifically with 
NIFI-4650, would give you the flexibility you appear to need and doesn't seem 
to come with the same introduced alternative problem.  So this is probably a 
good path to pursue.

Regarding the comments about a particular vendor distribution obviously 
this community cannot speak to that.

Hopefully that helps you see what is being suggested is not recommending 
you do any forking.

Thanks


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank To be clear this is the Apache NiFi open source community.  There 
are no customers here - just an open community.  

Regarding this contribution: The contribution as provided thus far creates 
an alternative problem, for which, a couple of committers have said we should 
find a better way.  What Joey is showing/suggesting above, specifically with 
NIFI-4650, would give you the flexibility you appear to need and doesn't seem 
to come with the same introduced alternative problem.  So this is probably a 
good path to pursue.

Regarding the comments about a particular vendor distribution obviously 
this community cannot speak to that.

Hopefully that helps you see what is being suggested is not recommending 
you do any forking.

Thanks


---

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272959#comment-16272959
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user baank commented on the issue:

https://github.com/apache/nifi/pull/2293
  
People like myself who represent very large enterprise customers are 
looking to run a supported platform in production i.e. HDF. So custom builds is 
a very big change in direction.

Can I request a final decision whether this will be merged or not ?



> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[baank]

Github user baank commented on the issue:

https://github.com/apache/nifi/pull/2293
  
People like myself who represent very large enterprise customers are 
looking to run a supported platform in production i.e. HDF. So custom builds is 
a very big change in direction.

Can I request a final decision whether this will be merged or not ?



---

[jira] [Commented] (MINIFICPP-336) With default GetFile settings dot files are not getting ignored on linux systems as they should

[bqiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272942#comment-16272942
 ] 

bqiu commented on MINIFICPP-336:


yes, the code section pointed by Marc has the problem.

> With default GetFile settings dot files are not getting ignored on linux 
> systems as they should
> ---
>
> Key: MINIFICPP-336
> URL: https://issues.apache.org/jira/browse/MINIFICPP-336
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0, 0.1.0, 0.2.0
>Reporter: Joseph Witt
>Assignee: marco polo
> Fix For: 0.4.0
>
>
> With this config
> Processors:
> - name: GetFile
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> Batch Size: '10'
> File Filter: '[^\.].*'
> Ignore Hidden Files: 'true'
> Input Directory: test/input
> Keep Source File: 'false'
> Maximum File Age:
> Maximum File Size:
> Minimum File Age: 0 sec
> Minimum File Size: 0 B
> Path Filter:
> Polling Interval: 0 sec
> Recurse Subdirectories: 'true'
> The minifi flow picks up any files starting with '.' character right away.  I 
> believe this is causing duplication to occur when NiFi writes to that 
> directory being watched, for example, because it writes the files a 
> hidden/dot notation then renamed it when done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3472) Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase processors

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272934#comment-16272934
 ] 

Jorge Machado commented on NIFI-3472:
-

[~jtstorck] sure that is only with ticket cache? I think we could call 
UserGroupInformation.getLoginUser() this will internally call the 
spawnAutoRenewalThreadForUserCreds if I'm not mistaken or read bad the code. 
What would be really great is if the Hadoop team had a how to proper use 
UserGroupInformation, because it seems that everyone tries to do his thing... 
But yes the abstract class from Nifi should Have a thread that takes care of 
renewing the token. 

> Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase 
> processors
> --
>
> Key: NIFI-3472
> URL: https://issues.apache.org/jira/browse/NIFI-3472
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>
> PutHDFS is not able to relogin if the ticket expires.
> NiFi, running locally as standalone, was sending files to HDFS.  After 
> suspending the system for the weekend, when the flow attempted to continue to 
> process flowfiles, the following exception occurred:
> {code}2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10] 
> org.apache.hadoop.ipc.Client Exception encountered while connecting to the 
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
> 2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10] 
> o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of 
> class ClientNamenodeProtocolTranslatorPB over [host:port] after 3 fail over 
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: 
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]; Host Details : local host is: "[host:port]"; destination 
> host is: [host:port];
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1479) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1412) 
> ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>  ~[hadoop-hdfs-2.7.3.jar:na]
>   at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source) 
> ~[na:na]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_102]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>  ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
>   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) 
> [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>  [hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262) 
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
>   at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.8.0_102]
>   at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
>  [hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) 
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
>   at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>  

[jira] [Commented] (NIFI-4565) PutParquet doesn't support S3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272932#comment-16272932
 ] 

ASF GitHub Bot commented on NIFI-4565:
--

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank So, we still wouldn't need to update the Hadoop version on 
everything yet, because in principle you can just do a build of NiFi overriding 
the hadoop.version property and use 2.8.x. For example:

```sh
$ mvn -T 2.0C clean install -Dhadoop.version=2.8.2 
-Dhadoop.guava.version=12.0.1 -Dhadoop.http.client.version=4.5.2 
-Dhadoop.http.core.version=4.4.4 -DskipTests
```
That said, this is a little bit of a lie because in later versions of 
HttpComponents HttpClient and HttpCore aren't versioned identically and we 
currently only use a single property hadoop.http.client.version for these; 
i.e., the hadoop.http.core.version property doesn't exist yet. See 
[NIFI-4650](https://issues.apache.org/jira/browse/NIFI-4650) though.

So, I did the build above with the new property and tested with the 
following jars and things seem to work:

```
aws-java-sdk-core-1.11.68.jar
aws-java-sdk-kms-1.11.68.jar
aws-java-sdk-s3-1.11.68.jar
hadoop-aws-2.8.2.jar
hadoop-common-2.8.2.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
jackson-annotations-2.6.0.jar
jackson-core-2.6.1.jar
jackson-databind-2.6.1.jar
joda-time-2.8.2.jar
```

We're trying to be very cautious about updating the default to the next 
major version of Hadoop so it might be best to stick with this still being a 
property override.


> PutParquet doesn't support S3
> -
>
> Key: NIFI-4565
> URL: https://issues.apache.org/jira/browse/NIFI-4565
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Franco
>Assignee: Joey Frazee
>
> PutParquet doesn't support S3 targets due to the lack of the hadoop-aws 
> dependency.
> To recreate it:
> 1) Modify core-site.xml to change fs.defaultName to one starting with s3://
> 2) Add a PutParquet 
> 3) It will fail to run. The logs will show the missing hadoop-aws dependency.
> Simple fix is just to add hadoop-aws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2293: NIFI-4565 - PutParquet doesn't support S3

[jfrazee]

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/2293
  
@baank So, we still wouldn't need to update the Hadoop version on 
everything yet, because in principle you can just do a build of NiFi overriding 
the hadoop.version property and use 2.8.x. For example:

```sh
$ mvn -T 2.0C clean install -Dhadoop.version=2.8.2 
-Dhadoop.guava.version=12.0.1 -Dhadoop.http.client.version=4.5.2 
-Dhadoop.http.core.version=4.4.4 -DskipTests
```
That said, this is a little bit of a lie because in later versions of 
HttpComponents HttpClient and HttpCore aren't versioned identically and we 
currently only use a single property hadoop.http.client.version for these; 
i.e., the hadoop.http.core.version property doesn't exist yet. See 
[NIFI-4650](https://issues.apache.org/jira/browse/NIFI-4650) though.

So, I did the build above with the new property and tested with the 
following jars and things seem to work:

```
aws-java-sdk-core-1.11.68.jar
aws-java-sdk-kms-1.11.68.jar
aws-java-sdk-s3-1.11.68.jar
hadoop-aws-2.8.2.jar
hadoop-common-2.8.2.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
jackson-annotations-2.6.0.jar
jackson-core-2.6.1.jar
jackson-databind-2.6.1.jar
joda-time-2.8.2.jar
```

We're trying to be very cautious about updating the default to the next 
major version of Hadoop so it might be best to stick with this still being a 
property override.


---

[jira] [Commented] (MINIFICPP-49) NiFi Expression Language support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272925#comment-16272925
 ] 

ASF GitHub Bot commented on MINIFICPP-49:
-

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/188
  
looks good to me +1


> NiFi Expression Language support
> 
>
> Key: MINIFICPP-49
> URL: https://issues.apache.org/jira/browse/MINIFICPP-49
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Andrew Christianson
>
> An implementation of NiFi's expression language is needed for the cpp version 
> of the project. The feature should support all of the same EL syntax that 
> NiFi supports so that any flow developed in NiFi and then ran through the 
> minifi-toolkit will generate a valid .yml configuration file. Currently if 
> there are any expression language values in the flow.yml file the agent will 
> not act as anticipated or crash.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-49) NiFi Expression Language support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272924#comment-16272924
 ] 

ASF GitHub Bot commented on MINIFICPP-49:
-

Github user minifirocks commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/188#discussion_r154135525
  
--- Diff: extensions/expression-language/ProcessContextExpr.cpp ---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include 
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+
+bool ProcessContext::getProperty(const std::string , std::string 
,
+ const std::shared_ptr 
_file) {
+  if (expressions_.find(name) == expressions_.end()) {
--- End diff --

OK.


> NiFi Expression Language support
> 
>
> Key: MINIFICPP-49
> URL: https://issues.apache.org/jira/browse/MINIFICPP-49
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Andrew Christianson
>
> An implementation of NiFi's expression language is needed for the cpp version 
> of the project. The feature should support all of the same EL syntax that 
> NiFi supports so that any flow developed in NiFi and then ran through the 
> minifi-toolkit will generate a valid .yml configuration file. Currently if 
> there are any expression language values in the flow.yml file the agent will 
> not act as anticipated or crash.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp pull request #188: MINIFICPP-49 Added initial implementation...

[minifirocks]

Github user minifirocks commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/188#discussion_r154135525
  
--- Diff: extensions/expression-language/ProcessContextExpr.cpp ---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include 
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+
+bool ProcessContext::getProperty(const std::string , std::string 
,
+ const std::shared_ptr 
_file) {
+  if (expressions_.find(name) == expressions_.end()) {
--- End diff --

OK.


---

[GitHub] nifi-minifi-cpp issue #188: MINIFICPP-49 Added initial implementation of NiF...

[minifirocks]

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/188
  
looks good to me +1


---

[jira] [Created] (NIFI-4650) Add Hadoop HttpCore property independent of the HttpClient property

[Joey Frazee (JIRA)]

Joey Frazee created NIFI-4650:
-

 Summary: Add Hadoop HttpCore property independent of the 
HttpClient property
 Key: NIFI-4650
 URL: https://issues.apache.org/jira/browse/NIFI-4650
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joey Frazee
Assignee: Joey Frazee
Priority: Minor


Apache HttpComponents versions HttpClient and HttpCore independently (e.g., 
4.5.2 and 4.4.4 used in Hadoop 2.8.x) so overriding hadoop.http.client.version 
with certain versions of HttpClient or HttpCore isn't currently possible. We 
should add a separate hadoop.http.core.version to pom.xml and 
nifi-nar-bundles/nifi-hadoop-libraries-bundle/pom.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4649) FlattenJson processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272875#comment-16272875
 ] 

ASF GitHub Bot commented on NIFI-4649:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2307

NIFI-4649 Added FlattenJson processor.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4649

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2307.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2307


commit 72b45e6fd9a1250fd5f20da7632fd78538af6f00
Author: Mike Thomsen 
Date:   2017-11-30T12:23:53Z

NIFI-4649 Added FlattenJson processor.




> FlattenJson processor
> -
>
> Key: NIFI-4649
> URL: https://issues.apache.org/jira/browse/NIFI-4649
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>
> Users have requested a processor that can flatten a JSON record into a simple 
> key/value pair document no matter how nested it is.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2307: NIFI-4649 Added FlattenJson processor.

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2307

NIFI-4649 Added FlattenJson processor.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4649

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2307.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2307


commit 72b45e6fd9a1250fd5f20da7632fd78538af6f00
Author: Mike Thomsen 
Date:   2017-11-30T12:23:53Z

NIFI-4649 Added FlattenJson processor.




---

[jira] [Updated] (NIFI-4606) TailFile has a property named "Rolling Strategy" but it is never used

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4606:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master

> TailFile has a property named "Rolling Strategy" but it is never used
> -
>
> Key: NIFI-4606
> URL: https://issues.apache.org/jira/browse/NIFI-4606
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The TailFile processor has a property that is named "Rolling Strategy." It 
> has two allowable values "Fixed name" or "Changing name." I was unable to 
> understand from the documentation what the intent of the property was, so I 
> began to read the code in order to understand so that I could better update 
> the documentation. However, looking through the code, I found that the 
> property was only ever looked at in the customValidate method and never used 
> for any logic within the processor.
> Therefore, this property needs to be removed from the processor, since it 
> adds complexity to the processor's configuration and doesn't actually affect 
> the behavior of the processor itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4602) TailFile does not recover cluster-wide state when Primary Node changes

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4602:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master

> TailFile does not recover cluster-wide state when Primary Node changes
> --
>
> Key: NIFI-4602
> URL: https://issues.apache.org/jira/browse/NIFI-4602
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> I created a TailFile processor and set the "File Location" property to 
> "Remote." I then added some text to a file and started the TailFile processor 
> (Primary Node Only).
> I disconnected the primary node from the cluster. A second node became the 
> new Primary Node and continued tailing the data as expected. As new data was 
> added, the node continued to emit FlowFiles.
> Then, I added Node 1 back to the cluster. Verified that Node 2 continued to 
> poll data. I then disconnected Node 2 so that Node 1 once again became the 
> Primary Node. At this point, Node 1 continued tailing from where it last left 
> off, not where Node 2 left off.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4607) MonitorMemory has problematic unit tests

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4607:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master

> MonitorMemory has problematic unit tests
> 
>
> Key: NIFI-4607
> URL: https://issues.apache.org/jira/browse/NIFI-4607
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The MonitorMemoryTest class contains 3 unit tests. Two of them are ignored 
> with @Ignore due to them depending on environmental condition. The third unit 
> test depends on FlowController and is testing the framework as much as it is 
> testing the Reporting Task itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-3975) TailFile stopped tailing file

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3975:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master

> TailFile stopped tailing file
> -
>
> Key: NIFI-3975
> URL: https://issues.apache.org/jira/browse/NIFI-3975
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
> Environment: Linux
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> I have a TailFile processor running on a 3-node cluster. It is tailing 
> logs/nifi-app.log. Two of the nodes seem to be working well. The third node 
> is not picking up any data. I enabled DEBUG logging and see the following 
> logs:
> {code}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] No data to consume; removed 
> created FlowFile
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Returning LOCAL State: 
> StandardStateMap[version=442100928, values={file.0.length=0, 
> file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}]
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Setting LOCAL State to 
> {file.0.length=0, file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] No data to consume; removed 
> created FlowFile
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Returning LOCAL State: 
> StandardStateMap[version=442100929, values={file.0.length=0, 
> file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}]
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Setting LOCAL State to 
> {file.0.length=0, file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] No data to consume; removed 
> created FlowFile
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Returning LOCAL State: 
> StandardStateMap[version=442100930, values={file.0.length=0, 
> file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}]
> {code}
> The file is approximately 33 MB right now. But the state indicates that the 
> position it is at in the file is over 105 MB; also, according to state, the 
> checksum is 0 and the file length is 0. This likely indicates that the file 
> was about 105 MB at one point and then it rolled over and stopped processing 
> properly, given that it thinks the checksum is 0.

[jira] [Commented] (NIFI-3975) TailFile stopped tailing file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272861#comment-16272861
 ] 

ASF subversion and git services commented on NIFI-3975:
---

Commit 45df23b1e09f340fb2c7f3c3234462570ddb5f6b in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=45df23b ]

NIFI-4607, NIFI-3975, NIFI-4602, NIFI-4606: This closes #2272. Fixed bug in 
TailFile that caused new Primary Node to not pull current Clustered State when 
File Location was set to Remote. Fixed bug that caused TailFile to occasionally 
become 'stuck' when the file it is tailing is renamed and a new file is created 
with the same name. Removed the 'Rolling Strategy' property from TailFile 
because it is not actually used in the processor anymore. Deleted 
MonitorMemoryTest because the unit test was testing the behavior of 
FlowController more than the behavior of the reporting task itself and in order 
to do so had a dependency in the pom.xml on nifi-framework-core, which means 
that it no longer compiles when FlowController is modified.

Signed-off-by: joewitt 


> TailFile stopped tailing file
> -
>
> Key: NIFI-3975
> URL: https://issues.apache.org/jira/browse/NIFI-3975
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
> Environment: Linux
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> I have a TailFile processor running on a 3-node cluster. It is tailing 
> logs/nifi-app.log. Two of the nodes seem to be working well. The third node 
> is not picking up any data. I enabled DEBUG logging and see the following 
> logs:
> {code}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] No data to consume; removed 
> created FlowFile
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Returning LOCAL State: 
> StandardStateMap[version=442100928, values={file.0.length=0, 
> file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}]
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-48] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Setting LOCAL State to 
> {file.0.length=0, file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] No data to consume; removed 
> created FlowFile
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Returning LOCAL State: 
> StandardStateMap[version=442100929, values={file.0.length=0, 
> file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}]
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Setting LOCAL State to 
> {file.0.length=0, file.0.position=105133313, file.0.checksum=0, 
> file.0.filename=logs/nifi-app.log, file.0.timestamp=1495637757000}
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Recovering Rolled Off 
> Files; total number of files rolled off = 0
> 2017-05-25 13:00:32,563 DEBUG [Timer-Driven Process Thread-10] 
> o.a.nifi.processors.standard.TailFile 
> TailFile[id=47f4a068-015a-1000-81b7-7d991bbbf80f] Reading lines starting at 
> position 105133313
> 

[jira] [Commented] (NIFI-4602) TailFile does not recover cluster-wide state when Primary Node changes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272862#comment-16272862
 ] 

ASF subversion and git services commented on NIFI-4602:
---

Commit 45df23b1e09f340fb2c7f3c3234462570ddb5f6b in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=45df23b ]

NIFI-4607, NIFI-3975, NIFI-4602, NIFI-4606: This closes #2272. Fixed bug in 
TailFile that caused new Primary Node to not pull current Clustered State when 
File Location was set to Remote. Fixed bug that caused TailFile to occasionally 
become 'stuck' when the file it is tailing is renamed and a new file is created 
with the same name. Removed the 'Rolling Strategy' property from TailFile 
because it is not actually used in the processor anymore. Deleted 
MonitorMemoryTest because the unit test was testing the behavior of 
FlowController more than the behavior of the reporting task itself and in order 
to do so had a dependency in the pom.xml on nifi-framework-core, which means 
that it no longer compiles when FlowController is modified.

Signed-off-by: joewitt 


> TailFile does not recover cluster-wide state when Primary Node changes
> --
>
> Key: NIFI-4602
> URL: https://issues.apache.org/jira/browse/NIFI-4602
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> I created a TailFile processor and set the "File Location" property to 
> "Remote." I then added some text to a file and started the TailFile processor 
> (Primary Node Only).
> I disconnected the primary node from the cluster. A second node became the 
> new Primary Node and continued tailing the data as expected. As new data was 
> added, the node continued to emit FlowFiles.
> Then, I added Node 1 back to the cluster. Verified that Node 2 continued to 
> poll data. I then disconnected Node 2 so that Node 1 once again became the 
> Primary Node. At this point, Node 1 continued tailing from where it last left 
> off, not where Node 2 left off.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4606) TailFile has a property named "Rolling Strategy" but it is never used

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272863#comment-16272863
 ] 

ASF subversion and git services commented on NIFI-4606:
---

Commit 45df23b1e09f340fb2c7f3c3234462570ddb5f6b in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=45df23b ]

NIFI-4607, NIFI-3975, NIFI-4602, NIFI-4606: This closes #2272. Fixed bug in 
TailFile that caused new Primary Node to not pull current Clustered State when 
File Location was set to Remote. Fixed bug that caused TailFile to occasionally 
become 'stuck' when the file it is tailing is renamed and a new file is created 
with the same name. Removed the 'Rolling Strategy' property from TailFile 
because it is not actually used in the processor anymore. Deleted 
MonitorMemoryTest because the unit test was testing the behavior of 
FlowController more than the behavior of the reporting task itself and in order 
to do so had a dependency in the pom.xml on nifi-framework-core, which means 
that it no longer compiles when FlowController is modified.

Signed-off-by: joewitt 


> TailFile has a property named "Rolling Strategy" but it is never used
> -
>
> Key: NIFI-4606
> URL: https://issues.apache.org/jira/browse/NIFI-4606
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The TailFile processor has a property that is named "Rolling Strategy." It 
> has two allowable values "Fixed name" or "Changing name." I was unable to 
> understand from the documentation what the intent of the property was, so I 
> began to read the code in order to understand so that I could better update 
> the documentation. However, looking through the code, I found that the 
> property was only ever looked at in the customValidate method and never used 
> for any logic within the processor.
> Therefore, this property needs to be removed from the processor, since it 
> adds complexity to the processor's configuration and doesn't actually affect 
> the behavior of the processor itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4607) MonitorMemory has problematic unit tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272860#comment-16272860
 ] 

ASF subversion and git services commented on NIFI-4607:
---

Commit 45df23b1e09f340fb2c7f3c3234462570ddb5f6b in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=45df23b ]

NIFI-4607, NIFI-3975, NIFI-4602, NIFI-4606: This closes #2272. Fixed bug in 
TailFile that caused new Primary Node to not pull current Clustered State when 
File Location was set to Remote. Fixed bug that caused TailFile to occasionally 
become 'stuck' when the file it is tailing is renamed and a new file is created 
with the same name. Removed the 'Rolling Strategy' property from TailFile 
because it is not actually used in the processor anymore. Deleted 
MonitorMemoryTest because the unit test was testing the behavior of 
FlowController more than the behavior of the reporting task itself and in order 
to do so had a dependency in the pom.xml on nifi-framework-core, which means 
that it no longer compiles when FlowController is modified.

Signed-off-by: joewitt 


> MonitorMemory has problematic unit tests
> 
>
> Key: NIFI-4607
> URL: https://issues.apache.org/jira/browse/NIFI-4607
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The MonitorMemoryTest class contains 3 unit tests. Two of them are ignored 
> with @Ignore due to them depending on environmental condition. The third unit 
> test depends on FlowController and is testing the framework as much as it is 
> testing the Reporting Task itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4607) MonitorMemory has problematic unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272866#comment-16272866
 ] 

ASF GitHub Bot commented on NIFI-4607:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2272


> MonitorMemory has problematic unit tests
> 
>
> Key: NIFI-4607
> URL: https://issues.apache.org/jira/browse/NIFI-4607
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The MonitorMemoryTest class contains 3 unit tests. Two of them are ignored 
> with @Ignore due to them depending on environmental condition. The third unit 
> test depends on FlowController and is testing the framework as much as it is 
> testing the Reporting Task itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2272: NIFI-4607, NIFI-3975, NIFI-4602, NIFI-4606: Fixed b...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2272


---

[jira] [Commented] (NIFI-3472) Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase processors

[Jeff Storck (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272821#comment-16272821
 ] 

Jeff Storck commented on NIFI-3472:
---

I've reproduced the tgt expiration issue outside of NiFi in a small test 
utility that I wrote.  The Hadoop team is looking into this issue.  There are a 
few Hadoop JIRAs regarding the UGI class that are being worked on concurrently, 
so we should see a fix for this issue once those JIRAs have been resolved.

[~jomach] After a quick look at the code, spawnAutoRenewalThreadForUserCreds is 
meant for renewing credentials from the ticket cache, and the thread doesn't 
get started if keytabs are being used.  Since it's a private method, NiFi 
wouldn't be able to make use of this explicitly, and we still need to find a 
way to renew the TGT whether the ticket cache or keytabs are being used.  
Hopefully, this will be transparent to NiFi.  hadoop-client will implicitly 
relogin if an operation is performed before expiration of the TGT, but won't 
acquire another TGT if the current one has expired.  Hopefully that'll be part 
of the fix provided by the Hadoop team.

> Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase 
> processors
> --
>
> Key: NIFI-3472
> URL: https://issues.apache.org/jira/browse/NIFI-3472
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>
> PutHDFS is not able to relogin if the ticket expires.
> NiFi, running locally as standalone, was sending files to HDFS.  After 
> suspending the system for the weekend, when the flow attempted to continue to 
> process flowfiles, the following exception occurred:
> {code}2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10] 
> org.apache.hadoop.ipc.Client Exception encountered while connecting to the 
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
> 2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10] 
> o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of 
> class ClientNamenodeProtocolTranslatorPB over [host:port] after 3 fail over 
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: 
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]; Host Details : local host is: "[host:port]"; destination 
> host is: [host:port];
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1479) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1412) 
> ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>  ~[hadoop-hdfs-2.7.3.jar:na]
>   at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source) 
> ~[na:na]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_102]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>  ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
>   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) 
> [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>  [hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262) 
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
>   at 

[jira] [Commented] (NIFI-4630) Implement Satori RTM processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272797#comment-16272797
 ] 

ASF GitHub Bot commented on NIFI-4630:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2286#discussion_r154106542
  
--- Diff: 
nifi-nar-bundles/nifi-satori-bundle/nifi-satori-bundle-processors/pom.xml ---
@@ -0,0 +1,63 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-satori-bundle
+1.5.0-SNAPSHOT
+
+
+nifi-satori-bundle-processors
+jar
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.satori
+satori-sdk-java
--- End diff --

Need to make sure this LICENSE is ok. Please file a question to Apache 
Legal so we can determine which category this license is under, thanks in 
advance!


> Implement Satori RTM processors
> ---
>
> Key: NIFI-4630
> URL: https://issues.apache.org/jira/browse/NIFI-4630
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Laurence Da Luz
>
> Satori is a cloud-based data platform that provides a publish-subscribe 
> messaging service called RTM:
> https://www.satori.com/docs/using-satori/overview
> This JIRA is to cover two new NiFi processors for communicating with the 
> Satori RTM messaging service:
> * *ConsumeSatoriRTM* - used to consume a live data feed from a Satori RTM 
> channel
> * *PublishSatoriRTM* - used to publish messages to a Satori RTM channel



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2286: NIFI-4630 Added Satori RTM processor bundle (Consum...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2286#discussion_r154106542
  
--- Diff: 
nifi-nar-bundles/nifi-satori-bundle/nifi-satori-bundle-processors/pom.xml ---
@@ -0,0 +1,63 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-satori-bundle
+1.5.0-SNAPSHOT
+
+
+nifi-satori-bundle-processors
+jar
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.satori
+satori-sdk-java
--- End diff --

Need to make sure this LICENSE is ok. Please file a question to Apache 
Legal so we can determine which category this license is under, thanks in 
advance!


---

[jira] [Updated] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4559:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
> Fix For: 1.5.0
>
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4559:
---
Fix Version/s: 1.5.0

> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
> Fix For: 1.5.0
>
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272785#comment-16272785
 ] 

ASF GitHub Bot commented on NIFI-4559:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2246


> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272783#comment-16272783
 ] 

ASF subversion and git services commented on NIFI-4559:
---

Commit 3b15ed855cc4319844dac13ba3244dc8ddde6207 in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3b15ed8 ]

NIFI-4559: Add non-zero status relationship to ExecuteStreamCommand

NIFI-4559: Removed Penalize Non-zero Status property and updated doc per review 
comments
Signed-off-by: Matthew Burgess 

This closes #2246


> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272782#comment-16272782
 ] 

ASF subversion and git services commented on NIFI-4559:
---

Commit 3b15ed855cc4319844dac13ba3244dc8ddde6207 in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3b15ed8 ]

NIFI-4559: Add non-zero status relationship to ExecuteStreamCommand

NIFI-4559: Removed Penalize Non-zero Status property and updated doc per review 
comments
Signed-off-by: Matthew Burgess 

This closes #2246


> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2246: NIFI-4559: Add non-zero status relationship to Exec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2246


---

[jira] [Resolved] (NIFI-4597) Add a property to override HTTP return code in ListenHTTP processor

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-4597.
---
Resolution: Fixed

corrected checkstyle problems, unit test failures since validator not set, and 
added imports.

+1 merged to master.

> Add a property to override HTTP return code in ListenHTTP processor
> ---
>
> Key: NIFI-4597
> URL: https://issues.apache.org/jira/browse/NIFI-4597
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Sébastien Bouchex Bellomié
> Fix For: 1.5.0
>
>
> Current ListenHTTP processor is always returning 200 as HTTP return code but 
> is some case, 204 may be required.
> I know that I may use the HandlerHttpRequest and HandlerHttpResponse couple 
> but that's a bit overkill to this.
> The idea is to have a property (default value is 200) giving the ability to 
> override the return code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4597) Add a property to override HTTP return code in ListenHTTP processor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272772#comment-16272772
 ] 

ASF subversion and git services commented on NIFI-4597:
---

Commit 33281300cd90842da426d6824e942c43d4cf4916 in nifi's branch 
refs/heads/master from [~sbouc...@infovista.com]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3328130 ]

NIFI-4597 This closes #2283. correcting imports, checkstyle issues, and adding 
validator
Add a property to override HTTP return code in ListenHTTP processor

Signed-off-by: joewitt 


> Add a property to override HTTP return code in ListenHTTP processor
> ---
>
> Key: NIFI-4597
> URL: https://issues.apache.org/jira/browse/NIFI-4597
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Sébastien Bouchex Bellomié
> Fix For: 1.5.0
>
>
> Current ListenHTTP processor is always returning 200 as HTTP return code but 
> is some case, 204 may be required.
> I know that I may use the HandlerHttpRequest and HandlerHttpResponse couple 
> but that's a bit overkill to this.
> The idea is to have a property (default value is 200) giving the ability to 
> override the return code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4597) Add a property to override HTTP return code in ListenHTTP processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272774#comment-16272774
 ] 

ASF GitHub Bot commented on NIFI-4597:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2283


> Add a property to override HTTP return code in ListenHTTP processor
> ---
>
> Key: NIFI-4597
> URL: https://issues.apache.org/jira/browse/NIFI-4597
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Sébastien Bouchex Bellomié
> Fix For: 1.5.0
>
>
> Current ListenHTTP processor is always returning 200 as HTTP return code but 
> is some case, 204 may be required.
> I know that I may use the HandlerHttpRequest and HandlerHttpResponse couple 
> but that's a bit overkill to this.
> The idea is to have a property (default value is 200) giving the ability to 
> override the return code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2283: NIFI-4597 Add a property to override HTTP return co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2283


---

[jira] [Commented] (NIFI-4559) ExecuteStreamCommand should have a failure relationship

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272737#comment-16272737
 ] 

ASF GitHub Bot commented on NIFI-4559:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2246
  
@MikeThomsen Thank you for your review! I think I tested it fairly well 
also, so with your +1 I can merge this in.


> ExecuteStreamCommand should have a failure relationship
> ---
>
> Key: NIFI-4559
> URL: https://issues.apache.org/jira/browse/NIFI-4559
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> ExecuteStreamCommand only has a success relationship.  The gist is it 
> executes the command, captures the output stream, code, etc.. and stores the 
> necessary content and attributes.
> However, for those wanting to see the status code as an indicator of failure 
> they have to create a flow which checks, loops, etc..  This is more work than 
> necessary since the processor can provide a better indicator.  By adding a 
> failure relationship we can route results and status codes for non-zero to 
> this relationship.  We'll have to consider backward compatibility friendly 
> ideas but the gist is to make this common failure/retry pattern easier to 
> configure and track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2246: NIFI-4559: Add non-zero status relationship to ExecuteStre...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2246
  
@MikeThomsen Thank you for your review! I think I tested it fairly well 
also, so with your +1 I can merge this in.


---

[jira] [Commented] (NIFI-4648) Max Bin Age within MergeContent has misleading description

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272707#comment-16272707
 ] 

Joseph Witt commented on NIFI-4648:
---

[~nadenf] have you verified there is actually a problem or were you just 
noticing the code?  I ask because that 'asTimePeriod' mechanism takes the 
underlying string value of say (5 mins) and converts it to the seconds integer 
necessary.  This way a processor can be built against whatever time unit it 
wants/needs and all the logic is in a single point for interpreting that from 
what the user supplied.



> Max Bin Age within MergeContent has misleading description
> --
>
> Key: NIFI-4648
> URL: https://issues.apache.org/jira/browse/NIFI-4648
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Franco
>
> The Max Bin Age attribute within MergeContent has the description as such:
> "The maximum age of a Bin that will trigger a Bin to be complete. Expected 
> format is   where  is a positive integer and 
> time unit is one of seconds, minutes, hours"
> Except that it only supports seconds as you can tell here: 
> binManager.setMaxBinAge(context.getProperty(MAX_BIN_AGE).asTimePeriod(TimeUnit.SECONDS).intValue());



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4646) AbstractRecordProcessor: Report reason for SchemaNotFoundException

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272703#comment-16272703
 ] 

ASF subversion and git services commented on NIFI-4646:
---

Commit 3df3ff6c610760c3adb1dffdd75d8cbaed1b471b in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3df3ff6 ]

NIFI-4646: This closes #2306.
- Ensuring the reason behind the SchemaNotFoundException is reported when 
interacting with the configured SchemaRegistry.

Signed-off-by: joewitt 


> AbstractRecordProcessor: Report reason for SchemaNotFoundException
> --
>
> Key: NIFI-4646
> URL: https://issues.apache.org/jira/browse/NIFI-4646
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Trivial
> Fix For: 1.5.0
>
>
> When a schema registry is unable to retrieve a schema, the reason is reported 
> in the corresponding exception. However, when that exception is handled in 
> the corresponding record processor, the reasoning is not included in the 
> ProcessException that is thrown for the framework to catch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4646) AbstractRecordProcessor: Report reason for SchemaNotFoundException

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4646:
--
   Resolution: Fixed
Fix Version/s: 1.5.0
   Status: Resolved  (was: Patch Available)

> AbstractRecordProcessor: Report reason for SchemaNotFoundException
> --
>
> Key: NIFI-4646
> URL: https://issues.apache.org/jira/browse/NIFI-4646
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Trivial
> Fix For: 1.5.0
>
>
> When a schema registry is unable to retrieve a schema, the reason is reported 
> in the corresponding exception. However, when that exception is handled in 
> the corresponding record processor, the reasoning is not included in the 
> ProcessException that is thrown for the framework to catch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4646) AbstractRecordProcessor: Report reason for SchemaNotFoundException

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272705#comment-16272705
 ] 

Joseph Witt commented on NIFI-4646:
---

+1 merged to master

> AbstractRecordProcessor: Report reason for SchemaNotFoundException
> --
>
> Key: NIFI-4646
> URL: https://issues.apache.org/jira/browse/NIFI-4646
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Trivial
> Fix For: 1.5.0
>
>
> When a schema registry is unable to retrieve a schema, the reason is reported 
> in the corresponding exception. However, when that exception is handled in 
> the corresponding record processor, the reasoning is not included in the 
> ProcessException that is thrown for the framework to catch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4646) AbstractRecordProcessor: Report reason for SchemaNotFoundException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272704#comment-16272704
 ] 

ASF GitHub Bot commented on NIFI-4646:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2306


> AbstractRecordProcessor: Report reason for SchemaNotFoundException
> --
>
> Key: NIFI-4646
> URL: https://issues.apache.org/jira/browse/NIFI-4646
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Trivial
> Fix For: 1.5.0
>
>
> When a schema registry is unable to retrieve a schema, the reason is reported 
> in the corresponding exception. However, when that exception is handled in 
> the corresponding record processor, the reasoning is not included in the 
> ProcessException that is thrown for the framework to catch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2306: NIFI-4646: AbstractRecordProcessor: Report reason f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2306


---

[jira] [Updated] (NIFI-3538) Add DeleteHBase processor(s)

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-3538:
---
Status: Patch Available  (was: Open)

> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (MINIFICPP-336) With default GetFile settings dot files are not getting ignored on linux systems as they should

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo reassigned MINIFICPP-336:


Assignee: marco polo

> With default GetFile settings dot files are not getting ignored on linux 
> systems as they should
> ---
>
> Key: MINIFICPP-336
> URL: https://issues.apache.org/jira/browse/MINIFICPP-336
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0, 0.1.0, 0.2.0
>Reporter: Joseph Witt
>Assignee: marco polo
> Fix For: 0.4.0
>
>
> With this config
> Processors:
> - name: GetFile
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> Batch Size: '10'
> File Filter: '[^\.].*'
> Ignore Hidden Files: 'true'
> Input Directory: test/input
> Keep Source File: 'false'
> Maximum File Age:
> Maximum File Size:
> Minimum File Age: 0 sec
> Minimum File Size: 0 B
> Path Filter:
> Polling Interval: 0 sec
> Recurse Subdirectories: 'true'
> The minifi flow picks up any files starting with '.' character right away.  I 
> believe this is causing duplication to occur when NiFi writes to that 
> directory being watched, for example, because it writes the files a 
> hidden/dot notation then renamed it when done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3472) Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase processors

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272642#comment-16272642
 ] 

Jorge Machado commented on NIFI-3472:
-

[~jtstorck]  How is this going ? The UserGroupInformation spwans the Thread 
that renews the ticket for us. Have use used that ? Check: 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1037

> Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase 
> processors
> --
>
> Key: NIFI-3472
> URL: https://issues.apache.org/jira/browse/NIFI-3472
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>
> PutHDFS is not able to relogin if the ticket expires.
> NiFi, running locally as standalone, was sending files to HDFS.  After 
> suspending the system for the weekend, when the flow attempted to continue to 
> process flowfiles, the following exception occurred:
> {code}2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10] 
> org.apache.hadoop.ipc.Client Exception encountered while connecting to the 
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
> 2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10] 
> o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of 
> class ClientNamenodeProtocolTranslatorPB over [host:port] after 3 fail over 
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: 
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]; Host Details : local host is: "[host:port]"; destination 
> host is: [host:port];
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1479) 
> ~[hadoop-common-2.7.3.jar:na]
>   at org.apache.hadoop.ipc.Client.call(Client.java:1412) 
> ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>  ~[hadoop-hdfs-2.7.3.jar:na]
>   at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source) 
> ~[na:na]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_102]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>  ~[hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>  ~[hadoop-common-2.7.3.jar:na]
>   at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
>   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) 
> [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>  [hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
>  [hadoop-hdfs-2.7.3.jar:na]
>   at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262) 
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
>   at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.8.0_102]
>   at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
>  [hadoop-common-2.7.3.jar:na]
>   at 
> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) 
> [nifi-hdfs-processors-1.1.1.jar:1.1.1]
>   at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>  [nifi-api-1.1.1.jar:1.1.1]
>   at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
>  [nifi-framework-core-1.1.1.jar:1.1.1]
>   at 
> 

[jira] [Updated] (MINIFICPP-336) With default GetFile settings dot files are not getting ignored on linux systems as they should

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo updated MINIFICPP-336:
-
Fix Version/s: 0.4.0

> With default GetFile settings dot files are not getting ignored on linux 
> systems as they should
> ---
>
> Key: MINIFICPP-336
> URL: https://issues.apache.org/jira/browse/MINIFICPP-336
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0, 0.1.0, 0.2.0
>Reporter: Joseph Witt
> Fix For: 0.4.0
>
>
> With this config
> Processors:
> - name: GetFile
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> Batch Size: '10'
> File Filter: '[^\.].*'
> Ignore Hidden Files: 'true'
> Input Directory: test/input
> Keep Source File: 'false'
> Maximum File Age:
> Maximum File Size:
> Minimum File Age: 0 sec
> Minimum File Size: 0 B
> Path Filter:
> Polling Interval: 0 sec
> Recurse Subdirectories: 'true'
> The minifi flow picks up any files starting with '.' character right away.  I 
> believe this is causing duplication to occur when NiFi writes to that 
> directory being watched, for example, because it writes the files a 
> hidden/dot notation then renamed it when done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (MINIFICPP-336) With default GetFile settings dot files are not getting ignored on linux systems as they should

[marco polo (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272589#comment-16272589
 ] 

marco polo edited comment on MINIFICPP-336 at 11/30/17 12:17 PM:
-

offending code is here:


{code:java}
 if (request.ignoreHiddenFile && fullName.c_str()[0] == '.')
  return false;

{code}

Should be


{code:java}
if (request.ignoreHiddenFile && name.c_str()[0] == '.')
  return false;
{code}



was (Author: phrocker):
offending code is here:


{code:cpp}
 if (request.ignoreHiddenFile && fullName.c_str()[0] == '.')
  return false;

{code}

Should be


{code:cpp}
if (request.ignoreHiddenFile && name.c_str()[0] == '.')
  return false;
{code}


> With default GetFile settings dot files are not getting ignored on linux 
> systems as they should
> ---
>
> Key: MINIFICPP-336
> URL: https://issues.apache.org/jira/browse/MINIFICPP-336
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0, 0.1.0, 0.2.0
>Reporter: Joseph Witt
> Fix For: 0.4.0
>
>
> With this config
> Processors:
> - name: GetFile
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> Batch Size: '10'
> File Filter: '[^\.].*'
> Ignore Hidden Files: 'true'
> Input Directory: test/input
> Keep Source File: 'false'
> Maximum File Age:
> Maximum File Size:
> Minimum File Age: 0 sec
> Minimum File Size: 0 B
> Path Filter:
> Polling Interval: 0 sec
> Recurse Subdirectories: 'true'
> The minifi flow picks up any files starting with '.' character right away.  I 
> believe this is causing duplication to occur when NiFi writes to that 
> directory being watched, for example, because it writes the files a 
> hidden/dot notation then renamed it when done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-336) With default GetFile settings dot files are not getting ignored on linux systems as they should

[marco polo (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272589#comment-16272589
 ] 

marco polo commented on MINIFICPP-336:
--

offending code is here:


{code:cpp}
 if (request.ignoreHiddenFile && fullName.c_str()[0] == '.')
  return false;

{code}

Should be


{code:cpp}
if (request.ignoreHiddenFile && name.c_str()[0] == '.')
  return false;
{code}


> With default GetFile settings dot files are not getting ignored on linux 
> systems as they should
> ---
>
> Key: MINIFICPP-336
> URL: https://issues.apache.org/jira/browse/MINIFICPP-336
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.3.0, 0.1.0, 0.2.0
>Reporter: Joseph Witt
>
> With this config
> Processors:
> - name: GetFile
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> Batch Size: '10'
> File Filter: '[^\.].*'
> Ignore Hidden Files: 'true'
> Input Directory: test/input
> Keep Source File: 'false'
> Maximum File Age:
> Maximum File Size:
> Minimum File Age: 0 sec
> Minimum File Size: 0 B
> Path Filter:
> Polling Interval: 0 sec
> Recurse Subdirectories: 'true'
> The minifi flow picks up any files starting with '.' character right away.  I 
> believe this is causing duplication to occur when NiFi writes to that 
> directory being watched, for example, because it writes the files a 
> hidden/dot notation then renamed it when done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)