[jira] [Commented] (NIFI-4069) ListXXX processors can miss files those created while the processor is listing and filesystem does not provide timestamp milliseconds precision

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051262#comment-16051262
 ] 

ASF GitHub Bot commented on NIFI-4069:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1915
  
Fixed test issues, and added a property to specify timestamp precision. 
Let's see if all test passes on Travis.


> ListXXX processors can miss files those created while the processor is 
> listing and filesystem does not provide timestamp milliseconds precision
> ---
>
> Key: NIFI-4069
> URL: https://issues.apache.org/jira/browse/NIFI-4069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: ListFilesWithoutMilliseconds.png
>
>
> For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3 
> are known that only support timestamp in seconds precision. Also some FTP 
> server is reported that it can only provides timestamp precision in minutes.
> This can cause files to NOT be listed as ListXXX processors logic expects 
> timestamps in milliseconds.
> Specifically, if generate several files in one second, not all files will be 
> listened.
> Steps to reproduce:
> 1. start processor ListFile
> 2. generate 1 zero size files with following command:
> {code}
> for i in {1..1}; do touch ./test_$i; done
> {code}
> 3. see processor stats: out 3952 (0 bytes)
> Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and 
> postponing the files those have the latest timestamp within a listing 
> iteration to next iteration, however with those filesystem without 
> milliseconds precision, these logics do not work as expected.
> This issue is originally reported at nifi-dev ML. 
> http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1915: NIFI-4069: Make ListXXX work with timestamp precision in s...

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1915
  
Fixed test issues, and added a property to specify timestamp precision. 
Let's see if all test passes on Travis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4081) GrokReader - add the option to keep raw message in a dedicated field

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4081:
-
Status: Patch Available  (was: Open)

> GrokReader - add the option to keep raw message in a dedicated field
> 
>
> Key: NIFI-4081
> URL: https://issues.apache.org/jira/browse/NIFI-4081
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> In some use cases, it can be useful to keep the raw message in the record. I 
> propose to add a parameter to the GrokReader allowing the user to store in 
> the field {{rawMessage}} the raw content of the record (stack trace included 
> if any).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4081) GrokReader - add the option to keep raw message in a dedicated field

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051147#comment-16051147
 ] 

ASF GitHub Bot commented on NIFI-4081:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1921

NIFI-4081 - Added raw message option in GrokReader

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1921


commit ad7e8ee657d30c80ad16de9c3550837d645b574b
Author: Pierre Villard 
Date:   2017-06-15T22:09:45Z

NIFI-4081 - Added raw message option in GrokReader




> GrokReader - add the option to keep raw message in a dedicated field
> 
>
> Key: NIFI-4081
> URL: https://issues.apache.org/jira/browse/NIFI-4081
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> In some use cases, it can be useful to keep the raw message in the record. I 
> propose to add a parameter to the GrokReader allowing the user to store in 
> the field {{rawMessage}} the raw content of the record (stack trace included 
> if any).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1921: NIFI-4081 - Added raw message option in GrokReader

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1921

NIFI-4081 - Added raw message option in GrokReader

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1921


commit ad7e8ee657d30c80ad16de9c3550837d645b574b
Author: Pierre Villard 
Date:   2017-06-15T22:09:45Z

NIFI-4081 - Added raw message option in GrokReader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4081) GrokReader - add the option to keep raw message in a dedicated field

[Pierre Villard (JIRA)]

Pierre Villard created NIFI-4081:


 Summary: GrokReader - add the option to keep raw message in a 
dedicated field
 Key: NIFI-4081
 URL: https://issues.apache.org/jira/browse/NIFI-4081
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.3.0
Reporter: Pierre Villard
Assignee: Pierre Villard
Priority: Minor


In some use cases, it can be useful to keep the raw message in the record. I 
propose to add a parameter to the GrokReader allowing the user to store in the 
field {{rawMessage}} the raw content of the record (stack trace included if 
any).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4080) ValidateCSV - Add support for Expression Language

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-4080:
--

 Summary: ValidateCSV - Add support for Expression Language 
 Key: NIFI-4080
 URL: https://issues.apache.org/jira/browse/NIFI-4080
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Matt Burgess


The ValidateCSV processor could benefit if the following fields supported 
Expression Language evaluation:

- Schema
- Quote character
- Delimiter character
- End of line symbols



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4079) Add concat function in Record Path Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051051#comment-16051051
 ] 

ASF GitHub Bot commented on NIFI-4079:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1920

NIFI-4079 Add concat in Record Path Guide

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4079

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1920


commit 7c64de3acf05e80a6d454fa7267b1a5a672308c4
Author: Pierre Villard 
Date:   2017-06-15T20:52:28Z

NIFI-4079 Add concat in Record Path Guide




> Add concat function in Record Path Guide
> 
>
> Key: NIFI-4079
> URL: https://issues.apache.org/jira/browse/NIFI-4079
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> In the Record Path Guide, the documentation for the standalone function 
> {{concat}} is missing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4079) Add concat function in Record Path Guide

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4079:
-
Status: Patch Available  (was: Open)

> Add concat function in Record Path Guide
> 
>
> Key: NIFI-4079
> URL: https://issues.apache.org/jira/browse/NIFI-4079
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> In the Record Path Guide, the documentation for the standalone function 
> {{concat}} is missing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1920: NIFI-4079 Add concat in Record Path Guide

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1920

NIFI-4079 Add concat in Record Path Guide

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4079

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1920


commit 7c64de3acf05e80a6d454fa7267b1a5a672308c4
Author: Pierre Villard 
Date:   2017-06-15T20:52:28Z

NIFI-4079 Add concat in Record Path Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4079) Add concat function in Record Path Guide

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4079:
-
Summary: Add concat function in Record Path Guide  (was: Add concat 
function in Recrod Path Guide)

> Add concat function in Record Path Guide
> 
>
> Key: NIFI-4079
> URL: https://issues.apache.org/jira/browse/NIFI-4079
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> In the Record Path Guide, the documentation for the standalone function 
> {{concat}} is missing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4079) Add concat function in Recrod Path Guide

[Pierre Villard (JIRA)]

Pierre Villard created NIFI-4079:


 Summary: Add concat function in Recrod Path Guide
 Key: NIFI-4079
 URL: https://issues.apache.org/jira/browse/NIFI-4079
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Documentation & Website
Affects Versions: 1.3.0
Reporter: Pierre Villard
Assignee: Pierre Villard
Priority: Trivial


In the Record Path Guide, the documentation for the standalone function 
{{concat}} is missing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4078) Provide capability for record conversion inspection

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-4078:
---

 Summary: Provide capability for record conversion inspection
 Key: NIFI-4078
 URL: https://issues.apache.org/jira/browse/NIFI-4078
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Affects Versions: 1.3.0
Reporter: Andy LoPresto


The {{ConvertRecord}} and related processors are a great new feature. I do 
think a useful feature would be inter-step debug for record conversion. In the 
same way you can start a processor and not the follow-on and inspect the 
flowfiles in the queue for correctness, we should be able to do that during 
record conversion. So if I have input format {{X}} and output format {{Y}}, I 
want to see what the ephemeral value is after {{X}} has been parsed but before 
{{Y}} has been generated. This will allow for better resolution of situations 
where the output is not as expected. This could be because the input is not 
being read/parsed correctly, or because the output is not being generated 
correctly. Being able to bisect that issue will greatly help with solving the 
problem. 

I anticipate a context menu option to inspect flowfiles being processed by the 
component when it is enabled and running. While there is no internal queue for 
the processor, logically, it is the same as {{List Flowfile}} being executed on 
a queue. Depending on the processing volume, this time slice window may be able 
to hold flowfiles for some time following processing (a few flowfiles in a few 
seconds) or it may need to rapidly refresh the "queue" but maintain the 
flowfile that was under examination to avoid confusing the user. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-1586) embedded zookeeper disk utilization grows unbounded

[Jeff Storck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Storck updated NIFI-1586:
--
Affects Version/s: (was: 0.4.1)
   1.3.0

> embedded zookeeper disk utilization grows unbounded
> ---
>
> Key: NIFI-1586
> URL: https://issues.apache.org/jira/browse/NIFI-1586
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
> Environment: latest 0.5.1 release
>Reporter: Matthew Clarke
>
> Observed that embedded NiFi zookeeper disk utilization will grow unbounded.  
> Zookeeper will occasional create snapshots but at no time will it ever purge 
> any of those snapshots it creates. This behavior is documented here:
> https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_administering
> It is the operators responsibility to purge old snapshot files. NiFi needs to 
> provide a configuration that will automate this pruge.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-1586) embedded zookeeper disk utilization grows unbounded

[Jeff Storck (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050942#comment-16050942
 ] 

Jeff Storck commented on NIFI-1586:
---

This issue has started to be observed again.  After doing some research on 
ZooKeeper's auto purge functionality, it looks like we need to start the 
ZK-provided DatadirCleanupManager when the embedded ZK server is started.  I'll 
look into this a bit more to see if there's a better solution for ZK 3.4.6 
(which we're using currently), but otherwise the DatadirCleanupManager will 
solve this issue.

> embedded zookeeper disk utilization grows unbounded
> ---
>
> Key: NIFI-1586
> URL: https://issues.apache.org/jira/browse/NIFI-1586
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
> Environment: latest 0.5.1 release
>Reporter: Matthew Clarke
>
> Observed that embedded NiFi zookeeper disk utilization will grow unbounded.  
> Zookeeper will occasional create snapshots but at no time will it ever purge 
> any of those snapshots it creates. This behavior is documented here:
> https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_administering
> It is the operators responsibility to purge old snapshot files. NiFi needs to 
> provide a configuration that will automate this pruge.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4031) Nullable Array

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050928#comment-16050928
 ] 

ASF GitHub Bot commented on NIFI-4031:
--

GitHub user champagst opened a pull request:

https://github.com/apache/nifi/pull/1919

NIFI-4031: Fix Avro nullable arrays

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/champagst/nifi nifi-4031

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1919.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1919


commit 9d2c18480de7ae96f41c4b91e947439f492f297c
Author: Steve Champagne 
Date:   2017-06-15T18:24:24Z

NIFI-4031: Fix Avro nullable arrays




> Nullable Array
> --
>
> Key: NIFI-4031
> URL: https://issues.apache.org/jira/browse/NIFI-4031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Steve Champagne
> Attachments: NullableArray.xml
>
>
> I'm getting an error when I try to use a nullable array. I've attached an 
> example template.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1919: NIFI-4031: Fix Avro nullable arrays

[champagst]

GitHub user champagst opened a pull request:

https://github.com/apache/nifi/pull/1919

NIFI-4031: Fix Avro nullable arrays

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/champagst/nifi nifi-4031

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1919.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1919


commit 9d2c18480de7ae96f41c4b91e947439f492f297c
Author: Steve Champagne 
Date:   2017-06-15T18:24:24Z

NIFI-4031: Fix Avro nullable arrays




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4031) Nullable Array

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050926#comment-16050926
 ] 

ASF GitHub Bot commented on NIFI-4031:
--

Github user champagst commented on the issue:

https://github.com/apache/nifi/pull/1895
  
Ah, yeah, that does make sense. I will make the change. Thanks for the 
feedback! 

I had messed up my fork after I submitted this and decided to start from 
scratch, so it might come in as a new pull request. Sorry for any 
inconvenience. 


> Nullable Array
> --
>
> Key: NIFI-4031
> URL: https://issues.apache.org/jira/browse/NIFI-4031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Steve Champagne
> Attachments: NullableArray.xml
>
>
> I'm getting an error when I try to use a nullable array. I've attached an 
> example template.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1895: NIFI-4031: Allow nullable arrays

[champagst]

Github user champagst commented on the issue:

https://github.com/apache/nifi/pull/1895
  
Ah, yeah, that does make sense. I will make the change. Thanks for the 
feedback! 

I had messed up my fork after I submitted this and decided to start from 
scratch, so it might come in as a new pull request. Sorry for any 
inconvenience. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4077) o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from store

[Joseph Gresock (JIRA)]

Joseph Gresock created NIFI-4077:


 Summary: o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve 
Provenance Events from store
 Key: NIFI-4077
 URL: https://issues.apache.org/jira/browse/NIFI-4077
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Joseph Gresock


I get this error when trying to display Data Provenance on a 4-node cluster:

2017-06-15 18:28:23,945 ERROR [Provenance Query-2] 
o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from 
store
java.io.FileNotFoundException: Unable to locate file 
/data/nifi/provenance_repository/85309321.prov
at 
org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:119)
at 
org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$46(WriteAheadProvenanceRepository.java:125)
at 
org.apache.nifi.provenance.store.iterator.SelectiveRecordReaderEventIterator.nextEvent(SelectiveRecordReaderEventIterator.java:137)
at 
org.apache.nifi.provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
at 
org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:192)
at 
org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:167)
at 
org.apache.nifi.provenance.index.lucene.LuceneEventIndex.lambda$submitQuery$58(LuceneEventIndex.java:442)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

I don't know if this is related, but I also see this error in the same logs:

2017-06-15 18:28:42,923 ERROR [Compress Provenance Logs-1-thread-1] 
o.a.n.p.s.EventFileCompressor Failed to read TOC File 
/data/nifi/provenance_repository/toc/87247412.toc
java.io.EOFException: null
at 
org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
at 
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4019) Add support for X-Forward-* headers

[Jeff Storck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Storck updated NIFI-4019:
--
   Resolution: Fixed
Fix Version/s: 1.4.0
   Status: Resolved  (was: Patch Available)

> Add support for X-Forward-* headers
> ---
>
> Key: NIFI-4019
> URL: https://issues.apache.org/jira/browse/NIFI-4019
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Jeff Storck
>Assignee: Matt Gilman
> Fix For: 1.4.0
>
>
> Extend NiFi proxy support by allowing the usage of the following headers:
> || Header || Purpose ||
> | X-Forwarded-Proto | the scheme used to connect to the proxy
> | X-Forwarded-Host | the host of the proxy
> | X-Forwarded-Port | the port on which the proxy is listening
> | X-Forwarded-Context | the the context path of the request to the proxy
> A mapping exists for the X-Forwarded-* headers to the currently supported 
> X-Proxy* headers:
> || X-Forwarded Header || X-Proxy Header ||
> | X-Forwarded-Proto | X-ProxyScheme
> | X-Forwarded-Host | X-ProxyHost
> | X-Forwarded-Port | X-ProxyPort
> | X-Forwarded-Context | X-ProxyContextPath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4019) Add support for X-Forward-* headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050913#comment-16050913
 ] 

ASF GitHub Bot commented on NIFI-4019:
--

Github user mcgilman closed the pull request at:

https://github.com/apache/nifi/pull/1900


> Add support for X-Forward-* headers
> ---
>
> Key: NIFI-4019
> URL: https://issues.apache.org/jira/browse/NIFI-4019
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Jeff Storck
>Assignee: Matt Gilman
>
> Extend NiFi proxy support by allowing the usage of the following headers:
> || Header || Purpose ||
> | X-Forwarded-Proto | the scheme used to connect to the proxy
> | X-Forwarded-Host | the host of the proxy
> | X-Forwarded-Port | the port on which the proxy is listening
> | X-Forwarded-Context | the the context path of the request to the proxy
> A mapping exists for the X-Forwarded-* headers to the currently supported 
> X-Proxy* headers:
> || X-Forwarded Header || X-Proxy Header ||
> | X-Forwarded-Proto | X-ProxyScheme
> | X-Forwarded-Host | X-ProxyHost
> | X-Forwarded-Port | X-ProxyPort
> | X-Forwarded-Context | X-ProxyContextPath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4019) Add support for X-Forward-* headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050912#comment-16050912
 ] 

ASF GitHub Bot commented on NIFI-4019:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1900
  
Yup. Thanks again!


> Add support for X-Forward-* headers
> ---
>
> Key: NIFI-4019
> URL: https://issues.apache.org/jira/browse/NIFI-4019
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Jeff Storck
>Assignee: Matt Gilman
>
> Extend NiFi proxy support by allowing the usage of the following headers:
> || Header || Purpose ||
> | X-Forwarded-Proto | the scheme used to connect to the proxy
> | X-Forwarded-Host | the host of the proxy
> | X-Forwarded-Port | the port on which the proxy is listening
> | X-Forwarded-Context | the the context path of the request to the proxy
> A mapping exists for the X-Forwarded-* headers to the currently supported 
> X-Proxy* headers:
> || X-Forwarded Header || X-Proxy Header ||
> | X-Forwarded-Proto | X-ProxyScheme
> | X-Forwarded-Host | X-ProxyHost
> | X-Forwarded-Port | X-ProxyPort
> | X-Forwarded-Context | X-ProxyContextPath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1900: NIFI-4019: Added support for X-Forwarded-* headers

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1900
  
Yup. Thanks again!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1900: NIFI-4019: Added support for X-Forwarded-* headers

[mcgilman]

Github user mcgilman closed the pull request at:

https://github.com/apache/nifi/pull/1900


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4019) Add support for X-Forward-* headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050908#comment-16050908
 ] 

ASF GitHub Bot commented on NIFI-4019:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1900
  
These changes look good with a couple extra fixes, merged to master.


> Add support for X-Forward-* headers
> ---
>
> Key: NIFI-4019
> URL: https://issues.apache.org/jira/browse/NIFI-4019
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Jeff Storck
>Assignee: Matt Gilman
>
> Extend NiFi proxy support by allowing the usage of the following headers:
> || Header || Purpose ||
> | X-Forwarded-Proto | the scheme used to connect to the proxy
> | X-Forwarded-Host | the host of the proxy
> | X-Forwarded-Port | the port on which the proxy is listening
> | X-Forwarded-Context | the the context path of the request to the proxy
> A mapping exists for the X-Forwarded-* headers to the currently supported 
> X-Proxy* headers:
> || X-Forwarded Header || X-Proxy Header ||
> | X-Forwarded-Proto | X-ProxyScheme
> | X-Forwarded-Host | X-ProxyHost
> | X-Forwarded-Port | X-ProxyPort
> | X-Forwarded-Context | X-ProxyContextPath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1900: NIFI-4019: Added support for X-Forwarded-* headers

[jtstorck]

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1900
  
These changes look good with a couple extra fixes, merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4076) Cluster screen 'Versions' tab does not update "Showing X of Y" properly

[Mark Payne (JIRA)]

Mark Payne created NIFI-4076:


 Summary: Cluster screen 'Versions' tab does not update "Showing X 
of Y" properly
 Key: NIFI-4076
 URL: https://issues.apache.org/jira/browse/NIFI-4076
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core UI
Reporter: Mark Payne


I have a cluster of 3 nodes. Each node has 2 different Content Repositories 
configured. If I open the Cluster dialog, on the Nodes tab, it says "Displaying 
3 of 3," which is correct. If I then click on "Versions" It shows "Displaying 3 
of 3." When I click "Content Storage," it indicates "Displaying 6 of 6." Again, 
all of this is correct. However, if I change the tab from "Content Storage" to 
"Versions" I see "Displaying 3 of 6" instead of "Displaying 3 of 3."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4019) Add support for X-Forward-* headers

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050903#comment-16050903
 ] 

ASF subversion and git services commented on NIFI-4019:
---

Commit 490e1da5dbfcf4a39e43c1559d5e6ca75f021011 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=490e1da ]

NIFI-4019:
- Adding support for X-Forwarded-* headers.
- Unrelated code clean up.
- Addressing additional proxying issues.
- Addressing mistyped landing page.
- Handled trailing slashes in proxy headers for context path


> Add support for X-Forward-* headers
> ---
>
> Key: NIFI-4019
> URL: https://issues.apache.org/jira/browse/NIFI-4019
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Jeff Storck
>Assignee: Matt Gilman
>
> Extend NiFi proxy support by allowing the usage of the following headers:
> || Header || Purpose ||
> | X-Forwarded-Proto | the scheme used to connect to the proxy
> | X-Forwarded-Host | the host of the proxy
> | X-Forwarded-Port | the port on which the proxy is listening
> | X-Forwarded-Context | the the context path of the request to the proxy
> A mapping exists for the X-Forwarded-* headers to the currently supported 
> X-Proxy* headers:
> || X-Forwarded Header || X-Proxy Header ||
> | X-Forwarded-Proto | X-ProxyScheme
> | X-Forwarded-Host | X-ProxyHost
> | X-Forwarded-Port | X-ProxyPort
> | X-Forwarded-Context | X-ProxyContextPath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4061) Implement a StateProvider backed by Redis

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-4061:
--
Status: Patch Available  (was: Open)

> Implement a StateProvider backed by Redis
> -
>
> Key: NIFI-4061
> URL: https://issues.apache.org/jira/browse/NIFI-4061
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> We currently have only one clustered state provider which is a ZooKeeper 
> implementation. Redis would make a good candidate to provide another option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4061) Implement a StateProvider backed by Redis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050835#comment-16050835
 ] 

ASF GitHub Bot commented on NIFI-4061:
--

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/1918

NIFI-4061 Add a RedisStateProvider

This PR adds a RedisStateProvider that can be used as an alternative to the 
ZooKeeperStateProvider for clustered state.

This PR includes the Redis work from NIFI-4043, so this could be merged to 
include both NIFI-4043 and NIFI-4061, or the other one could be merged first 
and then this branch could be rebased.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-4061

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1918.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1918


commit 3cfe414eabc82192797fbd5d39c6c27860dc3eaa
Author: Bryan Bende 
Date:   2017-06-12T19:53:20Z

NIFI-4043 Initial commit of nifi-redis-bundle

commit 1581cd1662bf6239465fe6fefa472903788a255e
Author: Bryan Bende 
Date:   2017-06-13T19:57:22Z

NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in 
StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all 
the time




> Implement a StateProvider backed by Redis
> -
>
> Key: NIFI-4061
> URL: https://issues.apache.org/jira/browse/NIFI-4061
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> We currently have only one clustered state provider which is a ZooKeeper 
> implementation. Redis would make a good candidate to provide another option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1918: NIFI-4061 Add a RedisStateProvider

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/1918

NIFI-4061 Add a RedisStateProvider

This PR adds a RedisStateProvider that can be used as an alternative to the 
ZooKeeperStateProvider for clustered state.

This PR includes the Redis work from NIFI-4043, so this could be merged to 
include both NIFI-4043 and NIFI-4061, or the other one could be merged first 
and then this branch could be rebased.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-4061

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1918.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1918


commit 3cfe414eabc82192797fbd5d39c6c27860dc3eaa
Author: Bryan Bende 
Date:   2017-06-12T19:53:20Z

NIFI-4043 Initial commit of nifi-redis-bundle

commit 1581cd1662bf6239465fe6fefa472903788a255e
Author: Bryan Bende 
Date:   2017-06-13T19:57:22Z

NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in 
StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all 
the time




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4031) Nullable Array

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050812#comment-16050812
 ] 

ASF GitHub Bot commented on NIFI-4031:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1895
  
Hey @champagst sorry for the delay in getting this reviewed. I think the 
solution that you provided here will work for this use case, but I'm a little 
bit concerned with this approach. I say that because if a field has a type of 
ARRAY, then the object should be array, not a List. Avro, however, comes with 
some oddities, and so in the AvroTypeUtil class, there is a method named 
isCompatibleDataType(). It checks if the "avro form" of an object is compatible 
with a given data type. So I think this is where we should add this logic. 
Adding it to DataTypeUtils may result in a different writer expecting an array 
but getting a List object. If we instead just update 
AvroTypeUtil#isCompatibleDataType to allow for a List in the case of Arrays, 
then I think we will still have the desired outcome but will prevent passing 
List objects around generically for ARRAY data types. Does that make sense?


> Nullable Array
> --
>
> Key: NIFI-4031
> URL: https://issues.apache.org/jira/browse/NIFI-4031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Steve Champagne
> Attachments: NullableArray.xml
>
>
> I'm getting an error when I try to use a nullable array. I've attached an 
> example template.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1895: NIFI-4031: Allow nullable arrays

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1895
  
Hey @champagst sorry for the delay in getting this reviewed. I think the 
solution that you provided here will work for this use case, but I'm a little 
bit concerned with this approach. I say that because if a field has a type of 
ARRAY, then the object should be array, not a List. Avro, however, comes with 
some oddities, and so in the AvroTypeUtil class, there is a method named 
isCompatibleDataType(). It checks if the "avro form" of an object is compatible 
with a given data type. So I think this is where we should add this logic. 
Adding it to DataTypeUtils may result in a different writer expecting an array 
but getting a List object. If we instead just update 
AvroTypeUtil#isCompatibleDataType to allow for a List in the case of Arrays, 
then I think we will still have the desired outcome but will prevent passing 
List objects around generically for ARRAY data types. Does that make sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4075) Inconsistent State for RPGs

[Bryan Bende (JIRA)]

Bryan Bende created NIFI-4075:
-

 Summary: Inconsistent State for RPGs
 Key: NIFI-4075
 URL: https://issues.apache.org/jira/browse/NIFI-4075
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Bryan Bende
Priority: Blocker
 Fix For: 1.4.0


It was reported on the mailing list that RPGs were shown as 
enabled/transmitting in the UI, but were not actually running.

After investigating, it appears this could be related to NIFI-3900 that was 
addressed in 1.3.0.

That fix introduced a ScheduledStateLookUp which was used for determining the 
state of processors and ports, for example:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/StandardFlowSerializer.java#L354

But it appears to not be used for RemoteGroupPorts:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/StandardFlowSerializer.java#L314



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4074) ListHDFS recursive search with regex

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050416#comment-16050416
 ] 

ASF GitHub Bot commented on NIFI-4074:
--

GitHub user champagst opened a pull request:

https://github.com/apache/nifi/pull/1917

NIFI-4074: Fix recursive regex search

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/champagst/nifi nifi-4074

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1917


commit ca6984f8050a2915c8a02d79e0f6ab4d7aed3652
Author: Steve Champagne 
Date:   2017-06-15T12:41:23Z

NIFI-4074: Fix recursive regex search




> ListHDFS recursive search with regex
> 
>
> Key: NIFI-4074
> URL: https://issues.apache.org/jira/browse/NIFI-4074
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Steve Champagne
>
> ListHDFS isn't picking up nested files correctly when you use a regex.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1917: NIFI-4074: Fix recursive regex search

[champagst]

GitHub user champagst opened a pull request:

https://github.com/apache/nifi/pull/1917

NIFI-4074: Fix recursive regex search

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/champagst/nifi nifi-4074

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1917


commit ca6984f8050a2915c8a02d79e0f6ab4d7aed3652
Author: Steve Champagne 
Date:   2017-06-15T12:41:23Z

NIFI-4074: Fix recursive regex search




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4074) ListHDFS recursive search with regex

[Steve Champagne (JIRA)]

Steve Champagne created NIFI-4074:
-

 Summary: ListHDFS recursive search with regex
 Key: NIFI-4074
 URL: https://issues.apache.org/jira/browse/NIFI-4074
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Steve Champagne


ListHDFS isn't picking up nested files correctly when you use a regex.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4073) Duplicated stack trace in ThreadPoolRequestReplicator

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4073:
-
Status: Patch Available  (was: Open)

> Duplicated stack trace in ThreadPoolRequestReplicator
> -
>
> Key: NIFI-4073
> URL: https://issues.apache.org/jira/browse/NIFI-4073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> There is a duplicated stack trace due to the following part.
> {code:title=ThreadPoolRequestReplicator.java|borderStyle=solid}
> try {
> // create and send the request
> final WebResource.Builder resourceBuilder = createResourceBuilder();
> final String requestId = headers.get("x-nifi-request-id");
> logger.debug("Replicating request {} {} to {}", method, uri.getPath(), 
> nodeId);
> nodeResponse = replicateRequest(resourceBuilder, nodeId, method, uri, 
> requestId, headers, clusterResponse);
> } catch (final Exception e) {
> nodeResponse = new NodeResponse(nodeId, method, uri, e);
> logger.warn("Failed to replicate request {} {} to {} due to {}", method, 
> uri.getPath(), nodeId, e);
> logger.warn("", e);
> }
> {code}
> It just requires to use {{e.toString()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4073) Duplicated stack trace in ThreadPoolRequestReplicator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050300#comment-16050300
 ] 

ASF GitHub Bot commented on NIFI-4073:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1916

NIFI-4073 - fix duplicated stack trace

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4073

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1916


commit d5512207948dc84f4b5e3b09e77e862bbd2f7b64
Author: Pierre Villard 
Date:   2017-06-15T10:32:42Z

NIFI-4073 - fix duplicated stack trace




> Duplicated stack trace in ThreadPoolRequestReplicator
> -
>
> Key: NIFI-4073
> URL: https://issues.apache.org/jira/browse/NIFI-4073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> There is a duplicated stack trace due to the following part.
> {code:title=ThreadPoolRequestReplicator.java|borderStyle=solid}
> try {
> // create and send the request
> final WebResource.Builder resourceBuilder = createResourceBuilder();
> final String requestId = headers.get("x-nifi-request-id");
> logger.debug("Replicating request {} {} to {}", method, uri.getPath(), 
> nodeId);
> nodeResponse = replicateRequest(resourceBuilder, nodeId, method, uri, 
> requestId, headers, clusterResponse);
> } catch (final Exception e) {
> nodeResponse = new NodeResponse(nodeId, method, uri, e);
> logger.warn("Failed to replicate request {} {} to {} due to {}", method, 
> uri.getPath(), nodeId, e);
> logger.warn("", e);
> }
> {code}
> It just requires to use {{e.toString()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1916: NIFI-4073 - fix duplicated stack trace

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1916

NIFI-4073 - fix duplicated stack trace

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4073

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1916


commit d5512207948dc84f4b5e3b09e77e862bbd2f7b64
Author: Pierre Villard 
Date:   2017-06-15T10:32:42Z

NIFI-4073 - fix duplicated stack trace




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4073) Duplicated stack trace in ThreadPoolRequestReplicator

[Pierre Villard (JIRA)]

Pierre Villard created NIFI-4073:


 Summary: Duplicated stack trace in ThreadPoolRequestReplicator
 Key: NIFI-4073
 URL: https://issues.apache.org/jira/browse/NIFI-4073
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.3.0
Reporter: Pierre Villard
Assignee: Pierre Villard
Priority: Trivial


There is a duplicated stack trace due to the following part.

{code:title=ThreadPoolRequestReplicator.java|borderStyle=solid}
try {
// create and send the request
final WebResource.Builder resourceBuilder = createResourceBuilder();
final String requestId = headers.get("x-nifi-request-id");

logger.debug("Replicating request {} {} to {}", method, uri.getPath(), 
nodeId);
nodeResponse = replicateRequest(resourceBuilder, nodeId, method, uri, 
requestId, headers, clusterResponse);
} catch (final Exception e) {
nodeResponse = new NodeResponse(nodeId, method, uri, e);
logger.warn("Failed to replicate request {} {} to {} due to {}", method, 
uri.getPath(), nodeId, e);
logger.warn("", e);
}
{code}

It just requires to use {{e.toString()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4069) ListXXX processors can miss files those created while the processor is listing and filesystem does not provide timestamp milliseconds precision

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050060#comment-16050060
 ] 

ASF GitHub Bot commented on NIFI-4069:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1915
  
Unit tests fail occasionally. Still investigating the cause.


> ListXXX processors can miss files those created while the processor is 
> listing and filesystem does not provide timestamp milliseconds precision
> ---
>
> Key: NIFI-4069
> URL: https://issues.apache.org/jira/browse/NIFI-4069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: ListFilesWithoutMilliseconds.png
>
>
> For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3 
> are known that only support timestamp in seconds precision. Also some FTP 
> server is reported that it can only provides timestamp precision in minutes.
> This can cause files to NOT be listed as ListXXX processors logic expects 
> timestamps in milliseconds.
> Specifically, if generate several files in one second, not all files will be 
> listened.
> Steps to reproduce:
> 1. start processor ListFile
> 2. generate 1 zero size files with following command:
> {code}
> for i in {1..1}; do touch ./test_$i; done
> {code}
> 3. see processor stats: out 3952 (0 bytes)
> Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and 
> postponing the files those have the latest timestamp within a listing 
> iteration to next iteration, however with those filesystem without 
> milliseconds precision, these logics do not work as expected.
> This issue is originally reported at nifi-dev ML. 
> http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1915: NIFI-4069: Make ListXXX work with timestamp precision in s...

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1915
  
Unit tests fail occasionally. Still investigating the cause.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---