[jira] [Updated] (NIFI-2709) Email processors with Exchange don't output to RFC2822 format

[Andre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andre updated NIFI-2709:

Attachment: screenshot-1.png

> Email processors with Exchange don't output to RFC2822 format
> -
>
> Key: NIFI-2709
> URL: https://issues.apache.org/jira/browse/NIFI-2709
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
> Environment: Ubuntu 16.04 LTS with openjdk-8-jre-headless
>Reporter: Emil Frank
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> When using the new ConsumeIMAP and ConsumePOP3 processors with a Microsoft 
> Exchange 2013 IMAP server the flowfiles which are produced are simple HTML 
> messages with no RFC2822 headers. I have also tried setting Exchange to force 
> emails to be text only, sadly only the body with some Content-Type: fields 
> are outputed.
> This mean that ExtractEmailHeaders and ExtractEmailAttachments cannot be used 
> directly with these processors.
> In Python, I can force Exchange to output the headers by specify RFC822 in 
> the connection settings:
> - https://docs.python.org/3/library/imaplib.html#imap4-example
> Is a similar option available for the spring mail framework?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2709) Email processors with Exchange don't output to RFC2822 format

[Andre (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515502#comment-15515502
 ] 

Andre commented on NIFI-2709:
-

[~lightraven24]

Thanks for raising this issue. 

Can you confirm if setting up the message retrieval format as documented in 
[here|https://technet.microsoft.com/en-us/library/aa997869(v=exchg.150).aspx] 
works?



> Email processors with Exchange don't output to RFC2822 format
> -
>
> Key: NIFI-2709
> URL: https://issues.apache.org/jira/browse/NIFI-2709
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
> Environment: Ubuntu 16.04 LTS with openjdk-8-jre-headless
>Reporter: Emil Frank
>Priority: Minor
>
> When using the new ConsumeIMAP and ConsumePOP3 processors with a Microsoft 
> Exchange 2013 IMAP server the flowfiles which are produced are simple HTML 
> messages with no RFC2822 headers. I have also tried setting Exchange to force 
> emails to be text only, sadly only the body with some Content-Type: fields 
> are outputed.
> This mean that ExtractEmailHeaders and ExtractEmailAttachments cannot be used 
> directly with these processors.
> In Python, I can force Exchange to output the headers by specify RFC822 in 
> the connection settings:
> - https://docs.python.org/3/library/imaplib.html#imap4-example
> Is a similar option available for the spring mail framework?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #38: MINIFI-86 Adding explicit checks for any unsup...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/38


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2810) Allow Content Type to be set in PutS3Object processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514617#comment-15514617
 ] 

ASF GitHub Bot commented on NIFI-2810:
--

Github user vegaed commented on the issue:

https://github.com/apache/nifi/pull/1034
  
@pvillard31 I think I did it right should be good to go now.


> Allow Content Type to be set in PutS3Object processor
> -
>
> Key: NIFI-2810
> URL: https://issues.apache.org/jira/browse/NIFI-2810
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.1.0
>Reporter: Edgardo Vega
>Priority: Minor
> Fix For: 1.1.0
>
>
> The PutS3 Processor does not allow you to set the Content Type. Add this 
> ability in this processor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1034: [NIFI-2810] Allow Content Type to be set in PutS3Object pr...

[vegaed]

Github user vegaed commented on the issue:

https://github.com/apache/nifi/pull/1034
  
@pvillard31 I think I did it right should be good to go now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-2690) Label color is incorrect after template import

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2690:
-

Assignee: Scott Aslan

> Label color is incorrect after template import
> --
>
> Key: NIFI-2690
> URL: https://issues.apache.org/jira/browse/NIFI-2690
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Minor
> Attachments: WebCrawler.xml
>
>
> Steps to reproduce:
> 1. Import the template WebCrawler.xml (attached)
> 2. Add this template to the canvas
> 3. The label "This is configured to use yahoo mail.  You will need to change 
> it for your email server and address." should have the color #ffde93, but is 
> #fff7d7 instead.
> This is a regression/change from 0.7.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2810) Allow Content Type to be set in PutS3Object processor

[Edgardo Vega (JIRA)]

Edgardo Vega created NIFI-2810:
--

 Summary: Allow Content Type to be set in PutS3Object processor
 Key: NIFI-2810
 URL: https://issues.apache.org/jira/browse/NIFI-2810
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Affects Versions: 1.1.0
Reporter: Edgardo Vega
Priority: Minor
 Fix For: 1.1.0


The PutS3 Processor does not allow you to set the Content Type. Add this 
ability in this processor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[Joseph Gresock (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Gresock closed NIFI-2417.


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514391#comment-15514391
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user gresockj commented on the issue:

https://github.com/apache/nifi/pull/733
  
Thanks for the help!


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514392#comment-15514392
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user gresockj closed the pull request at:

https://github.com/apache/nifi/pull/733


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[gresockj]

Github user gresockj closed the pull request at:

https://github.com/apache/nifi/pull/733


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #733: NIFI-2417: Implementing QueryElasticsearchHttp and ScrollEl...

[gresockj]

Github user gresockj commented on the issue:

https://github.com/apache/nifi/pull/733
  
Thanks for the help!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1034: Allow Content Type to be set in PutS3Object processor

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1034
  
@vegaed LGTM. Do you have an associated JIRA for this? If not, could you 
create one and link it to this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2805) Add fragment attributes for SplitAvro

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-2805:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Build with contrib-check, verified attributes in a basic workflow, merged into 
master.

> Add fragment attributes for SplitAvro
> -
>
> Key: NIFI-2805
> URL: https://issues.apache.org/jira/browse/NIFI-2805
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitAvro processor should write these 
> same attributes for its split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2805) Add fragment attributes for SplitAvro

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514305#comment-15514305
 ] 

ASF subversion and git services commented on NIFI-2805:
---

Commit 49297b725d4770040c2cc2c01514ef49b2680c88 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=49297b7 ]

NIFI-2805: Add fragment attributes to SplitAvro

This closes #1046.


> Add fragment attributes for SplitAvro
> -
>
> Key: NIFI-2805
> URL: https://issues.apache.org/jira/browse/NIFI-2805
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitAvro processor should write these 
> same attributes for its split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1046: NIFI-2805: Add fragment attributes to SplitAvro

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1046


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2071) Support repeating capture groups in ExtractText

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-2071:
-
 Assignee: Pierre Villard
Fix Version/s: 1.1.0
   Status: Patch Available  (was: Open)

> Support repeating capture groups in ExtractText
> ---
>
> Key: NIFI-2071
> URL: https://issues.apache.org/jira/browse/NIFI-2071
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Pierre Villard
> Fix For: 1.1.0
>
>
> ExtractText doesn't currently support repeating capture groups so any 
> repeating patterns have to specified by hand and can only be repeated a fixed 
> number of times.
> I think this is because it only uses find() and not findAll() in its pattern 
> matching [1].
> 1. 
> https://github.com/apache/nifi/blob/1bd2cf0d09a7111bcecffd0f473aa71c25a69845/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractText.java#L324



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1050: NIFI-2071 - Support repeating capture groups in Ext...

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1050

NIFI-2071 - Support repeating capture groups in ExtractText



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI2071

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1050.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1050


commit 1c30a97b404b10629317cf3ab928925eaad3e73e
Author: Pierre Villard 
Date:   2016-09-22T18:43:42Z

NIFI-2071 - Support repeating capture groups in ExtractText




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514195#comment-15514195
 ] 

ASF subversion and git services commented on NIFI-2417:
---

Commit 00412f6e97fdc077cf9c8c0d365ebe4876a33305 in nifi's branch 
refs/heads/master from Joe Gresock
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=00412f6 ]

NIFI-2417: Adding Query and Scroll processors for Elasticsearch

Signed-off-by: Matt Burgess 


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514198#comment-15514198
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/733
  
Oops, forgot to rebase and add the comment that auto-closes this PR. Do you 
mind closing it? I'll Resolve the Jira. Thanks!


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess resolved NIFI-2417.

Resolution: Fixed

> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514196#comment-15514196
 ] 

ASF subversion and git services commented on NIFI-2417:
---

Commit 937dc71aec107cdd0367d75d91435eaa5c7e73d9 in nifi's branch 
refs/heads/master from Joe Gresock
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=937dc71 ]

NIFI-2417: Addressing code review comments

Signed-off-by: Matt Burgess 


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #733: NIFI-2417: Implementing QueryElasticsearchHttp and ScrollEl...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/733
  
Oops, forgot to rebase and add the comment that auto-closes this PR. Do you 
mind closing it? I'll Resolve the Jira. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514191#comment-15514191
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/733
  
+1 LGTM, Thanks for the great contribution! Lots of folks have asked about 
this capability, glad to have it in NiFi :) Merging to master


> Implement Query and Scroll processors for ElasticSearch
> ---
>
> Key: NIFI-2417
> URL: https://issues.apache.org/jira/browse/NIFI-2417
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Minor
> Fix For: 1.1.0
>
>
> FetchElasticsearchHttp allows users to select a single document from 
> Elasticsearch in NiFi, but there is no way to run a query to retrieve 
> multiple documents.
> We should add a QueryElasticsearchHttp processor for running a query and 
> returning a flow file per result, for small result sets.  This should allow 
> both input and non-input execution.  
> A separate ScrollElasticsearchHttp processor would also be useful for 
> scrolling through a huge result set.  This should use the state manager to 
> maintain the scroll_id value, and use this as input to the next scroll page.  
> As a result, this processor should not allow flow file input, but should 
> retrieve one page per run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514180#comment-15514180
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80112353
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final 

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80112353
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FINISHED_QUERY_STATE = "finishedQuery";
+private static final String SCROLL_ID_STATE = "scrollId";
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String 

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514174#comment-15514174
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user gresockj commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80111838
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String 

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[gresockj]

Github user gresockj commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80111838
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FINISHED_QUERY_STATE = "finishedQuery";
+private static final String SCROLL_ID_STATE = "scrollId";
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String 

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514167#comment-15514167
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user gresockj commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80111550
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String 

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[gresockj]

Github user gresockj commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80111550
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FINISHED_QUERY_STATE = "finishedQuery";
+private static final String SCROLL_ID_STATE = "scrollId";
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String 

[jira] [Commented] (NIFI-2781) Broken bower install

[Matt Gilman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514165#comment-15514165
 ] 

Matt Gilman commented on NIFI-2781:
---

[~apsaltis] Scott's PR addressed the build issues I was seeing on my Windows 
VM. I've also verified offline builds and merged this to master. Can you pull 
down the latest and see if it addresses the issues you were seeing too? Would 
like to verify prior to closing out the JIRA you opened. Thanks!

> Broken bower install
> 
>
> Key: NIFI-2781
> URL: https://issues.apache.org/jira/browse/NIFI-2781
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Psaltis
>Assignee: Scott Aslan
>
> Often times when building the NiFi source, the installation of bower fails 
> and thus the whole build fails. 
> Looking at a mvn debug log, appears to show this:
> [INFO] Running 'npm install bower' in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [WARNING] npm WARN package.json mqlight@1.0.2016051011 No repository field.
> [ERROR] npm http GET https://registry.npmjs.org/bower
> [ERROR] npm http 200 https://registry.npmjs.org/bower
> [INFO] bower@1.7.9 ../../../../../../../../../node_modules/bower
> [INFO] 
> [INFO] --- frontend-maven-plugin:1.0:bower (bower-install) @ nifi-web-ui ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.0:bower 
> from plugin realm 
> ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.0, parent: 
> sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo 
> 'com.github.eirslett:frontend-maven-plugin:1.0:bower' with basic configurator 
> -->
> [DEBUG]   (f) arguments = install  
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets
> [DEBUG]   (f) installDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) project = MavenProject: 
> org.apache.nifi:nifi-web-ui:1.0.0-SNAPSHOT @ 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
> [DEBUG]   (f) repositorySystemSession = 
> org.eclipse.aether.DefaultRepositorySystemSession@315edc50
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@4052c8c2
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = false
> [DEBUG]   (f) workingDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.0:bower 
> {execution: bower-install}
> [DEBUG] -- end configuration --
> [INFO] Running 'bower install 
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets'
>  in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [ERROR] 
> [ERROR] module.js:340
> [ERROR] throw err;
> [ERROR]   ^
> [ERROR] Error: Cannot find module 
> '/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node_modules/bower/bin/bower'
> [ERROR] at Function.Module._resolveFilename (module.js:338:15)
> [ERROR] at Function.Module._load (module.js:280:25)
> [ERROR] at Function.Module.runMain (module.js:497:10)
> [ERROR] at startup (node.js:119:16)
> [ERROR] at node.js:901:3
> [INFO] 
> 
> In essence bower is not installed. This is reproducible in my environment, 
> which looks like this:
> $ printenv
> TERM_PROGRAM=iTerm.app
> TERM=xterm
> SHELL=/bin/bash
> TMPDIR=/var/folders/5l/xlzh00b10_lf16cd06dqwvs4gn/T/
> Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jdxbKLHhHZ/Render
> TERM_PROGRAM_VERSION=3.0.7
> OLDPWD=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui
> TERM_SESSION_ID=w0t0p1:4401C1E6-AF27-43E2-91BF-7814B628BEC1
> USER=apsaltis
> SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.VHh1dBoww3/Listeners
> __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
> PATH=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sb...
> 

[jira] [Commented] (NIFI-2781) Broken bower install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514160#comment-15514160
 ] 

ASF GitHub Bot commented on NIFI-2781:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1030
  
Thanks @scottyaslan! This looks good and addressed the build issues I was 
seeing in my Windows VM. Also verified offline build.

This has been merged to master.


> Broken bower install
> 
>
> Key: NIFI-2781
> URL: https://issues.apache.org/jira/browse/NIFI-2781
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Psaltis
>Assignee: Scott Aslan
>
> Often times when building the NiFi source, the installation of bower fails 
> and thus the whole build fails. 
> Looking at a mvn debug log, appears to show this:
> [INFO] Running 'npm install bower' in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [WARNING] npm WARN package.json mqlight@1.0.2016051011 No repository field.
> [ERROR] npm http GET https://registry.npmjs.org/bower
> [ERROR] npm http 200 https://registry.npmjs.org/bower
> [INFO] bower@1.7.9 ../../../../../../../../../node_modules/bower
> [INFO] 
> [INFO] --- frontend-maven-plugin:1.0:bower (bower-install) @ nifi-web-ui ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.0:bower 
> from plugin realm 
> ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.0, parent: 
> sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo 
> 'com.github.eirslett:frontend-maven-plugin:1.0:bower' with basic configurator 
> -->
> [DEBUG]   (f) arguments = install  
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets
> [DEBUG]   (f) installDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) project = MavenProject: 
> org.apache.nifi:nifi-web-ui:1.0.0-SNAPSHOT @ 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
> [DEBUG]   (f) repositorySystemSession = 
> org.eclipse.aether.DefaultRepositorySystemSession@315edc50
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@4052c8c2
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = false
> [DEBUG]   (f) workingDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.0:bower 
> {execution: bower-install}
> [DEBUG] -- end configuration --
> [INFO] Running 'bower install 
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets'
>  in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [ERROR] 
> [ERROR] module.js:340
> [ERROR] throw err;
> [ERROR]   ^
> [ERROR] Error: Cannot find module 
> '/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node_modules/bower/bin/bower'
> [ERROR] at Function.Module._resolveFilename (module.js:338:15)
> [ERROR] at Function.Module._load (module.js:280:25)
> [ERROR] at Function.Module.runMain (module.js:497:10)
> [ERROR] at startup (node.js:119:16)
> [ERROR] at node.js:901:3
> [INFO] 
> 
> In essence bower is not installed. This is reproducible in my environment, 
> which looks like this:
> $ printenv
> TERM_PROGRAM=iTerm.app
> TERM=xterm
> SHELL=/bin/bash
> TMPDIR=/var/folders/5l/xlzh00b10_lf16cd06dqwvs4gn/T/
> Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jdxbKLHhHZ/Render
> TERM_PROGRAM_VERSION=3.0.7
> OLDPWD=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui
> TERM_SESSION_ID=w0t0p1:4401C1E6-AF27-43E2-91BF-7814B628BEC1
> USER=apsaltis
> SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.VHh1dBoww3/Listeners
> __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
> PATH=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sb...
> PWD=/Users/apsaltis/development/forked-nifi
> 

[GitHub] nifi issue #1030: [NIFI-2781] use npm to install bower

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1030
  
Thanks @scottyaslan! This looks good and addressed the build issues I was 
seeing in my Windows VM. Also verified offline build.

This has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2781) Broken bower install

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514154#comment-15514154
 ] 

ASF subversion and git services commented on NIFI-2781:
---

Commit e9738747930406cf1611ec9f08031f4c3c0e5fee in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e973874 ]

[NIFI-2781] removing bower and allowing npm to manage client side resources in 
nifi-web-ui and nifi-jolt-transform-json-ui. This closes #1030


> Broken bower install
> 
>
> Key: NIFI-2781
> URL: https://issues.apache.org/jira/browse/NIFI-2781
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Psaltis
>Assignee: Scott Aslan
>
> Often times when building the NiFi source, the installation of bower fails 
> and thus the whole build fails. 
> Looking at a mvn debug log, appears to show this:
> [INFO] Running 'npm install bower' in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [WARNING] npm WARN package.json mqlight@1.0.2016051011 No repository field.
> [ERROR] npm http GET https://registry.npmjs.org/bower
> [ERROR] npm http 200 https://registry.npmjs.org/bower
> [INFO] bower@1.7.9 ../../../../../../../../../node_modules/bower
> [INFO] 
> [INFO] --- frontend-maven-plugin:1.0:bower (bower-install) @ nifi-web-ui ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.0:bower 
> from plugin realm 
> ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.0, parent: 
> sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo 
> 'com.github.eirslett:frontend-maven-plugin:1.0:bower' with basic configurator 
> -->
> [DEBUG]   (f) arguments = install  
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets
> [DEBUG]   (f) installDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) project = MavenProject: 
> org.apache.nifi:nifi-web-ui:1.0.0-SNAPSHOT @ 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
> [DEBUG]   (f) repositorySystemSession = 
> org.eclipse.aether.DefaultRepositorySystemSession@315edc50
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@4052c8c2
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = false
> [DEBUG]   (f) workingDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.0:bower 
> {execution: bower-install}
> [DEBUG] -- end configuration --
> [INFO] Running 'bower install 
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets'
>  in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [ERROR] 
> [ERROR] module.js:340
> [ERROR] throw err;
> [ERROR]   ^
> [ERROR] Error: Cannot find module 
> '/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node_modules/bower/bin/bower'
> [ERROR] at Function.Module._resolveFilename (module.js:338:15)
> [ERROR] at Function.Module._load (module.js:280:25)
> [ERROR] at Function.Module.runMain (module.js:497:10)
> [ERROR] at startup (node.js:119:16)
> [ERROR] at node.js:901:3
> [INFO] 
> 
> In essence bower is not installed. This is reproducible in my environment, 
> which looks like this:
> $ printenv
> TERM_PROGRAM=iTerm.app
> TERM=xterm
> SHELL=/bin/bash
> TMPDIR=/var/folders/5l/xlzh00b10_lf16cd06dqwvs4gn/T/
> Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jdxbKLHhHZ/Render
> TERM_PROGRAM_VERSION=3.0.7
> OLDPWD=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui
> TERM_SESSION_ID=w0t0p1:4401C1E6-AF27-43E2-91BF-7814B628BEC1
> USER=apsaltis
> SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.VHh1dBoww3/Listeners
> __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
> 

[jira] [Commented] (NIFI-2781) Broken bower install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514156#comment-15514156
 ] 

ASF GitHub Bot commented on NIFI-2781:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1030


> Broken bower install
> 
>
> Key: NIFI-2781
> URL: https://issues.apache.org/jira/browse/NIFI-2781
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Psaltis
>Assignee: Scott Aslan
>
> Often times when building the NiFi source, the installation of bower fails 
> and thus the whole build fails. 
> Looking at a mvn debug log, appears to show this:
> [INFO] Running 'npm install bower' in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [WARNING] npm WARN package.json mqlight@1.0.2016051011 No repository field.
> [ERROR] npm http GET https://registry.npmjs.org/bower
> [ERROR] npm http 200 https://registry.npmjs.org/bower
> [INFO] bower@1.7.9 ../../../../../../../../../node_modules/bower
> [INFO] 
> [INFO] --- frontend-maven-plugin:1.0:bower (bower-install) @ nifi-web-ui ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.0:bower 
> from plugin realm 
> ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.0, parent: 
> sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo 
> 'com.github.eirslett:frontend-maven-plugin:1.0:bower' with basic configurator 
> -->
> [DEBUG]   (f) arguments = install  
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets
> [DEBUG]   (f) installDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) project = MavenProject: 
> org.apache.nifi:nifi-web-ui:1.0.0-SNAPSHOT @ 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
> [DEBUG]   (f) repositorySystemSession = 
> org.eclipse.aether.DefaultRepositorySystemSession@315edc50
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@4052c8c2
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = false
> [DEBUG]   (f) workingDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.0:bower 
> {execution: bower-install}
> [DEBUG] -- end configuration --
> [INFO] Running 'bower install 
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets'
>  in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [ERROR] 
> [ERROR] module.js:340
> [ERROR] throw err;
> [ERROR]   ^
> [ERROR] Error: Cannot find module 
> '/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node_modules/bower/bin/bower'
> [ERROR] at Function.Module._resolveFilename (module.js:338:15)
> [ERROR] at Function.Module._load (module.js:280:25)
> [ERROR] at Function.Module.runMain (module.js:497:10)
> [ERROR] at startup (node.js:119:16)
> [ERROR] at node.js:901:3
> [INFO] 
> 
> In essence bower is not installed. This is reproducible in my environment, 
> which looks like this:
> $ printenv
> TERM_PROGRAM=iTerm.app
> TERM=xterm
> SHELL=/bin/bash
> TMPDIR=/var/folders/5l/xlzh00b10_lf16cd06dqwvs4gn/T/
> Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jdxbKLHhHZ/Render
> TERM_PROGRAM_VERSION=3.0.7
> OLDPWD=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui
> TERM_SESSION_ID=w0t0p1:4401C1E6-AF27-43E2-91BF-7814B628BEC1
> USER=apsaltis
> SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.VHh1dBoww3/Listeners
> __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
> PATH=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sb...
> PWD=/Users/apsaltis/development/forked-nifi
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home
> LANG=en_US.UTF-8
> ITERM_PROFILE=Default
> XPC_FLAGS=0x0
> XPC_SERVICE_NAME=0
> SHLVL=1
> HOME=/Users/apsaltis
> 

[GitHub] nifi pull request #1030: [NIFI-2781] use npm to install bower

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1030


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #34: MINIFI-82 Adding support for Processor 'Annota...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/34


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2636) UnpackContent has concurrent thread safety issue, causes flowfiles to fail

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514084#comment-15514084
 ] 

ASF subversion and git services commented on NIFI-2636:
---

Commit a952dc96a23a418e25f9dc49c8874b59adc34654 in nifi's branch refs/heads/0.x 
from [~boardm26]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a952dc9 ]

NIFI-2636 resolve thread safety problem in UnpackContent

Signed-off-by: Joe Skora 


> UnpackContent has concurrent thread safety issue, causes flowfiles to fail
> --
>
> Key: NIFI-2636
> URL: https://issues.apache.org/jira/browse/NIFI-2636
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Michael Moser
>Assignee: Michael Moser
> Fix For: 1.1.0, 0.8.0
>
>
> Shortly after merging NIFI-2611 I took a last look at the code and noticed 
> that each onTrigger() call, when the Packaging Format property is set to "use 
> mime.type attribute", that the class instance variable "private Unpacker 
> unpacker" can change.  When UnpackContent is set to > 1 concurrent task, this 
> isn't thread safe.  Thread A can set the unpacker to the TarUnpacker, but 
> before it gets a chance to unpack its tar file, Thread B changes the unpacker 
> to a FlowFileUnpackagerV3 which causes Thread A to fail its unpack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2781) Broken bower install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514051#comment-15514051
 ] 

ASF GitHub Bot commented on NIFI-2781:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1030
  
Reviewing...


> Broken bower install
> 
>
> Key: NIFI-2781
> URL: https://issues.apache.org/jira/browse/NIFI-2781
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Psaltis
>Assignee: Scott Aslan
>
> Often times when building the NiFi source, the installation of bower fails 
> and thus the whole build fails. 
> Looking at a mvn debug log, appears to show this:
> [INFO] Running 'npm install bower' in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [WARNING] npm WARN package.json mqlight@1.0.2016051011 No repository field.
> [ERROR] npm http GET https://registry.npmjs.org/bower
> [ERROR] npm http 200 https://registry.npmjs.org/bower
> [INFO] bower@1.7.9 ../../../../../../../../../node_modules/bower
> [INFO] 
> [INFO] --- frontend-maven-plugin:1.0:bower (bower-install) @ nifi-web-ui ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.0:bower 
> from plugin realm 
> ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.0, parent: 
> sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo 
> 'com.github.eirslett:frontend-maven-plugin:1.0:bower' with basic configurator 
> -->
> [DEBUG]   (f) arguments = install  
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets
> [DEBUG]   (f) installDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) project = MavenProject: 
> org.apache.nifi:nifi-web-ui:1.0.0-SNAPSHOT @ 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
> [DEBUG]   (f) repositorySystemSession = 
> org.eclipse.aether.DefaultRepositorySystemSession@315edc50
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@4052c8c2
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = false
> [DEBUG]   (f) workingDirectory = 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.0:bower 
> {execution: bower-install}
> [DEBUG] -- end configuration --
> [INFO] Running 'bower install 
> --config.directory=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/nifi-web-ui-1.0.0-SNAPSHOT/assets'
>  in 
> /Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
> [ERROR] 
> [ERROR] module.js:340
> [ERROR] throw err;
> [ERROR]   ^
> [ERROR] Error: Cannot find module 
> '/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node_modules/bower/bin/bower'
> [ERROR] at Function.Module._resolveFilename (module.js:338:15)
> [ERROR] at Function.Module._load (module.js:280:25)
> [ERROR] at Function.Module.runMain (module.js:497:10)
> [ERROR] at startup (node.js:119:16)
> [ERROR] at node.js:901:3
> [INFO] 
> 
> In essence bower is not installed. This is reproducible in my environment, 
> which looks like this:
> $ printenv
> TERM_PROGRAM=iTerm.app
> TERM=xterm
> SHELL=/bin/bash
> TMPDIR=/var/folders/5l/xlzh00b10_lf16cd06dqwvs4gn/T/
> Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.jdxbKLHhHZ/Render
> TERM_PROGRAM_VERSION=3.0.7
> OLDPWD=/Users/apsaltis/development/forked-nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui
> TERM_SESSION_ID=w0t0p1:4401C1E6-AF27-43E2-91BF-7814B628BEC1
> USER=apsaltis
> SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.VHh1dBoww3/Listeners
> __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
> PATH=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sb...
> PWD=/Users/apsaltis/development/forked-nifi
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home
> LANG=en_US.UTF-8
> ITERM_PROFILE=Default
> XPC_FLAGS=0x0
> XPC_SERVICE_NAME=0
> SHLVL=1
> 

[GitHub] nifi issue #1030: [NIFI-2781] use npm to install bower

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1030
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #34: MINIFI-82 Adding support for Processor 'Annota...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/34#discussion_r80102080
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/dto/ProcessorSchemaTest.java
 ---
@@ -45,6 +45,35 @@
 private final String testKey = "testKey";
 private final String testValue = "testValue";
 private final String testPenalizationPeriod = "55 s";
+private final String testAnnotationData = ">criteria\n" +
--- End diff --

Most certainly, just wanted to make sure I wasn't overlooking.  Will merge 
it in.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514029#comment-15514029
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80101565
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final 

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80101565
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FINISHED_QUERY_STATE = "finishedQuery";
+private static final String SCROLL_ID_STATE = "scrollId";
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String 

[GitHub] nifi-minifi pull request #34: MINIFI-82 Adding support for Processor 'Annota...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/34#discussion_r80100987
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/dto/ProcessorSchemaTest.java
 ---
@@ -45,6 +45,35 @@
 private final String testKey = "testKey";
 private final String testValue = "testValue";
 private final String testPenalizationPeriod = "55 s";
+private final String testAnnotationData = ">criteria\n" +
--- End diff --

Ah good catch, it should be removed. I mis-copied it from a the xml of a 
template. Could you just amend the commit to remove that? 

Sorry, I'm in the middle of debugging something else


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #38: MINIFI-86 Adding explicit checks for any unsup...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/38#discussion_r80092383
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/resources/TemplateWithFunnel.xml
 ---
@@ -0,0 +1 @@
+TemplateWithFunnele2c5b2f0-6740-4d57-a40b-20b27da784a35e4f1cc2-78e9-4e08-a0c2-11779c9efc172268.1912159884546695.393079631423509/21/2016
 19:08:15 EDT
--- End diff --

This and the other three test resources should have the license header.  
RAT is currently failing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80098194
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java
 ---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "scroll", "read", "get", "http" })
+@CapabilityDescription("Scrolls through an Elasticsearch query using the 
specified connection properties. "
++ "This processor is intended to be run on the primary node, and 
is designed for scrolling through "
++ "huge result sets, as in the case of a reindex.  The state must 
be cleared before another query "
++ "can be run.  Each page of results is returned, wrapped in a 
JSON object like so: { \"hits\" : [ , ,  ] }.  "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to a Flow File for transfer.")
+@WritesAttributes({
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type") })
+@Stateful(description = "After each successful scroll page, the latest 
scroll_id is persisted in scrollId as input for the next scroll call.  "
++ "Once the entire query is complete, finishedQuery state will be 
set to true, and the processor will not execute unless this is cleared.", 
scopes = { Scope.LOCAL })
+public class ScrollElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FINISHED_QUERY_STATE = "finishedQuery";
+private static final String SCROLL_ID_STATE = "scrollId";
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String 

[jira] [Commented] (NIFI-2417) Implement Query and Scroll processors for ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513959#comment-15513959
 ] 

ASF GitHub Bot commented on NIFI-2417:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80096977
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
 ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "read", "get", "http" })
+@CapabilityDescription("Queries Elasticsearch using the specified 
connection properties. "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to Flow Files for transfer.  Also note that the 
Elasticsearch max_result_window index "
++ "setting is the upper bound on the number of records that can be 
retrieved using this query.  "
++ "To retrieve more records, use the ScrollElasticsearchHttp 
processor.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type"),
+@WritesAttribute(attribute = "es.result.*", description = "If 
Target is 'Flow file attributes', the JSON attributes of "
++ "each result will be placed into corresponding 
attributes with this prefix.") })
+public class QueryElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String QUERY_QUERY_PARAM = "q";
+private static final String SORT_QUERY_PARAM = "sort";
+private static final String FROM_QUERY_PARAM = "from";
+private static final String 

[GitHub] nifi pull request #733: NIFI-2417: Implementing QueryElasticsearchHttp and S...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/733#discussion_r80096977
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
 ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.ByteArrayInputStream;
+import org.codehaus.jackson.JsonNode;
+
+import okhttp3.HttpUrl;
+import okhttp3.OkHttpClient;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@SupportsBatching
+@Tags({ "elasticsearch", "query", "read", "get", "http" })
+@CapabilityDescription("Queries Elasticsearch using the specified 
connection properties. "
++ "Note that the full body of each page of documents will be read 
into memory before being "
++ "written to Flow Files for transfer.  Also note that the 
Elasticsearch max_result_window index "
++ "setting is the upper bound on the number of records that can be 
retrieved using this query.  "
++ "To retrieve more records, use the ScrollElasticsearchHttp 
processor.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type"),
+@WritesAttribute(attribute = "es.result.*", description = "If 
Target is 'Flow file attributes', the JSON attributes of "
++ "each result will be placed into corresponding 
attributes with this prefix.") })
+public class QueryElasticsearchHttp extends 
AbstractElasticsearchHttpProcessor {
+
+private static final String FIELD_INCLUDE_QUERY_PARAM = 
"_source_include";
+private static final String QUERY_QUERY_PARAM = "q";
+private static final String SORT_QUERY_PARAM = "sort";
+private static final String FROM_QUERY_PARAM = "from";
+private static final String SIZE_QUERY_PARAM = "size";
+
+public static final String TARGET_FLOW_FILE_CONTENT = "Flow file 
content";
+public static final String TARGET_FLOW_FILE_ATTRIBUTES = "Flow file 
attributes";
+private static final String 

[GitHub] nifi-minifi pull request #38: MINIFI-86 Adding explicit checks for any unsup...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/38#discussion_r80093282
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMainTest.java
 ---
@@ -173,6 +174,46 @@ public void 
testTransformRoundTripStressTestFramework() throws IOException, JAXB
 transformRoundTrip("StressTestFramework");
 }
 
+@Test
--- End diff --

if anticipating a exception, should likely use the @Test(expected = ) in 
lieu of the try-catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #38: MINIFI-86 Adding explicit checks for any unsup...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/38#discussion_r80093463
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMain.java
 ---
@@ -251,6 +268,10 @@ public int transform(String[] args) {
 System.out.println();
 printTransformUsage();
 return ERR_UNABLE_TO_TRANFORM_TEMPLATE;
--- End diff --

realize this is already existing, but perhaps we should adjust to correct 
spelling error while in this code area


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #37: MINIFI-110 Adding ' ' (space) to the list of c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/37


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-2720) UI - in view configuration textarea is not fully resizable

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2720:
-

Assignee: Scott Aslan

> UI - in view configuration textarea is not fully resizable
> --
>
> Key: NIFI-2720
> URL: https://issues.apache.org/jira/browse/NIFI-2720
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Pierre Villard
>Assignee: Scott Aslan
>Priority: Minor
>
> When a processor is started, accessing 'view configuration', and clicking on 
> a property to display its content, the text area is only vertically 
> resizable. For some processors like ExecuteScript it'd better to be able to 
> fully resize the text area (as it is now when editing the configuration while 
> the processor is stopped).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #36: MINIFI-105 Fixing BootstrapCodec tertiary comm...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/36


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #35: MINIFI-59 Removing exclude statement for Quart...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/35


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information

[James Wing (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513824#comment-15513824
 ] 

James Wing commented on NIFI-2115:
--

Will do... rebase :)

> Enhanced About Box Version Information
> --
>
> Key: NIFI-2115
> URL: https://issues.apache.org/jira/browse/NIFI-2115
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Tools and Build
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Attachments: screenshot-1-of-4-about-expands.png, 
> screenshot-2-of-4-git-build.png, screenshot-3-of-4-release-build.png, 
> screenshot-4-of-4-source-build.png
>
>
> The UI's About dialog and underlying API provide the version of NiFi, like 
> "0.7.0-SNAPSHOT".  For many bug reports and troubleshooting requests, this is 
> not very precise, especially around rapidly changing code or 
> platform-dependent behavior.  It would help if NiFi captured and displayed 
> additional information:  
> * NiFi build commit hash
> * NiFi build branch
> * NiFi build date/time
> * Java Version
> * Java Vendor (Oracle, OpenJDK)
> * OS
> Then a simple copy/paste from the about box would provide very specific 
> information about a particular NiFi installation and which code it was 
> derived from.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2809) Add processors for Google Cloud Storage Fetch/Put/Delete

[Frank Maritato (JIRA)]

Frank Maritato created NIFI-2809:


 Summary: Add processors for Google Cloud Storage Fetch/Put/Delete
 Key: NIFI-2809
 URL: https://issues.apache.org/jira/browse/NIFI-2809
 Project: Apache NiFi
  Issue Type: New Feature
Affects Versions: 1.1.0
Reporter: Frank Maritato


Creating a ticket to add nifi processors for Google Cloud Storage similar to 
how the AWS S3 processors are done:

* FetchGCSObject
* PutGCSObject
* DeleteGCSObject

I have the code mostly written and will attach the patch file when complete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #34: MINIFI-82 Adding support for Processor 'Annota...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/34#discussion_r80084132
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/dto/ProcessorSchemaTest.java
 ---
@@ -45,6 +45,35 @@
 private final String testKey = "testKey";
 private final String testValue = "testValue";
 private final String testPenalizationPeriod = "55 s";
+private final String testAnnotationData = ">criteria\n" +
--- End diff --

Very minor, but Is the beginning '>' a typo? I realize it's just test data, 
but seems like it would be invalid.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1049: Fix for invalid M2_HOME directory in Appveyor

[vegaed]

GitHub user vegaed opened a pull request:

https://github.com/apache/nifi/pull/1049

Fix for invalid M2_HOME directory in Appveyor

Appveyor was given the following error:
**ERROR: M2_HOME is set to an invalid directory.

M2_HOME = "C:\Program Files (x86)\Apache\Maven"

Please set the M2_HOME variable in your environment to match the location 
of the Maven installation**

Fixed by explicitly defining it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vegaed/nifi appveyor_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1049.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1049


commit 7ef2782a4508f88e455c7814800c53d569db7899
Author: Edgardo 
Date:   2016-09-22T16:17:47Z

Fix for invalid M2_HOME directory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1048: Appveyor maven home fix

[vegaed]

Github user vegaed closed the pull request at:

https://github.com/apache/nifi/pull/1048


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1048: Appveyor maven home fix

[vegaed]

GitHub user vegaed opened a pull request:

https://github.com/apache/nifi/pull/1048

Appveyor maven home fix

Appveyor was given the following error:
**ERROR: M2_HOME is set to an invalid directory.

M2_HOME = "C:\Program Files (x86)\Apache\Maven"

Please set the M2_HOME variable in your environment to match the location 
of the Maven installation**

Fixed by explicitly defining it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vegaed/nifi appveyor_maven_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1048.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1048


commit 3901afe0619ae4568a13fb7b4a071d1e7ef97f82
Author: Edgardo 
Date:   2016-09-20T15:08:39Z

Allow Content Type to be set in PutS3Object processor

commit 2a3ca5fb9db0ad8203609871ba70f510ba4df097
Author: Edgardo 
Date:   2016-09-21T14:25:39Z

Merge remote-tracking branch 'upstream/master'

commit dba8cc2d291d77b2b2deaae9c0b63e34c974fbc9
Author: Edgardo 
Date:   2016-09-22T16:11:36Z

Fix for Appveyor M2_HOME invalid directory error




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1034: Allow Content Type to be set in PutS3Object processor

[vegaed]

Github user vegaed commented on the issue:

https://github.com/apache/nifi/pull/1034
  
@trixpan moved it out of this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1034: Allow Content Type to be set in PutS3Object processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1034
  
@vegaed

Thanks for looking at appveyor. Do you mind cherry picking 7a2fc08 into a 
separate PR?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513683#comment-15513683
 ] 

Andrew Lim commented on NIFI-2801:
--

[~alopresto] Happy to take this on.

> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2808) tls-toolkit should generate nifi.properties in client mode

[Bryan Rosander (JIRA)]

Bryan Rosander created NIFI-2808:


 Summary: tls-toolkit should generate nifi.properties in client mode
 Key: NIFI-2808
 URL: https://issues.apache.org/jira/browse/NIFI-2808
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Bryan Rosander


Client/server mode would be more useful if the client was able to generate 
nifi.properties in a similar way to standalone mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2807) tls-toolkit should support using externally generated CA

[Bryan Rosander (JIRA)]

Bryan Rosander created NIFI-2807:


 Summary: tls-toolkit should support using externally generated CA
 Key: NIFI-2807
 URL: https://issues.apache.org/jira/browse/NIFI-2807
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Bryan Rosander


Many users will probably want to bring their own certificates that are either 
signed by a real CA or by their own internal CA.  This isn't currently 
supported in the tls-toolkit but should be.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513669#comment-15513669
 ] 

ASF GitHub Bot commented on NIFI-2115:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/583
  
Looks amazing but I would add some flashing/blinking HTML and a 
construction sign as homage to the good old GeoCities pages. :-)

On a more serious note, @jvwing could you rebase?


> Enhanced About Box Version Information
> --
>
> Key: NIFI-2115
> URL: https://issues.apache.org/jira/browse/NIFI-2115
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Tools and Build
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Attachments: screenshot-1-of-4-about-expands.png, 
> screenshot-2-of-4-git-build.png, screenshot-3-of-4-release-build.png, 
> screenshot-4-of-4-source-build.png
>
>
> The UI's About dialog and underlying API provide the version of NiFi, like 
> "0.7.0-SNAPSHOT".  For many bug reports and troubleshooting requests, this is 
> not very precise, especially around rapidly changing code or 
> platform-dependent behavior.  It would help if NiFi captured and displayed 
> additional information:  
> * NiFi build commit hash
> * NiFi build branch
> * NiFi build date/time
> * Java Version
> * Java Vendor (Oracle, OpenJDK)
> * OS
> Then a simple copy/paste from the about box would provide very specific 
> information about a particular NiFi installation and which code it was 
> derived from.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2806) tls-toolkit client mode specify password

[Bryan Rosander (JIRA)]

Bryan Rosander created NIFI-2806:


 Summary: tls-toolkit client mode specify password
 Key: NIFI-2806
 URL: https://issues.apache.org/jira/browse/NIFI-2806
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Bryan Rosander


When running the tls-toolkit in client mode, it would be useful to be able to 
specify the passwords for keystore, key, truststore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #583: NIFI-2115 Detailed Version Info for About Box

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/583
  
Looks amazing but I would add some flashing/blinking HTML and a 
construction sign as homage to the good old GeoCities pages. :-)

On a more serious note, @jvwing could you rebase?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2806) tls-toolkit client mode specify password

[Bryan Rosander (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Rosander updated NIFI-2806:
-
Labels: tls-toolkit  (was: )

> tls-toolkit client mode specify password
> 
>
> Key: NIFI-2806
> URL: https://issues.apache.org/jira/browse/NIFI-2806
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Rosander
>  Labels: tls-toolkit
>
> When running the tls-toolkit in client mode, it would be useful to be able to 
> specify the passwords for keystore, key, truststore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2786) TLS Toolkit doesn't support -outputDirectory property in all modes

[Bryan Rosander (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Rosander updated NIFI-2786:
-
Labels: tls-toolkit  (was: )

> TLS Toolkit doesn't support -outputDirectory property in all modes
> --
>
> Key: NIFI-2786
> URL: https://issues.apache.org/jira/browse/NIFI-2786
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andrew Grande
>Priority: Minor
>  Labels: tls-toolkit
>
> TLS toolkit has 3 modes:
> # standalone
> # server (host a CA authority server)
> # client (remotely interact with CA authority server)
> There is an -o option to dump all artifacts (config, keystore, truststore) in 
> a dedicated directory (default is . ). However, it's not a supported property 
> for server and client mode.
> This is most useful for 'client' mode. When running on a remote workstation 
> to create new certs signed by a NiFi CA authority, quite a few files are 
> saved. If I need to generate multiple certs for multiple people I would have 
> to manually move files out of the way (otherwise they are overwritten).
> Expected: add -o/--outputDirectory to the client mode (maybe server too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1710) Resolve path name to nifi.sh in start script

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513647#comment-15513647
 ] 

ASF GitHub Bot commented on NIFI-1710:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/315
  
@jfrazee 

could you rebase this PR?


> Resolve path name to nifi.sh in start script
> 
>
> Key: NIFI-1710
> URL: https://issues.apache.org/jira/browse/NIFI-1710
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Priority: Trivial
>
> Currently if you symlink to nifi.sh the start script will not work because $0 
> doesn't evaluate to the physical location and the script assumes 
> bootstrap.conf is relative to the script.
> This means you can't, for example, symlink nifi.sh to /etc/init.d and use the 
> respective service commands to control NiFi.
> Adding realpath or readlink to the script should fix this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #315: NIFI-1710 Resolve path name to nifi.sh in start script

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/315
  
@jfrazee 

could you rebase this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1047: Fix creating NiFi RPM

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1047
  
@fidget83 thank you for the contribution, can you please create a JIRA that 
describes the problem this is trying to fix? and then update this PR to include 
the JIRA number. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-1870) Restore go to source/destination capability

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-1870:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Restore go to source/destination capability
> ---
>
> Key: NIFI-1870
> URL: https://issues.apache.org/jira/browse/NIFI-1870
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1870) Restore go to source/destination capability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513617#comment-15513617
 ] 

ASF GitHub Bot commented on NIFI-1870:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1028


> Restore go to source/destination capability
> ---
>
> Key: NIFI-1870
> URL: https://issues.apache.org/jira/browse/NIFI-1870
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1870) Restore go to source/destination capability

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513614#comment-15513614
 ] 

ASF subversion and git services commented on NIFI-1870:
---

Commit 0218bc61ef6a8f2e2bd1147477eeaa967d29607f in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0218bc6 ]

NIFI-1870: - Restoring the upstream/downstream connections dialog. - Correcting 
input port label when showing downstream components.

This closes #1028.

Signed-off-by: Bryan Bende 


> Restore go to source/destination capability
> ---
>
> Key: NIFI-1870
> URL: https://issues.apache.org/jira/browse/NIFI-1870
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1028: Restore upstream/downstream connections dialog

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1028


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1004: NIFI-2756 - Add displayName to maven archetypes

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1004#discussion_r80064941
  
--- Diff: 
nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
 ---
@@ -41,7 +51,8 @@
 public class MyProcessor extends AbstractProcessor {
 
 public static final PropertyDescriptor MY_PROPERTY = new 
PropertyDescriptor
-.Builder().name("My Property")
+.Builder().name("MY_PROPERTY")
--- End diff --

H... luckily you are democratic enough to review my code with those 
ugly capital letters used on "friendly" names. :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2756) nifi-processor-bundle-archetype lacks displayName property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513547#comment-15513547
 ] 

ASF GitHub Bot commented on NIFI-2756:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1004#discussion_r80064941
  
--- Diff: 
nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
 ---
@@ -41,7 +51,8 @@
 public class MyProcessor extends AbstractProcessor {
 
 public static final PropertyDescriptor MY_PROPERTY = new 
PropertyDescriptor
-.Builder().name("My Property")
+.Builder().name("MY_PROPERTY")
--- End diff --

H... luckily you are democratic enough to review my code with those 
ugly capital letters used on "friendly" names. :-)


> nifi-processor-bundle-archetype lacks displayName property
> --
>
> Key: NIFI-2756
> URL: https://issues.apache.org/jira/browse/NIFI-2756
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Andre
>Assignee: Andre
>
> When using {{nifi-processor-bundle-archetype}} to create a new bundle, the 
> resulting code lacks displayName leading to a number of PR and contributions 
> to arrive to peer review without displayName configured



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2756) nifi-processor-bundle-archetype lacks displayName property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513559#comment-15513559
 ] 

ASF GitHub Bot commented on NIFI-2756:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1004#discussion_r80066292
  
--- Diff: 
nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
 ---
@@ -29,9 +28,20 @@
 import org.apache.nifi.annotation.documentation.SeeAlso;
 import org.apache.nifi.annotation.documentation.Tags;
 import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.util.StandardValidators;
 
-import java.util.*;
+
+
+import java.util.ArrayList;
--- End diff --

addressed


> nifi-processor-bundle-archetype lacks displayName property
> --
>
> Key: NIFI-2756
> URL: https://issues.apache.org/jira/browse/NIFI-2756
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Andre
>Assignee: Andre
>
> When using {{nifi-processor-bundle-archetype}} to create a new bundle, the 
> resulting code lacks displayName leading to a number of PR and contributions 
> to arrive to peer review without displayName configured



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1004: NIFI-2756 - Add displayName to maven archetypes

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1004#discussion_r80066292
  
--- Diff: 
nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
 ---
@@ -29,9 +28,20 @@
 import org.apache.nifi.annotation.documentation.SeeAlso;
 import org.apache.nifi.annotation.documentation.Tags;
 import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
 import org.apache.nifi.processor.util.StandardValidators;
 
-import java.util.*;
+
+
+import java.util.ArrayList;
--- End diff --

addressed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1047: Fix creating NiFi RPM

[fidget83]

GitHub user fidget83 opened a pull request:

https://github.com/apache/nifi/pull/1047

Fix creating NiFi RPM

I've added script nifi_env.sh to bin directory and fixed jars in /lib and 
/lib/bootstrap directories.
Now if  I installed NiFi from rpm, its run correctly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fidget83/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1047.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1047


commit fb0d054332470ff7331ddb90782b34aeb1b833d8
Author: Artem Ermakov 
Date:   2016-09-22T15:15:21Z

Fix creating NiFi RPM




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1004: NIFI-2756 - Add displayName to maven archetypes

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1004#discussion_r80064544
  
--- Diff: 
nifi-maven-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-processors/src/main/java/MyProcessor.java
 ---
@@ -19,7 +19,6 @@
 import org.apache.nifi.components.PropertyDescriptor;
 import org.apache.nifi.components.PropertyValue;
 import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.processor.*;
--- End diff --

welcome


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2805) Add fragment attributes for SplitAvro

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2805:
---
Fix Version/s: 1.1.0
   Status: Patch Available  (was: In Progress)

> Add fragment attributes for SplitAvro
> -
>
> Key: NIFI-2805
> URL: https://issues.apache.org/jira/browse/NIFI-2805
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitAvro processor should write these 
> same attributes for its split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2805) Add fragment attributes for SplitAvro

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513523#comment-15513523
 ] 

ASF GitHub Bot commented on NIFI-2805:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1046

NIFI-2805: Add fragment attributes to SplitAvro

Same idea as #919 but for SplitAvro

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2805

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1046


commit a7b165d4b42961b7eb5cc2b1b6ed99ef6bc33521
Author: Matt Burgess 
Date:   2016-09-22T15:00:11Z

NIFI-2805: Add fragment attributes to SplitAvro




> Add fragment attributes for SplitAvro
> -
>
> Key: NIFI-2805
> URL: https://issues.apache.org/jira/browse/NIFI-2805
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitAvro processor should write these 
> same attributes for its split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1046: NIFI-2805: Add fragment attributes to SplitAvro

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1046

NIFI-2805: Add fragment attributes to SplitAvro

Same idea as #919 but for SplitAvro

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2805

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1046


commit a7b165d4b42961b7eb5cc2b1b6ed99ef6bc33521
Author: Matt Burgess 
Date:   2016-09-22T15:00:11Z

NIFI-2805: Add fragment attributes to SplitAvro




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-1029) Need to add a new attriibute to ExecuteSQL to indicate how many rows were selected

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-1029:
---
Fix Version/s: 0.4.0

> Need to add a new attriibute to ExecuteSQL to indicate how many rows were 
> selected
> --
>
> Key: NIFI-1029
> URL: https://issues.apache.org/jira/browse/NIFI-1029
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.3.0
>Reporter: Scott
>Priority: Critical
> Fix For: 0.4.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When running a select we need to provide a rows.count attribute so we can 
> detect if there is any data. This will prevent passing no data to the 
> ConvertAvroToJson and getting an error because there is no data.
> Relates too NIFI-972, but I cannot figure out how to add that in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2805) Add fragment attributes for SplitAvro

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-2805:
--

 Summary: Add fragment attributes for SplitAvro
 Key: NIFI-2805
 URL: https://issues.apache.org/jira/browse/NIFI-2805
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Matt Burgess
Assignee: Matt Burgess


Some "splitting" processors such as SplitText and SplitContent write attributes 
to the split flow files indicating their fragment index, the total count, and 
the filename of the original flow file. This is done to support a form of 
"micro-batching" and/or a split-join pattern in a data flow (i.e. split the 
original file, do work on the individual pieces, and possibly merge them 
together later).
For consistency and capability, the SplitAvro processor should write these same 
attributes for its split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1458) Add scriptable ReportingTask

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-1458:
---
Status: Patch Available  (was: In Progress)

> Add scriptable ReportingTask
> 
>
> Key: NIFI-1458
> URL: https://issues.apache.org/jira/browse/NIFI-1458
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that NIFI-210 adds scriptable Processors (and scripted onTrigger bodies), 
> a great extension would be to add a scriptable ReportingTask. This would 
> enable users to script their own ReportingTasks using the various supported 
> scripting languages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #31: MINIFI-58 Adjusting toolkit assembly name

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/31


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1458) Add scriptable ReportingTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513404#comment-15513404
 ] 

ASF GitHub Bot commented on NIFI-1458:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1045

NIFI-1458: Added ScriptedReportingTask

I refactored AbstractScriptProcessor into ScriptUtils, so the functionality 
could be shared between scripting processors and scripting reporting tasks. I 
admit the use of ScriptUtils is not pure OO (public members, e.g.) but I wanted 
to keep the spirit of what the processors did when this was part of its parent 
class (since we don't have traits or multiple inheritance).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-1458

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1045.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1045


commit cd5a523b41cbe81a602e4d0f10df38b6a58a6779
Author: Matt Burgess 
Date:   2016-09-22T14:13:40Z

NIFI-1458: Added ScriptedReportingTask




> Add scriptable ReportingTask
> 
>
> Key: NIFI-1458
> URL: https://issues.apache.org/jira/browse/NIFI-1458
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that NIFI-210 adds scriptable Processors (and scripted onTrigger bodies), 
> a great extension would be to add a scriptable ReportingTask. This would 
> enable users to script their own ReportingTasks using the various supported 
> scripting languages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1045: NIFI-1458: Added ScriptedReportingTask

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1045

NIFI-1458: Added ScriptedReportingTask

I refactored AbstractScriptProcessor into ScriptUtils, so the functionality 
could be shared between scripting processors and scripting reporting tasks. I 
admit the use of ScriptUtils is not pure OO (public members, e.g.) but I wanted 
to keep the spirit of what the processors did when this was part of its parent 
class (since we don't have traits or multiple inheritance).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-1458

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1045.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1045


commit cd5a523b41cbe81a602e4d0f10df38b6a58a6779
Author: Matt Burgess 
Date:   2016-09-22T14:13:40Z

NIFI-1458: Added ScriptedReportingTask




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1028: Restore upstream/downstream connections dialog

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1028
  
Good catch @bbende. Will update shortly... Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2804) Support resume in PutSFTP for SFTP file transfers

[Joey Frazee (JIRA)]

Joey Frazee created NIFI-2804:
-

 Summary: Support resume in PutSFTP for SFTP file transfers
 Key: NIFI-2804
 URL: https://issues.apache.org/jira/browse/NIFI-2804
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joey Frazee
Assignee: Joey Frazee
Priority: Minor


SFTP has some limited support for resuming file transfers and this is exposed 
in the underlying JSch ChannelSftp object, so PutSFTP should support 
_optionally_ resuming file transfers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-1458) Add scriptable ReportingTask

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-1458:
--

Assignee: Matt Burgess

> Add scriptable ReportingTask
> 
>
> Key: NIFI-1458
> URL: https://issues.apache.org/jira/browse/NIFI-1458
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that NIFI-210 adds scriptable Processors (and scripted onTrigger bodies), 
> a great extension would be to add a scriptable ReportingTask. This would 
> enable users to script their own ReportingTasks using the various supported 
> scripting languages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1893) Add processor for validating JSON

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513009#comment-15513009
 ] 

ASF GitHub Bot commented on NIFI-1893:
--

Github user bartoszjkwozniak commented on the issue:

https://github.com/apache/nifi/pull/1037
  
@pvillard31 you're welcome! Thank you too and @mattyb149 for valuable code 
review.


> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Matt Burgess
> Fix For: 1.1.0
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a 
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
>   name: "Test",
>   timestamp: 1463499695,
>   tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
>   },
>   fields: {
> "cpu": 10.2,
> "load": 15.6
>   }
> }
> Could be validated successfully against the following "schema":
> {
>   "type": "object",
>   "required": ["name", "tags", "timestamp", "fields"],
>   "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
>   }
> }
> There is at least one ASF-friendly library that could be used for 
> implementation: https://github.com/everit-org/json-schema



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1037: NIFI-1893 Add processor for validating JSON

[bartoszjkwozniak]

Github user bartoszjkwozniak commented on the issue:

https://github.com/apache/nifi/pull/1037
  
@pvillard31 you're welcome! Thank you too and @mattyb149 for valuable code 
review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1037: NIFI-1893 Add processor for validating JSON

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1037
  
@bartoszjkwozniak I merged it to master, thanks for your contribution!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-1893) Add processor for validating JSON

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1893.
--
   Resolution: Fixed
Fix Version/s: 1.1.0

> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Matt Burgess
> Fix For: 1.1.0
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a 
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
>   name: "Test",
>   timestamp: 1463499695,
>   tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
>   },
>   fields: {
> "cpu": 10.2,
> "load": 15.6
>   }
> }
> Could be validated successfully against the following "schema":
> {
>   "type": "object",
>   "required": ["name", "tags", "timestamp", "fields"],
>   "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
>   }
> }
> There is at least one ASF-friendly library that could be used for 
> implementation: https://github.com/everit-org/json-schema



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1893) Add processor for validating JSON

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512986#comment-15512986
 ] 

ASF GitHub Bot commented on NIFI-1893:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1037
  
@bartoszjkwozniak I merged it to master, thanks for your contribution!


> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Matt Burgess
> Fix For: 1.1.0
>
>
> NiFi has a ValidateXml processor to validate incoming XML files against a 
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
>   name: "Test",
>   timestamp: 1463499695,
>   tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
>   },
>   fields: {
> "cpu": 10.2,
> "load": 15.6
>   }
> }
> Could be validated successfully against the following "schema":
> {
>   "type": "object",
>   "required": ["name", "tags", "timestamp", "fields"],
>   "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
>   }
> }
> There is at least one ASF-friendly library that could be used for 
> implementation: https://github.com/everit-org/json-schema



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)