[jira] [Commented] (NIFI-4825) Processor is not shutting down normally

[kim teak jin (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344609#comment-16344609
 ] 

kim teak jin commented on NIFI-4825:


Hello.
I'm a writer.
I would like to report additional information regarding

I have solved the issues that are currently in operation, but I think I need to 
solve the ones that I have opened.

 

As to the method for resolving this issue first, the text processor used to 
change the time value contained in the collected log message to the 
corresponding time value (in fact, the nifi put mongo did not support the bson 
time format)
I think the next update should be made with reference to that.
I hope you are always happy engineer.

 

Please note the following process that doesn't parsing regular expressions, I 
have previously changed from JSON.stringify(content) to 
JSON.stringify(content,null,'\t') for make \n(new line) that is like 
json.pretty() in mongo script

> Processor is not shutting down normally
> ---
>
> Key: NIFI-4825
> URL: https://issues.apache.org/jira/browse/NIFI-4825
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Affects Versions: 1.4.0, 1.5.0
> Environment: Clustering with 3 Node and Seperated 3 zookeeper
>Reporter: kim teak jin
>Priority: Critical
> Attachments: error_template (1).xml, 
> image-2018-01-29-13-55-07-904.png, image-2018-01-29-13-55-56-745.png
>
>
> Hi dear Nifi users and Developers.
> If Json type time, created by default, is inserted, the problem did not 
> recognize the type as ISODate in MongoDB and is using ReplaceText processor 
> to resolve this.
> Now the input to MongoDB is working well, but there is another problem.
> So was v1.4, and I currently have updated to v1.5.
> The same phenomenon is occurring later but when the processor is not 
> operational anymore, the input queue becomes full approximately one day after 
> it starts on the ReplaceText processor.
> So I stopped the processor and tried to erase the queue and the processor in 
> front of it, but that didn't work.
> The following picture is a message that happens when I try to clear the queue.
> !image-2018-01-29-13-55-56-745.png!
> Now I think this is very important to me and if it continues to happen, I 
> think I will have to take a different solution.
> This is because it occurs continuously even if the queue can be deleted 
> normally after stopping and re-enabled successfully.
> So, temporarily, I copy the processor and know that it is not the right 
> solution though I am doing first aid such as photo.
> !image-2018-01-29-13-55-07-904.png!
> Currently, the left processor shows up as an inactive halt, but buttons such 
> as setting window and starting to stop do not appear.
>  I am attaching my template here, so please do not hesitate to deal with the 
> issue quickly.
>  
> We look forward to a speedy handling of the issues involved.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4828) MergeContent only processes one bin even if there are multiple ready bins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344586#comment-16344586
 ] 

ASF GitHub Bot commented on NIFI-4828:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2444

NIFI-4828: Fix MergeContent to process all ready bins

## NOTE to reviewers

This pull request has 3 commits intentionally. The first one is sufficient 
to address NIFI-4828. The other two commits are further refactoring those are 
not required, but should be incorporated, too. I didn't create dedicated JIRAs 
for those since they're too trivial.

Please squash commits and merge if all of those look fine. Or remove the 
2nd or 3rd commit if it looks over engineered. Thank you!

---

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4828

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2444






> MergeContent only processes one bin even if there are multiple ready bins
> -
>
> Key: NIFI-4828
> URL: https://issues.apache.org/jira/browse/NIFI-4828
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: mergecontent-multi-bins.xml
>
>
> [BinFiles.processBins|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java#L219]
>  is expected to loop through all ready bins, but it only process the first 
> bin. This incurs larger latency for FlowFiles to be merged.
> For example, if there are two FlowFiles FF1 and FF2 queued for a MergeContent 
> processor, each has an attribute named 'group'. FF1.group = 'a', and 
> FF2.group = 'b'. MergeContent is configured to use 'Correlation Attribute 
> Name' as 'group'.
> MergeContent takes FF1 and FF2 from its input queue, then correctly creates 
> two bins for group a and b, each having FF1 and FF2 respectively.
> Bug BinFiles.processBins only processes the first bin, which can be either 
> the bin for group a or b. The other bin is left unprocessed.
> The attached flow template has a flow to reproduce this.
> Expected behavior is MergeContent to process all queued FlowFiles at a single 
> onTrigger run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4828) MergeContent only processes one bin even if there are multiple ready bins

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-4828:

Status: Patch Available  (was: Open)

> MergeContent only processes one bin even if there are multiple ready bins
> -
>
> Key: NIFI-4828
> URL: https://issues.apache.org/jira/browse/NIFI-4828
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: mergecontent-multi-bins.xml
>
>
> [BinFiles.processBins|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java#L219]
>  is expected to loop through all ready bins, but it only process the first 
> bin. This incurs larger latency for FlowFiles to be merged.
> For example, if there are two FlowFiles FF1 and FF2 queued for a MergeContent 
> processor, each has an attribute named 'group'. FF1.group = 'a', and 
> FF2.group = 'b'. MergeContent is configured to use 'Correlation Attribute 
> Name' as 'group'.
> MergeContent takes FF1 and FF2 from its input queue, then correctly creates 
> two bins for group a and b, each having FF1 and FF2 respectively.
> Bug BinFiles.processBins only processes the first bin, which can be either 
> the bin for group a or b. The other bin is left unprocessed.
> The attached flow template has a flow to reproduce this.
> Expected behavior is MergeContent to process all queued FlowFiles at a single 
> onTrigger run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2444: NIFI-4828: Fix MergeContent to process all ready bi...

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2444

NIFI-4828: Fix MergeContent to process all ready bins

## NOTE to reviewers

This pull request has 3 commits intentionally. The first one is sufficient 
to address NIFI-4828. The other two commits are further refactoring those are 
not required, but should be incorporated, too. I didn't create dedicated JIRAs 
for those since they're too trivial.

Please squash commits and merge if all of those look fine. Or remove the 
2nd or 3rd commit if it looks over engineered. Thank you!

---

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4828

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2444






---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344468#comment-16344468
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164636057
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
--- End diff --

I now recall my reasoning on this.   The common way to use grok is to have 
most of the regexes in one or more grok pattern files, where they are easier to 
test then in an attribute property box. The expressions are then referenced by 
name in the expression passed in for evaluation, as in %{EX_1}. or even 
{EX_1} %{EX_2}.  If someone really does want to do an interesting regex 
in line, the delimiter can be  multiple characters, so it is easy to make a 
unique one like "<>".  If I wanted to locally define a set of 
regexes, I would could also also use the ExtractText processor. 
On the other hand, if we go with putting each expression in a separate 
attributes, the attributes need to be named in a way that can be sorted since 
we commit to order of evaluation.  If  the user wants to insert a new 
expression into the middle of the list, they have to (on average) recreate half 
of the rest of the attributes with new names.  In my own use case of building a 
classifier, i am often tweaking the list, so having to recreate my attributes 
would be a nuisance.  

PS: I will make the delimiter optional, with no default.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164636057
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
--- End diff --

I now recall my reasoning on this.   The common way to use grok is to have 
most of the regexes in one or more grok pattern files, where they are easier to 
test then in an attribute property box. The expressions are then referenced by 
name in the expression passed in for evaluation, as in %{EX_1}. or even 
{EX_1} %{EX_2}.  If someone really does want to do an interesting regex 
in line, the delimiter can be  multiple characters, so it is easy to make a 
unique one like "<>".  If I wanted to locally define a set of 
regexes, I would could also also use the ExtractText processor. 
On the other hand, if we go with putting each expression in a separate 
attributes, the attributes need to be named in a way that can be sorted since 
we commit to order of evaluation.  If  the user wants to insert a new 
expression into the middle of the list, they have to (on average) recreate half 
of the rest of the attributes with new names.  In my own use case of building a 
classifier, i am often tweaking the list, so having to recreate my attributes 
would be a nuisance.  

PS: I will make the delimiter optional, with no default.


---

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1633#comment-1633
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@mattyb149 

I've updated the code based on your review feedback (added MIT license,  
updated property description, added expression language support for db name and 
size properties, and added connection timeout property, updated documentation 
regarding line protocol). I've also added retry relation (for handling socket 
timeout), and max size exceed relation to handle if records size exceeds limit. 
 Unit/integration tests are also updated for these changes.

Regarding max size limit - this is limit on the size of records being 
posted to InfluxDB.  The processor can push a batch of records separated by new 
line to InfluxDB. I wanted to give the admin some control over how much data is 
being posted to InfluxDB.  If the size is exceeded, the size exceed relation 
can be used to handle this scenario.

Please let me know if you have any additional recommendations.

Thanks again for your thoughtful feedback.




> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2101: NIFI-4289 - InfluxDB put processor

[mans2singh]

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@mattyb149 

I've updated the code based on your review feedback (added MIT license,  
updated property description, added expression language support for db name and 
size properties, and added connection timeout property, updated documentation 
regarding line protocol). I've also added retry relation (for handling socket 
timeout), and max size exceed relation to handle if records size exceeds limit. 
 Unit/integration tests are also updated for these changes.

Regarding max size limit - this is limit on the size of records being 
posted to InfluxDB.  The processor can push a batch of records separated by new 
line to InfluxDB. I wanted to give the admin some control over how much data is 
being posted to InfluxDB.  If the size is exceeded, the size exceed relation 
can be used to handle this scenario.

Please let me know if you have any additional recommendations.

Thanks again for your thoughtful feedback.




---

[jira] [Created] (NIFI-4828) MergeContent only processes one bin even if there are multiple ready bins

[Koji Kawamura (JIRA)]

Koji Kawamura created NIFI-4828:
---

 Summary: MergeContent only processes one bin even if there are 
multiple ready bins
 Key: NIFI-4828
 URL: https://issues.apache.org/jira/browse/NIFI-4828
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.0.0
Reporter: Koji Kawamura
Assignee: Koji Kawamura
 Attachments: mergecontent-multi-bins.xml

[BinFiles.processBins|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java#L219]
 is expected to loop through all ready bins, but it only process the first bin. 
This incurs larger latency for FlowFiles to be merged.

For example, if there are two FlowFiles FF1 and FF2 queued for a MergeContent 
processor, each has an attribute named 'group'. FF1.group = 'a', and FF2.group 
= 'b'. MergeContent is configured to use 'Correlation Attribute Name' as 
'group'.
MergeContent takes FF1 and FF2 from its input queue, then correctly creates two 
bins for group a and b, each having FF1 and FF2 respectively.
Bug BinFiles.processBins only processes the first bin, which can be either the 
bin for group a or b. The other bin is left unprocessed.
The attached flow template has a flow to reproduce this.

Expected behavior is MergeContent to process all queued FlowFiles at a single 
onTrigger run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344264#comment-16344264
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164608075
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

I like the idea of having test flows, but maybe not in the codebase since 
they aren't executed? I often put test flows up as a Gist on GitHub and refer 
to it in the PR, so the reviewer can go there for a sample flow (which I try to 
document inline with labels to explain what's going on)


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164608075
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

I like the idea of having test flows, but maybe not in the codebase since 
they aren't executed? I often put test flows up as a Gist on GitHub and refer 
to it in the PR, so the reviewer can go there for a sample flow (which I try to 
document inline with labels to explain what's going on)


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344262#comment-16344262
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164607830
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

We can ask the users list to get UX and other feedback, for ExecuteSQL it 
was natural because the query used to be required, then we relaxed it and added 
doc to the property saying if left blank, the contents of the flow file are 
expected to have the SQL to execute. In this case, a blank setting is already 
meaningful, so we may need two properties. "Explicit is better" is a good rule 
of thumb, but now we have two properties that interact with each other in 
different ways, which can also be confusing (see my other comments on when 
things are set to various value combos)


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164607830
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

We can ask the users list to get UX and other feedback, for ExecuteSQL it 
was natural because the query used to be required, then we relaxed it and added 
doc to the property saying if left blank, the contents of the flow file are 
expected to have the SQL to execute. In this case, a blank setting is already 
meaningful, so we may need two properties. "Explicit is better" is a good rule 
of thumb, but now we have two properties that interact with each other in 
different ways, which can also be confusing (see my other comments on when 
things are set to various value combos)


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344257#comment-16344257
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164607381
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

Correct, just thinking of other processors where this happens, their doc 
says something like what I put above, with no incoming connections the 
relationship is unused.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164607381
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

Correct, just thinking of other processors where this happens, their doc 
says something like what I put above, with no incoming connections the 
relationship is unused.


---

[jira] [Comment Edited] (NIFI-2735) Add processor to perform simple aggregations

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344254#comment-16344254
 ] 

Matt Burgess edited comment on NIFI-2735 at 1/30/18 12:20 AM:
--

Probably not, the QueryRecord processor can do the simple aggregates pretty 
well using SQL. Although I don't know if we can specify output schemas yet so 
maybe not...


was (Author: mattyb149):
Probably not, the QueryRecord processor can do the simple aggregates pretty 
well using SQL

> Add processor to perform simple aggregations
> 
>
> Key: NIFI-2735
> URL: https://issues.apache.org/jira/browse/NIFI-2735
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> This is a proposal for a new processor (AggregateValues, for example) that 
> can perform simple aggregation operations such as count, sum, average, min, 
> max, and concatenate, over a set of "related" flow files. For example, when a 
> JSON file is split on an array (using the SplitJson processor), the total 
> count of the splits, the index of each split, and the unique identifier 
> (shared by each split) are stored as attributes in each flow file sent to the 
> "splits" relationship:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html
> These attributes are the "fragment.*" attributes in the documentation for 
> SplitText, SplitXml, and SplitJson, for example.
> Such a processor could perform these operations for each flow file split from 
> the original document, and when all documents from a split have been 
> processed, a flow file could be transferred to an "aggregate" relationship 
> containing attributes for the operation, aggregate value, etc.
> An interesting application of this (besides the actual aggregation 
> operations) is that you can use the "aggregate" relationship as an event 
> trigger. For example if you need to wait until all files from a group are 
> processed, you can use AggregateValues and the "aggregate" relationship to 
> indicate downstream that the entire group has been processed. If there is not 
> a Split processor upstream, then the attributes (fragment.*) would have to be 
> manipulated by the data flow designer, but this can be accomplished with 
> other processors (including the scripting processors if necessary). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-2735) Add processor to perform simple aggregations

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344254#comment-16344254
 ] 

Matt Burgess commented on NIFI-2735:


Probably not, the QueryRecord processor can do the simple aggregates pretty 
well using SQL

> Add processor to perform simple aggregations
> 
>
> Key: NIFI-2735
> URL: https://issues.apache.org/jira/browse/NIFI-2735
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> This is a proposal for a new processor (AggregateValues, for example) that 
> can perform simple aggregation operations such as count, sum, average, min, 
> max, and concatenate, over a set of "related" flow files. For example, when a 
> JSON file is split on an array (using the SplitJson processor), the total 
> count of the splits, the index of each split, and the unique identifier 
> (shared by each split) are stored as attributes in each flow file sent to the 
> "splits" relationship:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html
> These attributes are the "fragment.*" attributes in the documentation for 
> SplitText, SplitXml, and SplitJson, for example.
> Such a processor could perform these operations for each flow file split from 
> the original document, and when all documents from a split have been 
> processed, a flow file could be transferred to an "aggregate" relationship 
> containing attributes for the operation, aggregate value, etc.
> An interesting application of this (besides the actual aggregation 
> operations) is that you can use the "aggregate" relationship as an event 
> trigger. For example if you need to wait until all files from a group are 
> processed, you can use AggregateValues and the "aggregate" relationship to 
> indicate downstream that the entire group has been processed. If there is not 
> a Split processor upstream, then the attributes (fragment.*) would have to be 
> manipulated by the data flow designer, but this can be accomplished with 
> other processors (including the scripting processors if necessary). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-2735) Add processor to perform simple aggregations

[Andre F de Miranda (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344236#comment-16344236
 ] 

Andre F de Miranda commented on NIFI-2735:
--

[~mattyb149]

 

Any plans to rework this? :)

> Add processor to perform simple aggregations
> 
>
> Key: NIFI-2735
> URL: https://issues.apache.org/jira/browse/NIFI-2735
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> This is a proposal for a new processor (AggregateValues, for example) that 
> can perform simple aggregation operations such as count, sum, average, min, 
> max, and concatenate, over a set of "related" flow files. For example, when a 
> JSON file is split on an array (using the SplitJson processor), the total 
> count of the splits, the index of each split, and the unique identifier 
> (shared by each split) are stored as attributes in each flow file sent to the 
> "splits" relationship:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html
> These attributes are the "fragment.*" attributes in the documentation for 
> SplitText, SplitXml, and SplitJson, for example.
> Such a processor could perform these operations for each flow file split from 
> the original document, and when all documents from a split have been 
> processed, a flow file could be transferred to an "aggregate" relationship 
> containing attributes for the operation, aggregate value, etc.
> An interesting application of this (besides the actual aggregation 
> operations) is that you can use the "aggregate" relationship as an event 
> trigger. For example if you need to wait until all files from a group are 
> processed, you can use AggregateValues and the "aggregate" relationship to 
> indicate downstream that the entire group has been processed. If there is not 
> a Split processor upstream, then the attributes (fragment.*) would have to be 
> manipulated by the data flow designer, but this can be accomplished with 
> other processors (including the scripting processors if necessary). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344215#comment-16344215
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602936
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

Everything is tied to that parameter (or should be tied to it). One of the 
things I'm learning from having to train someone on NiFi is that "explicit is 
better" for a lot of people. I think the meaning of a blank query wouldn't be 
obvious to a lot of users, and you cannot trust that they'll read the 
description closely rather than cursing out the product because it seems to be 
"mysteriously misbehaving."


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602936
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

Everything is tied to that parameter (or should be tied to it). One of the 
things I'm learning from having to train someone on NiFi is that "explicit is 
better" for a lot of people. I think the meaning of a blank query wouldn't be 
obvious to a lot of users, and you cannot trust that they'll read the 
description closely rather than cursing out the product because it seems to be 
"mysteriously misbehaving."


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344213#comment-16344213
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602408
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

So you're saying it should just be more explicit in the documentation, and 
not be some sort of dynamic relationship a la RouteOnAttribute, right?


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602408
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

So you're saying it should just be more explicit in the documentation, and 
not be some sort of dynamic relationship a la RouteOnAttribute, right?


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344212#comment-16344212
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602170
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

It's a miniature test flow that's meant to show the behavior if someone has 
questions. I got the idea when I was reviewing the PutInfluxDb processor. On 
nifi-dev, I think @joewitt mentioned that a major problem with some of these 
bundles is the people who actually know a particular system might be highly 
limited in the community. I think getting into a habit of sharing test flows 
that use something like GenerateFlowFile could ease the learning curve on 
others. Particularly if a docker-compose file or something like that is added 
with it.

I can remove it, but I think it might be a good practice.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164602170
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

It's a miniature test flow that's meant to show the behavior if someone has 
questions. I got the idea when I was reviewing the PutInfluxDb processor. On 
nifi-dev, I think @joewitt mentioned that a major problem with some of these 
bundles is the people who actually know a particular system might be highly 
limited in the community. I think getting into a habit of sharing test flows 
that use something like GenerateFlowFile could ease the learning curve on 
others. Particularly if a docker-compose file or something like that is added 
with it.

I can remove it, but I think it might be a good practice.


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344097#comment-16344097
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164576841
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

Failure and Original only apply when there are incoming flow files, I see 
that you mention "input flowfiles" here but I wonder if it should be even more 
explicit, such as pointing out that if there are no incoming connections then 
the relationship will be ignored/unused?


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344096#comment-16344096
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164563821
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -302,15 +348,19 @@ public void process(OutputStream out) throws 
IOException {
 }
 }
 
-session.commit();
+if (input != null) {
+session.transfer(input, REL_ORIGINAL);
--- End diff --

If you set Query Location to Query Parameter, but there are input flow 
files, then the "original" relationship might be a bit confusing. If I schedule 
GetMongo to run once a second, and upstream it is generating flow files once 
every 2 seconds, then if using Query Parameter, the first GetMongo will not 
output along the "original" relationship, but the second one will, causing 
twice as many success FlowFiles as originals. This is in line with the 
documented behavior, but it seems a bit awkward to me, what do you think?

An alternative is to not run if there is an incoming connection but no flow 
file, basically the processor behaves in an event-driven fashion if there is an 
incoming connection, or a timer-driven fashion if there is no incoming 
(non-loop) connection. ExecuteSQL has this code at the beginning of onTrigger():
```
FlowFile fileToProcess = null;
if (context.hasIncomingConnection()) {
 fileToProcess = session.get();
 // If we have no FlowFile, and all incoming connections are self-loops 
then we can continue on.
 // However, if we have no FlowFile and we have connections coming from 
other Processors, then
 // we know that we should run only if we have a FlowFile.
 if (fileToProcess == null && context.hasNonLoopConnection()) {
  return;
 }
 }
```
You could do other runtime checking (unfortunately, connections are not 
currently known to the `ValidationContext` or `ProcessorInitializationContext` 
so must be done in onTrigger) such as making sure that if there is an incoming 
connection, then the Query Location must be Body, if that's the kind of 
behavior you want. Thoughts?



> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344098#comment-16344098
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164561209
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

It's a bummer we need an additional property here to specify where the 
query is coming from; seems to me that's only because you can leave Query blank 
and it means something. In other processors like ExecuteSQL, you either provide 
a Query in the property (and it is used), or you leave it blank and then it is 
expected that the body of the incoming flow file contains the query.

What is the behavior when Query is filled in but the Query Location is 
Body? Will Query Parameter be ignored or should the processor be marked 
invalid? It looks from the doc that it will be ignored, but I wonder if the 
description below should add something like 

> If 'body' is selected, it will come from the flowfile instead of the 
query parameter, whether the query parameter is set or not

That might just be me though, if it seems clear as written then I'm ok


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344099#comment-16344099
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164577827
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

I believe this file needs to be removed.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164576841
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -81,6 +82,15 @@ public ValidationResult validate(final String subject, 
final String value, final
 };
 
 static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All input flowfiles that are part of a failed 
query execution go here.")
--- End diff --

Failure and Original only apply when there are incoming flow files, I see 
that you mention "input flowfiles" here but I wonder if it should be even more 
explicit, such as pointing out that if there are no incoming connections then 
the relationship will be ignored/unused?


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164577827
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/test-flows/NIFI_4827.xml
 ---
@@ -0,0 +1,635 @@
+
--- End diff --

I believe this file needs to be removed.


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164561209
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -89,6 +99,17 @@ public ValidationResult validate(final String subject, 
final String value, final
 .expressionLanguageSupported(true)
 .addValidator(DOCUMENT_VALIDATOR)
 .build();
+
+static final AllowableValue LOC_BODY = new AllowableValue("body", 
"Body");
+static final AllowableValue LOC_PARAM = new AllowableValue("param", 
"Query Parameter");
+static final PropertyDescriptor QUERY_LOC = new 
PropertyDescriptor.Builder()
--- End diff --

It's a bummer we need an additional property here to specify where the 
query is coming from; seems to me that's only because you can leave Query blank 
and it means something. In other processors like ExecuteSQL, you either provide 
a Query in the property (and it is used), or you leave it blank and then it is 
expected that the body of the incoming flow file contains the query.

What is the behavior when Query is filled in but the Query Location is 
Body? Will Query Parameter be ignored or should the processor be marked 
invalid? It looks from the doc that it will be ignored, but I wonder if the 
description below should add something like 

> If 'body' is selected, it will come from the flowfile instead of the 
query parameter, whether the query parameter is set or not

That might just be me though, if it seems clear as written then I'm ok


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r164563821
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -302,15 +348,19 @@ public void process(OutputStream out) throws 
IOException {
 }
 }
 
-session.commit();
+if (input != null) {
+session.transfer(input, REL_ORIGINAL);
--- End diff --

If you set Query Location to Query Parameter, but there are input flow 
files, then the "original" relationship might be a bit confusing. If I schedule 
GetMongo to run once a second, and upstream it is generating flow files once 
every 2 seconds, then if using Query Parameter, the first GetMongo will not 
output along the "original" relationship, but the second one will, causing 
twice as many success FlowFiles as originals. This is in line with the 
documented behavior, but it seems a bit awkward to me, what do you think?

An alternative is to not run if there is an incoming connection but no flow 
file, basically the processor behaves in an event-driven fashion if there is an 
incoming connection, or a timer-driven fashion if there is no incoming 
(non-loop) connection. ExecuteSQL has this code at the beginning of onTrigger():
```
FlowFile fileToProcess = null;
if (context.hasIncomingConnection()) {
 fileToProcess = session.get();
 // If we have no FlowFile, and all incoming connections are self-loops 
then we can continue on.
 // However, if we have no FlowFile and we have connections coming from 
other Processors, then
 // we know that we should run only if we have a FlowFile.
 if (fileToProcess == null && context.hasNonLoopConnection()) {
  return;
 }
 }
```
You could do other runtime checking (unfortunately, connections are not 
currently known to the `ValidationContext` or `ProcessorInitializationContext` 
so must be done in onTrigger) such as making sure that if there is an incoming 
connection, then the Query Location must be Body, if that's the kind of 
behavior you want. Thoughts?



---

[jira] [Updated] (NIFIREG-132) Various REST API endpoints missing swagger annotations for body parameters.

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran updated NIFIREG-132:

Description: 
The following endpoints are missing the swagger {{@ApiParam}} annotation for 
the body parameter, which creates incorrect documentation/spec in the 
{{swagger.json}} file in the build output:

* create flow
* update flow
* create flow version

Thanks to [~chaffelson] for help identifying and testing this issue.

  was:
The following endpoints are missing the swagger {{@ApiParam}} annotation for 
the body parameter, which creates incorrect documentation/spec in the 
{{swagger.json}} file in the build output:

* create flow
* update flow
* create flow version




> Various REST API endpoints missing swagger annotations for body parameters.
> ---
>
> Key: NIFIREG-132
> URL: https://issues.apache.org/jira/browse/NIFIREG-132
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Kevin Doran
>Assignee: Kevin Doran
>Priority: Major
> Fix For: 0.2.0
>
>
> The following endpoints are missing the swagger {{@ApiParam}} annotation for 
> the body parameter, which creates incorrect documentation/spec in the 
> {{swagger.json}} file in the build output:
> * create flow
> * update flow
> * create flow version
> Thanks to [~chaffelson] for help identifying and testing this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFIREG-132) Various REST API endpoints missing swagger annotations for body parameters.

[Kevin Doran (JIRA)]

Kevin Doran created NIFIREG-132:
---

 Summary: Various REST API endpoints missing swagger annotations 
for body parameters.
 Key: NIFIREG-132
 URL: https://issues.apache.org/jira/browse/NIFIREG-132
 Project: NiFi Registry
  Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Kevin Doran
Assignee: Kevin Doran
 Fix For: 0.2.0


The following endpoints are missing the swagger {{@ApiParam}} annotation for 
the body parameter, which creates incorrect documentation/spec in the 
{{swagger.json}} file in the build output:

* create flow
* update flow
* create flow version





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344032#comment-16344032
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164564814
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -209,78 +287,94 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 
 try {
 final byte[] byteBuffer = buffer;
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(InputStream in) throws IOException {
-StreamUtils.fillBuffer(in, byteBuffer, false);
-}
-});
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
byteBuffer, false));
 final long len = Math.min(byteBuffer.length, 
flowFile.getSize());
 contentString = new String(byteBuffer, 0, (int) len, charset);
 } finally {
 bufferQueue.offer(buffer);
 }
 
-final Match gm = grok.match(contentString);
-gm.captures();
-
-if (gm.toMap().isEmpty()) {
-session.transfer(flowFile, REL_NO_MATCH);
-getLogger().info("Did not match any Grok Expressions for 
FlowFile {}", new Object[]{flowFile});
-return;
-}
-
-final ObjectMapper objectMapper = new ObjectMapper();
-switch (context.getProperty(DESTINATION).getValue()) {
-case FLOWFILE_ATTRIBUTE:
-Map grokResults = new HashMap<>();
-for (Map.Entry entry : 
gm.toMap().entrySet()) {
-if (null != entry.getValue()) {
-grokResults.put("grok." + entry.getKey(), 
entry.getValue().toString());
+try{
+for (Grok grok : grokList) {
+final Match gm = grok.match(contentString);
+gm.captures();
+final Map localResults = gm.toMap();
+if (!localResults.isEmpty()) {
+
matchedExpressionList.add(grok.getOriginalGrokPattern());
+results.putAll(localResults);
+if (breakOnFirstMatch) {
+break;
 }
 }
+}
 
-flowFile = session.putAllAttributes(flowFile, grokResults);
-session.getProvenanceReporter().modifyAttributes(flowFile);
-session.transfer(flowFile, REL_MATCH);
-getLogger().info("Matched {} Grok Expressions and added 
attributes to FlowFile {}", new Object[]{grokResults.size(), flowFile});
-
-break;
-case FLOWFILE_CONTENT:
-FlowFile conFlowfile = session.write(flowFile, new 
StreamCallback() {
-@Override
-public void process(InputStream in, OutputStream out) 
throws IOException {
-
out.write(objectMapper.writeValueAsBytes(gm.toMap()));
-}
-});
-conFlowfile = session.putAttribute(conFlowfile, 
CoreAttributes.MIME_TYPE.key(), APPLICATION_JSON);
-session.getProvenanceReporter().modifyContent(conFlowfile, 
"Replaced content with parsed Grok fields and values", 
stopWatch.getElapsed(TimeUnit.MILLISECONDS));
-session.transfer(conFlowfile, REL_MATCH);
+if (results.isEmpty()) {
+session.transfer(flowFile, REL_NO_MATCH);
+getLogger().debug("Did not match any Grok Expression for 
FlowFile {}", new Object[]{flowFile});
+return;
+}
+
+String matchedExpressions = 
StringUtils.join(matchedExpressionList, expressionSeparator);
+flowFile = session.putAttribute(flowFile, 
matchedExpressionAttribute, matchedExpressions);
 
-break;
+switch (context.getProperty(DESTINATION).getValue()) {
+case FLOWFILE_ATTRIBUTE:
+Map grokResults = new HashMap<>();
+for (Map.Entry entry : 
results.entrySet()) {
+if (null != entry.getValue()) {
+grokResults.put(resultPrefix + entry.getKey(), 
entry.getValue().toString());
+}
+ 

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164564814
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -209,78 +287,94 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 
 try {
 final byte[] byteBuffer = buffer;
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(InputStream in) throws IOException {
-StreamUtils.fillBuffer(in, byteBuffer, false);
-}
-});
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
byteBuffer, false));
 final long len = Math.min(byteBuffer.length, 
flowFile.getSize());
 contentString = new String(byteBuffer, 0, (int) len, charset);
 } finally {
 bufferQueue.offer(buffer);
 }
 
-final Match gm = grok.match(contentString);
-gm.captures();
-
-if (gm.toMap().isEmpty()) {
-session.transfer(flowFile, REL_NO_MATCH);
-getLogger().info("Did not match any Grok Expressions for 
FlowFile {}", new Object[]{flowFile});
-return;
-}
-
-final ObjectMapper objectMapper = new ObjectMapper();
-switch (context.getProperty(DESTINATION).getValue()) {
-case FLOWFILE_ATTRIBUTE:
-Map grokResults = new HashMap<>();
-for (Map.Entry entry : 
gm.toMap().entrySet()) {
-if (null != entry.getValue()) {
-grokResults.put("grok." + entry.getKey(), 
entry.getValue().toString());
+try{
+for (Grok grok : grokList) {
+final Match gm = grok.match(contentString);
+gm.captures();
+final Map localResults = gm.toMap();
+if (!localResults.isEmpty()) {
+
matchedExpressionList.add(grok.getOriginalGrokPattern());
+results.putAll(localResults);
+if (breakOnFirstMatch) {
+break;
 }
 }
+}
 
-flowFile = session.putAllAttributes(flowFile, grokResults);
-session.getProvenanceReporter().modifyAttributes(flowFile);
-session.transfer(flowFile, REL_MATCH);
-getLogger().info("Matched {} Grok Expressions and added 
attributes to FlowFile {}", new Object[]{grokResults.size(), flowFile});
-
-break;
-case FLOWFILE_CONTENT:
-FlowFile conFlowfile = session.write(flowFile, new 
StreamCallback() {
-@Override
-public void process(InputStream in, OutputStream out) 
throws IOException {
-
out.write(objectMapper.writeValueAsBytes(gm.toMap()));
-}
-});
-conFlowfile = session.putAttribute(conFlowfile, 
CoreAttributes.MIME_TYPE.key(), APPLICATION_JSON);
-session.getProvenanceReporter().modifyContent(conFlowfile, 
"Replaced content with parsed Grok fields and values", 
stopWatch.getElapsed(TimeUnit.MILLISECONDS));
-session.transfer(conFlowfile, REL_MATCH);
+if (results.isEmpty()) {
+session.transfer(flowFile, REL_NO_MATCH);
+getLogger().debug("Did not match any Grok Expression for 
FlowFile {}", new Object[]{flowFile});
+return;
+}
+
+String matchedExpressions = 
StringUtils.join(matchedExpressionList, expressionSeparator);
+flowFile = session.putAttribute(flowFile, 
matchedExpressionAttribute, matchedExpressions);
 
-break;
+switch (context.getProperty(DESTINATION).getValue()) {
+case FLOWFILE_ATTRIBUTE:
+Map grokResults = new HashMap<>();
+for (Map.Entry entry : 
results.entrySet()) {
+if (null != entry.getValue()) {
+grokResults.put(resultPrefix + entry.getKey(), 
entry.getValue().toString());
+}
+}
+flowFile = session.putAllAttributes(flowFile, 
grokResults);
+
session.getProvenanceReporter().modifyAttributes(flowFile);
+session.transfer(flowFile, 

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344031#comment-16344031
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164564264
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -181,15 +243,28 @@ public void onStopped() {
 
 @OnScheduled
 public void onScheduled(final ProcessContext context) throws 
GrokException {
+grokList.clear();
 for (int i = 0; i < context.getMaxConcurrentTasks(); i++) {
 final int maxBufferSize = 
context.getProperty(MAX_BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
 final byte[] buffer = new byte[maxBufferSize];
 bufferQueue.add(buffer);
 }
 
-grok = new Grok();
-
grok.addPatternFromFile(context.getProperty(GROK_PATTERN_FILE).getValue());
-grok.compile(context.getProperty(GROK_EXPRESSION).getValue(), 
context.getProperty(NAMED_CAPTURES_ONLY).asBoolean());
+resultPrefix = context.getProperty(RESULT_PREFIX).getValue();
+breakOnFirstMatch = 
context.getProperty(BREAK_ON_FIRST_MATCH).asBoolean() ;
+matchedExpressionAttribute = 
context.getProperty(MATCHED_EXP_ATTR).getValue();
+expressionSeparator = 
context.getProperty(EXPRESSION_SEPARATOR).getValue();
+
+String patterns  = context.getProperty(GROK_EXPRESSION).getValue();
+for (String patternName : patterns.split(expressionSeparator)) {
+Grok grok = new Grok();
+final String patternFileListString = 
context.getProperty(GROK_PATTERN_FILE).getValue();
+for (String patternFile : 
patternFileListString.split(PATTERN_FILE_LIST_SEPARATOR)) {
+grok.addPatternFromFile(patternFile);
--- End diff --

hmmm... ok. My feeling would be if they want to put spaces in their list 
they should include them in separator, but your way is more tolerant. Will fix.



> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164564264
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -181,15 +243,28 @@ public void onStopped() {
 
 @OnScheduled
 public void onScheduled(final ProcessContext context) throws 
GrokException {
+grokList.clear();
 for (int i = 0; i < context.getMaxConcurrentTasks(); i++) {
 final int maxBufferSize = 
context.getProperty(MAX_BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
 final byte[] buffer = new byte[maxBufferSize];
 bufferQueue.add(buffer);
 }
 
-grok = new Grok();
-
grok.addPatternFromFile(context.getProperty(GROK_PATTERN_FILE).getValue());
-grok.compile(context.getProperty(GROK_EXPRESSION).getValue(), 
context.getProperty(NAMED_CAPTURES_ONLY).asBoolean());
+resultPrefix = context.getProperty(RESULT_PREFIX).getValue();
+breakOnFirstMatch = 
context.getProperty(BREAK_ON_FIRST_MATCH).asBoolean() ;
+matchedExpressionAttribute = 
context.getProperty(MATCHED_EXP_ATTR).getValue();
+expressionSeparator = 
context.getProperty(EXPRESSION_SEPARATOR).getValue();
+
+String patterns  = context.getProperty(GROK_EXPRESSION).getValue();
+for (String patternName : patterns.split(expressionSeparator)) {
+Grok grok = new Grok();
+final String patternFileListString = 
context.getProperty(GROK_PATTERN_FILE).getValue();
+for (String patternFile : 
patternFileListString.split(PATTERN_FILE_LIST_SEPARATOR)) {
+grok.addPatternFromFile(patternFile);
--- End diff --

hmmm... ok. My feeling would be if they want to put spaces in their list 
they should include them in separator, but your way is more tolerant. Will fix.



---

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164562467
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -156,14 +209,23 @@
 
 final List _descriptors = new ArrayList<>();
 _descriptors.add(GROK_EXPRESSION);
+_descriptors.add(EXPRESSION_SEPARATOR);
 _descriptors.add(GROK_PATTERN_FILE);
 _descriptors.add(DESTINATION);
 _descriptors.add(CHARACTER_SET);
 _descriptors.add(MAX_BUFFER_SIZE);
 _descriptors.add(NAMED_CAPTURES_ONLY);
+_descriptors.add(RESULT_PREFIX);
+_descriptors.add(BREAK_ON_FIRST_MATCH);
+_descriptors.add(MATCHED_EXP_ATTR);
 descriptors = Collections.unmodifiableList(_descriptors);
 }
 
+private String resultPrefix = "";
--- End diff --

I agree I should conform to project style. will fix.


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344017#comment-16344017
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164562467
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -156,14 +209,23 @@
 
 final List _descriptors = new ArrayList<>();
 _descriptors.add(GROK_EXPRESSION);
+_descriptors.add(EXPRESSION_SEPARATOR);
 _descriptors.add(GROK_PATTERN_FILE);
 _descriptors.add(DESTINATION);
 _descriptors.add(CHARACTER_SET);
 _descriptors.add(MAX_BUFFER_SIZE);
 _descriptors.add(NAMED_CAPTURES_ONLY);
+_descriptors.add(RESULT_PREFIX);
+_descriptors.add(BREAK_ON_FIRST_MATCH);
+_descriptors.add(MATCHED_EXP_ATTR);
 descriptors = Collections.unmodifiableList(_descriptors);
 }
 
+private String resultPrefix = "";
--- End diff --

I agree I should conform to project style. will fix.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344012#comment-16344012
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164561259
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -142,10 +194,11 @@
 .description("FlowFiles are routed to this relationship when 
no provided Grok Expression matches the content of the FlowFile")
 .build();
 
+
 private final static List descriptors;
 private final static Set relationships;
 
-private volatile Grok grok = new Grok();
+private volatile List grokList = new ArrayList<>();
--- End diff --

ok, i will create a new list.  In practice, it is never modified while 
running multi-threaded. but it is an easy and computationally cheap fix for 
safer code.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164561259
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -142,10 +194,11 @@
 .description("FlowFiles are routed to this relationship when 
no provided Grok Expression matches the content of the FlowFile")
 .build();
 
+
 private final static List descriptors;
 private final static Set relationships;
 
-private volatile Grok grok = new Grok();
+private volatile List grokList = new ArrayList<>();
--- End diff --

ok, i will create a new list.  In practice, it is never modified while 
running multi-threaded. but it is an easy and computationally cheap fix for 
safer code.


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344010#comment-16344010
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164560737
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
+.name(EXP_SEPARATOR_KEY)
+.description("character to use to separate multiple grok 
expressions ")
+.required(true)
+.defaultValue(",")
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+
+public static final PropertyDescriptor MATCHED_EXP_ATTR = new 
PropertyDescriptor.Builder()
+.name(MATCHED_EXP_ATTR_KEY)
+.description("Name of attribute to receive the name(s) of the 
matched expression(s).")
+.required(true)
+.defaultValue("matched_expression")
+.addValidator(Validator.VALID)
--- End diff --

oops.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344005#comment-16344005
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164560550
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
--- End diff --

good point about the backward compatibility. I had imagined that users 
could select a separator if needed in that case.I now realize that is not 
reasonable.   I did not originally go with user-defined properties cause I was 
not sure how they would be ordered.   Are they delivered in the same order 
declared, or do I need to tell the user to name them in sortable order?


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164560737
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
+.name(EXP_SEPARATOR_KEY)
+.description("character to use to separate multiple grok 
expressions ")
+.required(true)
+.defaultValue(",")
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+
+public static final PropertyDescriptor MATCHED_EXP_ATTR = new 
PropertyDescriptor.Builder()
+.name(MATCHED_EXP_ATTR_KEY)
+.description("Name of attribute to receive the name(s) of the 
matched expression(s).")
+.required(true)
+.defaultValue("matched_expression")
+.addValidator(Validator.VALID)
--- End diff --

oops.


---

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164560550
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
+"\n\t (Does not apply when results returned as content)" +
+"\n\t (May be empty, the dot (.) separator is not 
implied)")
+.required(true)
+.defaultValue("grok.")
+.addValidator(Validator.VALID)
+.build();
+
+
+public static final PropertyDescriptor EXPRESSION_SEPARATOR = new 
PropertyDescriptor.Builder()
--- End diff --

good point about the backward compatibility. I had imagined that users 
could select a separator if needed in that case.I now realize that is not 
reasonable.   I did not originally go with user-defined properties cause I was 
not sure how they would be ordered.   Are they delivered in the same order 
declared, or do I need to tell the user to name them in sortable order?


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343999#comment-16343999
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559686
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
--- End diff --

yep br


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559686
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
--- End diff --

yep br


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343998#comment-16343998
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559463
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
--- End diff --

glad to get early warning that it was not intelligible.  I will add a more 
complete just FYI: in my use case. i am using this processor as a classifier. I 
arrange my expressions, so  that the first match it makes will be the best 
match.. 


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559463
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
--- End diff --

glad to get early warning that it was not intelligible.  I will add a more 
complete just FYI: in my use case. i am using this processor as a classifier. I 
arrange my expressions, so  that the first match it makes will be the best 
match.. 


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343995#comment-16343995
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164558925
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -70,33 +65,51 @@
 @Tags({"grok", "log", "text", "parse", "delimit", "extract"})
 @CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
 "adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
-"notation of the matched content")
+"notation of the matched content\n" +
+"uses https://github.com/thekrakken/java-grok.;)
 @WritesAttributes({
-@WritesAttribute(attribute = "grok.XXX", description = "When operating 
in flowfile-attribute mode, each of the Grok identifier that is matched in the 
flowfile " +
-"will be added as an attribute, prefixed with \"grok.\" For 
example," +
-"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"grok.timestamp\"")})
+@WritesAttribute(attribute = "{result prefix}XXX", description = "When 
operating in flowfile-attribute mode, each of the Grok identifier that is 
matched in the flowfile " +
+"will be added as an attribute, prefixed with \"{result prefix}\" 
For example," +
+"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"{result prefix}timestamp\""),
+
+@WritesAttribute(attribute = "ExtractGrok.exception", description 
= "if an error occurs, an exception will be written to this attribute, " +
+"and the flow routed to 'unmatched' ")
+})
 public class ExtractGrok extends AbstractProcessor {
 
 public static final String FLOWFILE_ATTRIBUTE = "flowfile-attribute";
 public static final String FLOWFILE_CONTENT = "flowfile-content";
-private static final String APPLICATION_JSON = "application/json";
-
+public static final String APPLICATION_JSON = "application/json";
+public static final String GROK_EXPRESSION_KEY = "Grok Expression";
+public static final String GROK_PATTERN_FILE_KEY = "Grok Pattern file";
+public static final String DESTINATION_KEY = "Destination";
+public static final String CHARACTER_SET_KEY = "Character Set";
+public static final String MAXIMUM_BUFFER_SIZE_KEY = "Maximum Buffer 
Size";
+public static final String NAMED_CAPTURES_ONLY_KEY = "Named captures 
only";
+public static final String SINGLE_MATCH_KEY = "Single Match";
+public static final String RESULT_PREFIX_KEY = "result prefix";
+public static final String MATCHED_EXP_ATTR_KEY = "matched expression 
attribute";
+public static final String EXP_SEPARATOR_KEY = "expression-separator";
+public static final String PATTERN_FILE_LIST_SEPARATOR = ",";
+
+//properties
 public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor.Builder()
-.name("Grok Expression")
-.description("Grok expression")
+.name(GROK_EXPRESSION_KEY)
--- End diff --

just because sometimes (maybe not in this processor) the literals for the 
properties do get re-used. I like consistency, even if in a particular 
situation it might not add much.   If the NIFI style is to not to do it this 
way, I will happily use literals.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343996#comment-16343996
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559047
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
--- End diff --

my bad. i will do something more expressive.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164559047
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
--- End diff --

my bad. i will do something more expressive.


---

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164558925
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -70,33 +65,51 @@
 @Tags({"grok", "log", "text", "parse", "delimit", "extract"})
 @CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
 "adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
-"notation of the matched content")
+"notation of the matched content\n" +
+"uses https://github.com/thekrakken/java-grok.;)
 @WritesAttributes({
-@WritesAttribute(attribute = "grok.XXX", description = "When operating 
in flowfile-attribute mode, each of the Grok identifier that is matched in the 
flowfile " +
-"will be added as an attribute, prefixed with \"grok.\" For 
example," +
-"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"grok.timestamp\"")})
+@WritesAttribute(attribute = "{result prefix}XXX", description = "When 
operating in flowfile-attribute mode, each of the Grok identifier that is 
matched in the flowfile " +
+"will be added as an attribute, prefixed with \"{result prefix}\" 
For example," +
+"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"{result prefix}timestamp\""),
+
+@WritesAttribute(attribute = "ExtractGrok.exception", description 
= "if an error occurs, an exception will be written to this attribute, " +
+"and the flow routed to 'unmatched' ")
+})
 public class ExtractGrok extends AbstractProcessor {
 
 public static final String FLOWFILE_ATTRIBUTE = "flowfile-attribute";
 public static final String FLOWFILE_CONTENT = "flowfile-content";
-private static final String APPLICATION_JSON = "application/json";
-
+public static final String APPLICATION_JSON = "application/json";
+public static final String GROK_EXPRESSION_KEY = "Grok Expression";
+public static final String GROK_PATTERN_FILE_KEY = "Grok Pattern file";
+public static final String DESTINATION_KEY = "Destination";
+public static final String CHARACTER_SET_KEY = "Character Set";
+public static final String MAXIMUM_BUFFER_SIZE_KEY = "Maximum Buffer 
Size";
+public static final String NAMED_CAPTURES_ONLY_KEY = "Named captures 
only";
+public static final String SINGLE_MATCH_KEY = "Single Match";
+public static final String RESULT_PREFIX_KEY = "result prefix";
+public static final String MATCHED_EXP_ATTR_KEY = "matched expression 
attribute";
+public static final String EXP_SEPARATOR_KEY = "expression-separator";
+public static final String PATTERN_FILE_LIST_SEPARATOR = ",";
+
+//properties
 public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor.Builder()
-.name("Grok Expression")
-.description("Grok expression")
+.name(GROK_EXPRESSION_KEY)
--- End diff --

just because sometimes (maybe not in this processor) the literals for the 
properties do get re-used. I like consistency, even if in a particular 
situation it might not add much.   If the NIFI style is to not to do it this 
way, I will happily use literals.


---

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343991#comment-16343991
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164558402
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -70,33 +65,51 @@
 @Tags({"grok", "log", "text", "parse", "delimit", "extract"})
 @CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
 "adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
-"notation of the matched content")
+"notation of the matched content\n" +
--- End diff --

thanks, i will use  instead of \n. and add the notice.  (BTW: It is 
same implementation as was being used before, without credit)


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2411: NIFI-4789 Extract grok multi pattern support

[charlesporter]

Github user charlesporter commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164558402
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -70,33 +65,51 @@
 @Tags({"grok", "log", "text", "parse", "delimit", "extract"})
 @CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
 "adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
-"notation of the matched content")
+"notation of the matched content\n" +
--- End diff --

thanks, i will use  instead of \n. and add the notice.  (BTW: It is 
same implementation as was being used before, without credit)


---

[jira] [Commented] (NIFI-4805) allow delayed transfer

[Martin Mucha (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343956#comment-16343956
 ] 

Martin Mucha commented on NIFI-4805:


I understand what you're saying and we can do that. Somewhat. The problem is, 
that retry 10 seconds is extremely long for first re-connection attempt and 
extremely short for fifth re-connection attempt. Using tools at hand, we can at 
most use penalization property as a 'waiting-basis' and multiply it: we will 
have processor with self loop, where I will loop only each second to get 
reasonable number of loops. And we have to keep track for each flowfile via 
nonsensical flowfile attributes(how long it already waited, how long it should 
wait before next retry) and ... it still waste cpu and it's ugly and laborious 
processor users considering otherwise trivial task.

 

> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Martin Mucha
>Priority: Minor
> Attachments: retry.xml
>
>
> Nifi has concept of penalization, but this penalization has fixed delay, and 
> there isn't way how to change it dynamically. 
> If we want to implement retry flow, where FlowFile flows in loop, we can 
> either lower performance of Processor via yielding it, or we can do active 
> waiting. And this is actually recommended as a correct way how to do that.
> It seems, that we can easily implement better RetryProcessor, all we missing 
> is `session.penalize` which accepts `penalizationPeriod`. Processor then can 
> gradually prolong waiting time after each failure.
>  
> Would it be possible to make such method visible?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3642) Specify preferred host for web API when clustered

[Prasad Rao Alokam (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343955#comment-16343955
 ] 

Prasad Rao Alokam commented on NIFI-3642:
-

It is an important feature for the container based like DC/OS. We have an issue 
with intra-cluster communication as 2 Jetty instances in a cluster are not able 
to communicate without the overlay network. It is sure becoming a problem.

Thanks

> Specify preferred host for web API when clustered
> -
>
> Key: NIFI-3642
> URL: https://issues.apache.org/jira/browse/NIFI-3642
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.1.1, 1.0.1
>Reporter: Bryan Bende
>Priority: Minor
>
> When clustered, each node calculates two pairs of hosts and ports:
> 1) The cluster socket address and cluster socket port
> (nifi.cluster.node.address and nifi.cluster.node.protocol.port) which
> are used for the clustering protocol
> 2) The node API address and node API port which is the web
> address/port to use when web-requests are being replicated across the
> cluster
> In a secure cluster, each node would have nifi.web.https.host specified in 
> order match the hostname in the certificate being used, but in an unsecure 
> cluster nifi.web.http.host could be left blank.
> When  nifi.web.http.host, Jetty will bind to all available network 
> interfaces, but the node will calculate its API address as localhost because 
> it does have any other information to use. This will cause a node to try and 
> replicate a request to another node using a localhost address which obviously 
> won't work.
> We should provide an additional, optional property like 
> nifi.cluster.preferred.web.api.host which would be used to determine the web 
> API hostname to report when unsecured, and when no http host is provided. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343926#comment-16343926
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164469623
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
--- End diff --

Please add a property for Connection Timeout (check the standard-nar for 
examples, I think ExecuteSQL has one). Unfortunately there is not a client-side 
Query Timeout that I could find, or else that should've been added too. The 
Connection Timeout should be set upon the underlying OkHttp client.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343929#comment-16343929
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164460497
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
--- End diff --

Is this a "soft limit", meaning if I had 99 records that were .9 MB in size 
and the 100th record was .2 in size, then all 100 records would be batched 
together (even though the size is 1.1 MB)? In either case it may be helpful to 
add more details to the description.


> Implement put processor for InfluxDB
> 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343925#comment-16343925
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164459404
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
--- End diff --

Would prefer Database Name here (I know DB is fairly obvious but Database 
is more descriptive, consistent, and the same as in the description)


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343931#comment-16343931
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164544734
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
--- End diff --

Consider supporting Expression Language here. Although it might not make 
sense to use FlowFile attributes (since it would be less efficient to have a 
connection per execution), supporting EL would allow the use of the Variable 
Registry, Process Group variables, etc. to set the URL, which helps when 
promoting flows from dev -> test -> production for example.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFIREG-131) Add logging authentication failure cases

[Kevin Doran (JIRA)]

Kevin Doran created NIFIREG-131:
---

 Summary: Add logging authentication failure cases
 Key: NIFIREG-131
 URL: https://issues.apache.org/jira/browse/NIFIREG-131
 Project: NiFi Registry
  Issue Type: Improvement
Affects Versions: 0.1.0
Reporter: Kevin Doran
Assignee: Kevin Doran
 Fix For: 0.2.0


The handling for unauthenticated requests made to a secured Registry instance 
swallows authentication exceptions with a vague error message: 
"AuthenticationEntryPoint invoked as no user identity credentials were found in 
the request." and an HTTP 401 response is returned with no body to aide in 
client troubleshooting.

We should add more useful troubleshooting information (including the root 
cause) to the logs and some guidance information for the client to the response 
body.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343930#comment-16343930
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164544898
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343923#comment-16343923
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164459116
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
--- End diff --

Perhaps an example of an InfluxDB connection URL here? Also URL should be 
capitalized.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343932#comment-16343932
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164460715
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
--- End diff --

Also is data size the appropriate thing to use here, or should it be number 
of records, or both? Any reason why Expression Language would not be supported?


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343935#comment-16343935
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164545775
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343922#comment-16343922
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164456208
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,209 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343936#comment-16343936
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164547117
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/PutInfluxDB.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"influxdb", "measurement","insert", "write", "put", "timeseries"})
+@CapabilityDescription("Processor to write the content of a FlowFile (in 
line protocol 
https://docs.influxdata.com/influxdb/v1.3/write_protocols/line_protocol_tutorial/)
 to InfluxDB (https://www.influxdb.com/). "
++ "  The flow file can contain single measurement point or 
multiple measurement points separated by line seperator.  The timestamp (last 
field) should be in nano-seconds resolution.")
+@WritesAttributes({
+@WritesAttribute(attribute = 
AbstractInfluxDBProcessor.INFLUX_DB_ERROR_MESSAGE, description = "InfluxDB 
error message"),
+})
+public class PutInfluxDB extends AbstractInfluxDBProcessor {
+
+public static AllowableValue CONSISTENCY_LEVEL_ALL = new 
AllowableValue("ALL", "All", "Return success when all nodes have responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_ANY = new 
AllowableValue("ANY", "Any", "Return success when any nodes have responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_ONE = new 
AllowableValue("ONE", "One", "Return success when one node has responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_QUORUM = new 
AllowableValue("QUORUM", "Quorum", "Return success when a majority of nodes 
have responded with write success");
+
+public static final PropertyDescriptor CONSISTENCY_LEVEL = new 
PropertyDescriptor.Builder()
+.name("influxdb-consistency-level")
+.displayName("Consistency Level")
+.description("InfluxDB consistency level")
+.required(true)
+.defaultValue(CONSISTENCY_LEVEL_ONE.getValue())
+.expressionLanguageSupported(true)
+.allowableValues(CONSISTENCY_LEVEL_ONE, CONSISTENCY_LEVEL_ANY, 
CONSISTENCY_LEVEL_ALL, CONSISTENCY_LEVEL_QUORUM)
+.build();
+
+public static final PropertyDescriptor RETENTION_POLICY = new 
PropertyDescriptor.Builder()

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164547117
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/PutInfluxDB.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"influxdb", "measurement","insert", "write", "put", "timeseries"})
+@CapabilityDescription("Processor to write the content of a FlowFile (in 
line protocol 
https://docs.influxdata.com/influxdb/v1.3/write_protocols/line_protocol_tutorial/)
 to InfluxDB (https://www.influxdb.com/). "
++ "  The flow file can contain single measurement point or 
multiple measurement points separated by line seperator.  The timestamp (last 
field) should be in nano-seconds resolution.")
+@WritesAttributes({
+@WritesAttribute(attribute = 
AbstractInfluxDBProcessor.INFLUX_DB_ERROR_MESSAGE, description = "InfluxDB 
error message"),
+})
+public class PutInfluxDB extends AbstractInfluxDBProcessor {
+
+public static AllowableValue CONSISTENCY_LEVEL_ALL = new 
AllowableValue("ALL", "All", "Return success when all nodes have responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_ANY = new 
AllowableValue("ANY", "Any", "Return success when any nodes have responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_ONE = new 
AllowableValue("ONE", "One", "Return success when one node has responded with 
write success");
+public static AllowableValue CONSISTENCY_LEVEL_QUORUM = new 
AllowableValue("QUORUM", "Quorum", "Return success when a majority of nodes 
have responded with write success");
+
+public static final PropertyDescriptor CONSISTENCY_LEVEL = new 
PropertyDescriptor.Builder()
+.name("influxdb-consistency-level")
+.displayName("Consistency Level")
+.description("InfluxDB consistency level")
+.required(true)
+.defaultValue(CONSISTENCY_LEVEL_ONE.getValue())
+.expressionLanguageSupported(true)
+.allowableValues(CONSISTENCY_LEVEL_ONE, CONSISTENCY_LEVEL_ANY, 
CONSISTENCY_LEVEL_ALL, CONSISTENCY_LEVEL_QUORUM)
+.build();
+
+public static final PropertyDescriptor RETENTION_POLICY = new 
PropertyDescriptor.Builder()
+.name("influxdb-retention-policy")
+.displayName("Retention Policy")
+.description("Retention policy for the saving the records")
+.defaultValue("autogen")
+  

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164461734
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
--- End diff --

Typo in "Successful", but you may want to specify what a "successful 
FlowFile is", is it a flow file whose contents have been successfully 
processed? I can see wanting to reuse the success relationship 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343924#comment-16343924
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164463448
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164546479
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("Failed FlowFiles are routed to this 
relationship").build();
+
+public static final String 

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164456208
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,209 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those 

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164462826
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("Failed FlowFiles are routed to this 
relationship").build();
--- End diff --

Same comment here 

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164463448
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("Failed FlowFiles are routed to this 
relationship").build();
+
--- End diff --

Consider 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343934#comment-16343934
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164546479
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164544898
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("Failed FlowFiles are routed to this 
relationship").build();
+
+public static final String 

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343927#comment-16343927
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164462826
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164460715
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
--- End diff --

Also is data size the appropriate thing to use here, or should it be number 
of records, or both? Any reason why Expression Language would not be supported?


---

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343928#comment-16343928
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164461734
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
  

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164460497
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
--- End diff --

Is this a "soft limit", meaning if I had 99 records that were .9 MB in size 
and the 100th record was .2 in size, then all 100 records would be batched 
together (even though the size is 1.1 MB)? In either case it may be helpful to 
add more details to the description.


---

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164469623
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
--- End diff --

Please add a property for Connection Timeout (check the standard-nar for 
examples, I think ExecuteSQL has one). Unfortunately there is not a client-side 
Query Timeout that I could find, or else that should've been added too. The 
Connection Timeout should be set upon the underlying OkHttp client.


---

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164459404
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
--- End diff --

Would prefer Database Name here (I know DB is fairly obvious but Database 
is more descriptive, consistent, and the same as in the description)


---

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164459116
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
--- End diff --

Perhaps an example of an InfluxDB connection URL here? Also URL should be 
capitalized.


---

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164544734
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
--- End diff --

Consider supporting Expression Language here. Although it might not make 
sense to use FlowFile attributes (since it would be less efficient to have a 
connection per execution), supporting EL would allow the use of the Variable 
Registry, Process Group variables, etc. to set the URL, which helps when 
promoting flows from dev -> test -> production for example.


---

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164545775
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/AbstractInfluxDBProcessor.java
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+
+/**
+ * Abstract base class for InfluxDB processors
+ */
+abstract class AbstractInfluxDBProcessor extends AbstractProcessor {
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("influxdb-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor INFLUX_DB_URL = new 
PropertyDescriptor.Builder()
+.name("influxdb-url")
+.displayName("InfluxDB connection url")
+.description("InfluxDB url to connect to")
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_NAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-dbname")
+.displayName("DB Name")
+.description("InfluxDB database to connect to")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("influxdb-username")
+.displayName("Username")
+.required(false)
+.description("Username for accessing InfluxDB")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("influxdb-password")
+.displayName("Password")
+.required(false)
+.description("Password for user")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+protected static final PropertyDescriptor MAX_RECORDS_SIZE = new 
PropertyDescriptor.Builder()
+.name("influxdb-max-records-size")
+.displayName("Max size of records")
+.description("Maximum size of records allowed to be posted in 
one batch")
+.defaultValue("1 MB")
+.required(true)
+.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+.build();
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("Sucessful FlowFiles are routed to this 
relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("Failed FlowFiles are routed to this 
relationship").build();
+
+public static final String 

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r164546743
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/PutInfluxDB.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.influxdb;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.influxdb.InfluxDB;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"influxdb", "measurement","insert", "write", "put", "timeseries"})
+@CapabilityDescription("Processor to write the content of a FlowFile (in 
line protocol 
https://docs.influxdata.com/influxdb/v1.3/write_protocols/line_protocol_tutorial/)
 to InfluxDB (https://www.influxdb.com/). "
--- End diff --

The URL can/will become a dead link in time, perhaps refer to the Line 
Protocol and say "consult InfluxDB documentation for more details" or something 
like that.


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343869#comment-16343869
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2443

NIFI-4827 Added support for reading queries from the flowfile body to…

… GetMongo.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4827

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2443


commit 53ccfec329c09a18cf87d562482195ddf179b73d
Author: Mike Thomsen 
Date:   2018-01-29T19:18:28Z

NIFI-4827 Added support for reading queries from the flowfile body to 
GetMongo.




> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343868#comment-16343868
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@mans2singh LGTM, but we need a committer to sign off now.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2443

NIFI-4827 Added support for reading queries from the flowfile body to…

… GetMongo.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4827

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2443


commit 53ccfec329c09a18cf87d562482195ddf179b73d
Author: Mike Thomsen 
Date:   2018-01-29T19:18:28Z

NIFI-4827 Added support for reading queries from the flowfile body to 
GetMongo.




---

[GitHub] nifi issue #2101: NIFI-4289 - InfluxDB put processor

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@mans2singh LGTM, but we need a committer to sign off now.


---

[jira] [Commented] (NIFI-4798) UpdateAttribute should allow empty string values without requiring state management

[Mark Bean (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343800#comment-16343800
 ] 

Mark Bean commented on NIFI-4798:
-

This exact use case may be a holdover from long ago when removing an attribute 
was not an option. And, to do so now first requires identifying all downstream 
flows/customers and verifying there will be no adverse affects. In the end, 
this is probably a good idea.

Having said that, I believe this ticket is still valid. While there are many 
ways to achieve the same goal, the 'Set empty string' checkbox should still be 
one of the options available.

> UpdateAttribute should allow empty string values without requiring state 
> management
> ---
>
> Key: NIFI-4798
> URL: https://issues.apache.org/jira/browse/NIFI-4798
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Extensions
>Affects Versions: 1.5.0
>Reporter: Mark Bean
>Priority: Minor
>  Labels: easyfix
>
> The conditional validation of dynamic properties in UpdateAttribute use the 
> NON_EMPTY_STRING validator when state management is not enabled. This prevent 
> the "Set empty string" checkbox option. The introduction of state management 
> to this processor changed this behavior which previously allowed an empty 
> string for an attribute value. The validator for the non-stateful case should 
> be changed to return the original behavior.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4823) Make GetMongo JSON pretty printing optional

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343777#comment-16343777
 ] 

ASF GitHub Bot commented on NIFI-4823:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2441
  
@markap14 Thanks for the feedback. I merged them into the commit and also 
found along the way a bug in how I was handling it. So should be good to go now 
once the build finishes.


> Make GetMongo JSON pretty printing optional
> ---
>
> Key: NIFI-4823
> URL: https://issues.apache.org/jira/browse/NIFI-4823
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> Pretty printed JSON can explode the size of a MongoDB result set, 
> particularly on large queries. Right now, pretty printing is done all the 
> time and isn't configurable. It should be configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2441: NIFI-4823 Made pretty printing configurable in GetMongo.

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2441
  
@markap14 Thanks for the feedback. I merged them into the commit and also 
found along the way a bug in how I was handling it. So should be good to go now 
once the build finishes.


---

[GitHub] nifi issue #2133: NIFI-4361: Fixing early recovery shutdown due to EOF in wa...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2133
  
@gresockj Sorry I must have not seen your comment on here. This PR has been 
around for quite a while. However, I wonder if 
https://github.com/apache/nifi/pull/2416 makes this no longer relevant? I 
recently implemented a new version of the write-ahead log due to an issue that 
was discovered (NIFI-4774). The new implementation actually avoids using 
multiple partitions at all so this, I believe, becomes a non-issue. If you 
agree, then is it okay for you to close the PR? If you don't agree, can you 
help me understand?


---

[jira] [Commented] (NIFI-4361) Server fails to start during recovery upon full disk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343772#comment-16343772
 ] 

ASF GitHub Bot commented on NIFI-4361:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2133
  
@gresockj Sorry I must have not seen your comment on here. This PR has been 
around for quite a while. However, I wonder if 
https://github.com/apache/nifi/pull/2416 makes this no longer relevant? I 
recently implemented a new version of the write-ahead log due to an issue that 
was discovered (NIFI-4774). The new implementation actually avoids using 
multiple partitions at all so this, I believe, becomes a non-issue. If you 
agree, then is it okay for you to close the PR? If you don't agree, can you 
help me understand?


> Server fails to start during recovery upon full disk
> 
>
> Key: NIFI-4361
> URL: https://issues.apache.org/jira/browse/NIFI-4361
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.1.0, 1.2.0, 1.3.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>Priority: Major
>
> Our disk filled up -- we then freed up some space and restarted, but the 
> server failed to start up due to:
> ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from 
> cluster due to: org.apache.nifi.cluster.ConnectionException: Failed to 
> connect node to cluster due to: java.lang.IllegalStateException: Signaled end 
> to recovery, but there are more recovery files for Partition in directory 
> /data/nifi/flowfile_repository/partition-8
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.endRecovery(MinimalLockingWriteAheadLog.java:1047)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0]
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:487)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0]
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343719#comment-16343719
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164503277
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -156,14 +209,23 @@
 
 final List _descriptors = new ArrayList<>();
 _descriptors.add(GROK_EXPRESSION);
+_descriptors.add(EXPRESSION_SEPARATOR);
 _descriptors.add(GROK_PATTERN_FILE);
 _descriptors.add(DESTINATION);
 _descriptors.add(CHARACTER_SET);
 _descriptors.add(MAX_BUFFER_SIZE);
 _descriptors.add(NAMED_CAPTURES_ONLY);
+_descriptors.add(RESULT_PREFIX);
+_descriptors.add(BREAK_ON_FIRST_MATCH);
+_descriptors.add(MATCHED_EXP_ATTR);
 descriptors = Collections.unmodifiableList(_descriptors);
 }
 
+private String resultPrefix = "";
--- End diff --

This is not thread-safe. Since each invocation of the Processor may use a 
different thread, we must ensure that any member variable is stored in a 
thread-safe manner. Given that these variables appear to be set only in the 
@OnScheduled method and are not mutable objects, it should be sufficient to 
mark them as volatile. It is probably best, though, to move them up in the 
codebase with the other member variables, as that's the de-facto standard with 
nifi processors.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343712#comment-16343712
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164499982
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -70,33 +65,51 @@
 @Tags({"grok", "log", "text", "parse", "delimit", "extract"})
 @CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
 "adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
-"notation of the matched content")
+"notation of the matched content\n" +
+"uses https://github.com/thekrakken/java-grok.;)
 @WritesAttributes({
-@WritesAttribute(attribute = "grok.XXX", description = "When operating 
in flowfile-attribute mode, each of the Grok identifier that is matched in the 
flowfile " +
-"will be added as an attribute, prefixed with \"grok.\" For 
example," +
-"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"grok.timestamp\"")})
+@WritesAttribute(attribute = "{result prefix}XXX", description = "When 
operating in flowfile-attribute mode, each of the Grok identifier that is 
matched in the flowfile " +
+"will be added as an attribute, prefixed with \"{result prefix}\" 
For example," +
+"if the grok identifier \"timestamp\" is matched, then the value 
will be added to an attribute named \"{result prefix}timestamp\""),
+
+@WritesAttribute(attribute = "ExtractGrok.exception", description 
= "if an error occurs, an exception will be written to this attribute, " +
+"and the flow routed to 'unmatched' ")
+})
 public class ExtractGrok extends AbstractProcessor {
 
 public static final String FLOWFILE_ATTRIBUTE = "flowfile-attribute";
 public static final String FLOWFILE_CONTENT = "flowfile-content";
-private static final String APPLICATION_JSON = "application/json";
-
+public static final String APPLICATION_JSON = "application/json";
+public static final String GROK_EXPRESSION_KEY = "Grok Expression";
+public static final String GROK_PATTERN_FILE_KEY = "Grok Pattern file";
+public static final String DESTINATION_KEY = "Destination";
+public static final String CHARACTER_SET_KEY = "Character Set";
+public static final String MAXIMUM_BUFFER_SIZE_KEY = "Maximum Buffer 
Size";
+public static final String NAMED_CAPTURES_ONLY_KEY = "Named captures 
only";
+public static final String SINGLE_MATCH_KEY = "Single Match";
+public static final String RESULT_PREFIX_KEY = "result prefix";
+public static final String MATCHED_EXP_ATTR_KEY = "matched expression 
attribute";
+public static final String EXP_SEPARATOR_KEY = "expression-separator";
+public static final String PATTERN_FILE_LIST_SEPARATOR = ",";
+
+//properties
 public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor.Builder()
-.name("Grok Expression")
-.description("Grok expression")
+.name(GROK_EXPRESSION_KEY)
--- End diff --

Can you help me understand the idea behind pulling the string literal out 
into a member variable? Since the PropertyDescriptor's name is generally not 
referenced elsewhere, it's usually a lot cleaner to have the name inlined.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4789) Enhance ExtractGrok processor to handle multiple grok expressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343721#comment-16343721
 ] 

ASF GitHub Bot commented on NIFI-4789:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2411#discussion_r164502184
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -107,31 +120,70 @@
 .build();
 
 public static final PropertyDescriptor CHARACTER_SET = new 
PropertyDescriptor.Builder()
-.name("Character Set")
+.name(CHARACTER_SET_KEY)
 .description("The Character Set in which the file is encoded")
 .required(true)
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .defaultValue("UTF-8")
 .build();
 
 public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
-.name("Maximum Buffer Size")
+.name(MAXIMUM_BUFFER_SIZE_KEY)
 .description("Specifies the maximum amount of data to buffer (per 
file) in order to apply the Grok expressions. Files larger than the specified 
maximum will not be fully evaluated.")
 .required(true)
 .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
 .addValidator(StandardValidators.createDataSizeBoundsValidator(0, 
Integer.MAX_VALUE))
 .defaultValue("1 MB")
 .build();
 
-public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
-.name("Named captures only")
-.description("Only store named captures from grok")
+ public static final PropertyDescriptor NAMED_CAPTURES_ONLY = new 
PropertyDescriptor.Builder()
+.name(NAMED_CAPTURES_ONLY_KEY)
+.description("Only store named captures from grokList")
 .required(true)
 .allowableValues("true", "false")
 .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
 .defaultValue("false")
 .build();
 
+public static final PropertyDescriptor BREAK_ON_FIRST_MATCH = new 
PropertyDescriptor.Builder()
+.name(SINGLE_MATCH_KEY)
+.description("Stop on first matched expression.")
+.required(true)
+.allowableValues("true", "false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.defaultValue("true")
+.build();
+
+public static final PropertyDescriptor RESULT_PREFIX = new 
PropertyDescriptor.Builder()
+.name(RESULT_PREFIX_KEY)
+.description("Value to prefix attribute results with (avoid 
collisions with existing properties)" +
--- End diff --

We should probably avoid using \n and \t in the description here and 
instead just write in more of a paragraph form. Since this is rendered in HTML 
and in a couple of different places, there is no guarantee that this will be 
rendered the way that it's intended.


> Enhance ExtractGrok processor to handle multiple grok expressions
> -
>
> Key: NIFI-4789
> URL: https://issues.apache.org/jira/browse/NIFI-4789
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.5.0
> Environment: all
>Reporter: Charles Porter
>Priority: Minor
>  Labels: features
>
> Many flows require running several grok expressions against an input to 
> correctly tag and extract data. using many separate grok processors to 
> accomplish this is unwieldy and hard to maintain.  Supporting multiple grok 
> expressions delimited by comma or user selected delimiter greatly simplifies 
> this.  
> Feature is coded and tested, ready for pull request, if feature is approved



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)